Methods and systems for single-photon imaging

Abstract

A system and method for single-photon imaging are provided. The method includes receiving a plurality of asynchronous or partially asynchronous single-photon detections and absolute timestamps associated with each of the received single-photon detections; performing probing measurements to determine the sum of the values ​​of a probing function for the single-photon detection values ​​at each of the absolute timestamps; reconstructing a continuous-time flux function from the probing measurements; and outputting the reconstructed continuous-time flux function or its integral.

Description

[Technical Field]

[0001] The following relates generally to photon processing and photonics, and more specifically to methods and systems for single-photon imaging. [Background technology]

[0002] A fundamental rule of thumb in high-speed imaging is that speed requires light; that is, the faster the scene changes, the more light is needed to accurately image without excessive noise or motion blur. High-speed light sources, high-speed cameras, and depth sensors, with the help of actively controlled light sources and synchronization, enable imaging of dynamic phenomena occurring at increasingly shorter time intervals. By operating the camera and light source in lockstep at repetition rates of MHz or higher to collect enough light, the same picosecond or nanosecond-scale event can be imaged millions of times. Unfortunately, while these techniques capture ultrafast events, they cannot simultaneously capture slower events because time resets to zero at the end of each synchronization cycle. As a result, when using such techniques, events spanning multiple synchronization cycles are essentially decomposed into single-cycle events and then summed up. This blurs anything that occurs over a time span longer than the synchronization cycle. [Overview of the Initiative]

[0003] In one embodiment, a processor implementation method for single-photon imaging is provided, the method comprising: receiving a plurality of asynchronous or partially asynchronous single-photon detections and an absolute timestamp associated with each of the received single-photon detections; performing a probing measurement to determine the sum of the values ​​of a probing function for the single-photon detection values ​​at each of the absolute timestamps; reconstructing a continuous-time flux function from the probing measurement; and outputting the reconstructed continuous-time flux function or its integral.

[0004] In certain cases of the method, the probing function is performed for frequencies from 0 to the maximum recoverable frequency, with each frequency separated by a frequency step.

[0005] In another case of the method, the frequency step is determined as 0.6 / t, where t is the total acquisition time.

[0006] In yet another case of the method, the probing function approximates the flux function to an additive noise function.

[0007] In yet another case of the method, the probing function is a continuous-time probing function.

[0008] In yet another case of the method, the probing function is a Fourier basis function, and the continuous-time flux function is reconstructed from the amplitude and phase of the detected frequencies.

[0009] In yet another case of the method, the Fourier basis function is p f (t)=e -j2πft That is the case.

[0010] In yet another case of the method, flux frequencies greater than 1 / 2q are ignored, and q is a given timing resolution.

[0011] In yet another case of the method, the method further includes using a constant false alarm rate (CFAR) detector to identify and remove noisy frequencies based on a predetermined probability of false alarms.

[0012] In yet another case of the method, a predetermined probability of a false alarm is determined by detecting whether a given probing frequency contributes to the flux function by determining whether the corresponding amplitude is greater than a predetermined threshold.

[0013] In yet another case of the method, reconstructing the continuous-time flux function involves summing corresponding Fourier basis functions scaled by the amplitude and shifted by the associated phase over the detected frequencies.

[0014] In yet another case of the method, reconstructing the continuous-time flux function

Number

[0015] In yet another case of the method, the reconstructed continuous-time flux function or its integral is used in computer vision or image processing tasks.

[0016] In yet another case of the method, the absolute time stamp is the frame number within the sequence captured by the image sensor.

[0017] In another aspect, a system for single photon imaging is provided, the system comprising one or more processors and data storage, the data storage receiving from a single photon detector multiple asynchronous or partially asynchronous single photon detections and absolute time stamps associated with each of the received single photon detections, a probing module for performing a probing measurement to determine the sum of the values of a probing function for the single photon detection values at each of the absolute time stamps, a reconstruction module for reconstructing a continuous-time flux function from the probing measurement, and an output module for outputting the reconstructed continuous-time flux function or its integral, and including instructions for causing the one or more processors to execute.

[0018] In a specific case of the system, the probing function is performed for frequencies from 0 to the maximum recoverable frequency, with each frequency separated by a frequency step.

[0019] In another case of the system, the probing function is a continuous-time probing function.

[0020] In yet another case of the system, the probing function is a Fourier basis function, and the continuous-time flux function is reconstructed from the amplitude and phase of the detected frequencies.

[0021] In yet another case of the system, one or more processors further execute a noise module to implement a constant false alarm rate (CFAR) detector to identify and eliminate noisy frequencies based on a predetermined probability of false alarms.

[0022] In yet another case of the system, reconstructing the continuous-time flux function involves summing the amplitudes and cosines of the relevant phases over the detected frequencies.

[0023] In yet another case of the system, reconstructing the continuous-time flux function is

number

[0024] In yet another case of the system, receiving multiple asynchronous single-photon detections from a single-photon detector involves performing a scene scan.

[0025] In yet another case of the system, the single-photon detector comprises a one-dimensional or two-dimensional array of single-photon avalanche diodes.

[0026] In yet another case of the system, the reconstructed continuous-time flux function or its integral is used for computer vision or image processing tasks.

[0027] In yet another case of the system, the absolute timestamp is the frame number in the sequence captured by the image sensor.

[0028] In another embodiment, a processor implementation method for single-photon imaging is provided, the method comprising: receiving a plurality of asynchronous or partially asynchronous single-photon detections and absolute timestamps associated with each of the received single-photon detections; performing interval probing using a heterogeneous fast Fourier transform to determine frequency components; determining an interval band using the frequency components determined using the heterogeneous fast Fourier transform; performing probing measurements using only the specified interval band to determine the sum of the values ​​of the probing function for the single-photon detection values ​​at each of the absolute timestamps; reconstructing a continuous-time flux function from the probing measurements; and outputting the reconstructed continuous-time flux function or its integral.

[0029] These and other embodiments are discussed and described herein. It will be understood that the above summary of the invention presents representative embodiments of the system and method to help a skilled reader understand the detailed description below.

[0030] A deeper understanding of the examples can be obtained by referring to the following diagram. [Brief explanation of the drawing]

[0031] [Figure 1A] This diagram illustrates a configuration for an exemplary experiment on passive ultra-broadband sensing using a single-pixel single-photon detector. [Figure 1B]This figure illustrates charts for reconstructed time-varying fluxes across various timescales for an exemplary experiment on passive ultra-broadband sensing using a single-pixel single-photon detector. [Figure 1C] This figure illustrates a reconstructed video frame for an exemplary experiment on passive ultra-broadband sensing using a single-pixel single-photon detector. [Figure 1D] This figure illustrates the reconstructed flux function spectrum resulting from multiple simultaneous phenomena in an exemplary experiment for passive ultra-broadband sensing using a single-pixel single-photon detector. [Figure 2] This is a conceptual diagram illustrating a system for passive single-photon imaging using an example. [Figure 3] This diagram illustrates a graph showing an example of the relationship between a stream of absolute timestamps, a counting process, a flux function, and its integral. [Figure 4] This is a conceptual flowchart illustrating a method for single-photon imaging using examples. [Figure 5] This graph illustrates an exemplary technique of the method shown in Figure 4, which is applied to photon timestamp data from a single pixel. [Figure 6A] This photograph illustrates the simultaneous reconstruction of fan rotation and light propagation. [Figure 6B] This is a photograph illustrating the experimental setup for acquiring 2D ultra-wideband video. [Figure 6C] This is a photograph illustrating an experimental setup for NLOS video imaging in a scene with multiple light sources. [Figure 6D] These are reconstructed video frames and charts illustrating the flux function visualized on two time scales for different points in a scene simultaneously illuminated by a pulsed laser and a flashing light bulb. [Figure 6E]These are reconstructed video frames and charts illustrating the flux function visualized on two time scales for different points in a scene simultaneously illuminated by a pulsed laser and a flashing light bulb. [Figure 6F] This photograph illustrates passive NLOS video acquisition using a raster-scanning laser projector and reconstructed video frames along with true-value frames. [Figure 6G] This photograph illustrates passive NLOS video acquisition using a raster-scanning laser projector and reconstructed video frames along with true-value frames. [Figure 6H] This figure illustrates a comparison between the system shown in Figure 2 and the use of SPAD array data with an alternative method for reconstructing flux per pixel. [Figure 6J] This figure illustrates a comparison between the system shown in Figure 2 and the use of SPAD array data with an alternative method for reconstructing flux per pixel. [Figure 7] (a) is a diagram illustrating a comparison of imaging techniques using photon time stamps between the passive imaging method, the active imaging method, and the method shown in Figure 4; (b) is a diagram illustrating a comparison of imaging techniques using photon time stamps between the passive imaging method, the active imaging method, and the method shown in Figure 4; and (c) is a diagram illustrating a comparison of imaging techniques using photon time stamps between the passive imaging method, the active imaging method, and the method shown in Figure 4. [Modes for carrying out the invention]

[0032] Examples will be described here with reference to the drawings. For the sake of brevity and clarity of the description, reference numbers may be repeated between figures to point to corresponding or similar elements, where appropriate. In addition, numerous specific details are given to provide a full understanding of the examples described herein. However, it will be understood by those skilled in the art that the examples described herein may be practiced without these specific details. In other examples, well-known methods, procedures, and components are not described in detail so as not to obscure the examples described herein. Furthermore, the description should not be considered to limit the scope of the examples described herein.

[0033] The various terms used throughout this description may be read and understood as follows, unless the context specifically indicates otherwise: “or,” when used throughout, is as comprehensive as when written as “and / or”; singular articles and pronouns, when used throughout, include their plural forms and vice versa; similarly, gender pronouns include their corresponding pronouns; therefore, pronouns should not be understood as being limited, for example, to single-gender use, implementation, or practice in any of the descriptions herein; and “exemplary” should be understood as “illustrating” or “showing examples,” and not necessarily “preferred” over other embodiments. Further definitions of terms may be presented herein, and these may apply to the examples before and after those terms, as can be understood from reading this description.

[0034] Modules, units, components, servers, computers, terminals, engines, or devices exemplifying this specification that execute instructions may include, or otherwise have access to, computer-readable media such as storage media, computer storage media, or data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROMs, digital versatile disks (DVDs), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by applications, modules, or both. Any such computer storage media may be part of a device, or may be accessible or connectable to a device. Furthermore, unless the context explicitly indicates otherwise, any processor or controller presented herein may be implemented as a single processor or as multiple processors. Multiple processors may be in an array or distributed, and the processing functions referred to herein may be executed by one or more processors, even if a single processor is sometimes exemplified. Any method, application, or module described herein may be implemented using computer-readable / executable instructions, which may be stored or otherwise held in such computer-readable media, and may be executed by one or more processors.

[0035] The following relates generally to photon processing and photonics, and more specifically to methods and systems for passive single-photon imaging.

[0036] Imaging highly dynamic scenes passively (i.e., without any controlled light source, synchronization, or large amounts of light) at both low and ultrafast speeds is a very challenging problem. This problem is particularly problematic for existing models for passive low-light imaging because they break down on timescales much shorter than the time span between photon arrivals. As a result, ultrafast imaging at low light has remained unattainable with current passive methods.

[0037] This embodiment effectively bridges two forms: active imaging and passive imaging. Specifically, in active single-photon imaging, a periodic light source synchronized with a photon detector acquires photons and times-stamps their detection against the start time of the most recent synchronization period. In this embodiment, a passive single-photon imaging technique is provided that eliminates synchronization between the camera and the light source in the scene. In this embodiment, each photon sensor (camera) pixel can times-stamp the detected photons using an internal clock that follows a time arrow, eliminating the need for any external timing signals.

[0038] Passive (synchronization-free) imaging is advantageously more powerful than active imaging at low flux settings. Specifically, without a periodic timing signal from the light source, time never rewinds in the synchronization period. Ultrafast scenes can be imaged over arbitrarily long time spans. Furthermore, flux fluctuations occurring simultaneously over 12 orders of magnitude (picoseconds to seconds), and flux fluctuations involving numerous unknown sources, can be recorded using just one camera.

[0039] In the embodiments of this disclosure, since photon timestamps at all light sources and all time scales can be recorded simultaneously, the selection of a time scale for display and the selection of light sources for use in visual processing can be advantageously performed after acquisition (as illustrated in Figures 1A–1D, 6A, 6D, and 6E). Thus, just as a light field camera enables post-capture refocusing in space, this embodiment enables post-capture refocusing in time, from transient time scales to everyday time scales. The technique of this embodiment may informally be referred to as passive ultra-broadband imaging.

[0040] Reconstructing the flux function from an asynchronous absolute photon timestamp stream using an ultra-broadband spectrum (e.g., DC ~ 10 GHz) is a considerable challenge. To address this challenge, this embodiment uses a flux probing method developed by the inventors, which employs stochastic analysis to relate the Fourier series decomposition of the time-varying flux function to the timestamp realization of the underlying stochastic process.

[0041] In passive settings, other techniques for estimating flux from photon data generally rely on various flux invariance assumptions and, as a result, are not applicable to the ultra-broadband configuration of this embodiment. In active settings, other single-photon imaging techniques can process photons from only one light source and generally rely exclusively on light source synchronization and synchronization-relative timestamps, losing information about sub-MHz flux fluctuations. Apart from single-photon imaging, other techniques for active ultrafast imaging assume that only one synchronized source emits light and generally rely on hetero-dining to measure flux at a specific modulation frequency. In contrast to these techniques, this embodiment has substantially improved optical efficiency and ultra-broadband capability. Furthermore, this embodiment eliminates the need for light source synchronization and can process photons from multiple artificial or natural light sources that emit light simultaneously.

[0042] Referring here to Figure 2, a system 100 for passive single-photon imaging according to an embodiment is shown. As will be understood by those skilled in the art, in some cases some components of system 100 may run on separate hardware implementations. In other cases some components of system 100 may be implemented on one or more general-purpose processors, which may be distributed locally or remotely, or on the processor / hardware of the image sensor itself.

[0043] Figure 2 shows various physical and logical components of an embodiment of System 100. As shown, System 100 has several physical and logical components, including one or more processors 102, data storage 104, output interface 106, and input interface 110, and a local bus 108 that enables the components to communicate with each other. One or more processors 102 may include one or more central processing units, one or more graphics processing units, microprocessors, dedicated hardware, logic arrays, or other integrated processing circuits. Data storage 104 may store programs, instructions, and / or operating systems, including computer executable instructions for implementing the methods described herein, as well as derived or related data. Although Figure 2 describes System 100 implemented on a single computing device, it is understood that the processing or any functions performed by System 100 may be distributed across multiple computing devices, for example, in a cloud or distributed computing environment.

[0044] In one embodiment, one or more processors 102 may be configured to run several conceptual modules, for example, an imaging module 112, a probing module 114, a reconstruction module 116, a noise module 118, and an output module 120. In further cases, the functions of the above modules may be combined with other modules or run on other modules. In some cases, the functions of the above modules may be run on remote computing devices such as centralized servers and cloud computing resources that communicate via a network module 176.

[0045] The output interface 106 allows another electronic or computing device to transmit data or receive output from system 100, as described herein. In some embodiments, the output interface 106 allows a user to view such output, for example, via a display or monitor. In some cases, the output from system 100 may also be stored in data storage 104. In other cases, the output from system 100 may undergo further processing by system 100 or other devices, for example, for use in 3D imaging or passive extreme computer vision. The input interface 110 can communicate with certain devices, such as an image sensor 130, which may be inside or outside system 100, either alone or in combination with the output interface 106. The image sensor 130 may be any suitable device capable of capturing and storing single photons and their associated arrival times, such as an asynchronous single-photon avalanche diode (SPAD), or a one-dimensional or two-dimensional array of SPADs.

[0046] In this embodiment, the system 100 can be used in both situations where the system 100 exercises control over the appearance of a scene and situations where the system does not exercise control over the appearance of a scene. Thus, the light sources of the scene can be natural light, artificial light, or both, and their number, operating principles, and time-varying characteristics can be unknown and unrestricted. It can also be assumed that no electronic timing signals such as triggers or synchronization pulses are received.

[0047] According to standard radiometric conventions, the incident light at a pixel is an unknown time-varying function representing the instantaneous flux of the pixel at time t ≥ 0

Number

Number

[0048] The system 100 can attempt to reconstruct a flux function having an ultra-wideband with frequency components covering the entire range from, for example, a constant flux (f min = 0 Hz) to an extreme time scale of flight time (f max ≥ 10 GHz). Furthermore,

Number

[0049] System 100 can be applied to single-photon imaging where the time span between the arrival of consecutive photons is not negligible. In this setting,

number

number

[0050] SPADs can generally detect and timestamp the arrival of individual photons with extremely high time precision (usually tens of picoseconds). However, SPADs can exhibit four major non-idealities: quantum efficiency, dead time, timestamp quantization, and jitter. Quantum efficiency refers to the probability that a pixel actually detects a photon while the pixel is in an active state. This probability can be far below 1, depending on the wavelength, and can be thought of as a scaling of the flux function.

number

number

[0051] Photon timestamps are subject to quantization from the time-to-digital conversion process and jitter, i.e., instability in timing electronics. Both can decrease to a few picoseconds for SPADs in the visible light range. Generally, the resolution and precision of a timestamp can be assumed to be identical, and therefore the bin size q of the timestamp also takes jitter into account. When the precision of the timestamp is lower than the resolution of the timestamp, the number of effective bits of the timestamp decreases. System 100 can use a timestamp quantized to, for example, 4 picoseconds, but the standard deviation of the jitter is 16 picoseconds, and therefore q=16 can be conservatively used for performance modeling.

[0052] Since an external timing signal is not available to serve as a reference, it can be assumed that the photon detection timestamp follows the time arrow and increases monotonically according to the SPAD's internal clock. This is the timestamp stream.

number

[0053] Other approaches to passive low-flux imaging using SPADs generally treat the time span between consecutive detections as a noisy sample of the scene's flux. This implicitly assumes that the flux does not fluctuate during that time span, and the photon detection rate is set to f max This sets the boundary to a (loose) upper limit. As a result, these methods allow f max It is limited to low speeds of around several tens of kHz.

[0054] Methods using synchronous light sources generally occupy the other extreme frequency range. Their basic principles are based on the known frequency f sync The purpose is to timestamp the detection against the synchronization signal, and therefore all timestamps are the same short interval [0, 1 / f], regardless of the actual time span between them. sync This wraps the photons in a relatively small number of time bins, usually thousands, and scales them into a time-wrapped flux function.

number

[0055] These photon histogram techniques can achieve very fast imaging speeds, but their dependence on relative timestamps is...

number

[0056] The synchronization signals used in these techniques are typically in the low MHz range for single-photon imaging applications with pulsed sources. This choice balances the improvement in the signal-to-noise ratio (faster synchronization means more laser pulses, more photons detected in each histogram bin, and fewer time bins for photon accumulation) with the possibility of missing photons due to dead time or photons arriving "late" because time has already elapsed. At such MHz synchronization frequencies, the memory and computational cost of reconstructing the histogram from the timestamps can be significant, and several methods have been proposed for just-in-time processing of photon timestamps (for synchronization).

[0057] Advantageously, system 100 considers the relationship between the photon detection rate and the maximum reconstructible frequency. Also advantageously, system 100 can be considered as the limit case of a photon histogram where the frequency of the synchronization signal decreases completely to zero. Specifically, f sync As it decreases, the interval [0, 1 / f sync ] increases, and more histogram bins are needed to reach it. Also, fewer photons reach each bin. Furthermore, the time-wrapped flux function can represent fluctuations that occur over longer time spans. Furthermore, the reconstructible frequency space (e.g., f sync The integer multiples of f expand. sync At the limit where is exactly zero, there is generally no synchronization at all, and every photon reaches its own inherent time bin, so the timestamp becomes absolute. Importantly, all frequencies from direct current (DC) to the Nyquist limit, and from any light source, may potentially be reconstructible.

[0058] Mathematical modeling of the extreme case is not easy because the histogram is distorted due to the complete absence of photons, and the contents of any given bin, 0 or 1, provide little information about the flux function. Computationally, the entire acquisition interval [0,t] is effectively divided into time bins with the timing resolution of the SPAD, and therefore acquisitions longer than 1 second can involve trillions of time bins (most of which are empty). Fortunately, the inventors have found that all of these problems can be overcome with f sync We determined that this could be overcome by formulating flux reconstruction from the perspective of a photon counting process that does not degenerate even when = 0.

[0059] System 100 establishes a direct mathematical link between the stream of absolute timestamps detected by pixels, no matter how few or how far apart they may be, and the flux function that generated them. This link allows System 100 to "probing" the Fourier spectrum of an unknown flux function over the entire range, for example, DC to GHz, for frequencies that have statistically significant support in the timestamp data. To address flux reconstruction, flux probing techniques are provided. For generality, timestamps can be considered as continuous-value random variables into which quantization may be incorporated.

[0060] Even if a single absolute timestamp provides little to no information about the underlying flux function for a photon count, the stream of absolute timestamps as a whole contains relevant and useful information. The inventors determined that certain relationships between the two can be determined using probabilistic analysis. Specifically, a stream of real-valued absolute timestamps in the continuous-time domain.

number

number

number

[0061] The function n(t) in equation (1) counts the photons received up to time t,

number

[0062] Figure 3 illustrates a graph showing an example of the relationship between a stream of absolute timestamps, a counting process, and the flux function and its integral. In this example, the timestamps are the flux function described herein.

number

[0063] System 100 is generally, • The highest possible frequency f that can be reconstructed by a passive single-photon imaging system that outputs a quantized absolute timestamp. max What is it? How can a noise model be derived that enables efficient detection and removal of spurious frequencies within an achievable bandwidth, and quantifies the accuracy of the actual frequency as a function of the acquired timestamp stream? To at least address this, we use tools from stochastic analysis.

[0064] The first proposition is that system 100 can constantly probe the flux function and reconstruct (noisy) measurements of its integral with almost any other function. Furthermore, probing can be efficiently calculated from a timestamp stream and can be considered a continuous-time and synchronization-free generalization of a compressed acquisition method for photon count histograms. In particular,

number

number

number

number

number

number

[0065] Fourier basis function p f (t)=e -j2πft In the special case of probing using f>1 / 2q, probing with f>1 / 2q produces aliased measurement results that "wrap around" the frequency spectrum, which are identical to and indistinguishable from low-frequency measurement results.

[0066] Therefore, equation (1) establishes a relationship between the derivative of the counting process n(t) and the right-hand side of equation (1). This relationship is used to derive equation (2) from equation (1) and provides the probing described. Informally, the derivative of n(t) is zero except at the moment of photon arrival, at which point it is 1; that is, n(t) is used to count photons and therefore increases by 1 when a photon arrives, and remains constant otherwise. Thus, the integral of the probing function p() is equal to the sum of the values ​​of p() at the photon arrival timestamps and is expressed by the left-hand side of equation (2).

[0067] The second proposition is that, given a time resolution q, the maximum recoverable frequency is

number

[0068] System 100 can use a noise model to handle the common case of real-valued timestamps, taking into account the heterogeneous Poisson nature of photon detection. The noise model is usable for any flux level within the low-flux form and remains valid for low-count acquisitions (e.g., around 10 photons). More specifically, the third proposition below indicates that the noise in a probing measurement has a distribution that can be estimated from the timestamp stream through another probing operation. Thus, probing provides both a means to observe the flux function and a means to quantify the uncertainty of that observation.

[0069] The third proposition is the probing measurement results.

number

number

number

[0070] The following conclusions enable system 100 to quantify the accuracy with which a particular flux frequency can be estimated from a given timestamp stream.

[0071] In the first conclusion, the Fourier probing measurement results

number

number

[0072] In the second consequence, the normalized energy of the Fourier basis probing measurement is,

number

number

[0073] The inventors determined that unbiased estimates of the parameters of the above distribution can be obtained through probing.

[0074] Given an estimated distribution of Fourier probing energies, a constant false alarm rate (CFAR) detector can be used to identify and remove noisy frequencies based on a desired probability α of false alarms. False alarms occur when the system is f and

number

number

number

number

[0075] Equation (7) is therefore a noise model that enables efficient detection and discarding of spurious frequencies for frequencies within an achievable bandwidth, and quantification of the actual frequency accuracy as a function of the acquired timestamp stream.

[0076] For a fixed α, the probability of detecting a frequency is proportional to the total number of photons detected.

number

[0077] The second proposition indicates that imaging with less synchronization using absolute timestamps does not hinder but rather provides a significant bandwidth advantage over single-photon sensor systems. For example, using a resolution of 16 picoseconds, system 100 can simultaneously acquire flux variations caused by any number of independently operating unknown light sources across the entire frequency range of DC to 31 GHz. This bandwidth is orders of magnitude wider than what was generally thought to be directly achievable by intensity cameras, SPADs, or other methods without relying on homodyne or heterodyne detection schemes.

[0078] The second proposition explains reconstructibility (for example, frequencies beyond the limit are not reconstructible), while the third proposition and equation (7) provide insights into the accuracy and detectability of flux variations at different frequencies.

[0079] Exemplary experiments conducted by the inventors demonstrate several real-world applications of simultaneous imaging at DC ~ 10 GHz under very harsh illumination conditions. These experiments validate the noise models of equations (4) to (6) and at least partially confirm the theoretical boundaries.

[0080] The probing technique described herein provides the ability to reconstruct the Fourier transform of a flux function by frequency scanning the entire bandwidth from DC to GHz. Unlike other techniques such as Fourier domain histogram techniques, this technique provides a principled method for estimating frequency uncertainty in an acquired timestamp stream, and is therefore applicable to streams consisting of absolute photon timestamps acquired without any synchronization, enabling manageable operation in forms involving potentially billions of candidate frequencies (e.g., 1Hz resolution scans from DC to 10GHz) by eliminating spurious frequencies and reducing storage requirements. Frequency detection involves flux probing, where α is set, for example, so that the expected number of false alarms is less than 1. While this disclosure generally describes reconstructing a flux function from a Fourier transform, it will be understood that any suitable reconstruction technique, such as least squares or training a neural network to represent the flux function, can be used.

[0081] Figure 4 illustrates a flowchart of Method 200 for single-photon imaging according to an embodiment. In some cases, the method can be used for passive imaging. However, in other cases, the method can be carried out for active imaging, for example, the environment may be illuminated by an actively controlled source such as a laser. In either imaging scenario, the light source does not need to be synchronized with the image sensor 130. Furthermore, Method 200 can be carried out in the presence of multiple light sources, including natural light sources or artificial light sources such as lasers and projectors.

[0082] In block 202, the imaging module 112 receives multiple asynchronous or partially asynchronous single-photon detections from the image sensor 130 that captures the scene. The imaging module 112 also receives an absolute timestamp associated with each of the received single-photon signals. Multiple asynchronous single-photon detections may be received from the single-photon image sensor 130 from a scan of the scene, or from a one-dimensional or two-dimensional array of single-photon detectors such as the image sensor 130, or from an array of the image sensor 130 with or without scanning. In certain cases, the single-photon image sensor 130 may be a single-photon avalanche diode (SPAD) sensor.

[0083] "Asynchronous" is understood to mean that the image sensor 130 and system 100 operate without synchronization with the light source. "Partially asynchronous" is understood to mean the operation of system 100 between fully asynchronous and fully synchronous modes, such as between fully asynchronous mode and frame-based mode. In an example of partially asynchronous operation, some 2D SPAD sensors output 2D "frames" of timestamps at a rate of 50,000 to 500,000 frames per second, with each frame recording up to one photon timestamp per pixel. In another example of partially asynchronous operation, some cameras output a sequence of binary frames, with each bit indicating whether or not a photon was detected at a particular pixel during the frame's exposure time. In such cases, photon detection is considered to be synchronized across the image sensors but does not have to be synchronized with the external light source and may therefore be partially asynchronous.

[0084] "Absolute" is understood to mean that time is measured relative to a clock which may be inside or outside system 100 and / or image sensor 130. Absolute refers to elapsed time from the start of single-photon signal acquisition, universal time, or any other fixed point in time. Thus, "absolute" is understood to mean that elapsed time is measured relative to a temporally fixed point and not as an offset from a periodic timing signal (e.g., relative to the start of a pulse from a laser, or relative to the rising or falling edge of a square wave).

[0085] In block 204, the probing module 114 performs a probing measurement to determine the sum of single-photon detection values ​​of the probing function at each absolute timestamp. In most cases, the probing function p(t) is a continuous-time probing function. The use of a continuous-time probing function offers substantial advantages over, for example, the use of discrete vectors, due to a substantial improvement in the accuracy of the measurement results output by system 100.

[0086] In block 206, in some cases, the noise module 118 identifies and removes noisy frequencies based on a predetermined probability of false alarms, and removes such frequencies. In certain cases, the noise module 118 may implement a constant false alarm rate (CFAR) detector, but any other suitable detection method that identifies statistically significant frequencies may be used.

[0087] In block 208, the reconstruction module 116 reconstructs the continuous-time flux function from the amplitude and phase of the frequencies detected in the probing measurement.

[0088] In block 210, output module 122 outputs the reconstructed continuous-time flux function or its integral to data storage 104, output interface 106, or both. The reconstructed continuous-time flux function or its integral can be used for several applications, such as video with widely varying speeds, passive 3D sensing or imaging, computer vision at different time scales, scientific imaging (e.g., used in telescopes to measure pulsars and other ultrafast signals), biological imaging of ultrafast structures, navigation in low light, or in combination with other single-photon sensors (each representing a pixel).

[0089] In some cases, the output module 122 can decompose the reconstructed continuous-time flux function into multiple component flux functions, for example, one for each individual light source, as described with reference to Figures 1B, 1C, and 1D. In some cases, the system 100 can perform further computational processing on the reconstructed continuous-time flux function for use in applications such as 3D imaging, laser analysis, or optical visual communication.

[0090] For example, System 100 uses the following pseudocode technique:

number

[0091] In the above, the frequency step Δf may be determined in advance or adaptively.

[0092] Figure 5 shows a visual explanation of the pseudocode described above. The flux function for finite spectral support can be represented as a sum of sinuses and can generate a stream of absolute timestamps. Frequency scanning can be used to probe the flux function using Fourier bases and measure the response at each frequency. Frequency detection can be performed by detecting whether each of the frequencies being probed contributes to the flux function if the corresponding amplitude is greater than a threshold (upper graph), the threshold is selected to achieve a desired probability of false alarms (lower graph). Flux reconstruction can be used to reconstruct the continuous-time flux function from the amplitude and phase of the detected frequencies. In some cases, the reconstruction module 116 can identify one or more dominant frequencies within the set of detected frequencies and can add the harmonics of the dominant frequencies to the set of detected frequencies used.

[0093] The inventors verified the substantial benefits of this embodiment using exemplary experiments for passive ultra-wideband imaging. This includes the use of passive ultra-wideband sensing for both 1D intensity signals and 2D video signals ranging from DC to 10 GHz, passive non-line-of-sight (NLOS) video via MHz-rate flux function reconstruction, and generalization to 2D SPAD arrays for high-speed video imaging.

[0094] For passive ultra-wideband sensing, exemplary experiments reconstructed signals across a frequency scale of approximately nine orders of magnitude, from DC to 5 GHz, as illustrated in Figures 1A–1D. Single-pixel SPADs were placed in the scene to capture flux variations from (1) pulse-width modulation of an incandescent light bulb (900 Hz), (2) backscattered light from a raster-scan laser projector (60 Hz–10 MHz), and (3) two asynchronous picosecond lasers (40 MHz–10 GHz). Remarkably, the flux function was reconstructed from only 77,000 photon timestamps. System 100 reconstructed time-varying fluxes across billions of frequencies from this minuscule set of photons.

[0095] Figures 1A–1D illustrate experimental results for passive ultra-broadband imaging using a single free-running SPAD pixel. As shown in Figures 1A and 1B, in the captured experiments, an asynchronous single-photon avalanche diode (SPAD) passively records indirect light coming from multiple sources operating asynchronously with respect to each other (such as asynchronous picosecond laser projectors). The incident flux exhibits simultaneous intensity fluctuations with a bandwidth of approximately nine orders of magnitude in frequency. As shown in Figure 1D, several simultaneously occurring phenomena in the flux function can be identified after acquisition: video flicker (58 Hz), pulse-width modulation of an LED bulb (900 Hz), a movie projected onto a nearby wall by a raster-scan laser projector (up to 5 MHz), and two asynchronous picosecond lasers (40 MHz–10 GHz). As illustrated in Figure 1C, by reconstructing the time-varying flux function of the laser projector, a video frame is reconstructed at a resolution of 1280 × 720 using approximately 450 to 4500 photons collected during each 1 / 58 second frame.

[0096] Figures 6A–6J illustrate exemplary experiments of passive ultra-broadband imaging conducted by the inventors. Figure 6A illustrates the simultaneous reconstruction of fan rotation and light propagation. Rendering the flux function at 1Kfps reveals the fan's movement (indicated by the arrow), but not the light propagation. At 250Gfps, light propagation is visible, but the fan is stopped. A conventional histogram synchronized with the laser (right) fails to reconstruct the (unsynchronized) fan rotation. Figures 6B and 6C illustrate experimental configurations for ultra-broadband video acquisition (Figure 6B) and NLOS video imaging in a scene with multiple light sources (Figure 6C). Figures 6D and 6E illustrate flux function images at two time scales for different points in a scene illuminated by a pulsed laser and a blinking light bulb. The three peaks in B, C, and D correspond to the light pulse incident on the bottle (B), propagating to the cap (C), and reflecting back (D). Figures 6F and 6G illustrate passive NLOS acquisition using a raster-scanning laser projector, along with the true and reconstructed frames. Figures 6H and 6J illustrate a comparison of this system 100 with the use of SPAD array data using an alternative technique for reconstructing flux per pixel. In this alternative technique, a piecewise constant flux is not maintained, even for this simple scene of a rotating fan.

[0097] An exemplary experiment demonstrated that ultra-wideband video (as described in Figure 6A) is feasible by raster scanning a scene in which a pulsed laser with a repetition rate of 20 MHz and a full width at half maximum (FWHM) of 80 ps illuminates a fan rotating at 108 Hz. System 100 detected frequencies from DC to 10 GHz and rendered flux functions at 1 Kfps and 250 Gfps, showing both the rotation of the fan blades and the propagation of the laser pulses. In contrast, other methods reconstruct the scene at only one of the aforementioned frame rates, temporally blurring either slow or fast events. Furthermore, System 100 can render the flux at any appropriate timescale, effectively freezing time at all timescales. In this experiment, only a single-pixel SPAD was used, so timestamps were collected by scanning across the SPAD's field of view. In other cases, a two-dimensional array of SPAD sensors could be used, in which case scanning would not be necessary. No synchronization signal was used to reconstruct the flux function; therefore, the exemplary experiment demonstrates the significant advantage of reconstructing the appearance of the scene as it appears in each laser pulse. This differs from estimating the average appearance of the flux function over time using synchronization and histograms. The images in the exemplary experiment were rendered by integrating the flux function over exposures of each frame. Thus, these images not only exhibit a high dynamic range, but their intensity is expressed in physical units of photons, thereby ensuring the calibration of the radiometric measurement in an essential sense.

[0098] Figure 6B illustrates another video example where the same picosecond laser is used to illuminate a Coca-Cola bottle containing water and a small amount of milk, causing the light to scatter. In the same scene, a compact fluorescent lightbulb (CFL) blinks at 120 Hz. System 100 rendered the video at 10 Kfps and 200 Gfps to visualize the blinking of the CFL and the light pulses propagating through the bottle, as described in Figures 6D and 6E.

[0099] In an exemplary experiment, System 100 performed passive non-line-of-sight (NLOS) video reconstruction using light indirectly measured from a raster-scanning laser projector (see descriptions and photographs in Figures 1A–1D and 6C). During the raster scan of the projector, the SPAD collects indirect light from the projector by observing a single point on the wall. The video is reconstructed by reconstructing the 1D flux function of the raster scan. The results of the exemplary experiment demonstrate the results for both the multi-lighting setup and the single-projector setup shown in Figures 1A–1D. In the latter case, System 100 reconstructed fine details (described in Figures 6F–6H) from only 3000 photons collected during a 1 / 60 second raster scan of each video frame.

[0100] Exemplary experiments also demonstrated that System 100 can be directly applied to various SPAD sensors. The exemplary experiments compared the method performed by System 100 to another method that reconstructed high-speed video using a 32x32 SPAD array and identified consecutive sets of timestamps with the same flux, assuming piecewise invariance of flux. The other method averaged each set of timestamps to reconstruct a single estimate of flux (explained in Figure 6J). In contrast, System 100 reconstructed a time-varying flux function using flux probing (explained in Figure 6H). System 100 advantageously reconstructed a time-varying flux function that was more faithful to the periodic motion of a rotating fan.

[0101] Figures 7(a)–(c) illustrate a comparison of low-flux imaging techniques using photon timestamps with exemplary experiments. Figure 7(a) shows that other techniques for passive imaging generally assume that photons can be detected at a rate (much) higher than the highest flux frequency. Figure 7(b) shows that active techniques are designed to address the opposite case where photon detection occurs at a rate (much) lower than the frequency of the flux function. sync If it is not a multiple of (for example,

number

number

[0102] This embodiment addresses the substantial challenge of simultaneously imaging dynamic scenes over extremely wide timescales (e.g., seconds to picoseconds) passively, without large amounts of light, and without any timing signals radiated from a light source. Since other flux estimation techniques for single-photon cameras do not work in this form, the embodiment of this disclosure uses a flux probing technique to reconstruct the time-varying flux of pixels from a monotonically increasing stream of photon detection timestamps. This technique was used because (1) it utilizes a passive free-running SPAD camera with a frequency bandwidth spanning the entire DC to 31 GHz range under low flux conditions, (2) it performs Fourier-domain flux reconstruction to scan this range for frequencies that have statistically significant support in the timestamp data, and (3) it has a noise model that remains valid even for very low photon counts. The substantial advantages of System 100 have been experimentally demonstrated and shown to have the ability to at least (1) image a scene simultaneously illuminated by sources operating at very different speeds without synchronization (e.g., light bulbs, projectors, multiple pulsed lasers), (2) acquire passive out-of-line video, (3) record ultra-wideband video that can be played back later at 30 Hz to show everyday motion, but can also be played back at a billion times slower to show the propagation of light itself, and (4) reconstruct images with a much higher signal-to-noise ratio compared to existing techniques.

[0103] The sheer volume of data involved in sensing and probing timestamp streams is considerable; in low light, even a single pixel can output tens of thousands of timestamps per second, and ultra-wideband can necessitate probing at billions of frequencies. Advantageously, system 100 is not limited by speed and can detect motion orders of magnitude faster than the frame rate. Furthermore, since the image sensor 130 is not necessarily connected to or communicated with any light source that would require synchronization, system 100 can sense light from multiple sources, thus mitigating the need to synchronize with multiple light sources.

[0104] Figure 8 illustrates a flowchart of Method 800 for single-photon imaging using fast probing, according to another embodiment.

[0105] In block 802, the imaging module 112 receives multiple asynchronous or partially asynchronous single-photon detections from the image sensor 130 that captures the scene, as described herein.

[0106] In block 804, the probing module 114 performs spaced probing using a non-uniform fast Fourier transform, such as the Flatiron Institute Nonuniform Fast Fourier Transform (FINUFFT).

[0107] In block 806, the probing module 114 determines the frequency interval bandwidth using frequency components determined using a non-uniform fast Fourier transform.

[0108] In block 808, the probing module 114 performs probing measurements only on specified interval bands to determine the sum of single-photon detection values. Probing measurements within the specified interval bands can therefore be used to determine whether the flux contains a signal in each of the specified bands. In certain cases, the interval bands may be uniformly spaced intervals, e.g., 1 Hz, 10 Hz, or 100 Hz. In other cases, the intervals may be non-uniform. However, as the intervals increase, the CFAR boundary becomes stricter (i.e., frequencies are more likely to be missed because it is more difficult for frequencies to cross the boundary). By keeping the intervals fixed, system 100 can ensure that the detection likelihood is constant across the entire frequency range.

[0109] In block 810, in some cases, the noise module 118 identifies and removes noisy frequencies based on a predetermined probability of false alarms. In certain cases, the noise module 118 may implement a constant false alarm rate (CFAR) detector, but any other suitable detection method that identifies statistically significant frequencies may be used.

[0110] In block 812, the reconstruction module 116 reconstructs the continuous-time flux function from the amplitude and phase of the frequencies detected in the probing measurement.

[0111] In block 814, the output module 122 outputs the reconstructed continuous-time flux function or its integral to the data storage 104, the output interface 106, or both.

[0112] In method 800, the fast Fourier transform (FFT) of the rate function λ(t) at a certain frequency f is:

number

number

number

number

number

number

number

[0113] The above means that the function value is not 1 but g(t k ) is, t k This is equivalent to computing a Nonuniform Fast Fourier Transform (NUFFT) using the non-uniform points evaluated in . This sum provides an output and, favorably, must be computed only at frequencies equal to nw, for all n∈N, where w is the window size.

[0114] For the CFAR detector, the following was used to determine the desired noise estimate.

number

number

number

[0115] The frequency components are determined, for example, by determining frequencies such that f_min = 0 Hz ~ f_max ≥ 10 GHz, as described herein. However, in this method, probing is advantageously performed by using a non-uniform Fast Fourier Transform (nuFFT). The output of the nuFFT is identical or similar to that of Method 200, but the computation can be orders of magnitude faster. Method 200 has a computational complexity of the order N*F, where N is the number of photons and F is the number of frequencies being probed (i.e., the number of calculations required is of the order N*F). Since the number of photons can be in the hundreds of thousands and the number of frequencies being probed can be in the billions, this represents a considerable number of calculations.

[0116] The applicant determined that using the non-uniform fast Fourier transform could provide a speed improvement of four to six orders of magnitude, along with a computational efficiency improvement of more than 1,000 times.

[0117] For example, a typical probing step can include a step size of approximately 0.1 Hz for a 1-second exposure. Therefore, to probate an exemplary range of 0 Hz to 10 GHz, this would mean 100 billion separate probing operations (10 GHz / 0.1 Hz). In contrast, using the interval bandwidth of Method 800, the step size can be increased to prob at intervals of, for example, 1 Hz, 10 Hz, or even 100 Hz for the presence of frequencies above the CFAR threshold. Thus, scanning the range of 0 Hz to 10 GHz, for example, might require a total of 100 million interval probing operations (10 GHz / 100 Hz), which is 1 / 1000th of the original.

[0118] This embodiment uses absolute timestamps, which offer substantial advantages over using relative timestamps, because relative timestamps are generally only available when the image sensor (e.g., SPAD) is synchronized to a single pulsed laser source, which substantially limits the applicability of such systems.

[0119] This embodiment also has substantial advantages over techniques that use observations applied to binned data instead of timestamps. Furthermore, it has substantial advantages over techniques that simply determine the radiation period instead of determining the flux function, such as calculating the pulsation period of a single celestial object but not calculating the flux function, not discarding noisy frequencies, or not reconstructing the flux function, which substantially limits the applicability of such techniques. Moreover, such techniques would generally require the presence of a single light source emitting light.

[0120] For example, this embodiment for passive acquisition and processing of timestamp streams from a free-running SPAD has many applications for dynamic imaging, e.g., • Completely asynchronous single-shot observation of ultrafast phenomena using multiple light sources across different time scales. • Passive depth imaging using non-collaborative ambient light sources. • Compressed ultra-high-speed video recording using sparse photon counting, SPAD enables the monitoring of intensity fluctuations over time scales of approximately 10 orders of magnitude (e.g., DC to 31 GHz), which can be theoretically captured by a temporal "microscope". • Many other uses as understood by those skilled in the art It has.

[0121] Furthermore, this embodiment can be used in any appropriate computer vision technique or image processing technique based on flux data generated by a sensor, for example, in detection, tracking, recognition, reconstruction, or navigation.

[0122] In this embodiment, the probing function is generally described as a Fourier basis function, but in further cases, the probing function may be, for example, a wavelet basis function or a function learned by a deep neural network.

[0123] While this disclosure generally describes the reconstruction and processing of the flux function on a pixel-by-pixel basis, it should be understood that receiving multiple asynchronous or partially asynchronous single-photon detections and an absolute timestamp associated with each of the received single-photon detections includes receiving single-photon detections and their associated absolute timestamps from a group of pixels. A group of pixels includes neighborhoods of pixels on a sensor, patches of pixels on a sensor, or all pixels on a sensor. In such cases, the continuous-time flux function is a three-dimensional flux function that includes two-dimensional pixel coordinates and one-dimensional time.

[0124] It is understood that the system and method of this embodiment can be further integrated into other applications. For example, they can be integrated into systems for microscopes, telescopes, LiDAR (Light Detection and Ranging), or free-space optical communication. This embodiment can also receive detected photons from any suitable source, such as one or more lasers, light-emitting diodes, projectors, natural light, or indoor or outdoor artificial light sources.

[0125] Although the present invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended herein. The entire disclosure of all the references mentioned above is incorporated herein by reference.

Claims

1. A processor implementation method for single-photon imaging, Receiving multiple asynchronous or partially asynchronous single-photon detections and an absolute timestamp associated with each of the received single-photon detections, To determine the sum of the values ​​of the probing function for the single-photon detection value in each of the aforementioned absolute timestamps, Reconstructing the continuous-time flux function from the aforementioned probing measurements, To output the reconstructed continuous-time flux function or its integral. Methods that include...

2. The method according to claim 1, wherein the probing function is performed on frequencies from 0 to the maximum recoverable frequency, or a subset thereof, and each of the frequencies is separated by a frequency step.

3. The method according to claim 2, wherein the frequency step is determined as 0.6 / t, where t is the total acquisition time.

4. The method according to claim 1, wherein the probing function approximates the continuous-time flux function to an additive noise function.

5. The method according to claim 1, wherein the probing function is a continuous-time probing function.

6. The method according to claim 1, wherein the probing function is a Fourier basis function, and the continuous-time flux function is reconstructed from the amplitude and phase of the detected frequency.

7. The Fourier basis function is p ｆ (t) = e －ｊ２πｆｔ The method according to claim 6.

8. The method according to claim 1, wherein flux frequencies exceeding 1 / 2q are ignored, and q is a given timing resolution.

9. The method according to claim 1, further comprising identifying and removing noisy frequencies based on a predetermined probability of false alarms.

10. The method according to claim 9, wherein the predetermined probability of a false alarm is determined by detecting whether a given probing frequency contributes to the continuous-time flux function by determining whether the corresponding amplitude is greater than a predetermined threshold.

11. The method according to claim 1, wherein reconstructing the continuous-time flux function comprises summing the corresponding Fourier basis functions, scaled by amplitude and shifted by associated phase, over the detected frequency.

12. Reconstructing the continuous-time flux function means [Math 1] This includes determining the frequency f is the detected frequency, F ｕｓｅｄ This is the set of detected frequencies used, A ｆ φ is the amplitude at each frequency, ｆ The method according to claim 11, wherein is the phase at each frequency.

13. The method according to claim 12, further comprising identifying one or more dominant frequencies within the set of detected frequencies and adding harmonics of the dominant frequencies to the set of detected frequencies to be used.

14. The method according to claim 1, wherein receiving the plurality of asynchronous or partially asynchronous single-photon detections and the absolute timestamp associated with each of the received single-photon detections includes receiving single-photon detections and the associated absolute timestamps from a group of pixels, the continuous-time flux function includes a three-dimensional flux function.

15. The method according to claim 14, wherein the group of pixels includes a neighborhood of a pixel on the sensor, a patch of pixels on the sensor, or all pixels on the sensor.

16. The method according to claim 1, wherein the reconstructed continuous-time flux function or its integral is used for a computer vision or image processing task.

17. The method according to claim 1, wherein the absolute timestamp is the frame number in the sequence captured by the image sensor.

18. A system for single-photon imaging, the system comprising one or more processors and data storage, the data storage being, An imaging module for receiving multiple asynchronous or partially asynchronous single-photon detections and an absolute timestamp associated with each of the received single-photon detections from a single-photon detector, A probing module for performing probing measurements to determine the sum of the values ​​of the probing function for the single-photon detection value at each of the aforementioned absolute timestamps, A reconstruction module for reconstructing the continuous-time flux function from the aforementioned probing measurements, An output module for outputting the reconstructed continuous-time flux function or its integral, A system comprising instructions for which one or more processors execute.

19. The system according to claim 18, wherein the probing function is performed for frequencies from 0 to the maximum recoverable frequency, and each of the frequencies is separated by a frequency step.

20. The system according to claim 18, wherein the probing function is a continuous-time probing function.

21. The system according to claim 18, wherein the probing function is a Fourier basis function, and the continuous-time flux function is reconstructed from the amplitude and phase of the detected frequency.

22. The system according to claim 18, wherein the one or more processors further execute a noise module for identifying and removing noisy frequencies based on a predetermined probability of false alarms.

23. The system according to claim 18, wherein reconstructing the continuous-time flux function includes summing the cosines of the amplitude and associated phase over the detected frequency.

24. Reconstructing the continuous-time flux function means [Math 2] This includes determining the frequency f is the detected frequency, F ｕｓｅｄ This is the set of detected frequencies used, A ｆ φ is the amplitude at each frequency, ｆ The system according to claim 23, wherein is the phase at each frequency.

25. The system according to claim 18, wherein receiving a plurality of asynchronous single-photon detections from the single-photon detector includes performing a scene scan.

26. The system according to claim 18, wherein the single-photon detector comprises a one-dimensional or two-dimensional array of single-photon avalanche diodes.

27. The system according to claim 18, wherein the reconstructed continuous-time flux function or its integral is used for computer vision or image processing tasks.

28. The system according to claim 18, wherein the absolute timestamp is the frame number in the sequence captured by the image sensor.

29. A processor implementation method for single-photon imaging, Receiving multiple asynchronous or partially asynchronous single-photon detections and an absolute timestamp associated with each of the received single-photon detections, To determine the frequency components, we perform interval probing using the non-uniform fast Fourier transform, The interval band is determined using the frequency components determined using the aforementioned non-uniform fast Fourier transform, To determine the sum of the values ​​of the probing function for the single-photon detection value in each of the absolute timestamps, a probing measurement is performed using only the specified interval band, Reconstructing the continuous-time flux function from the aforementioned probing measurements, To output the reconstructed continuous-time flux function or its integral. Methods that include...

Images