System and method for event-based processing of photon flow data from single photon camera sensor

Abstract

A system and method for processing photon flow data received from a single photon avalanche diode (SPAD) image sensor is provided. The image processor receives a stream of binary pixel values from the SPAD image sensor and performs operations of (a) a continuous flux estimation process, (b) an event triggering process in response to the continuous flux estimation process, and (c) a communication process in response to the event triggering to send image data to the host. The continuous flux estimation process may be responsive to an exponential moving average (EMA) process, which may be defined based on an attenuation factor, and / or based on previous values relative to the flow stored in a buffer memory. The EMA may also be based on hyper-parameters including a variance attenuation factor and a flux attenuation factor. The event triggering process may determine when to send an event to the host based on the variation, randomness, and temporal resolution.

Description

Technical Field

[0001] The present invention relates to single-photon camera sensors and data processing of photon flow data generated thereby. Background Art

[0002] Event-based vision utilizes asynchronous updating of pixels in an image, rather than the traditional frame-based updating of image data. Each pixel operates independently and does not have to wait for the global exposure time of the frame before the pixel value is sent. Conventional event cameras typically produce one of three values ​​(+1, -1, or 0) for each pixel, with a high timing resolution of up to 1 microsecond. This value indicates whether the scene illumination has increased (+1), decreased (-1), or has not changed (0). The value is determined by comparing the change in scene illumination to a threshold. The advantages of such sensors are high dynamic range (120 dB compared to 60 dB for frame-based cameras), low power consumption by avoiding the transmission of redundant data, and high temporal resolution due to the independence of each pixel. For useful background information, see Event-Based Vision - A Survey on the World Wide Web at the URL https: / / ieeexplorer.ieee.org / abstract / document / 9138762. These characteristics make these sensors attractive for applications in robotics and wearable electronics, where latency, power consumption, and high dynamic range are important. They facilitate simultaneous localization and mapping (SLAM), optical flow estimation, object tracking, and other downstream algorithms.

[0003] Single-photon cameras based on Single Photon Avalanche Diode (SPAD) arrays (also referred to herein as "single-photon camera sensors" or "single-photon cameras") are sensitive to single photons arriving at the sensor. In binary photon counting mode, the value recorded by a SPAD pixel is either 1 (if the pixel detects at least one photon during the exposure time) or zero (if no photon is detected). The probability of a photon arriving at a pixel is proportional to the level of the light flux (or photon flux) at the principal point on the image plane. The arrival of photons follows a Poisson distribution and is affected by other types of noise such as dark counts and thermal noise.

[0004] Single-photon cameras outperform conventional event cameras and exceed them on various metrics such as temporal resolution (from microseconds to picoseconds), better performance in low light (due to higher photon sensitivity), and higher event density (106 events / s to 109 counts / s). For useful background information, see P. Lichtsteiner, C. Posch, and T. Delbruck. A 128×128 120dB 15μs Latency Asynchronous Temporal Contrast Vision Sensor. IEEE J. Solid-State Circuits, 43(2), 2008, and Arin Can Ulku, Claudio Bruschini, Ivan Michel Antolovic, Yung Kuo, Rinat Ankri, Shimon Weiss, Xavier Michalet, and Edoardo Charbon. A 512x512 SPAD image sensor with integrated gating for widefield. IEEE Journal of Selected Topics in Quantum Electronics, 25(1): 1–12, January 2019. Furthermore, event cameras utilize analog hardware to detect events rather than directly measuring static scene flux levels. This effectively means that event cameras remove the DC term and require additional electronics to sample intensity images directly from the photon well (as a non-limiting background example, see DAVIS346, Inivation Specification Sheet, available on the World Wide Web at URL https: / / inivation.com / wp-content / uploads / 222 / 08 / 2022-08-iniVation-devices-Specifications.pdf). In contrast, single-photon cameras can directly reconstruct a static scene by accumulating photon counts.

[0005] A limitation of single-photon cameras is the much higher amount of data required to be read from the sensor compared to event cameras. This higher data volume occurs because the data rate of a conventional event camera scales only with the amount of motion (brightness changes) in the scene, whereas the data rate of a single-photon camera scales with the absolute brightness of the scene. For example, a static scene will not result in any data transfer for a conventional event camera. This results in higher power consumption and creates an information bottleneck for single-photon cameras.

[0006] More specifically, some SPAD arrays can capture binary frames at very high frame rates (>100KHz). When this data is read out at 100k frames per second (fps), it results in a bandwidth requirement of about 4GB / s for a moderate resolution of 512x512 pixels, where each pixel is represented by 1 bit. The USB 3.0 standard for connecting the SPAD and the host supports 600MB / s, which is an order of magnitude lower than the requirement. Future developments in SPAD sensors take into account increases in resolution. Reaching 4-8 megapixels (MP) in the near future will further widen the gap between required and available bandwidth by at least another order of magnitude. Therefore, it is desirable to provide a system and method that can efficiently transfer data from the sensor to the host and obtain the full advantages of a single-photon camera. Summary of the Invention

[0007] The present invention overcomes the shortcomings of the prior art by providing systems and methods for event-based processing based on binary frames (photon streams) captured by a SPAD array, which significantly reduces redundant data transmission over the communication channel. The procedures herein run on the single-photon sensor chip before the data is sent off the chip, and can reduce the amount of data transmitted (and therefore, power requirements) and accommodate more data in the communication channel. In addition, the events generated by these procedures from single-photon cameras contain the actual brightness values ​​of the pixels and therefore contain substantially more information than conventional event cameras, while still benefiting from higher temporal resolution and better low-light performance. The method is designed to run a program that intelligently accumulates the binary observations to estimate the true brightness level of each pixel and determines when the brightness has changed significantly, thereby warranting its value to be transmitted from the sensor. The system and method also provide SPAD-specific modulation techniques to increase the dynamic range of images that the SPAD can capture.

[0008] In an illustrative embodiment, a system and method for processing data received from a single-photon avalanche diode (SPAD) image sensor is provided. An image and data processor receives a stream of binary pixel values ​​from the SPAD image sensor. The image and data processor operates a continuous flux estimation process, an event-triggered process responsive to the continuous flux estimation process, and a communication process for transmitting image data to a host in response to the event triggering. Exemplarily, the continuous flux estimation process may be responsive to an exponential moving average (EMA) process, wherein the EMA may be defined based on a decay factor and / or based on previous values ​​relative to the stream stored in a buffer memory. The EMA may be based on hyperparameters including a variance decay factor and a flux decay factor. The event-triggered process may be constructed and arranged to determine when to transmit an event from the sensor to the host based on variance, randomness, and temporal resolution. The event-triggered process may include an adaptive threshold that controls sparsity based on bandwidth and lux level to reduce drift and maintain a desired signal-to-noise ratio. Exemplarily, the host is adapted to utilize the image data for image motion tracking and optical flow estimation. Image motion estimation may be used for at least one of object tracking and SLAM. A continuous flux estimation process may include a recurrent neural network adapted to determine a continuous flux estimate and select when to send an update event. The continuous flux estimation process may be adapted to utilize local spatiotemporal context to calculate an EMA and when to trigger an update event. The continuous flux estimation process may include multiple EMA processes operating in parallel to provide the continuous flux estimate. The system and method may perform spatial preprocessing of the image data prior to operating at least one of the continuous flux estimation process and the event triggering process, wherein the output of the spatial preprocessing is used to determine thresholds and other parameters for the event triggering process. Alternatively or additionally, the preprocessing process may utilize a motion projection image that performs a temporal integration along at least a scene velocity direction. A data compression process may compress received data prior to operating at least one of the continuous flux estimation process and the event triggering process. The data compression process may include at least one of run length encoding, a wavelet transform, and a discrete cosine transform (DCT).

[0009] In an illustrative embodiment, a system and method for processing data received from an image sensor assembly is provided. An image and data processor receives a stream of binary pixel values ​​from the image sensor assembly, the image and data processor being operated. A continuous flux estimation process operates on the image data, and an event triggering process is responsive to the continuous flux estimation process. A communication process transmits the image data to a host in response to the event triggering. The image sensor assembly is one of various types of photon counting sensors. More specifically, the image sensor assembly may include at least one of a point-based device, a photomultiplier tube, a CMOS sensor, and a CCD sensor, as well as other types of components that derive the image data stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The following description of the invention refers to the accompanying drawings, in which:

[0011] Figure 1 is a schematic diagram of a typical SPAD camera arrangement for use with the systems and methods herein;

[0012] Figure 2 is a block diagram showing the overall process in the system and method in which binary observations from a photon cube are passed through a flux estimation process to track continuous flux estimates and then through an event triggering process to determine events;

[0013] Figure 3 is a block diagram illustrating a change point process including main functional modules to calculate flux and trigger events based on flux estimates and buffer values;

[0014] Figure 4 is a block diagram illustrating an exponential moving average (EMA) calculation function module for continuous flux estimation;

[0015] Figure 5 is a block diagram showing a continuous flux estimation functional module with a single EMA;

[0016] Figure 6 is a block diagram illustrating event triggering functional modules on a SPAD image sensor arrangement to determine when to send events from the sensor to a host; and

[0017] Figure 7 yes Figure 5 Block diagram of the transformation to logarithmic function used during flux estimation. DETAILED DESCRIPTION

[0018] I. Overview

[0019] Figure 1An exemplary single-photon avalanche diode (SPAD) camera arrangement 100 for use in conjunction with the systems and methods described herein is depicted. The camera 100 includes a conventional lens assembly 110 with a fixed or variable focus and / or aperture that focuses light from an imaged scene onto a sensor defining a pixel array that operates according to a known and / or custom SPAD process to deliver a binary image stream in the form of frames 122 to an image processor 130, which, in an exemplary embodiment, may comprise a field-programmable gate array (FPGA). The functional elements / modules of the processor 130 may be arranged in a manner apparent to one skilled in the art of image processor and camera design. Among the other functional modules / processes (processors), there is an event process (processor) 132 that implements the systems and methods described herein. The FPGA also includes or is linked to a data communication interface, such as a USB 3.0 hub 134. The hub 134 operates conventional and / or custom software / firmware that allows data associated with asynchronous events 136 to be communicated between the camera arrangement 100 and a host computing device 150. Host computing device 150 may be a general-purpose or specialized computer (e.g., a PC), server, laptop, tablet, and / or smartphone). Among other things, it operates various host processes / processors 152 associated with processing received asynchronous events and transmitting control data 138 to camera 100. A user may interact with host 150 and / or camera 160 via user interface UI 170 (e.g., a touch screen, keyboard, mouse, etc.).

[0020] The transmission of asynchronous events (136) involves detecting a change point in the photon flux at each pixel, defined as a spatiotemporal location in the photon cube (i.e., SPAD raw data, x, y spatial resolution, photon arrival time T, resulting in a data volume indexed by x, y, t) where the brightness value of the pixel changes significantly over time. An event is defined as an asynchronous update of the pixel value in the frame using the current flux estimate when a change point occurs. An illustrative implementation of the events can use sign-based events (-1, +1, 0), which are commonly used for event cameras, but can also transmit the current flux estimate at the pixel when the event occurs. By way of useful background, Gallego, Guillermo et al., "Event-based vision: Asurvey". IEEE Transactions on Pattern Analysis and Machine Intelligence 44.1 (2020): 154-180. Since each update of a SPAD-based event contains the current flux value with high precision (greater than 1 bit), it can be considered a "lossless" event camera. For conventional event cameras that emit only low-precision updates, accurately reconstructing the original grayscale values ​​(corresponding to the photon flux) on the image sensor is challenging and typically requires computationally expensive optimization and machine learning algorithms. Lossless asynchronous updating has been demonstrated in earlier asynchronous event-based cameras using analog circuits, but the use of digital hardware in SPAD electronics makes lossless asynchronous updating feasible (lossless event cameras require special analog circuits that are not present in commercial devices). For additional background, see Gallego (cited above) and C. Posch, D. Matolin, and R. Wohlgenannt, "An asynchronous time-based image sensor," 2008 IEEE International Standard for Circuits and Systems. 2008, pp. 2130-2133, doi:10.1109 / ISCAS.2008.4541871.

[0021] In the illustrative system and method, digital filters are applied instead of analog integrator and comparator logic to accumulate photons and detect events. The basic frame at the beginning triggers an event at each pixel. Given the basic frame and the event sequence, the current frame can be reconstructed by updating the pixel values ​​accordingly with the latest event arrival of the pixel value. Figure 2 and Figure 3 As shown, the detection of change points and the transmission of events occur in a two-step process.

[0022] refer to Figure 2, the process 200 executes binary observations from the photon cube 200, tracks a continuous flux estimate 222 through a flux estimation algorithm / process 220, and then determines an event 240 through an event triggering algorithm / process 230. The event 240 then performs a masking process 250 relative to the continuous flux estimate 222 to form an output 260, which is transmitted via a communication channel 270 (e.g., a USP 3.0 interface). In summary, the calculation of the continuous flux estimate (variations of which are described below) provides (a) a single-scale flux estimate; (b) a multi-scale flux estimate; (c) a fixed signal-to-noise ratio (SNR) flux estimate; and (d) a high dynamic range (HDR) flux estimate using SPAD gate modulation; and (e) triggering and transmission of events.

[0023] Reference Figure 3 , shows a change point process 300 according to an illustrative embodiment. It includes the main components for calculating flux 310 from a per-pixel data stream 312 and triggering an event 320 based on the flux estimate and the buffer value. A multiplier 330 combines the flux calculation 310 and the trigger event 320 to generate a flux event 340 (which may include a value of approximately 10 bits).

[0024] By way of further background, it should be noted that some variations of event logic have been implemented as prior implementations at the sensor level in event cameras [see Gallego et al., Posch et al.], where temporal pixel differences are thresholded to create binary event data, thereby compressing visual information. However, in these sensors, the logic is implemented on-chip in analog circuits, thereby limiting the flexibility of the computational components by “hardwiring” multiple design choices (e.g., integration time, event strategy, thresholds, and output data format, which is typically limited to binary event updates). A key difference of the illustrative systems and methods compared to the prior art is the ability to digitally integrate single-photon data “post-capture” in a flexible manner using simple computational components that can be computed in silicon close to each pixel within the same integrated circuit. Digital logic provides a number of benefits:

[0025] Adaptive Integration - After photons are collected, the integration time or exposure time can be modulated in digital hardware. This enables algorithms like the following to adaptively change the integration time used to accumulate photons for more effective and efficient change detection or event triggering under a wide range of lighting conditions.

[0026] Multi-buffering - Since the raw photon counts can be easily copied and manipulated digitally, algorithms with multiple simultaneous exposures can be used. This would require additional readout from the ADC using analog integration, which would be subject to additional read noise and increased circuit complexity.

[0027] Adaptive event trigger thresholds - Conventional event sensors have hardwired thresholds that the comparator uses to trigger on events that are not easily changed based on the logic around the pixel. Digital hardware supports trigger thresholds that can adapt to information such as expected "events per second", "flux variance", and other metrics that the digital hardware can easily calculate.

[0028] "Lossless" High-Precision Events - Conventional event sensors utilize a comparator between two analog signals to output low-precision events (+1, 0, -1), while digital hardware is capable of communicating high-precision changes in the grayscale value of a pixel within a bit depth limited only by the available digital logic in the integrated circuit. If low-precision events are desired, the illustrative system and method requirements can be easily adapted to provide these events as well.

[0029] II. Continuous Flux Estimation Process

[0030] A. Glossary—The following terminology applies to the systems and methods herein:

[0031] Binary frame: If no photons are detected while the SPAD gate is open, the frame contains a "0", and if at least 1 photon arrives within that window, the frame contains a "1" (additional photons do not increment, as this is a 1-bit counter).

[0032] Average counts per frame, μ c : Over some virtual integration time, the system can estimate the average photon count per frame, which is a number in the interval [0, 1]. For a count "c" over "N" frames, this is just However, the system can also use the exponential moving average (EMA) to generate a running average to estimate μ c .

[0033] Virtual Integration Time (VIT): Since photon counting sensors like SPADs can accumulate photon counts digitally, the VIT can be expressed as the number of frames N, which the system can use to calculate the average count, μ c EMA can be used as follows. For N binary frames, the effective integration time is α is the decay factor of EMA.

[0034] Binary frame variance, σ c : The expected variance of the binary frame values ​​due to shot noise and pixel gating. For binary frame based SPAD readout as described throughout this document, and p is the real photon flux when φ is the exposure time of each binary frame, η is the quantum efficiency of the SPAD sensor, and r qis the probability that no photon is detected when the dark count rate is given. Although φ is unknown, standard methods can be used to solve the equation for φ Estimated from photon counting data, as additional background, Sizhuo Ma, S Gupta, Arin C. Ulku, Claudio Bruschini, Edoardo Charbon, and Mohit Gupta, "Quanta Burst Photography," 2020 Proc. ACM SIGGRAPH and Ivan MichelAntolovic, Claudio Bruschini, Edoardo Charbon, "Dynamic range extension for photon counting arrays." 2018 Optics Express 26(17). Note that p can be estimated as μ c ,therefore

[0035] The variance estimate of the mean count per frame, Given μ c The system can calculate the variance of this estimate, i.e., the variance of the average count per frame, such as by taking a running average of the running counts. For example, if μ c is calculated as the average count over N binary frames, then the variance is

[0036] The continuous flux estimation algorithm attempts to calculate μ with sufficiently low variance c , so that one can observe how it changes with time and how it changes due to shot noise (from ) and changes more than expected. For the illustrative event-based update scheme, it is assumed that the underlying photon flux is constant, and it is only desired to send the updated flux when the flux constancy assumption is violated. Therefore, in the "Event Triggered Algorithm" section below, some thresholds can be provided to detect when the expected flux has changed significantly.

[0037] B. Single-scale flux estimation algorithm

[0038] The continuous flux estimation algorithm is adapted to calculate the instantaneous expected value of the flux level μ for each pixel using each new observation s Sampling these flux values ​​for all pixels at any given time step will provide the current flux expectation for the entire image at that time step. The program specifically uses exponential moving averages (EMA), first-order infinite impulse response (IIR) filters, such as Figure 4As shown in Figure 4, the EMA function 400 takes as input the previous value of the EMA 410, a damping factor 420 that modulates the weight assigned to the new value 430, and the error 440 between the new EMA value 430 and the previous value 410. A multiplier 450 operates on the damping factor 420 and the error 440 and adds 460 the product to the old value 410, generating the output, the new EMA value 430. The EMA has minimal memory overhead, which is necessary when running the algorithm in embedded systems and edge devices with limited memory. An independent EMA operating on each pixel requires only the memory overhead of a single buffer to store the result. The size of the buffer is equal to the frame and proportional to the precision used in the buffer's data type. It has a single parameter to be set: the damping factor α, which indicates how much weight each new observation should be given relative to past observations when calculating the expected value. Generally speaking, when sufficient memory is available, a high-order IIR filter can be used to calculate the flux estimate.

[0039] μ c,0 =c t

[0040] μ c,t =α t c t +(1-α t )μ c,t-1 (1)

[0041] where c t is the binary observation at time t.

[0042] The trade-off between a high attenuation factor and a low attenuation factor is that a low attenuation factor results in a higher signal-to-noise ratio (SNR) due to averaging more observations, while a high attenuation factor makes it more resilient to rapid changes in flux levels. Therefore, when the true flux level of the scene is constant (static scene) or changing slowly, a low attenuation factor will result in a reliable estimate. As the speed of motion increases, the light intensity changes more rapidly, resulting in increased motion blur in the frame. This is similar to the long exposures in conventional cameras. Similarly, when there is motion in the image, a high attenuation factor will result in a sharper estimate of the image, but will be more sensitive to the shot noise inherent in the image. This is similar in nature to the short exposures in conventional cameras.

[0043] EMA with an adaptive attenuation factor is used to mitigate motion blur while having a high signal-to-noise ratio. The attenuation factor indicates the length of the integration used (the number of time steps that effectively affect the EMA). The goal is to have long integration times (low attenuation factors) in areas of low pixel value variation and short integration times (high attenuation factors) in areas of high pixel value variation. The attenuation factor α for each pixel is recalculated at each time step and compared to the flux value σ at that time step. sThe idea is to effectively reset the flux estimate when there are sudden brightness variances, such as edges. When an edge is present, the variance of the incoming photon values ​​will be high, causing the flux EMA decay factor to increase. This will generally make it more sensitive to incoming photons, effectively resetting its value. To obtain a reliable variance estimate, an exponentially weighted moving variance (EWMV) similar to EMA is used for variance estimation, which is used to estimate σ c The EWMV has a fixed decay factor β determined experimentally. The single-scale flux estimation process 500 is Figure 5 , and more specifically, a continuous flux estimation with a single EMA. Function 500 takes as input a photon stream 510 and outputs a flux estimate 520. The hyperparameters 530 that are set are the variance attenuation factor 532 and the flux attenuation factor 534 for the EMA and EWMV, respectively. The default values ​​533 and 535 can be 0.8 and 1.0, respectively. The function also uses two buffer memories 540 for calculating (by the calculation module 550) the EMA and EWMV across time steps. Specifically, for a 4-bit input stream, the calculation module 550 converts the input stream 510 to logarithmic values ​​using the convert to log function 560 and subtracts 562 the stored flux estimate 563 from it. Figure 7 An example of a conversion to logarithmic function 570 is depicted. Its output is multiplied by 2 (566) and input, along with an 8-bit variance estimate of, for example, 0.25, to generate a variance EMA 570, which is stored in a buffer memory 540 as an updated variance estimate 564 and input to a square root operator 572 and a multiplier 574, which receives hyperparameters 530 as operands. This product is provided along with the subtraction product 562 and the stored flux estimate 563 to generate a flux EMA 580 which serves as a continuous flux output 520. This function 500 is run for each pixel and each time step at which a photon arrives at the sensor. The attenuation factor is modulated by the EWMV. As described above, the system initializes the EWMV to "0.25," but any other suitable value may be used in alternative embodiments.

[0044]

[0045] and the flux attenuation factor α is set to

[0046] α t =σ c,t , (3)

[0047] and cut α t , so that it remains between [0,1].

[0048] Brief Reference Figure 7, shows the conversion to a logarithmic function 570. This function receives the 4-bit pixel stream 510 and divides the value 710 by the maximum 4-bit value (e.g., 15). The result is provided to a subtractor 720, which subtracts it from 1 and provides it to a logarithmic function 730. The logarithmic product is again subtracted from 1 740 and provided to another logarithmic function 750 to provide an 8-bit logarithmic photon output 760.

[0049] C. Multi-scale flux estimation process

[0050] In the above procedure, a single EMA is used, and the attenuation factor is modulated by EWMV. Another approach is motivated by the desire to integrate the image for a long duration to improve SNR, and to use shorter integration times when there are intensity gradients (such as edges). Two parallel EMAs are calculated at each time step. EMA1 uses a smaller attenuation factor α1 and results in a longer integration time, and EMA2 uses a larger attenuation factor α2 and results in a shorter integration time. The two EMAs are defined by an attenuation spread that describes the ratio of small and large attenuation factors. Therefore, at each time step t, the procedure linearly combines EMA1 and EMA2 using a weighting factor w to calculate a continuous flux estimate μ at that pixel. c The weighting factor is calculated using the ratio of the cumulative error to the cumulative error in the time window N. The cumulative error B t The average difference between the current error and the error N steps ago is given. Cumulative error A t The absolute difference between the current error and the error N steps ago is given. The current error, ε, is defined by the difference between the current frame and the previous frame. This ratio is a way to normalize pixels and identify trend changes while being robust to small changes due to the variance of the Poisson distribution (photon arrivals). A trend change is a change in the true mean of photon arrivals, while the actual arrival values ​​have variance. When there is a large trend change, it is desirable to weigh shorter integration times more heavily, while when there is a small trend change, it is desirable to weigh longer integration times more heavily.

[0051] ε=c t -c t-1

[0052]

[0053]

[0054] μ c,t =w t EMA1+(1- w t )·EMA2 (4)

[0055] where μ c,tis the estimated mean count per frame at time step t.

[0056] D. Fixed Signal-to-Noise Ratio (SNR) Flux Estimation Process

[0057] A key advantage of the photon counting approach is that the system can easily modulate the virtual integration time of each pixel so that the expected SNR of the photon counts is the same across all pixels. The system can measure the observed photon flux in units of "counts (photons) per second," or CPS. Since the photon counts over a given time interval are a Poisson process, changing the integration time changes the expected error, also known as shot noise. Below, we describe how the integration time can be varied so that each pixel has the same SNR over a wide range of incident photon fluxes. This provides a natural way to choose the integration time so that downstream processing that is sensitive to SNR, such as object detectors, works reliably, while also minimizing motion blur caused by integration times that are longer than necessary.

[0058] In a high dynamic range scene, there may be sub-regions of the image with high and low photon flux. Regions of the frame with high photon flux can be integrated for a short period of time, resulting in little motion blur. Pixels observing low photon flux will be integrated for a longer period of time, so these pixels will produce an image with more motion blur.

[0059] In the context of event estimation, the advantage of fixed SNR flux estimation is the ability to apply a simple threshold to all pixels as a way of providing change point detection.

[0060] For a SPAD pixel, it returns “0” if no photon is detected and “1” if one or more photons are detected within a “binary frame” exposure time τ, which indicates that the total integration time for the flux estimate is “N”, which is the total number of binary frames used to produce the sum of observed photons.

[0061] To choose "N" that produces a desired SNR, consider the inverse of the SNR or the coefficient of variation The integration time "N" can be chosen to produce a CV for some value of δ. This value corresponds to the expected deviation if the real photon flux is constant for a given "N". For the desired sensitivity of the CPS, Δ, the process sets The updated value of "N" that produces the expected CV of δ is shown below. Write the corresponding update equation to describe the update given the new observation μ c,t In this case, how N should be updated, the following applies:

[0062]

[0063] therefore,

[0064]

[0065] Therefore, for each new μ c The new value of "N" is chosen so that the coefficient of change is close to δ. Each pixel calculates this value independently so that each pixel has its own value of "N" stored in a local buffer. In practice, μ c Low values ​​of produce very large "N", so "N max" is selected which prevents the system from integrating beyond a pre-selected maximum integration time.

[0066] In practice, the system uses EMA for each pixel, similar to the method described above. Therefore, the attenuation factor can be calculated from "N" as:

[0067]

[0068] and the flux estimate is calculated as:

[0069] μ c,t =αc t +(1-α)μ c,t-1 (8)

[0070] This is similar to the EMA above, where c t is the binary frame count.

[0071] The system can also derive the update equation and decay parameter for the “true count”, which uses μ c The maximum likelihood estimate (MLE) of the true photon count is:

[0072]

[0073] For additional background, see Sizhuo Ma, S Gupta, Arin C. Ulku, Claudio Bruschini, Edoardo Charbon, and Mohit Gupta, “Quanta Burst Photography,” 2020 Proc. ACM SIGGRAPH, and Ivan Michel Antolovic, Claudio Bruschini, Edoardo Charbon, “Dynamic range extension for photon counting arrays,” 2018 Optics Express 26(17). Therefore, the variance is:

[0074]

[0075] And "N" is the virtual integration time, similar to the above N t .

[0076] Therefore, the system can use the same procedure as above to calculate the update for N and write:

[0077]

[0078] The advantage of using the true count estimate (φ) is that the soft saturation effects of single-photon counting are properly accounted for. For additional background on soft saturation, see Atul Ingle, A Velten, and Mohit Gupta, “High Flux Passive Imaging with Single-Photon Sensors,” in Proc. IEEE CVPR (Conference on Computer Vision and Pattern Recognition), 2019.

[0079] E. High Dynamic Range Flux Estimation Using Gate Modulation

[0080] As described above, the illustrative flux estimation process has described a method for automatically estimating the photon flux or CPS using digital logic to dynamically change the "virtual integration time" using various metrics of the input photon detection flow. This virtual integration time describes how the per-pixel photon counter is updated when a photon is detected.

[0081] Some SPADs also have a gate that can be modulated in time. This gate controls when the SPAD pixel is sensitive to an incoming photon. In normal operation, the gate is open for the entire single-photon integration time, which is typically 10 microseconds (100k fps). Note that if 1 or more photons arrive during this integration time, the camera returns a "1". Therefore, for a high-throughput environment, it is very likely that multiple photons will arrive within this 10 microsecond window, saturating the detector. However, the gate can be modulated so that it is open for a small portion of the 10 microsecond window, limited only by the gate hardware (typically gate widths as low as 1-10ps). This gate width can be modulated for each exposure, allowing for an interleaved exposure scheme, further increasing the dynamic range.

[0082] In such an arrangement, photon detection can be integrated into buffers in a rotating fashion, with each buffer corresponding to a different gate width. For M gate widths, this process yields M buffers, which can be read out and combined to form an HDR image. Multi-exposure bracketed HDR methods are common in digital photography, but exposures must be performed sequentially. By modulating the gate at a frequency of, for example, 100 kHz, this illustrative system and method enables staggered exposures, avoiding artifacts present in sequential exposures and applicable only to single-photon imaging.

[0083] F. Event triggering process

[0084] When an update requires pixels to be sent from the sensor, an event is triggered. The update value can be the current flux estimate or a code representing the flux change compared to the last emitted flux value. The triggering is based on a strategy that aims to reduce image reconstruction artifacts and noise while increasing the sparsity of data transmission to achieve higher compression. The strategy is designed based on three modules, such as Figure 6 As shown in the program 600 of FIG, the program describes an event trigger function on the sensor to determine when to send an event from the sensor to the host. The function inputs a continuous flux estimate (e.g., a 10-bit value) 602 (via a subtraction module 604) and accordingly employs a calculation module 610 having three modules 612 (in conjunction with a random number generator 613), 614, and 616 based on the variation, randomness, and time resolution. The output is a Boolean value 630 indicating whether the event should be triggered. Hyperparameters 640 (e.g., consisting of a minimum time resolution 642, an event threshold 644, and a random event probability 646) are provided to configure the sensitivity of the module. A buffer memory 650 is provided in association with the calculation module 610 for storing the time step of the last event 652 (a time-related event) and the flux value 654 when the last event was sent. For logical operations, various OR (660), AND (662), and ADD (664) operators are provided. These operations interact (as shown) with a write module 670 that stores previous flux events 654 and a select module 672 that acts on the last event 652 stored in the buffer. The illustrative function 600 runs for each pixel and each time step that a photon arrives at the sensor.

[0085] 1. Flux estimate change (θ t )

[0086] When the continuous flux estimate changes significantly relative to the last event sent for that pixel, an event is triggered to update the reconstruction of the photon cube. It is based on a change threshold set as a hyperparameter, and the event is triggered when the absolute difference between the current estimate and the previous event update for that pixel exceeds this threshold.

[0087] 2. Random Events (R t )

[0088] To account for variability in flux estimates, events are also randomly triggered based on random numbers sampled from a uniform distribution and modulated by the randomness hyperparameter. This also counteracts the possibility of drift, a phenomenon where the flux change per time step is always below the aforementioned event threshold, and therefore no events are sent. By sending random events, it reduces the possibility of drift targeting the camera.

[0089] 3. Minimum time resolution (Tt )

[0090] While the SPAD's frame rate can be set very high, resulting in a maximum temporal resolution of motion in the frame equal to the frame rate, many downstream applications are designed to require lower temporal resolution. Since the algorithm requires sparsity in the transmission of events, a minimum temporal resolution hyperparameter is set to ensure that a pixel update can only be triggered after this minimum time has passed.

[0091] θ t =|μ c,t -μ c；t' |>Threshold

[0092] R t =RAND()<randomness, where RAND() is a random number generator

[0093] T t =(t-t')>time resolution-(9)

[0094] Here, the time step of the previous event that occurred in this pixel is represented by t'.

[0095] E t =(θ t or R t )and T t -(10)

[0096] Among them E t Indicates whether the event is triggered at time step t.

[0097] As mentioned above, each of the three modules has hyperparameters 640 that set the characteristics of the trigger and modulate the sensitivity of the event trigger. The threshold 644, randomness 646 and minimum time resolution 642 are configured based on the application and can be updated in real time.

[0098] III. Process Function

[0099] The following sections provide a step-by-step process for executing the various calculations performed herein. This article provides further descriptions of the above mathematical expressions and more closely reflects the actual runtime implementation of the systems and methods described herein.

[0100] A. Single-scale flux estimation process

[0101] The following steps are employed in the illustrative single-scale flux estimation process:

[0102] (1) Input binary observation value c at time step t t ;

[0103] (2) For each pixel, perform steps 3-8;

[0104] (3) Calculation error ε = c t -c t-1 ;

[0105] (4) Update EWMV using ε and β using the above equation (2) t ;

[0106] (5) Using the above equation (3) using EWMV t Calculate the flux attenuation factor α t ;

[0107] (6) Update EMA using the above equation (1) t ;

[0108] (7) Output flux estimation μ c,t =EMA t ;as well as

[0109] (8) Go to the event triggering process.

[0110] B. Multi-scale flux estimation process

[0111] (1) Input binary observation value c at time step t ;

[0112] (2) For each pixel, perform steps 3-10;

[0113] (3) Calculation error ε = c t -c t-1 ;

[0114] (4) Update EMA1 using ε and α using the above equation (1);

[0115] (4) Update EMA2 using ε and β using the above equation (2);

[0116] (5) Update the cumulative error B using the above equation (4) using ε t ;

[0117] (6) Update the cumulative error A using the above equation (4) using ε t ;

[0118] (7) Using the above equation (4) using B t and A t Calculate the scaling weight w;

[0119] (8) Use w to linearly combine EMA1 and EMA2;

[0120] (9) Output flux estimation μ c,t = linear combination; and

[0121] (10) Go to the event triggering process.

[0122] C. Fixed Signal-to-Noise Ratio (SNR) Flux Estimation Process

[0123] (1) Input binary observation value c at time step t t ;

[0124] (2) For each pixel, perform steps 3-6;

[0125] (3) Update N using the above equation (6) t ;

[0126] (4) Update μ using the above equations (7, 8) c ;

[0127] (5) output flux estimation; and

[0128] (6) Go to the event triggering process.

[0129] D. High Dynamic Range Flux Estimation Using Gate Modulation

[0130] (1) Specify a set of gate lengths: {G0, G1, …GM};

[0131] (2) For N iterations:

[0132] (a) For each gate length, create a local count buffer initialized to 0:

[0133] (i) Set the SPAD gate to the gate length;

[0134] (ii) reading the count value;

[0135] (iii) incrementing the count value in the corresponding buffer;

[0136] (3) Combining local buffers into HDR count estimates;

[0137] (a) Calculate the MLE (maximum likelihood estimate) of the true photon flux for each buffer;

[0138] (b) Take the weighted average of the MLEs, weighted for each estimated SNR:

[0139] (i) in

[0140] (ii)

[0141] (iii) take the SNR of each of the M buffers and take a weighted combination of the buffer flux estimates (e.g., according to SNR_M / sum(SNR));

[0142] (c) using the flux to recover the count estimate of the fully open gates, a real number between [0,1]; and

[0143] (4) Go to the flux estimation algorithm and replace c with the combined count estimate t .

[0144] E. Event triggering process

[0145] (1) Input flux estimate μ at time step t c,t ;

[0146] (2) For each pixel, perform steps 3-10;

[0147] (3) Retrieve μ from the buffer c,t ', where t' is the time step of the previous event;

[0148] (4) Use the above equation (9) to evaluate the change event θ t ;

[0149] (5) Use the above equation (9) to evaluate the random event R t ;

[0150] (6) Use the above equation (9) to evaluate the time event T t :

[0151] (7) Using the above equation (10) using θ t 、R t and T t Check if event E is triggered t ;

[0152] (8) If E t If true, go to step 9;

[0153] (9) Set t' = t; and

[0154] (10) Output event trigger E t wait.

[0155] F. Event Transmission

[0156] The output of the change point process described above is a set of sparse matrices that are expanded along the time domain to reconstruct the denoised photon cube. Asynchronous events can be sent as messages with coordinate components, the updated flux estimate, and the time step when the update occurred (called the Address Event Representation or AER). An alternative method of sending the message is to encode in a cube with a depth equal to the time step, with the updated flux estimate occupying the same coordinates as its source pixel. This will make the x-axis and y-axis equal to the image plane. The remaining values ​​are set to zero. This results in a sparse data cube, which can be further compressed in a spatial manner using techniques such as run length encoding, wavelet transform, or discrete cosine transform (DCT). The spatially compressed cube can then be sent when requested by the host.

[0157] G. Modifications to the above procedures AE

[0158] 1. Spatial preprocessing

[0159] Before continuous flux estimation or event triggering, the input frames can be spatially preprocessed. The output of spatial preprocessing can be used to determine event triggering thresholds and other parameters. An example of such an approach is to run JPEG compression, where the coefficients of the DCT can modulate the event triggering threshold. Since JPEG compression is generally not applicable to binary frames, they can be pre-integrated to save the sum of several frames. When high-frequency features are present, a higher threshold is required because the flux intensity in this area will vary more. Wavelet transforms can also be used to preprocess the frames to further increase their sparsity.

[0160] Alternatively, a pre-processing step can be used to perform a temporal integration of the motion projection image not along the time axis but along the scene velocity. This can reduce events from areas not in focus, thereby simultaneously increasing events and information flow from areas in focus. The focus is defined as the area or object that is clear in the motion projection because it moves in the direction and velocity set by the configuration used to calculate the motion projection.

[0161] 2. Multiple event streams

[0162] A modification to this process involves operating multiple EMAs in parallel for continuous flux estimation. In this example, each EMA has a different decay factor and corresponds to multiple integration times for the same pixel. Each of these EMAs is event-triggered, so the output is a channel of distinct event transmissions that can be combined by the host or used for downstream tasks.

[0163] 3. Adaptive Threshold

[0164] The threshold used to determine whether a change in the flux estimate is large enough to trigger an event and be sent can be adaptive, rather than remaining constant until manually changed. The adaptive threshold can be based on various factors:

[0165] a. Bandwidth-based sparsity control - The threshold can be dynamically changed based on the available transmission bandwidth. The bandwidth can be maximized by lowering the threshold until the upper limit is reached.

[0166] b. Lux Level - The threshold can be varied based on the brightness level in the scene. This is similar in spirit to how auto exposure works.

[0167] c. Avoid drift - The threshold can be modulated by the time since the last event, where the threshold can be lowered over time to increase the probability of an event. When an event occurs, the threshold can be reset to its maximum value.

[0168] d. Maintain SNR - The threshold can be designed to ensure that the same SNR is maintained throughout the frame, so that low signal pixels have a higher threshold to maintain the SNR of the event. This is further described above in the "Fixed SNR Flux Estimation Algorithm". The corresponding threshold is where γ is a parameter that determines the number of standard deviations of error that an event is expected to trigger.

[0169] e. True count flux threshold—In the same reasoning as (d), this threshold is used instead of the true count estimate (Φ). Thus, the threshold becomes and Δ = |Φ′ - Φ|, which is the absolute difference in the estimate of the true flux count. The standard deviation can be calculated using the variance defined above. Come discover

[0170] 4. Recurrent Neural Network (RNN)

[0171] Each pixel can use a neural network to determine a continuous flux estimate and choose when to send an update event. The network's input will be a binary observation at each time step, and the output will be a flux estimate if an event is triggered, or zero to indicate no trigger event. If the data stream is viewed as a time series, the neural network can use a recurrent architecture (e.g., a recurrent neural network, or RNN). RNNs incorporate temporal context, which is crucial for flux estimation and event triggering logic. They can track long-term dependencies, continuously updating internal memory maintained using cell state variables. The state variables are updated with each iteration, allowing a continuous flux estimate to be maintained internally even when no event is triggered (i.e., the output is zero). The update rate is modulated by the RNN's gates. These gates are typically an input gate, a forget gate, and an output gate. The gates tell how much weight to give to new observations in the flux estimate and when to trigger an event. Illustratively, they regulate the flow of information between the input and output.

[0172] The parameters of the RNN are learned through an iterative data-driven approach that optimizes the network output to achieve a goal. Optimization can be performed using a backpropagation method to converge to a minimum. In this case, the goal is to denoise the input stream, where the flux estimate at any given time step is equal to the mean of the true Poisson distribution of photon arrivals. It is expected to track trend changes by being sensitive to signal changes and insensitive to noise variance. The goal can also be set to maximize compression and send the minimum number of events to achieve maximum information transfer. The learning mechanism can also be designed in an end-to-end manner, where the objective function is defined based on a downstream task (such as object detection accuracy). Setting different objectives affects the underlying priors developed by the RNN, which determines the overall behavior of the network.

[0173] The data used to train the RNN can be obtained through video simulation or by collecting the captured data from a SPAD camera and performing continuous flux estimation using conventional denoising methods (such as BM3D). The trigger mechanism can be simulated to study the compression ratio, which can then be used to generate training data for the RNN.

[0174] 5. Block-based local context

[0175] While the above process has been designed and described to run independently on each pixel, the event-triggered algorithm can use local context to improve the triggering mechanism. The purpose of event triggering is to ensure that sharp temporal gradients are captured and reported. Depending on the context of the scene, the temporal gradient can arise from signal changes in the scene or noise changes in the scene. It may belong to interesting image features (such as edges) or noise caused by dark current, shot noise, or thermal noise. Ideally, event triggering should always send image features and never send noise variance. To meet this condition, utilizing local spatial context in small image patches rather than relying only on single pixel level context can help understand whether the change is due to signal or noise and send it accordingly.

[0176] 6. Scope and applicability of various sensors

[0177] Although the above methods and algorithms are presented in the context of SPAD sensors, they are equally applicable to raw data captured by a variety of photon counting sensors, including laptop-based devices and photomultiplier tubes, or even conventional CMOS / CCD cameras with an appropriate frame rate.

[0178] 7. Event Vision Application

[0179] In event-based vision, there is a growing body of work on downstream algorithms and tasks, either independently or in conjunction with conventional cameras. The following is a description of how event single-photon cameras can provide improvements in these applications.

[0180] 8. Applications in SLAM systems for robot navigation and camera tracking

[0181] Some robotics applications require sensors that can acquire information in dynamic brightness changes, motion, and low-light settings. For example, Simultaneous Localization and Mapping (SLAM), a classic robotics method required for robot navigation, can be improved using event cameras due to their high temporal resolution and low motion artifacts. However, since event cameras send little to no information in stationary scenes, a regular camera is required to run alongside the event camera. As useful background information, see the final SLAM available at the URL address https: / / rpg.ifi.uzb.cb / docs / RAL18_VidalRebecq.pdf. High temporal resolution events also help in visual-inertial odometry, where the inertial measurement unit sends at a much higher rate than a regular camera. Regular images with low temporal resolution help in computer vision tasks where high-level features in the image such as edges are required, such as object detection and tracking.

[0182] Event single-photon cameras can bring the advantages of both types of transmission by using periodic full frames (entire images) at low temporal resolution, with interval event updates at higher temporal resolution. Tasks such as object detection can be run on full frames, while the event data is combined with the IMU for better state estimation and fast inter-frame tracking of features. Using a single sensor also eliminates the need for external calibration, which is a major source of error in multi-sensor fusion.

[0183] 9. Optical Flow Estimation

[0184] Dense optical flow methods typically use the constant brightness assumption. Tracking edge motion involves finding image gradients and solving for parameters that estimate the direction and speed of motion based on the gradients (e.g., Lucas-Kanade). As described using the aforementioned algorithm, an event is triggered when the intensity in a pixel changes. The presence of the absolute value of the flux estimate, rather than just its sign, when emitting an event is well-suited to identifying motion priors and areas of the image that may have motion.

[0185] In motion compensation algorithms that process bursts of frames, inter-frame motion is estimated across the entire image to identify regions to warp and merge. By providing priors about the region and direction of motion, event transport can accelerate downstream motion compensation tasks by reducing the amount of compensation required.

[0186] 10. Run on embedded computing platforms

[0187] The process is defined to minimize memory usage and maintain a low computational count. The number of memory registers per pixel is actively constrained; for example, single-scale flux estimation requires two registers per pixel, and multi-scale requires four registers per pixel. Design decisions are made to achieve the goal of seamlessly running the algorithm on an embedded system that can reside on the sensor chip. Coupling the event logic to the sensor is crucial to fully exploit the full benefits of asynchronous updates, such as lower data rates and reduced power consumption.

[0188] IV. Conclusion

[0189] It should be clear that the above-described system and method provide an efficient and robust technique for processing SPAD sensor data using digital methods. This approach allows for desired resolution and novel features, such as robust operation in low light, high dynamic range, and high-speed motion, while avoiding the large amount of irrelevant photon stream data typically generated by SPAD arrays. This approach allows for communication of sensor data over conventional bus-based links and can employ conventional chip design techniques.

[0190] The exemplary embodiments of the present invention have been described in detail above. Various modifications and additions may be made without departing from the spirit and scope of the present invention. The features of each of the various embodiments described above may be appropriately combined with the features of the other described embodiments to provide a variety of feature combinations in associated new embodiments. In addition, although multiple separate embodiments of the apparatus and method of the present invention have been described above, what is described herein is merely illustrative of the application of the principles of the present invention. For example, as used herein, various direction and orientation terms (and their grammatical variations), such as "vertical," "horizontal," "upward," "downward," "bottom," "top," "side," "front," "back," "left," "right," "front," "back," etc., are used only as relative conventions and not as absolute orientations relative to a fixed coordinate system (such as the direction of gravity). In addition, the described processes or processors may be combined with other processes and / or processors or divided into various sub-processes or processors. According to the embodiments herein, such sub-processes and / or sub-processors may be combined in various ways. Similarly, it is expressly intended that any function, process, and / or processor herein may be implemented using electronic hardware, software consisting of a non-transitory computer-readable medium of program instructions, or a combination of hardware and software. Furthermore, limiting terms such as "substantially" and "approximately" are considered to allow for reasonable variations from the stated measurement or value, such as 1-5% variations, that may be employed in a manner such that the element retains the functionality contemplated herein. Accordingly, this description is intended to be illustrative only and does not otherwise limit the scope of the present invention.

Claims

1. A system for processing data received from a single-photon avalanche diode (SPAD) image sensor, comprising: An image and data processor receives a stream of binary pixel values ​​from the SPAD image sensor, the image and data processor performing the following operations: Continuous flux estimation process, an event triggering process responsive to said continuous flux estimation process, and A communication process is triggered to send image data to the host in response to an event.

2. The system according to claim 1, wherein: The continuous flux estimation process is responsive to an exponential moving average (EMA) process.

3. The system according to claim 2, wherein: EMA is defined based on a decay factor.

4. The system according to claim 3, wherein: The EMA is based on previous values ​​relative to the stream stored in a buffer memory.

5. The system according to claim 4, wherein: The EMA is based on hyperparameters including a variance decay factor and a flux decay factor.

6. The system according to claim 5, wherein: The event triggering process is constructed and arranged to determine when to send events from the sensor to the host based on variance, randomness, and time resolution.

7. The system according to claim 6, wherein: The event triggering process includes an adaptive threshold that controls the sparsity based on bandwidth and lux level, thereby reducing drift and maintaining the signal-to-noise ratio at a desired level.

8. The system of claim 1, wherein the host is adapted to use the image data for image motion tracking and optical flow estimation.

9. The system according to claim 8, wherein: The image motion estimation is used for at least one of object tracking and SLAM.

10. The system according to claim 1, wherein: The continuous flux estimation process includes a recurrent neural network adapted to determine a continuous flux estimate and to select when to send an update event.

11. The system according to claim 2, wherein: The continuous flux estimation process is adapted to exploit the local spatiotemporal context to calculate the EMA and when to trigger update events.

12. The system according to claim 2, wherein: The continuous flux estimation process includes a plurality of EMA processes operating in parallel to provide the continuous flux estimate.

13. The system of claim 1 , further comprising performing spatial pre-processing of the image data prior to operating at least one of the continuous flux estimation process and the event triggered process, wherein The output of the spatial pre-processing is used to determine the thresholds and other parameters of the event triggering process.

14. The system of claim 1 further comprising a pre-processing process that employs motion projection images to perform a temporal integration along at least a scene velocity direction.

15. The system of claim 1, further comprising a data compression process that compresses received data prior to operation of at least one of the continuous flux estimation process and the event triggering process.

16. The system of claim 15, wherein the data compression process comprises at least one of run length encoding, wavelet transform, and discrete cosine transform (DCT).

17. A system for processing data received from an image sensor assembly, comprising: An image and data processor receives a stream of binary pixel values ​​from the image sensor assembly, the image and data processor performing the following operations: Continuous flux estimation process, an event triggering process responsive to said continuous flux estimation process, and In response to an event triggering a communication process of sending image data to the host, The image sensor component is one of multiple types of photon counting sensors.

18. The system according to claim 17, wherein: The image sensor assembly includes at least one of a point-based device, a photomultiplier tube, a CMOS sensor, and a CCD sensor.

19. A method for processing data received from a single photon avalanche diode (SPAD) image sensor, comprising: receiving, with an image and data processor, a stream of binary pixel values ​​from the SPAD image sensor; estimating continuous flux using the image and data processor; estimating a triggering event based on the continuous flux using the image and data processor; as well as Image data is sent to the host in response to the triggered event.

20. The method according to claim 19, wherein The estimating step includes determining the continuous flux estimate using a recurrent neural network and selecting when to send an update event.

21. The method according to claim 19, wherein The estimating step is responsive to an exponential moving average (EMA) and further includes employing a local spatiotemporal context to calculate the EMA and trigger an update event.

22. The method according to claim 21, wherein The estimating step includes operating a plurality of EMA processes in parallel to provide the continuous flux estimate.

23. The method of claim 19, further comprising performing spatial preprocessing of the image data prior to operating at least one of the estimating step and the triggering step, and using an output of the spatial preprocessing to determine thresholds and other parameters for the triggering step.

24. The method of claim 19, further comprising pre-processing the motion projection images by performing a temporal integration along at least a scene velocity direction.

Images