Event detection circuit, event detection sensor, and event detection method

Abstract

The present disclosure generally relates to an event detection circuit configured to: acquire a photon detection signal indicating that a photon is incident on a photodetection unit; associating the photon detection signal with a photon detection time period; and detecting an event if it is detected that there is a difference between the photon detection signal and the number of associations of a photon detection time period and the number of associations of a previous event within a predetermined integration time period.

Description

Technical Field

[0001] This disclosure generally relates to event detection circuits, event detection sensors, and event detection methods. Background Technology

[0002] Typically, time-of-flight techniques are known. For example, in SPAD (single-photon avalanche diode)-based time-of-flight (ToF) technology, the distance to a scene or the acquisition of a depth image of a scene can be determined by measuring the round-trip time of the emitted light.

[0003] Event sensing (e.g., based on dynamic vision sensors (DVS)) can be used to determine events in a scene, that is, to determine whether the scene has changed.

[0004] When combining these two technologies, it is known to provide a hybrid sensor, namely a sensor with both ToF pixels and DVS pixels, all of which can be arranged in a predetermined pattern. Another way to combine these two technologies is to provide two separate sensors whose positions can be calibrated (i.e., to make them have the same field of view).

[0005] Although techniques exist for acquiring ToF measurements and for event sensing, it is generally preferable to provide an event detection circuit, an event detection sensor, and an event detection method. Summary of the Invention

[0006] According to the first aspect, this disclosure provides an event detection circuit configured to:

[0007] Acquire a photon detection signal, which indicates that a photon has been incident on the photoelectric detection unit;

[0008] Correlating the photon detection signal with the photon detection time period; and

[0009] An event is detected if, within a predetermined integration time period, a predetermined difference is found between the number of associations between a photon detection signal and a photon detection time period and the number of associations between previous events.

[0010] According to a second aspect, this disclosure provides an event detection sensor, comprising:

[0011] Multiple photoelectric detection units arranged in an array; and

[0012] The event detection circuit is configured as follows:

[0013] For multiple photoelectric detection units, a photon detection signal is acquired, which indicates that a photon is incident on one of the multiple photoelectric detection units;

[0014] Correlating the photon detection signal with the photon detection time period; and

[0015] For at least one of a plurality of photoelectric detection units, if a predetermined difference exists between the number of photon detection signals associated with a photon detection time period and the number of previous events within a predetermined integration time period, then an event is detected.

[0016] According to a third aspect, this disclosure provides an event detection method for operating an event detection circuit, the method comprising:

[0017] Acquire a photon detection signal, which indicates that a photon has been incident on the photoelectric detection unit;

[0018] Correlating the photon detection signal with the photon detection time period; and

[0019] An event is detected if, within a predetermined integration time period, a predetermined difference is found between the number of associations between a photon detection signal and a photon detection time period and the number of associations between previous events.

[0020] Other aspects are set forth in the dependent claims, the following description, and the accompanying drawings. Attached Figure Description

[0021] The implementation method is explained by way of example with reference to the accompanying drawings, in which:

[0022] Figure 1 An embodiment of a pixel circuit including an event detection circuit according to the present disclosure is described, using a digital time counting signal;

[0023] Figure 2 Depicting Figure 1 A more detailed view of the event detection circuit depicted in the image;

[0024] Figure 3 An embodiment of a pixel circuit including an event detection circuit according to the present disclosure, using an analog time counting signal, is described;

[0025] Figure 4 An implementation method for optimizing power consumption by utilizing information in signal evolution is described;

[0026] Figure 5 A timing diagram of the system operating under high light levels is depicted;

[0027] Figure 6 Another implementation of a pixel circuit using a digital time counting signal is described, wherein photon counting is also used in the event detection circuit.

[0028] Figure 7A flowchart of an event detection method according to this disclosure is depicted;

[0029] Figure 8 A flowchart of the imaging method according to this disclosure is depicted;

[0030] Figure 9 A timing diagram illustrating the sensing modes and their time consumption according to this disclosure is depicted;

[0031] Figure 10 A schematic illustration of a macropixel capable of performing TOF measurements according to this disclosure is depicted;

[0032] Figure 11 Two hybrid sensors are described, including pixels for TOF, imaging, and / or EVS, wherein the pixels are arranged in different patterns in the respective sensors;

[0033] Figure 12 Implementation methods of the event detection method according to this disclosure are described;

[0034] Figure 13 Another embodiment of the event detection method according to this disclosure is described, wherein time-of-flight imaging is performed after event detection. Detailed Implementation

[0035] From Figure 1 Before proceeding with a detailed description of the implementation methods, a general explanation will be given first.

[0036] As mentioned at the beginning, Time-of-Flight (ToF) technology and event sensing technology are generally known. However, it has been recognized that it may be desirable to implement different functions of the two technologies based on the same pixel structure. Furthermore, it has been recognized that it may also be desirable to implement more functions in the same sensor.

[0037] It has been recognized that event sensing can also be performed using, for example, SPAD-based photon detection / counting techniques, and then used for time-of-flight measurements.

[0038] Therefore, some implementations relate to an event detection circuit configured to: acquire a photon detection signal indicating that a photon is incident on a photodetector unit; associate the photon detection signal with a photon detection time period; and detect an event if, within a predetermined integration time period, a predetermined difference is detected between the number of associations between the photon detection signal and a photon detection time period and the number of associations with previous events.

[0039] The circuit can be based on any entity or multiple entities that can be configured to process signals from photodetector units (e.g., light sensors, pixels, etc.), such as one or more processors (e.g., CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), etc.), wherein combinations of different entities are also contemplated according to this disclosure.

[0040] In some embodiments, a photon detection signal is acquired, which indicates that a photon (i.e., one or more photons) has been incident on the photodetector unit. This means that the photon detection signal can come directly from the photodetector unit or can be processed before being acquired by the circuitry. In some embodiments, the photodetector unit may be included in the event detection circuitry, while in other embodiments, the event detection circuitry and the photodetector unit may be separate components.

[0041] The photoelectric detection unit can be based on any technology suitable for detecting (single) photons (or typically used to determine the number of incident photons), such as SPAD (single-photon avalanche diode), JOT pixel (small pixel with low noise), APD (avalanche photodiode), conventional photodiode, etc.

[0042] In some implementations, the photon detection signal can be associated with a photon detection time period (e.g., the time point or time interval at which detection occurs). For example, an event can be triggered when a predetermined number of photons are incident on the photodetector unit.

[0043] The photon detection time period can correspond to a predetermined time point within the integration time period where the photon detection signals during that period can be summed. When the number of photons detected during the predetermined integration time period exceeds a predetermined threshold, an event is detected.

[0044] The predetermined threshold can correspond to the difference between the number of associations between the photon detection signal and the photon detection time period and the number of associations with previous events.

[0045] In a non-limiting example, the memory stores previous measurements of ten photons. An event is generated when fewer or more than ten photons are incident on the photodetector unit within a predetermined integration time period (e.g., one frame). Therefore, each time a photon avalanche occurs (in the case of a SPAD), it can indicate that photons have been incident on the SPAD. The avalanche signal can thus be processed and associated with the photon detection time period (e.g., in CADTOWF, which will be discussed below). For example, an event can be detected when this process is performed more than a predetermined number of times within a given predetermined integration time period.

[0046] Therefore, according to this disclosure, a SPAD-based event detection circuit can be provided. Compared to known event detection devices, such event detection devices require simulated change detection and are typically noisy or slow-responding under low-light conditions. Furthermore, for example, event detection and imaging can be performed by the same device. For example, imaging can be performed after an event is detected.

[0047] In some implementations, the event detection circuit may be further configured to generate an event detection signal upon detecting an event. For example, this signal may be used to initiate the imaging phase of the photodetector unit.

[0048] In some implementations, the event detection circuit is further configured to (e.g., after an event is detected) utilize a photoelectric detection unit to acquire time-of-flight measurements, as discussed herein.

[0049] In some implementations, the photodetector unit includes a single-photon avalanche diode, as discussed herein.

[0050] In some implementations, the circuit includes at least two accumulators optimized for different frame rates.

[0051] In some implementations, events are detected based on a comparison of the corresponding output signals of at least two accumulators.

[0052] In some implementations, the circuit is further configured to select one output signal from the corresponding output signals of at least two accumulators based on the comparison.

[0053] In some implementations, if no event is detected, the output frame rate corresponds to the lower frame rate of at least two accumulators; and if an event is detected, the output frame rate corresponds to the higher frame rate of at least two accumulators.

[0054] Some implementations relate to an event detection sensor comprising a plurality of photodetector units arranged in an array, and an event detection circuit configured to: acquire a photon detection signal for each of the plurality of photodetector units, the photon detection signal indicating that a photon has been incident on one of the photodetector units; correlate the photon detection signal with a photon detection time period; and, for at least one of the plurality of photodetector units, detect an event, as discussed herein, if a predetermined difference is detected between the number of correlations between the photon detection signal and a photodetector time period and the number of correlations with a previous event within a predetermined integration time period.

[0055] In some implementations, the event detection sensor further includes a data path configured to transmit only signals from photoelectric detection units determined to have caused an event.

[0056] Typically, an event detection sensor may include or be associated with event detection circuitry, as described herein, and repeated descriptions are omitted. However, it should be understood that all aspects described for event detection circuitry are equally applicable to event detection sensors.

[0057] As is commonly known, multiple photodetector units arranged in an array can constitute an image sensor. Furthermore, some of these photodetector units can function as event detectors, while others can function as Time-of-Flight (ToF) imagers, thus enabling the sensor to function as a hybrid sensor.

[0058] Therefore, according to this disclosure, a combination of event sensing, viewing, PCT, and ToF can be performed without the need for hybrid sensors (i.e., sensors with different photodetector elements). Furthermore, compared to conventional ToF sensors, higher resolution can be achieved while reducing power consumption because ToF measurement is only triggered when an event occurs. Additionally, noise can be reduced.

[0059] An event can be triggered individually for each photoelectric detection unit, or if a photoelectric detection unit detects an event, the event can be triggered for all photoelectric detection units or a subset of photoelectric detection units, or it may require at least some photoelectric detection units to detect the event in order to detect the event.

[0060] In some implementations, each of the multiple photodetector units is a single-photon avalanche diode, as discussed herein.

[0061] In some implementations, the event detection sensor is further configured to acquire time-of-flight measurements using multiple photoelectric detection units after an event is detected, as discussed herein.

[0062] In some implementations, (e.g., after an event is detected) at least two modulated signals with different phase delays are applied as analog time codes to a plurality of photodetectors so as to use at least one of the photodetectors to obtain a time-of-flight measurement.

[0063] For example, a zero-degree phase signal can be applied to a first subset, and a ninety-degree phase signal can be applied to a first subset, thereby enabling a measurement. This technique may be referred to as CADTof, as described in patent application WO 2022043480A1, which is incorporated herein by reference.

[0064] It should be noted that the additional effect of the event detection according to this disclosure is compatibility with CADTof technology, as mentioned above, especially since CADTof originates from similar intra-pixel functions and circuitry.

[0065] Generally, it should be understood that CADTofF technology is generally applicable to this disclosure, or that the principles of this disclosure can be applied within a CADTofF system.

[0066] Some implementations relate to an event detection method (for operating an event detection sensor including event detection circuitry), the method comprising: acquiring a photon detection signal indicating that photons are incident on a photodetector unit; associating the photon detection signal with a photon detection time period; and detecting an event, as discussed herein, if a predetermined difference is detected between the number of associations between the photon detection signal and the photon detection time period and the number of associations with previous events within a predetermined integration time period.

[0067] In some embodiments, the method further includes generating an event detection signal after an event is detected, as discussed herein. In some embodiments, the method further includes acquiring a time-of-flight measurement using photodetector units, as discussed herein. In some embodiments, the photodetector units include single-photon avalanche diodes, as discussed herein. In some embodiments, the event detection circuit includes a plurality of photodetector units arranged in an array, wherein an event is detected if a difference is detected for at least one of the plurality of photodetector units, as discussed herein. In some embodiments, the method further includes acquiring a time-of-flight measurement using the plurality of photodetector units after an event is detected, as discussed herein. In some embodiments, modulation signals with at least two different phase delays are applied as analog time codes to the plurality of photodetector units to acquire a time-of-flight measurement using at least one of the plurality of photodetector units, as discussed herein. In some embodiments, the circuit includes at least two different accumulators optimized for different frame rates, as discussed herein. In some embodiments, the event is detected based on a comparison of the corresponding output signals of the at least two accumulators, as discussed herein. In some implementations, the method further includes selecting an output signal from the corresponding output signals of at least two accumulators based on the comparison. In some implementations, if no event is detected, the output frame rate corresponds to the lower frame rate of the at least two accumulators; and if an event is detected, the output frame rate corresponds to the higher frame rate of the at least two accumulators.

[0068] Some implementations relate to an image sensor based on a SPAD pixel array unit, comprising: a timecode generation unit configured to generate an initialized timecode (Tc) at the beginning of an exposure period; wherein each pixel includes a counting unit configured to count SPAD output pulses and output a count value during the exposure period; wherein each pixel further includes an acquisition unit configured to acquire a timecode indicating a time point in time when the count value reaches a threshold before the end of the exposure period; wherein the acquisition unit is configured to output a digital signal or analog signal representing a photon count.

[0069] In some implementations, the digital signal is obtained by extrapolating the photon count over the entire exposure period.

[0070] In some implementations, the sensor is further configured for at least one of event-based sensing (EVS), photon counting (PCT), time-of-flight sensing (TOF), direct time-of-flight sensing (DTOF), and conventional imaging (CIS).

[0071] In some implementations, each pixel also includes (e.g., 14-bit) memory configured to store a previous timecode (Tc-1) or a digital signal representing a previous photon count (Cph-1) recorded during a previous exposure period.

[0072] In some implementations, each pixel also includes a comparator configured to compare Tc-1 or Cph-1 stored in (14-bit) memory with Tc or Cph acquired during the current exposure period.

[0073] In some implementations, the comparator is further configured to compare the most significant memory bit of both Tc and Tc-1, or both Cph and Cph-1, and their adjacent high and low bits (+ / -1 bits). In some implementations, the comparator is further configured to output a signal indicating whether an event has occurred or not in the observed scene based on the comparison result.

[0074] In some implementations, the counting unit is a switched capacitor (switched capacitor device).

[0075] In some implementations, the switched capacitor is configured as a CADTofF circuit.

[0076] In some implementations, the counting unit is a digital counter or an analog counter.

[0077] In some implementations, the sensor is based on 16x16 SPAD pixel macropixels to share 256 counters and output dToF signals based on histograms.

[0078] In some implementations, each pixel includes an analog comparator configured to instruct the acquisition unit to output Tc or Cph based on a predetermined threshold of the detected light event, thereby changing the dynamic range of the sensor.

[0079] In some implementations, the sensor also includes an array of color filters patterned with 50% full pass (for imaging / event sensing) and 50% band pass (940 nm, for dToF / PCT).

[0080] The methods described herein are, in some embodiments, also implemented as computer programs that, when executed on a computer and / or processor, cause the computer and / or processor to perform the methods. In some embodiments, a non-transitory computer-readable recording medium is also provided, storing a computer program product that, when executed by a processor (such as the processor described above), causes the methods described herein to perform.

[0081] Back Figure 1 The figure depicts a pixel circuit 1 that includes an event detection circuit according to the present disclosure.

[0082] Pixel circuit 1 includes a plurality of SPADs 2 (or, in some embodiments, one or more SPADs; it should be understood that this is generally applicable to each embodiment of the present disclosure). When light is incident on a SPAD 2, the SPAD generates an avalanche signal 3, which is fed into a module having one or more counters 5. At least one counter 3 is set with a predetermined counting threshold corresponding to the number of photons to be counted. When the threshold is exceeded, an overflow signal 6 is generated and fed into an event comparison / latch circuit 7. Event comparison / latch circuit 7 further receives a digital time count signal 8 and, if necessary, an event flag reset signal 9 (in cases where the event should be reset and the event comparison / counting should restart).

[0083] Now refer to Figure 2 The event comparison / latch circuit 7 is further described in the form of a block diagram.

[0084] The event comparison / latch circuit 7 includes combinational logic 20 in which a digital time count signal 8 and a digital storage element 21 are compared to detect changes. At a predetermined time point (e.g., at startup or after resetting the event flag using a reset signal 9), the current measurement result is stored in the digital storage element 21.

[0085] From this point onward, change detection according to this disclosure can be performed in a variety of ways, and this disclosure should not be construed as limited to one of the solutions presented herein. For example, the digital time counting signal 8 can provide linear or logarithmic counting, and the combinational logic 20 can be configured to calculate its absolute difference with the digital storage element 21, and detect a change by checking whether such absolute difference is greater than a given event threshold. In another embodiment, the digital time counting signal 8 can be a one-bit rotation signal, and the combinational logic 20 can be configured to detect a change by checking whether the position difference (δ) between the stored value 21 and the instantaneous value 8 exceeds two bits.

[0086] If combinational logic 20 detects a change, it stores the change in logic element 22.

[0087] Logic element 22 is configured to output an event flag signal 10, indicating that a change has been detected for that pixel. This change signal 10 can be read out externally at fixed time intervals, allowing the sensor settings to be changed to acquire more frames, etc., after such a change is detected. After a change detection is read, it is reset via signal 9 to prepare for detecting the next change.

[0088] Figure 3 A pixel circuit 30 according to this disclosure is depicted. Similar to... Figure 1 The diagram shows multiple SPADs 31 (or, as mentioned above, one or more SPADs in some embodiments), but in this embodiment, an analog counting signal is used, and at least one analog counting signal is fed into comparator 33. It should be noted that the SPADs 31 and analog counters 32 can also be implemented as one or more conventional photodiodes with 4T (or other) readout circuitry. Such circuitry does indeed have an analog voltage that varies proportionally to the amount of incident photons; however, SPAD implementations are currently considered more sensitive and less noisy. Comparator 33 receives an external threshold signal 38, and if the threshold represented by the external threshold signal is exceeded, comparator 33 outputs an overflow signal 39, which is fed into event comparison / latch circuit 40. Unlike event comparison / latch circuit 7, event comparison / latch circuit 40 receives an analog time count signal 41 (instead of a digital signal). The analog time count signal 41 varies proportionally over the target exposure time, wherein the voltage is linearly (or logarithmically) related to time. In addition, the event comparison / latch circuit 40 also receives the event flag reset signal 42.

[0089] The event comparison / latch circuit 40 includes accumulators 34 to 37, which are driven to optimize their outputs for different frame rates.

[0090] In some implementations, optimization can be achieved by ensuring that the circuit converges within the frame rate period. For example, for a 1000fps output, if the accumulator is an analog memory (such as a sample-and-hold circuit), it can be achieved by sampling at least once every millisecond; or, if the accumulator is an average switched-capacitor circuit with a capacitance ratio of N, the circuit can have an approximate time constant N and can be optimized by using switched capacitors to perform at least five N samples per millisecond to achieve convergence within a 1000fps frame period of one millisecond.

[0091] Different circuits and frame rates can be optimized accordingly, ensuring convergence of the output signal within the frame rate cycle each time.

[0092] It should be understood that, depending on the specific circumstances, fewer or more accumulators can be implemented, and the frame rate can be different, or other parameters (such as integration time or counting threshold) can also be different.

[0093] In some implementations, the accumulator is any analog memory or other analog storage device, such as an average switched-capacitor circuit. The average switched-capacitor circuit will be discussed below for illustrative purposes. Such a circuit switches and discharges a small capacitor pre-charged with samples of analog code 41 to a larger capacitor based on an overflow signal 39. Therefore, at each overflow signal 39, the analog time count signal 41 is added to the larger capacitor at a factor of 1 / N (scaled down), where N is the ratio of the sum of the large and small capacitances of the switched capacitors to the small sampling capacitance. This forms an exponential moving average function of the analog code 41.

[0094] Accumulator 37 is designed with a very large ratio N for the lowest frame rate (5 frames per second in this embodiment), while accumulator 34 is designed with a small ratio for the highest frame rate (1000 fps in this embodiment). In this embodiment, accumulators 35 and 36 are designed to be between 50 fps and 200 fps, where these specific frame rates should not be considered as constraints. For example, accumulators 34 to 37 can be configured individually; for instance, they can be constructed as a sequence of four switching capacitors with a small ratio N, where 34 uses only the first switching capacitor, 35 is a combination of the first two switching capacitors, 36 is a combination of the first three switching capacitors, and 37 is a combination of all four switching capacitors, thus producing the highest accumulation ratio N.

[0095] Based on these four accumulators, change detection can be implemented. In this case, when comparators (e.g., Schmitt triggers) are placed on the outputs of accumulator 34 and accumulator 35, the comparators will change their outputs when the signal changes suddenly because accumulator 34 changes faster than accumulator 35. Therefore, the signal represents a change detected at 1000fps but not yet seen at 200fps. The change detection flag can be obtained by maintaining the peak or low peak values ​​of the outputs of such comparators.

[0096] As discussed above, such change flags can alternatively be read externally and reset after an action is taken or noticed. Similarly, comparators can be implemented between other accumulators 35, 36, and 37, enabling change detection even at lower frame rates. Based on such signals, an output selection switch can be added to select the currently valid signal. For example, if there is no change between 1000fps and 200fps, and also no change between 200fps and 50fps, but the comparator between 50fps and 5fps indicates a significant difference, the 5fps signal may be unreliable and still changing, so the output selection switch will select 50fps as the optimal output for the pixel. After a period of time, once 5fps also approaches the stable signal amplitude of 50fps, the comparator between 50fps and 5fps will indicate no further difference, so the output selection switch can automatically select the more average 5fps value as the output.

[0097] In some implementations, one way to read out such signals is to use a bus for each category of change flags in each row. If a pixel highlights a change, that pixel sets the bus of its row high, and a scan of that row is performed at the next possible moment. The pixel highlighting the change is converted to digital by an ADC and read out. Depending on the rate of change, fewer or more bits can be used. For example, 4 bits for 1000fps allows for a higher conversion speed, while 10 bits for 5fps provides greater accuracy but requires more time, thus optimizing the conversion time of the entire array. In this way, it is possible to achieve synchronous scanning of changes, for example, at a frame rate of 1000fps, while pixels continuously and asynchronously store an average version of the measured signal from 5fps to 1000fps, and read out these measured signals according to changes in the scene.

[0098] Figure 4An implementation of the signal evolution is described, which utilizes the above information to optimize system power consumption based on signals at 50fps and 1000fps. For example, if a target frame rate of 50fps is required, the system can reduce its internal frame rate whenever the comparator indication between 50fps and 1000fps remains unchanged, thereby requiring only one overflow measurement every ten milliseconds, comparing the overflow measurement value with the average value currently stored in the pixel. Therefore, the system only needs to consume one SPAD event (e.g., 200) and associated power consumption required for the overflow measurement every ten milliseconds.

[0099] Once a change is detected, the system can immediately resume data acquisition at a faster pace, allowing the circuitry to continuously acquire data at its normal internal frame rate (e.g., once every two milliseconds, or even faster, once every 200 μs or 20 μs, depending on the implementation) until convergence is achieved between 50 fps and 1000 fps. At this point, the system can switch back to performing an overflow measurement every ten milliseconds. This allows for two measurements within a 50 fps frame every twenty milliseconds, thus enabling half of that time to be used to converge the 50 fps output value (e.g., in cases where the change is detected only ten milliseconds after it occurred).

[0100] This method can achieve significant SPAD power savings while maintaining a readout speed of 50fps at the output.

[0101] Another effect of these variation indicators between measurements acquired at different frame rates is the ability to reduce the output bandwidth of the image sensor. This can be achieved by outputting only the changed pixels. At the host side, these changed pixels can be reassembled into a complete image based on the full set of varied images. If the variation in the image is limited, the sensing frame rate can be significantly increased to, for example, 1000 fps or even 10000 fps without requiring more ADC conversions and / or higher output stage bandwidth. Of course, the complete image can also be requested and transmitted if needed.

[0102] Figure 5An example of an analog time-counting signal 214 showing the relationship between logarithmic voltage and time is illustrated. The graph shown assumes a high illumination level, i.e., 20,000 photons incident within the same two-millisecond time period 240. In this embodiment, the rise rate of curve LowLL 215 corresponding to the internal analog counter is hundreds of times faster than in the case of 200 photons incident within a frame. Instead of two milliseconds, it takes only 20 μs to reach the threshold Vtrip 212. At the 20 μs mark, LowLL 215 reaches Vtrip 212, and the voltage on curve HighLL 217 is fixed and stored in analog memory. As mentioned above, such accumulators may additionally or alternatively include or be based on switched-capacitor circuitry.

[0103] At the end of the frame, the fixed voltage is read out at node HighLL 121, thus revealing the sampling time, and therefore the illumination level can be measured using only 200 photons after the start of period 240.

[0104] If the SPAD circuitry is configured to disable after reaching the threshold Vtrip, the number of SPAD triggers will be limited to 200 under this high light level. Similarly, if the light level is very high, for example, with one million photons incident every two milliseconds, the maximum count of 200 photons may be reached after 400 ns.

[0105] By according to Figure 5 The voltage of the analog time-counting signal 214 is allocated (starting at 1V and ending at 0V throughout the entire frame period 240), thus supporting high dynamic range operation. Assuming that the ADC in the readout circuit has an accuracy of 1mV (approximately 10 bits) at the 1V range, it can be achieved over the entire specified high illumination level range (i.e., 200 to 1,000,000 photons), and a quantization step size of 1mV may correspond to only 1% of the change in incident illumination level.

[0106] A single frame will still have considerable shot noise because each measurement is based on only 200 photons. The SNR at this point is... This means the standard deviation (σ) of the measured illumination level is approximately 7%. If the application requires a higher SNR, this can be achieved by increasing the maximum count, or by adding additional circuitry (such as a switched-capacitor accumulator) to average over multiple frames. Subsequently, the low illumination level voltage values ​​on node LowLL 118 can be averaged, or the high illumination level values ​​on node HighLL 121 can be averaged.

[0107] Figure 6 Another embodiment of the pixel circuit 7 is described, which is similar to... Figure 3 The implementation method is the same, but the difference is that, in addition to the overflow signal 6, a count signal 61 indicating the number of photons is also fed to the event comparison / latch circuit 7.

[0108] In this scenario, instead of using a comparator to detect threshold overflow at fixed intervals or exposure times, the count signal 61 is passed to the comparator / latch circuit 7, and again exponential averaging is performed based on different frame rates from 1000fps to 5fps. In this implementation, the lowest possible light intensity (up to complete darkness) may be required to achieve overflow, and this implementation is equally effective even if only one SPAD event is detected during the exposure time.

[0109] Figure 7 A flowchart of the event detection method 70 according to this disclosure is depicted.

[0110] At step 71, a threshold related to the number of photons is set, which may be analog or digital, as described herein.

[0111] At step 72, a start frame is captured. In the start frame, the timecode for reaching the threshold is measured, and then the timecode is stored in memory.

[0112] At step 73, an event frame is captured. If the change between this time frame and the stored timecode exceeds a threshold, an event flag is generated at step 74, and event information is output at step 75. If the timecode / threshold is not exceeded, another event frame is captured.

[0113] As shown in this article, event detection can be triggered to increase the internal frame rate or to capture depth images, etc. Figure 8 As shown, this will be described below.

[0114] Figure 8 A flowchart of an imaging method according to this disclosure is depicted.

[0115] At step 81, passive sensing is performed, i.e., event sensing as described herein is performed. In this embodiment, event detection is performed to detect whether a change has occurred in the scene. If no change is detected at step 82, passive sensing 81 is further performed. If a change is detected at step 82, active sensing is performed, i.e., ToF measurement is performed.

[0116] Figure 9 A timing diagram 90 is depicted to illustrate exemplary modes in an image sensor according to the present disclosure and the time consumption of these exemplary modes. In this embodiment, as in conjunction with Figure 9As described, passive sensing takes one millisecond. In this implementation, active sensing (e.g., time-of-flight measurement) takes ten milliseconds. After both passive and active sensing, the sensor has a 22-millisecond idle time. Therefore, a complete frame requires 33 milliseconds, resulting in a frame rate of approximately 30 fps.

[0117] Figure 10 A schematic illustration of a macropixel 100 capable of acquiring Time-of-Flight (TOF) measurements according to this disclosure is depicted. The macropixel 100 comprises four pixels 101 (any number of pixels can be used), each pixel 101 including event sensing circuitry comprising switched-capacitor circuitry and a SPAD, as described herein. In this case, the analog timecode used is a periodically modulated signal driven by different phase signals phase-shifted relative to each other by ninety degrees. If an event is detected, active sensing is performed, i.e., as is generally known in the field of time-of-flight, modulated light is emitted and detected using pixel 101 based on the phase signal, thereby acquiring I and Q values ​​as is generally known. Thus, the circuitry of each pixel is reused to acquire time-of-flight measurements in the macropixel configuration. This can be achieved, for example, by wiring the circuitry such that the SPAD signal can bypass the analog counter 32 and comparator 33 and go directly into accumulators 34 to 37, which can be connected to the periodically modulated analog timecode having different phase signals phase-shifted relative to each other by ninety degrees. Analog timecodes with different periodic modulated signals having different phase signals with a 90-degree phase shift relative to each other can be connected to multiple accumulators in a single pixel, or to accumulators distributed across different pixels. Those skilled in the art will appreciate that other modulated signals, such as PRBS or other phase shifts different from 90 degrees, can also be used instead of the 90-degree phase shift. The final implemented circuit is described in more detail in patent application WO 2022043480 A1, which is incorporated herein by reference.

[0118] Figure 11 The illustration depicts two hybrid sensors 110 and 120, each including pixels 111 and 112. Pixel 111 includes an all-pass filter for imaging / EVS, and pixel 112 includes a bandpass filter for ToF / imaging / EVS. Pixel 112 also includes an event detection circuit according to this disclosure. This means that different types of passive sensing can be performed simultaneously, or (after an event is detected) passive and active sensing can be performed simultaneously.

[0119] The hybrid sensor 110 includes pixels 112 grouped together in an adjacent manner, while the sensor 120 has a checkerboard pattern (each pixel 111 is adjacent to pixel 112).

[0120] Figure 12 The event detection method 120 according to this disclosure is depicted in the form of a block diagram.

[0121] At step 121, the photon detection signal is acquired, as discussed herein.

[0122] At step 122, the photon detection signal is correlated with the photon detection time period, as discussed herein.

[0123] At step 123, an event is detected when the number of associations exceeds a threshold, as discussed herein.

[0124] Figure 13 The event detection method 130 according to this disclosure is depicted in the form of a block diagram, the method being similar to... Figure 12 The difference with this method is that a ToF measurement is performed after an event is detected.

[0125] At step 121, the photon detection signal is acquired, as discussed herein.

[0126] At step 122, the photon detection signal is correlated with the photon detection time period, as discussed herein.

[0127] At step 123, an event is detected when the number of associations exceeds a threshold, as discussed herein.

[0128] At step 131, an event detection signal is generated, as discussed herein.

[0129] At step 132, a ToF measurement is obtained, as discussed herein.

[0130] It should be recognized that the implementation describes the method in an exemplary order of method steps. However, the specific order of the method steps is given for illustrative purposes only and should not be construed as restrictive. For example, Figure 7 The order of steps 71 and 72 in the implementation can be interchanged. Other variations in the order of method steps may be obvious to those skilled in the art.

[0131] It should be noted that the division of the event sensing circuit into units 20 to 22 or 34 to 43 is for illustrative purposes only, and this disclosure is not limited to any specific division of the functions in a particular unit. For example, the control event sensing circuit may be implemented by a corresponding programming processor, field-programmable gate array (FPGA), etc.

[0132] In some embodiments, a non-transitory computer-readable recording medium is also provided, wherein a computer program product is stored that, when executed by a processor (such as the processor described above), enables the described methods to be performed.

[0133] Unless otherwise stated, all units and entities described in this specification and claimed in the appended claims can be implemented (e.g., on a chip) as integrated circuit logic, and unless otherwise stated, the functionality provided by these units and entities can be implemented by software.

[0134] Given that the above-disclosed embodiments are implemented at least in part using software-controlled data processing equipment, it should be understood that computer programs providing such software control, as well as transmission media, storage media, or other media providing such computer programs, are considered aspects of this disclosure.

[0135] It should be noted that this technology can also be configured as described below.

[0136] (1) An event detection circuit, configured to:

[0137] Acquire a photon detection signal, which indicates that a photon has been incident on the photoelectric detection unit;

[0138] Correlating the photon detection signal with the photon detection time period; and

[0139] An event is detected if, within a predetermined integration time period, a predetermined difference is found between the number of associations between the photon detection signal and the photon detection time period and the number of associations with previous events.

[0140] (2) Based on the event detection circuit in (1), further configure it as follows:

[0141] An event detection signal is generated after an event is detected.

[0142] (3) The event detection circuit according to either (1) or (2) is further configured as follows:

[0143] After an event is detected, a photoelectric detection unit is used to obtain a time-of-flight measurement.

[0144] (4) An event detection circuit according to any one of (1) to (3), wherein the photoelectric detection unit includes a single-photon avalanche diode.

[0145] (5) An event detection circuit according to any one of (1) to (4), wherein the circuit includes at least two accumulators optimized for different frame rates.

[0146] (6) The event detection circuit according to (5) wherein the event is detected based on the comparison of the corresponding output signals of at least two accumulators.

[0147] (7) The event detection circuit according to (6), wherein the circuit is further configured to select one output signal from the corresponding output signals of at least two accumulators based on the comparison.

[0148] (8) According to the event detection circuit of (7), if no event is detected, the output frame rate corresponds to the lower frame rate of at least two accumulators; and if an event is detected, the output frame rate corresponds to the higher frame rate of at least two accumulators.

[0149] (9) An event detection sensor, comprising:

[0150] Multiple photoelectric detection units arranged in an array; and

[0151] The event detection circuit is configured as follows:

[0152] For multiple photoelectric detection units, a photon detection signal is acquired, which indicates that a photon is incident on one of the multiple photoelectric detection units;

[0153] Correlating the photon detection signal with the photon detection time period; and

[0154] For at least one of a plurality of photoelectric detection units, if a predetermined difference exists between the number of photon detection signals associated with a photon detection time period and the number of previous events within a predetermined integration time period, then an event is detected.

[0155] (10) The event detection sensor according to (9), wherein each of the multiple photoelectric detection units is a single-photon avalanche diode.

[0156] (11) The event detection sensor according to (9) or (10) also includes a data path configured to transmit only the signal of the photoelectric detection unit determined to have caused the event.

[0157] (12) An event detection sensor according to any one of (9) to (11), wherein signals with at least two different phase delays are applied as analog time codes to a plurality of photoelectric detection units so as to acquire time-of-flight measurements using at least one of the plurality of photoelectric detection units.

[0158] (13) An event detection method for operating an event detection circuit, the method comprising:

[0159] Acquire a photon detection signal, which indicates that a photon has been incident on the photoelectric detection unit;

[0160] Correlating the photon detection signal with the photon detection time period; and

[0161] An event is detected if, within a predetermined integration time period, a predetermined difference is found between the number of associations between the photon detection signal and the photon detection time period and the number of associations with previous events.

[0162] (14) According to the method of (13), it further includes:

[0163] An event detection signal is generated after an event is detected.

[0164] (15) According to the method of (13) or (14), wherein the circuit includes at least two accumulators optimized for different frame rates.

[0165] (16) According to the method of (15), the event is detected based on the comparison of the corresponding output signals of at least two accumulators.

[0166] (17) According to the method of (16), it further includes: based on the comparison, selecting an output signal from the corresponding output signals of at least two accumulators.

[0167] (18) According to the method of (17), wherein if no event is detected, the output frame rate corresponds to the lower frame rate of at least two accumulators; and wherein if an event is detected, the output frame rate corresponds to the higher frame rate of at least two accumulators.

[0168] (19) The method according to any one of (13) to (18) further includes:

[0169] The time of flight is measured using a photoelectric detection unit.

[0170] (20) According to the method of (13), wherein at least two different phase-delayed modulation signals are applied as analog codes to a plurality of photodetectors so as to acquire time-of-flight measurements using at least one of the plurality of photodetectors.

[0171] (21) A computer program including program code that, when executed on a computer, causes the computer to perform the method according to any one of (12) to (20).

[0172] (22) A non-transitory computer-readable recording medium storing a computer program product which, when executed by a processor, causes the method according to any one of (12) to (20) to be executed.

Claims

1. An event detection circuit, configured as follows: Acquire a photon detection signal, which indicates that a photon is incident on the photoelectric detection unit; The photon detection signal is correlated with the photon detection time period; and An event is detected if, within a predetermined integration time period, a predetermined difference is found between the number of associations between the photon detection signal and a photon detection time period and the number of associations with previous events.

2. The event detection circuit according to claim 1, further configured as follows: An event detection signal is generated after the event is detected.

3. The event detection circuit according to claim 1, further configured as follows: The photoelectric detection unit is used to obtain flight time measurements.

4. The event detection circuit according to claim 1, wherein, The photoelectric detection unit includes a single-photon avalanche diode.

5. The event detection circuit according to claim 1, wherein, The circuit includes at least two accumulators optimized for different frame rates.

6. The event detection circuit according to claim 5, wherein, Events are detected by comparing the corresponding output signals of the at least two accumulators.

7. The event detection circuit according to claim 6, further configured as follows: Based on the comparison, one output signal is selected from the corresponding output signals of the at least two accumulators.

8. The event detection circuit according to claim 7, wherein, If no event is detected, the output frame rate corresponds to the lower frame rate of the at least two accumulators, and if an event is detected, the output frame rate corresponds to the higher frame rate of the at least two accumulators.

9. An event detection sensor, comprising: Multiple photoelectric detection units arranged in an array; as well as Event detection circuit, the event detection circuit being configured to: For the plurality of photoelectric detection units, a photon detection signal is acquired, the photon detection signal indicating that a photon is incident on the photoelectric detection unit among the plurality of photoelectric detection units; The photon detection signal is correlated with the photon detection time period; and For at least one of the plurality of photoelectric detection units, if a predetermined difference is detected between the number of associations between the photon detection signal and a photon detection time period and the number of associations with previous events within a predetermined integration time period, then an event is detected.

10. The event detection sensor according to claim 8, wherein, Each of the plurality of photoelectric detection units is a single-photon avalanche diode.

11. The event detection sensor of claim 9, further comprising a data path configured to transmit only signals from photoelectric detection units determined to have caused an event.

12. The event detection sensor according to claim 9, wherein, At least two modulated signals with different phase delays are applied as analog time codes to the plurality of photodetectors so as to acquire time-of-flight measurements using at least one of the plurality of photodetectors.

13. An event detection method for operating an event detection circuit, the method comprising: Acquire a photon detection signal, which indicates that a photon is incident on the photoelectric detection unit; The photon detection signal is correlated with the photon detection time period; as well as An event is detected if, within a predetermined integration time period, a predetermined difference is found between the number of associations between the photon detection signal and a photon detection time period and the number of associations with previous events.

14. The method of claim 13, further comprising: An event detection signal is generated after the event is detected.

15. The method according to claim 13, wherein, The circuit includes at least two accumulators optimized for different frame rates.

16. The method according to claim 15, wherein, Events are detected by comparing the corresponding output signals of the at least two accumulators.

17. The method of claim 16, further comprising: Based on the comparison, one output signal is selected from the corresponding output signals of the at least two accumulators.

18. The method according to claim 17, wherein, If no event is detected, the output frame rate corresponds to the lower frame rate of the at least two accumulators, and if an event is detected, the output frame rate corresponds to the higher frame rate of the at least two accumulators.

19. The method of claim 13, further comprising: The photoelectric detection unit is used to obtain flight time measurements.

20. The method according to claim 13, wherein, At least two modulated signals with different phase delays are applied as analog time codes to a plurality of the photodetectors so as to acquire time-of-flight measurements using at least one of the plurality of photodetectors.

Images