Efficient single-photon depth sensing and three-dimensional imaging

Abstract

A distance measurement system and device leverage a laser emitting pulses toward a target and utilize single photon detection sensors to detect reflected photons. The system incorporates coincidence detection circuitry with a configurable threshold to identify simultaneous photon events within a specific time frame. Photon detection processing circuitry, including tracking circuitry with quantile tracking and photon counting capabilities, processes these events. The tracking circuitry may be structured with multi-stage configurations, capable of performing quantile value calculations or photon counting over defined gating windows. Iterative control circuits refine tracking results throughout transient distributions or around transient distribution peaks. The processed photon detection data is output on a per-pixel basis. The system enables precise distance measurements by managing and analyzing photon detection events through advanced quantile tracking and photon counting techniques.

Description

EFFICIENT SINGLE-PHOTON DEPTH SENSINGAND THREE-DIMENSIONAL IMAGINGInventor:Chih-Hsien ChouFIELD

[0001] The following is related generally to the field of depth sensing and three- dimensional (3D) imaging using time-of-flight sensors.BACKGROUND

[0002] Image sensors capable of capturing single photons, such as single-photon avalanche diode (SPAD) and silicon photomultiplier (SiPM) sensors, have gained popularity for a wide range of applications, including robotics, machine vision, autonomous driving, and augmented reality. One issue with these sensors is the amount of raw data generated makes processing such data in real time difficult. A single-photon camera (SPC) captures distance information using the direct time-of- flight (dToF) principle. As illustrated in FIG. 1 , a pulsed laser source illuminates each scene point and the corresponding camera pixel captures a stream of return events that represent the delays of the photon arrivals relative to the time the laser pulse was emitted. These return events capture not only the returning laser photons but also photons due to ambient light and other sources of noise.

[0003] Time of flight distance estimation is based on the time of flight to a target, multiplied by the speed of light, divided by two. To estimate the time-varying distribution of light intensity, called the transient distribution, current techniques digitize and store photon return events in the form of a histogram where each histogram bin represents a fixed time interval of photon arrivals. The location of the peak in this histogram provides an estimate of the true distance of the corresponding scene point. With a large number of pixels on an image sensor array to meet a spatial resolution requirement, and a large number of bins in each histogram to meet a depth resolution requirement, it is difficult to build a high-resolution histogram for each pixel on theimage sensor array due to the limited memory, data throughput, and computational resources available at each pixel.

[0004] Alternatives have been proposed using temporal encoding based on reflected-photon arrival times combined with histograms which use this temporal data to determine variable-width histograms which allow for estimation of this peak more accurately or more efficiently.SUMMARY

[0005] One general aspect includes a distance measurement system. The distance measurement system includes a plurality of single photon detection sensors each outputting photon detection signals, the photon detection signals at least partially based on photons reflected from a pulse emitted by a laser toward an object. The system also includes coincidence detection circuitry including a configurable coincidence threshold coupled to each of the plurality of single photon detection sensors in an array, the coincidence detection circuitry identifies simultaneous photon detection events within a defined spatial correlation window across the plurality of SPADs in the array during a temporal coincidence window and over the configurable coincidence threshold. The system also includes photon detection processing circuitry coupled to the coincidence detection circuitry, including: tracking circuitry detecting photon arrival events, the tracking circuitry including at least one of: a quantile tracking circuit and a photon counting circuit; and iteration control circuitry updating the tracking circuitry for one mode of: throughout a transient distribution and around a transient distribution peak. The system also includes pixel data output circuitry outputting per- pixel photon detection data.

[0006] Implementations may include the distance measurement system where the photon detection processing circuitry may include photon counting circuitry. Implementations may include the distance measurement system where the photon detection processing circuitry may include photon arrival timing circuitry. Implementations may include the distance measurement system where the tracking circuitry may include quantile value calculation circuitry. The quantile tracking circuitry may include a trisection circuitry outputting two quantile values. Implementations may include the distance measurement system where the quantile tracking circuitry mayinclude a quadrisection circuitry outputting three quantile values. Implementations may include the distance measurement system where the tracking circuitry may include a trisection circuitry tracking photon counts over two gating windows and outputting two photon count values. Implementations may include the distance measurement system where the tracking circuitry may include a quadrisection circuitry tracking photon counts over three gating windows and outputting three photon count values. Implementations may include the distance measurement system where the tracking circuitry is responsive to a gating window which tracks photon arrival timing or photon counting during the gating window. Implementations may include the distance measurement system where the tracking circuitry from adjacent pixels may be combined.

[0007] Implementations may include the distance measurement system where the tracking circuitry may include multiple stages, each stage may include one or more of a bisection tracking circuit, a trisection tracking circuit or a quadrisection tracking circuit. Implementations may include the distance measurement system where the tracking circuitry includes a feedforward full range mode to traverse all bin boundaries of a histogram in a tree structure. Implementations may include the distance measurement system where the system includes an iterative peak search mode to successively track histogram bin boundaries around a distribution peak. Implementations may include the distance measurement system of any foregoing implementation wherein the single photon detection sensors each comprise at least one single-photon avalanche diode (SPAD).

[0008] One general aspect includes a distance measurement device. The distance measurement device includes a laser configured to emit laser pulses toward an object. The device also includes pulse detection circuitry, may include a plurality of single photon detection sensors each outputting photon detection signals, the photon detection signals at least partially based on photons reflected from a pulse emitted by a laser toward an object; coincidence detection circuitry including a configurable coincidence threshold coupled to each of the plurality of single photon detection sensors in an array, the coincidence detection circuitry identifies simultaneous photon detection events within a defined spatial correlation window across the plurality of SPADs in the array during a temporal coincidence window and over the configurable coincidence threshold; photon detection processing circuitry coupled to thecoincidence detection circuitry, including: tracking circuitry detecting photon arrival events, the tracking circuitry including at least one of: a quantile tracking circuit and a photon counting circuit; and iteration control circuitry updating the tracking circuitry for one mode of: throughout a transient distribution and around a transient distribution peak; pixel data output circuitry outputting per-pixel photon detection data. The device also includes a non-transitory memory storage which may include instructions. The device also includes one or more processors in communication with the memory, where the one or more processors execute the instructions to control the laser output pulses and determine a distance to the object.

[0009] Implementations may include the distance measurement device where the photon detection processing circuitry may include photon counting circuitry. Implementations may include the distance measurement device where the photon detection processing circuitry may include photon arrival timing circuitry. Implementations may include the distance measurement device where the tracking circuitry may include quantile value calculation circuitry. Implementations may include the distance measurement device where the quantile tracking circuitry may include a trisection circuitry outputting two quantile values. Implementations may include the distance measurement device where the quantile tracking circuitry may include a quadrisection circuitry outputting three quantile values. The tracking circuitry may include a trisection circuitry tracking photon counts over two gating windows and outputting two photon count values. Implementations may include the distance measurement device where the tracking circuitry may include a quadrisection circuitry tracking photon counts over three gating windows and outputting three photon count values. Implementations may include the distance measurement device where the tracking circuitry may include multiple stages, each stage may include one or more of a bisection tracking circuit, a trisection tracking circuit or a quadrisection tracking circuit. Implementations may include the distance measurement device where the tracking circuitry includes a feedforward full range mode to traverse all bin boundaries of a histogram in a tree structure. Implementations may include the distance measurement device where the device includes an iterative peak search mode to successively track histogram bin boundaries around a distribution peak. Implementations may include the distance measurement device where the tracking circuitry is responsive to a gating window which tracks photon arrival timing or photoncounting during the gating window. Implementations may include the distance measurement device where the tracking circuitry from adjacent pixels may be combined. Implementations may include the distance measurement device of any foregoing implementation wherein the single photon detection sensors each comprise at least one single-photon avalanche diode (SPAD).

[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all disadvantages noted in the Background.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying Figures for which like references indicate identical or similar elements.

[0012] FIG. 1 illustrates a conventional single photon camera and pulsed laser arrangement.

[0013] FIG. 2 shows two types of histograms used in dToF systems.

[0014] FIG. 3 illustrates a single photon distance measurement system in accordance with the described embodiments.

[0015] FIG. 4 illustrates typical pixel patterns which may be programmed for spatial correlation logic.

[0016] FIGs. 5A and 5B illustrate two basic quantile tracker circuits for value-based signals (VBS).

[0017] FIG. 6 illustrates a first quantile tracker circuit for time-based signals (TBS).

[0018] FIG. 7 illustrates another quantile tracker circuit for time-based signals (TBS).

[0019] FIG. 8 are graphs of signal magnitude relative to time which illustrate examples of the ACT and TBS signals.

[0020] FIG. 9 illustrates a bisection TBS quantile tracker circuit configured for photon timing mode.

[0021] FIG. 10 illustrates a trisection TBS quantile tracker circuit configured for photon timing mode.

[0022] FIG. 11 illustrates a quadrisection TBS quantile tracker circuit configured for photon timing mode.

[0023] FIG. 12 illustrates a bisection quantile tracker circuit configured for photon counting mode.

[0024] FIG. 13 illustrates a trisection quantile tracker circuit configured for photon counting mode.

[0025] FIG. 14 illustrates a quadrisection quantile tracker circuit configured for photon counting mode.

[0026] FIG. 15A illustrates block inputs and outputs for bisection quantile tracker circuits (B Units).

[0027] FIG. 15B illustrates block inputs and outputs for trisection quantile tracker circuits (T Units).

[0028] FIG. 15C illustrates block inputs and outputs for quadrisection quantile tracker circuits (Q Units).

[0029] FIG. 16 is a block diagram showing a three-stage bisection quantile tracking circuit with a tree structure.

[0030] FIG. 17 is a block diagram showing a two-stage quadrisection quantile tracking circuit with a tree structure.

[0031] FIG. 18 is a block diagram showing both a two-stage and a three-stage trisection quantile tracking circuit with a tree structure.

[0032] FIG. 19 is a flowchart illustrating iterative peak search mode.

[0033] FIG. 20 is a flowchart illustrating a photon detection process.

[0034] FIG. 21 is a flowchart illustrating a quantile tracking process.

[0035] FIG. 22 is a flowchart illustrating a LiDAR in-pixel process.

[0036] FIG. 23 is a flowchart illustrating the overall method of operating the system of FIG. 3.

[0037] FIG. 24 illustrates another embodiment of a quantile tracking circuit with a recursive structure configured for peak search mode with gating windows.

[0038] FIGs. 25 and 26 are block diagrams illustrating a further embodiment of the system wherein an alternative methodology trades spatial resolution for depth resolution in full range mode and peak search mode, respectively.

[0039] FIG. 27 illustrates one application for a single photon dToF system such as that illustrated in FIG. 3.

[0040] FIG. 28 illustrates another application for a single photon dToF system such as that illustrated in FIG. 3.

[0041] FIG. 29 illustrates a network processing device 2900 which may be used to implement certain post processing operations in the embodiments discussed herein.DETAILED DESCRIPTION

[0042] Techniques are described to perform time-of-flight depth sensing with reduced in-pixel circuitry, lower power dissipation, increased fill factor, lower output data rate per pixel, and higher depth precision or wider depth range. The disclosed embodiments include a plurality of single photon detection sensors, such as SinglePhoton Avalanche Diode (SPAD) or Silicon Photomultiplier (SiPM) sensors and accompanying front-end circuitry. The single photon detection sensors receive incoming photons and generate photon arrival event signal pulses when photons are detected. A coincidence detection module counts detected photons from within an adjustable temporal coincidence window and triggers a coincident photon arrival event when an adjustable coincidence detection threshold is reached. A quantile tracking circuit tracks a predefined number of quantile values for the underlying transient distribution of photon arrival times by updating intermediate quantile values upon each coincident photon arrival event. An iteration control model updates the quantile values for the quantile tracking circuit to successively narrow the tracking range around the peak of transient distribution estimated from the previous iteration of quantile tracking. A coincidence threshold adjustment module automatically determines the adjustable coincidence threshold at the pixel-level to keep the coincident event rate within an optimal operating zone with no signal event loss while filtering out background noise. A timing and sequence control module controls firing of emitted laser pulses and operation of in-pixel processing circuitry. A pixel data output module transfers per-pixel photon detection data to downstream image processing or computer vision devices. A post-processing module uses per-pixel photon detection data from the pixel data output module to generate per-pixel depth values, or to integrate with downstream computer vision deep learning networks, such as depth estimation or 3D object detection.

[0043] Conventional systems require generation and storage of photon arrival timestamps. State of the art Light Detection and Ranging (LiDAR) systems suffer from relatively low spatial resolution as they require a large chip area of the in-pixel circuitry required to generate and store photon timestamps if on-chip photon arrival time histogram analysis is employed, thereby reducing the overall pixel fill factor. Such systems may also require a high data rate per pixel to transmit photon timestamps if off-chip photon arrival time histogram analysis is employed. Therefore, such systemsmay suffer from either lower depth resolution or narrower depth range due to number of histogram bins limited by pixel size (for on-chip photon arrival time histogram analysis) or data rate (for off-chip photon arrival time histogram analysis). Further, such systems may suffer from lower depth accuracy due to interferences of received photons from strong ambient or background lighting. The proposed system and methods support efficient depth sensing and 3D imaging using in-pixel processing implemented in analog, digital, or hybrid circuitry for direct time-of-flight (dToF) sensors with improved spatial and range resolution, with lower data rate and per-pixel power dissipation.

[0044] In contrast, the disclosed embodiments do not require storage or transmission of photon arrival timestamps, nor storage or updating of a histogram for photon arrival timestamps. This provides the advantages of smaller pixel area, lower operating power, and lower data throughput requirements. The disclosed embodiments provide higher angular resolution, higher depth resolution, wider depth range, better background suppression, and lower device cost for dToF sensors in, for example, LiDAR applications.

[0045] FIG. 1 illustrates a conventional single photon 3D camera and pulsed laser arrangement. In a typical dToF system, a pulsed laser 110 emits a series of timed pulses 115 based on a clock signal 130 toward a target objection 120 for which a distance D is to be determined. A detection sensor 125 (which may include a plurality of single photon detection sensors) captures a return stream of reflected pulses 122 reflected from the target. The reflected pulses 122 will have a delay (with respect to the corresponding timed pulses 115) determined by the time required for the pulse to travel the distance 2D at the speed of light. The sensor 125 will capture the stream of reflected pulses 122 originally emitted by laser 110 and then reflected by target object 120. The sensor 125 will also capture ambient photons generated by sources such as the sun 105.

[0046] As illustrated in FIG. 2, a dToF device may determine the time of flight of the laser pulses 115 using a histogram of reflected photon arrival times. FIG. 2 shows two types of histograms used in dToF systems. Traditional devices estimate the timevarying distribution of reflected photon arrival times, i.e. , the "transient distribution", by digitizing and storing photon return events in the form of a histogram 150 where eachhistogram bin represents a fixed time interval, such as shown at histogram 135 in FIG. 2. Some flash LiDARs utilize SPAD sensors with time-to-digital converters (TDC) to generate digitized photon arrival timestamps. A per-pixel histogram is computed from these timestamps either off-chip or on-chip to generate a photon arrival transient profile (i.e., distribution), where depth can then be estimated by locating the peak or centroid in the transient distribution. A traditional histogram 150, illustrated on the left of FIG. 2, is shown with five (5) bins (though it will be understood there are significantly more in practical applications), each with a different count magnitude. The location of the peak count magnitude 137 in this histogram can be used to provide an estimate of the true distance of the object 120. Alternatives have been proposed whereby a timebased measurement of photon arrival times is used to create histograms which converge to a median (or, more generally, to a k-quantile) of a distribution. This type of histogram is illustrated on the right of FIG. 2. Histogram 160 is shown with nine (9) bins with differing widths each representing detection bins having boundaries defined by photon arrival times. The narrower bins represent converging photon detection boundaries at time period 145, and the bin with the narrowest width among all the bins at 145 reflects a transient distribution peak 140 that can be used to provide an estimate of the true distance of the object 120.

[0047] The total range which may be measured by a ToF system depends on the detector, laser power, target reflectivity and background photons. Once a SPAD sensor detects a first incident photon in a laser cycle, it enters a dead time (typically 50 - 200 ns) during which it cannot detect further photons, causing a phenomenon known as “photon pile-up” which may distort the histogram created by subsequently detected photons. Background suppression may utilize multi-SPAD spatially correlated clusters supporting photon coincidence detection or operating in asynchronous mode for a higher signal-to-background ratio (SBR) while reducing range and / or increasing circuitry complexity.

[0048] FIG. 3 illustrates a single photon distance measurement system 300 with inpixel processing in accordance with the described embodiments. The system 300 supports efficient depth sensing and 3D imaging using in-pixel processing implemented in analog, digital, or hybrid circuitry for direct ToF sensors with improved spatial and range resolution, with lower data rate and power requirements compared to state-of-the-art systems.

[0049] The system 300 includes a plurality of SPAD and front-end units 305. Each SPAD and front-end unit 305 receives ambient and reflected photons 302 and generates photon arrival event signal pulses (not shown) when photons are detected. In FIG. 3, there are a number “K” of SPAD and front-end units 305 illustrated, with one unit including at least one SPAD sensor 310 and accompanying front-end circuitry. The front-end circuitry may include, for example, sensitivity control, quenching and monostable circuitry 315, and hot pixel masking circuitry 320. In embodiments, the front-end circuitry may also include timing electronics, analog-to-digital converters, and signal processing circuitry (not shown). Each SPAD and front-end unit 305 outputs raw photon arrival data 325 to a number “L” of its neighboring pixels and receives raw photon arrival data 330 from a number “M” of its neighboring pixels. This raw photon arrival data is aggregated by the spatial correlation logic 335 to analyze the arrival times of photons detected by different SPAD and front-end units 305 within the array of a pixel and its L neighboring pixels. Spatial correlation logic 335 aggregates the timing information (arrival times) of photons detected by different SPADs to provide insight into the properties of the detected photons around the spatial neighborhood. Spatial correlation logic 335 aggregates raw photon arrival data based on programmable pixel patterns (FIG. 4). The values of M and L are programmable and can be determined directly by the programmable pixel pattern used.

[0050] The system 300 also includes per-pixel circuitry 333 comprising a coincidence detection module 340 which counts detected and aggregated photon arrivals within an adjustable temporal coincidence window and triggers a coincident photon arrival output when an adjustable coincidence detection threshold is reached. Coincidence detection identifies simultaneous photon detection events within a defined spatial correlation window, which can be specified by the programmable pixel patterns (FIG. 4), across multiple SPADs in the array. Coincidence detection compares consecutive photon arrivals within the adjustable temporal coincidence window against the adjustable coincidence detection threshold. A coincidence threshold adjustment module 345 provides an automatically adjustable coincidence detection threshold at the pixel-level to maintain the temporal coincidence event rate within an optimal operative zone spatially for each image region and temporally for each frame. The temporal coincidence window setting is a programmable aspect of the system 300. In embodiments, a plurality of SPADs per pixel, the spatial correlation,and the temporal coincidence detection provide improved background photon suppression.

[0051] The output of the coincidence detection module 340 is provided to quantile tracking circuitry 350. The quantile tracking circuitry (various embodiments of which are disclosed herein) tracks a predefined number of quantile values for an underlying transient distribution of photon arrival times by updating intermediate quantile values upon each coincident photon arrival event. The quantile values are stored in a bin boundary memory 355. The quantile tracking circuitry 350 operates with the bin boundary memory 355 to track, update, and store histogram bin boundaries for photon arrival times in either a full range mode (for all bins throughout depth range) or a peak search mode (for denser bins around peak) as explained further below. The bin boundary memory 355 may be any of a number of suitable data memory types and structures.

[0052] An iteration control circuit 360 updates the quantile values for quantile tracking circuitry 350 to subsequently traverse a full-scale tracking range throughout a transient distribution estimated from the previous stage of quantile tracking (in full range mode), or successively narrow a tracking range around the peak of a transient distribution estimated from the previous iteration of quantile tracking (in peak search mode). A pixel data output module 365 transfers per-pixel photon detection data to downstream image processing or computer vision devices, represented by post processing module 370. In one embodiment, post-processing module 370 performs a calculation using per-pixel photon detection data from the pixel data output module 365 to generate per-pixel depth values or to integrate with downstream computer vision deep learning networks, such as depth estimation or 3D object detection.

[0053] Also shown in FIG. 3 is a timing and sequence controller 375 which controls the emission of laser pulses via a laser driver 380 to a laser 385. Generally, the timing and sequence controller 375, laser driver 380 and laser 385 are not included in the system 300, but timing information from the laser driver 380 may be used by the system 300.

[0054] FIG. 4 illustrates programmable pixel patterns 410, 420, 430, 440 which may be programmed for spatial correlation logic 335. In pattern 410, for example, the center pixel (black) is surrounded by a pattern of four (two vertical and two horizontal)adjacent correlating pixels. Pattern 420 shows eight correlating pixels, and patterns 430 and 440 show twelve and twenty correlating pixels, respectively. Any one of these patterns, as well as others, may be programmed into spatial correlation logic 335 as the defined spatial correlation window across multiple SPADs in the sensor array for coincidence detection.

[0055] Quantile values of the marginal distribution of a signal can be stochastically tracked using the self-convergent quantile tracker circuitry disclosed herein. FIGs. 5A and 5B illustrate two basic quantile tracker circuits for value-based signals. FIGs. 5A and 5B are value-based quantile trackers shown for comparison with respect to the time-based quantile trackers in FIGs. 6 and 7, used in the disclosed embodiments.

[0056] In the quantile tracker circuits of FIGs. 5A and 5B, an input signal is a valuebased signal (VBS) where the magnitude of the VBS is indicative of the information it conveys. In other words, the information is embedded in the value-based signal (VBS) as a signal value sampled at the triggering edge of a clock signal. Each VBS quantile tracker maintains a value threshold Q forcing the ratio between the numbers of sampled signal values less than and greater than Q to stochastically converge to the assigned ratio A / B, then Q will converge to the A / (A+B) quantile value of the input VBS.

[0057] In FIG. 5A, the VBS is one input to a comparator 505. A second input for the comparator 505 is provided from a latch 540 storing a value threshold QVal from a previous quantile tracking iteration. A clock signal (CLK) synchronizes edge-trigger timing between the counters 510 and 515 every time when a new signal value from the VBS is available at the input to the comparator 505. Based on the output of the comparator, either counter 510 or 515 will be incremented at the edge of the CLK signal. The outputs of counters 510 and 515 are passed through multipliers 520 and 525 and multiplied by predefined values A and B, respectively, to a summing node 530. Values from multiplier 520 will increase the incremental quantile value, while those from multiplier 525 will reduce the incremental quantile value at the output of the summing node 530. The incremental quantile value at the output of the summing node 530 will be scaled by a factor then accumulated to the current value threshold QVal stored in the latch 540 and the accumulated sum, i.e., the updated value threshold QVal, will be stored in the latch 540 only when an update signal UPD edge triggersthe latch 540 to update its stored value threshold QVal at the end of the current quantile tracking iteration. The update signal UPD also resets the counters 510 and 515 to 0 before the start of the next quantile tracking iteration by reset circuitry (not shown). Thus, a feedback loop is provided to self-adjust the value threshold QVal at the input of the comparator 505 to maintain a fixed ratio of A / B between the numbers of VBS values less than and greater than the value threshold QVal. Periodically, the UPD signal will update the quantile tracker for the value threshold QVal to converge to the A / (A+B) quantile value of the transient distribution of the VBS input.

[0058] When a VBS is received at comparator 505 and exceeds (or equals) the value threshold QVal and the CLK edge is received, counter 510 is incremented. When the value threshold QVal exceeds the VBS and the CLK edge is received, counter 515 is incremented instead. As for the scaling factor used while accumulating the incremental quantile value to the current value threshold QVal stored in the latch 540, larger scaling factor values usually lead to faster convergence with more fluctuations after convergence, while smaller scaling factor values usually lead to slower convergence with less fluctuations after convergence. In embodiments, variable scaling factors can be used to allow for faster convergence and more stable converged values.

[0059] FIG. 5B illustrates a simplified quantile tracker wherein like numbers represent and function as like elements to those in FIG. 5A. The tracker in FIG. 5B is similar to that in FIG. 5A but removes the counters 510 and 515, and multipliers 520, 525, in favor of a selector 545 and an accumulator 535. Selector 545 selects either A or -B input values based on the output of comparator 505. The accumulator 535 accumulates the selected values from the selector 545 to its intermediate sum at the edge of the CLK signal. The incremental quantile value at the output of the accumulator 535 will be scaled by a factor then accumulated to the current value threshold QVal stored in the latch 540 and the accumulated sum, i.e., the updated value threshold QVal, will be stored in the latch 540 only when an update signal UPD edge triggers the latch 540 to update its stored value threshold QVal at the end of the current quantile tracking iteration. The update signal UPD also resets the accumulator 535 to 0 before the start of the next quantile tracking iteration by reset circuitry (not shown).

[0060] FIGs. 6 and 7 illustrate quantile tracker circuits for time-based signals (TBS). The quantile tracker circuits for time-based signals can be used to produce a histogram such as histogram 160 in FIG. 2. For time-based signals , such as the output signal from the coincidence detection module 340 where photon detection information is embedded as signal timing within a clock window. In FIGs. 6 and 7, an action (ACT) signal is a regular periodic clock signal specifying the starting time of each clock window for a TBS. Each TBS quantile tracker maintains a timing threshold Q forcing the ratio between the numbers of measured signal pulses earlier and later than Q to stochastically converge to the assigned ratio A / B, and Q will converge to the A / (A+B) quantile value of the input TBS.

[0061] FIGs. 6 and 7 will be understood with reference to FIG. 8 which illustrates examples of ACT and TBS signals. In FIG. 6, the TBS is connected to the edge trigger inputs of counters 510 and 515. In FIG. 7, the TBS is connected to the edge trigger input of accumulator 535. The action signal ACT and previously stored timing threshold QVal serve as inputs to a temporal threshold generator 550. The temporal threshold generator 550 outputs low at the beginning of each laser cycle when edge- triggered by the ACT signal and changes to high after a certain time interval specified by the timing threshold QVal. The output of the temporal threshold generator 550 provides a binary timing reference signal to decide whether each pulse from the TBS is earlier or later than the timing threshold QVal. As in the detector of FIG. 5A, the counter outputs are sent through multipliers 520, 525, multiplied by predefined values A and B, summed at 530, and provided to latch 540. The incremental quantile value at the output of the summing node 530 will be scaled by a factor then accumulated to the current timing threshold QVal stored in the latch 540 and the accumulated sum, i.e. , the updated timing threshold QVal, will be stored in the latch 540 only when an update signal UPD edge triggers the latch 540 to update its stored timing threshold QVal at the end of the current quantile tracking iteration. The update signal UPD also resets the counters 510 and 515 to 0 before the start of the next quantile tracking iteration by reset circuitry (not shown).

[0062] FIG. 7 illustrates a simplified TBS quantile tracker similar to the VBS quantile tracker of FIG. 5B, but with the addition of a multiplexer 555. The quantile tracker of FIG. 7 uses selector 545 to select either the A or -B input values based on the output of temporal threshold generator 550. The accumulator 535 accumulates the selectedvalues from the selector 545 to its intermediate sum at the edge of the TBS signal. The incremental quantile value at the output of the accumulator 535 will be scaled by a factor then accumulated to the current timing threshold QVal stored in the latch 540 and the accumulated sum, i.e. , the updated timing threshold QVal, will be stored in the latch 540 only when an update signal UPD edge triggers the latch 540 to update its stored timing threshold QVal at the end of the current quantile tracking iteration. The update signal UPD also resets the accumulator 535 to 0 before the start of the next quantile tracking iteration by reset circuitry (not shown). As will be described below, the multiplexer 555 is used to divide the TBS into early and late signals for a subsequent stage of a quantile tracker. The output of the multiplexer 555 is a binary 1 or 0 dependent upon the input signal TBS and the multiplexing control output by the temporal threshold generator 550. As discussed below, the quantile tracker circuit can be crafted as multi-stage tracking circuits to increase photon detection temporal resolution, and multiplexer 555 enables these multi-stage circuits.

[0063] FIG. 8 are graphs of signal magnitude relative to time which illustrate examples of the ACT and TBS signals. As shown there, the ACT signal has a regular period (essentially acting as a clock signal) where each signal pulse represents the beginning of each cycle for laser pulse emission and reflected photon measurement while the TBS reflects arrival times of photons at the SPAD sensor, such as the output signal from the coincidence detection module 340. As shown at 805 and 815, certain time intervals will have multiple signal pulses when multiple coincident photon arrivals are detected, while others at 810 will have missing signal pulses when no coincident photon arrival is detected. This signal with embedded photon detection information is used to form photon arrival time histograms having bins with a varying width for further downstream processing.

[0064] The quantile trackers in FIGs. 5A - 7 illustrate bisectional trackers - outputting a single quantile value and bisecting the input time-based signal (TBS) over one timing threshold. In accordance with the described embodiments, FIGs. 9 - 11 illustrate bisection, trisection, and quadrisection TBS quantile tracker circuits, respectively. The trisection circuits provide two quantile values and trisect the input TBS over two timing thresholds, while the quadrisection circuits provide three quantile values and quadrisect the input TBS over three timing thresholds. Each of the tracker circuits in FIGs. 9 - 11 and 12 - 14 can serve as the building blocks for multi-stagetracker circuits, as discussed below. Quantile tracker circuits configured for photon timing mode (FIGs. 9 - 11 ) record photon arrival times to analyze the transient distribution for depth sensing and 3D imaging. Quantile tracker circuits configured for photon counting mode (FIGs. 12 - 14) count the number of coincident photon arrivals for each pixel during each intensity measurement interval.

[0065] Each of the accumulators in the quantile tracker circuits in FIGs. 9 - 11 for supporting photon timing mode (for depth sensing) can be configured as up counters only without utilizing temporal threshold generators to support photon counting mode (for intensity measuring), as in FIGs. 12 - 14. In photon timing mode, each QVal latch stores the corresponding quantile value QV for the photon arrival time histogram. In photon counting mode, each QVal latch stores the corresponding photon count PC during the gating window interval. In these embodiments, each accumulator may use a variable scaling factor which allows for faster convergence and more stable converged values. An adaptive one-pole filter can be employed in accumulators in each circuit to achieve more stable converged values.

[0066] In each of the embodiments of FIGs. 9 - 14, the control (CTR) signals are provided by a processing device, such as that illustrated in FIG. 29. This includes, in quantile tracker circuits of FIGs. 9 - 11 , the update (UPD) and action (ACT) (or clock) signals. In photon counting circuits of FIGs. 12 - 14, the CTR signals comprise the gating window (WIN) and the update (UPD) signals.

[0067] The embodiments of FIGs. 9 - 11 measure the quantile value QV to determine the distribution of the photon arrival times which can be used to estimate distance in dToF systems. FIG. 9 illustrates a bisection TBS quantile tracker circuit configured for photon timing mode. The TBS is connected to the input of a multiplexer 910 and the edge trigger input of an accumulator 920. The inputs to the selector 915 are set to “1” and “-1” and the output of which are controlled by the output of a temporal threshold generator 905. The ACT (clock) signal is provided to the temporal threshold generator 905, the output of which is based on the ACT signal and the current timing threshold QVal stored in a latch 925. The outputs of multiplexer 910 are two timebased signals. The multiplexer 910 splits input pulses into “later than threshold” pulses for its SL output or “earlier than threshold” pulses for its SE output based on whether a coincident photon arrival pulse is present at the input of the multiplexer 910 andwhether the output of the temporal threshold generator 905 is low (for input pulses routed to the SE output) or high (for input pulses routed to the SL output). As such, the circuit of FIG. 9 bisects the input TBS into pulses earlier and later than the current timing threshold QVal stored in the latch 925. Periodically, the UPD signal will update the quantile tracker for the timing threshold QVal to converge to the 1 / 2 quantile value of the transient distribution of the TBS input. This provides a fixed ratio of 1 / 1 , splitting the input pulses into one of the two outputs - later (SL) or earlier (SE) than the converged timing threshold QVal.

[0068] FIG. 10 illustrates a trisection TBS quantile tracker circuit configured for photon timing mode. As with the tracker in FIG. 9, in FIG. 10 the TBS is provided to a multiplexer 930a to split input pulses into “later than threshold” pulses for its SL output, “intermediate” pulses for its SM output, or “earlier than threshold” pulses for its SE output. The trisection quantile tracker includes two parallel stages. A first stage is comprised of threshold generator 905a, QVal latch 925a, accumulator 920a, and selector 915a. The fixed ratio at the inputs of the selector 915a are set at A = 2 and B = -1. The second stage is comprised of threshold generator 905b, QVal latch 925b, accumulator 920b, and selector 915b. The fixed ratio at the inputs of the selector 915b are set at A = 1 and B = -2. The circuit thus provides two output quantile values, QV2 and QV1 , stored in respective latches 925a and 925b, which control temporal threshold generators 905a and 905b, respectively. The circuit thus provides a higher resolution of the output quantile values, splitting QV into a 2 / 3 quantile value QV2 and a 1 / 3 quantile value QV1. A Thermometer-to-Binary (TM2B) encoder 935a converts the two-bit thermometer code output from threshold generators 905a and 905b into its binary equivalent to select the output of the multiplexer 930a.

[0069] FIG. 11 illustrates a quadrisection TBS quantile tracker circuit configured for photon timing mode. As with the tracker in FIG. 10, in FIG. 11 the TBS is provided to a multiplexer 930b to split input pulses into “later than threshold” pulses for its SL output, “later intermediate” pulses for its SC output, “earlier intermediate” pulses for its SY output, or “earlier than threshold” pulses for its SE output. The quadrisection quantile tracker includes three parallel stages. A first stage is comprised of threshold generator 905c, QVal latch 925c, accumulator 920c, and selector 915c. The fixed ratio at the inputs of the selector 915c are set at A = 3 and B = -1. The second stage is comprised of threshold generator 905d, QVal latch 925d, accumulator 920d, andselector 915d. The fixed ratio at the inputs of the selector 915d are set at A = 1 and B = -1. The third stage is comprised of threshold generator 905e, QVal latch 925e, accumulator 920e, and selector 915e. The fixed ratio at the inputs of the selector 915d are set at A = 1 and B = -3. The circuit thus provides three output quantile values, QV3, QV2 and QV1 , stored in respective latches 925c, 925d and 925e, which control temporal threshold generators 905c, 905d and 905e, respectively. The circuit thus provides a higher resolution of the output quantile values, splitting QV into a % quantile value QV3, a quantile value QV2, and a quantile value QV1 . A TM2B encoder 935b converts the three-bit thermometer code output from threshold generators 905c, 905d and 905e into its binary equivalent to select the output of the multiplexer 930b.

[0070] In each of the aforementioned quantile trackers of FIGs. 9 - 11 , the quantile tracking adapts to the arrival of the TBS and the value stored in each QVal latch will automatically converge to the pre-specified quantile values QV for the transient distribution of the pulse arrival timing as the trackers adapt to the pulse arrival times conveyed by the TBS.

[0071] FIGs. 12 - 14 illustrate tracking circuits configured for photon counting mode. As noted above, in photon counting mode, each QVal latch stores the corresponding photon count PC during the gating window interval. Photon counting mode provides an actual photon count during each assigned temporal window. In an example embodiment, the controller assigns timing and each pixel assigns the number of arrival photons during each assigned time interval. The tracking circuits of FIGs. 12 - 14 are similar to those of FIGs. 9 - 11 but include control signals comprising one or multiple gating window (WIN) signals.

[0072] FIG. 12 illustrates a bisection tracker circuit configured for photon counting mode. The embodiment of FIG. 12 still uses a TBS as input to a multiplexer 1210 and the edge trigger input of an accumulator 1220. However, there is no temporal threshold generator and feedback loop based on the QVal latch 1225, as in the embodiments of FIGs. 9 - 11. The selector 1215 takes as inputs a “1” or “0”, where the “1” is selected within the gating window (i.e. , when WIN signal is high), as the value is simply a count incremented by the accumulator 1220 upon each pulse of the TBS within the gating window. The gating window (WIN) signal activates both the multiplexer 1210 to select one of the two signal outputs SL or SE, and the accumulator to increment a count.When edge-triggered by the UPD signal, the QVal latch 1225 stores the corresponding photon count based on the accumulator count of pulses during the gating window. The output of the QVal latch 1225 is thus the photon count PC1 . The outputs of multiplexer 1210 are two time-based signals, splitting input pulses into “within window” pulses for its SL output or “beyond window” pulses for its SE output.

[0073] FIG. 13 illustrates a trisection tracker circuit configured for photon counting mode. The trisection tracker utilizes two gating window signals WINO and WIN1. As with the tracker in FIG. 12, in FIG. 13 the TBS is provided as the input to a multiplexer 1210a, with the multiplexer 1210a splitting input pulses into “within window 1” pulses for its SL output, “within window 0” pulses for its SM output, or “beyond window” pulses for its SE output based on the binary control code from TM2B 1235a and the TBS input. The trisection tracker of FIG. 13 includes two parallel stages. A first stage is comprised of QVal latch 1225a, accumulator 1220a, and selector 1215a. The inputs at selector 1215a are “1” and “0”. The second stage is comprised of QVal latch 1225b, accumulator 1220b, and selector 1215b, with the inputs at selector 1215b comprising “1” and “0”. Each selector 1215a and 1215b is controlled by a respective gating window signal WIN1 and WINO. A TM2B encoder 1235a converts the two-bit thermometer code from window signals WINO and WIN1 into its binary equivalent to select the output of the multiplexer 1210a. The PC1 and PC2 outputs thus provide photon counts during different gating windows, which are controlled by gating window signals WINO and WIN1 provided by a processing device such as that shown in FIG. 29.

[0074] FIG. 14 illustrates a quadrisection tracker circuit configured for photon counting mode. The quadrisection tracker utilizes three gating window signals WINO, WIN1 and WIN2. As with the tracker in FIG. 13, in FIG. 14 the TBS is provided as the input to a multiplexer 1210b, with the multiplexer 1210b splitting input pulses into “within window 2” pulses for its SL output, “within window 1” pulses for its SC output, “within window 0” pulses for its SY output, or “beyond window” pulses for its SE output based on the binary control code from TM2B 1235b and the TBS input. The quadrisection tracker of FIG. 14 includes three parallel stages. A first stage is comprised of QVal latch 1225c, accumulator 1220c, and selector 1215c. The second stage is comprised of QVal latch 1225d, accumulator 1220d, and selector 1215d. The third stage is comprised of QVal latch 1225e, accumulator 1220e, and selector 1215e. The inputs at selectors 1215c, 1215d and 1215e are “1” and “0”, with each selector1215c, 1215d and 1215e controlled by a respective gating window signal WIN2, WIN1 , and WINO. A TM2B encoder 1235b converts the three-bit thermometer code from window signals WINO, WIN1 , and WIN2 into its binary equivalent to select the output of the multiplexer 1210b. The PC1 , PC2, and PC3 outputs thus provide photon counts during different gating windows, which are controlled by gating window signals WINO, WIN1 , and WIN2 provided by a processing device such as that shown in FIG. 29

[0075] FIGs. 15A - 15C illustrate the bisection, trisection and quadrisection quantile tracker circuits described above as B UNIT, T UNIT, and Q UNIT blocks, respectively, each with their respective inputs and outputs. As discussed below, each of these fundamental quantile tracker circuits can be used as building blocks for multistage quantile tracking circuits configured for either photon timing mode or photon counting mode. Bisection tracker circuits bisect photon arrival pulses by splitting photon arrival time transient distribution by a 1 :1 quantile tracker, thus comprising a median tracker. Bisection tracker circuits take as input the control signals CTR and the time-based photon detection TBS, output SL and SE, and either the quantile value QV1 or photon count PC1. Trisection tracker circuits trisect photon arrival pulses by splitting photon arrival time transient distribution by a 1 :2 quantile tracker and a 2:1 quantile tracker. Trisection tracker circuits take as input the control signals CTR and the time-based photon detection TBS, output SL, SM and SE, and either the quantile values QV1 , QV2 or photon counts PC1 , PC2. Quadrisection tracker circuits quadrisect photon arrival pulses by splitting photon arrival time transient distribution by a 1 :3 quantile tracker, a 1 :1 median tracker, and a 3:1 quantile tracker. Quadrisection tracker circuits take as input the control signals CTR and the time-based photon detection TBS, output SL, SC, SY and SE, and either the quantile values QV1 , QV2, QV3 or photon counts PC1 , PC2, PC3.

[0076] FIGs. 16 - 18 show configurations of multi-stage tracking circuits built on the fundamental bisection, trisection and quadrisection tracker circuits disclosed herein. For each block (B UNIT, T UNIT or Q UNIT), the QV or PC output is referenced by a readout identifier B# which is used to illustrate distribution and intensity calculation. FIG. 16 shows a three-stage bisection quantile tracking circuit with a tree structure wherein the SL and SE outputs of one bisection tracker circuit in a stage comprise the TBS inputs to two corresponding bisection tracker circuits in the next stage. FIG. 17 shows a two-stage quadrisection quantile tracking circuit, and FIG. 18shows a three-stage trisection tracking circuit, with an alternative set of readout identifiers (in bold italics) representing the outputs of a two-stage trisection tracking circuit. In such multi-stage circuits, gating window signals are pre-determined globally for photon counting mode, while timing thresholds are tracked locally for photon timing mode.

[0077] Each of the multi-stage tree structures shown in FIGs. 16 - 18 may be operated in two different modes to detect histogram distribution peaks. In a feedforward full range mode, for each subsequent stage, all histogram bins from the previous stage are sectioned to construct a complete histogram tree. In an iterative peak search mode, for each iteration, the narrowest histogram bin from the current iteration is sectioned for more bins around the histogram distribution peak in subsequent iteration. Table 1 compares the hardware resources needed for each of the four embodiments illustrated in FIGs. 16 - 18, showing the number of stages, number of units (basic tracker circuits), number of trackers, number of histogram bins, number of time-gated slices and photon splitting.Table 1

[0078] In FIGs. 16 - 18, each fundamental tracking B UNIT, T UNIT, or Q UNIT is shown with each readout marked with corresponding identifiers B#. For photon timingmode, each readout B from the multi-stage tree structure represents a quantile value QV, i.e., bin boundary of the histogram. For photon counting mode, each readout B from the multi-stage tree structure represents a photon count PC, i.e., intensity value during the time-gated interval. The intensity values for each time-gated slice can be obtained with simple calculations listed in the tables below. For example, in a three- stage binary multi-stage tree structure, time-gated slice number 4 PC value is given by subtracting the readout values at B6 and B5 from B4. (Each of the dashes indicated in the tables below represents a subtraction sign.)Table 2Table 3

[0079] An alternative to the above-described feedforward full range mode for the multi-stage tree structures, is an iterative peak search mode for a recursive structure to be described below. FIG. 19 is a flowchart illustrating the iterative peak search mode. Generally, for each new iteration, the method sections the narrowest histogram bin from the current iteration for more bins around a distribution peak. In the beginning of an iterative peak search at step 1905, the system first initializes quantile values, an early bound, and a late bound of the histogram to define a pre-determined histogram search range by the early and late bounds and assign the quantile values evenly within the histogram search range in a quantile tracker circuit in the recursive structure. At step 1910, during a subsequent iteration, the laser pulses are repeated, photons are received, and the quantile tracker circuit is updated, for up to N cycles. At step 1912,the method calculates the lowest difference between the early bound, the quantile values, and the late bound. At step 1915, a determination is made as to whether or not the search results are convergent. The convergence conditions can be defined by the lowest difference calculated at step 1912 becoming lower than a pre-determined histogram resolution requirement, or by the number of repetitions at step 1910 exceeding a pre-determined maximum repetition limit. If search results are not convergent, at step 1915, the method updates quantile values, early bound and late bound at step 1920, and returns to step 1910 repeating the next laser pulse. If the search results are convergent, the process ends.

[0080] FIGs. 20 - 23 are flowcharts illustrating methods of operating the system of FIG. 3. The methods of FIGs. 20 - 23, and the system of FIG. 3, utilize multi-SPAD spatially correlated clusters supporting photon coincidence detection for a threshold number of photons arriving within a fixed or adjustable temporal coincidence window to substantially improve background suppression. The disclosed methods implement an automatically adjustable threshold at the pixel-level, by checking photon counting information, to keep the coincident event rate within an optimal operative zone (with no signal events loss, while filtering out background noise).

[0081] FIG. 20 is a flowchart illustrating a photon detection process. At step 2010, the SPAD is activated for photon detection. At step 2015, a determination is made as to whether a photon is detected and if so, the raw photon arrival data is forwarded to a number “L” of neighboring pixels and raw photon arrival data received from a number “M” of neighboring pixels at step 2020. At step 2025, photon coincidence is determined. Coincidence detection identifies simultaneous photon detection events within a defined spatial correlation window across multiple SPADs in the array. Coincidence detection compares consecutive photon arrivals within a temporal coincidence window against a coincidence detection threshold. At step 2030, a determination is made as to whether coincidence is detected and if so, a coincident photon arrival signal is output at step 2035. The coincident photon arrival output may refer to a temporal coincidence of photons detected within a predefined period of time and related to one of the M neighboring pixels from whom photons has been received.

[0082] FIG. 21 is a flowchart illustrating a quantile tracking process. As noted above, quantile tracking receives photon arrival times to analyze the transientdistribution for depth sensing and 3D imaging (for photon timing mode), or to measure the incoming light intensity (for photon counting mode). The process is executed at each stage of a multi-stage tracking circuit. At step 2110, the process of FIG. 20 for photon detection provides a coincident photon arrival signal. At step 2115, if a coincident photon arrival signal is received, then a determination is made at step 2117 as to whether the system is in photon counting mode or photon timing mode. If the quantile tracking circuit is operating in photon timing mode, then at step 2120 the quantile trackers are updated. If the tracking circuit is operating in photon counting mode, then at step 2125 the photon counters are updated. At step 2135, a determination is made as to whether the current stage is finished and if so, photon counts, or quantile values are stored at step 2140.

[0083] FIG. 22 is a flowchart illustrating a LiDAR in-pixel process, which is one application for the output of the quantile tracking process of FIG. 21 . At step 2215, the quantile tracking process of FIG. 21 should have resulted in storage of the photon counts or quantile values of the current stage if it has been finished. A determination is made at step 2220 as to whether the stage outputs are stored. If so, then at step 2225, a determination is made as to whether the operating mode is for a full range mode or a peak search mode. Full-range tree traversal measures a complete histogram throughout the full-scale depth range of a transient distribution and supports multiple reflections and semi-transparent objects with maximum measurement accuracy and full measurement time. Successive peak searching measures a partial histogram centered around the peak of a transient distribution and contributes to avoid background-induced saturation without decreasing the measurement accuracy or increasing the measurement time. In full range mode, the current stage results are updated at step 2230. In peak search mode, the current iteration results (FIG. 19) are updated at step 2235 (which is the same as updating the iteration results at step 1910 in the flowchart illustrated in FIG. 19). A determination is made at step 2240 as to whether all stages are finished in full range mode, or all iterations are finished in peak search mode (same as checking search convergence at steps 1912 and 1915 in the flowchart illustrated in FIG. 19). If so, then at step 2245, the photon counts, or quantile values are output to a memory module, such as the bin boundary memory 355 in the system 300, as illustrated in FIG. 3.

[0084] FIG. 23 is a flowchart illustrating the overall method of operating the system of FIG. 3. At step 2310, a coincidence detection threshold is assigned. The coincidence detection threshold may comprise a maximum allowable time difference between photon detections for them to be considered as occurring in coincidence. At step 2320, a laser pulse is emitted. At step 2325, the photon detection process of FIG. 22, the quantile tracking process of FIG. 21 , and the LiDAR in-pixel process of FIG. 22 occur. A determination is made at step 2330 as to whether the current laser cycle ends. If not, step 2325 will be repeated. When the current laser cycle ends, a determination is made at step 2335 as to whether photon measurement is complete. If not, steps 2320 and 2325 will be repeated. Once photon measurement is complete, per-pixel photon detection data (photon counts or quantile values) is output at step 2340. Coincidence threshold adjustment dynamically adjusts detection threshold for improved background suppression. A determination is made at step 2345 as to whether the coincidence detection threshold is to be updated and if so, the threshold is updated at step 2350.

[0085] FIG. 24 illustrates another embodiment of a quantile tracking circuit with a recursive structure configured for an iterative peak search mode with gating windows. In embodiments, one need not build out an entire multi-stage tree structure in a tracking circuit. Rather, one may implement only a portion of a tree structure and use an iterative peak search method to automatically search for histogram peaks from coarse to fine bin resolutions with a recursive structure in the circuit. This further saves circuit die space, footprint, memory size, and data readout rate. FIG. 24 illustrates a gating window controller 2405 providing gating window timing to a bisection, trisection or quadrisection tracker circuit block 2410. The B,T,Q UNIT 2410 may be any of the bisection, trisection or quadrisection tracker circuits disclosed herein, but for simplification, in FIG. 24 they are represented as one circuit block 2410. The QV read signal from the B,T,Q UNIT 2410 is provided to the bin boundary memory 2415 and the iteration control 2420. The iteration control 2420 outputs a gate high (GH) and gate low (GL) signals to gate latches 2425 and 2430, as well as the action signal (ACT) and update signal (UPD). In the iterative peak search mode, the quantile tracking circuit with a recursive structure will find the narrowest histogram bin as an indicator of distribution peak in current iteration and analyze it further in next iteration. Update schemes in one iteration for each aforementioned tracker circuit for the B,T,Q UNIT2410 are illustrated in the below pseudo code. The below pseudo code is a routine that searches for the sub-interval with narrowest bin width in the current iteration. After finding the narrowest bin width, the quantile values, the early and late bounds are assigned for the next iteration in the recursive structure. In FIG. 24, the symbol Vk(i) represents all the QV read signals for the quantile values or photon counts from the bisection, trisection or quadrisection tracker circuit in the B,T,Q UNIT 2410 in the current iteration i. The symbol Vk(i) represents V1 (i) for the bisection tracker circuit, V1 (i),V2(i) for the trisection tracker circuit, and V1 (i),V2(i),V3(i) for the quadrisection tracker circuit, respectively. Similarly, the symbol Vk(i+i) represents all the QV write signals for the quantile values or photon counts to the bisection, trisection or quadrisection tracker circuit in the B,T,Q UNIT 2410 for the next iteration i+1. The symbol Vk(i+1 ) represents V1 (i+1 ) for the bisection tracker circuit, V1 (i+1 ), V2(i+1 ) for the trisection tracker circuit, and V1 (i+1 ), V2(i+1 ), V3(i+1 ) for the quadrisection tracker circuit, respectively. The symbol V0(i+1 ) and Vx(i+1 ) represent the gate low (GL) write and gate high (GH) write signals to control the starting and ending time, respectively, of the gating window signal generated by the gating window controller 2405 for the next iteration i+1 . The symbol Vx(i+1 ) represents V2(i+ 1 ), V3(i+ 1 ), and V4(i+ 1 ) for the bisection, trisection or quadrisection tracker circuit, respectively. The below pseudo code illustrates how the iteration control module 2420 determines the values of V0(i+1 ) to Vx(i+1 ) for the next iteration i+1 from the values of V0(i) to Vx(i) in the current iteration i.

[0086] For a bisection tracker circuit:W1 = V2(i) - V1(i);W0 = V1(i) - V0(i); if W1 < W0 { V2(i+1 ) ~ V2(i);V1 (i+1 ) = [V2(i) + V1(i)] / 2;V0(i+1 ) = V1(i); } else{ V2(i+1 ) = V1 (i);V1 (i+1 ) = (V1(i) + V0(i)] / 2;V0(i+1) = V0(i): }.

[0087] For a trisection tracker circuit:W2 = V3(i) -V2(i);W1 = V2(i) -V1(i);WO = V1(i) -V0(i); if W2 < W1 & W2 < WO{ V3(i+1) = V3(i);V2(i+1) = (2 x V3(i) + V2(i)] / 3;V1 (i+1 ) = (2 x V2(i) + V3(i)] / 3;V0(i+1 ) = V2(i); } else if W1 < W2 & W1 < WO { V3(i+1) = V2(i);V2(i+1 ) = [2xV2(i) + V1(i)] / 3;V1 (i+1 ) = [2xV1(i) + V2(i)] / 3;V0(i+1) = V1(i); } else{ V3(i+1) = V1(i);V2(i+1) = [2 x V1 (i) + V0(i)] / 3;V1 (i+1 ) = [2 x V0(i) + V1(i)] / 3;V0(i+1) = V0(i); }.

[0088] And for a quadrisection tracker circuit:W3 = V4(i) - V3(i);W2 = V3(i) -V2(i);W1 = V2(i) - V1(i);WO = V1(i) -V0(i); if W3 < W2 & W3 < W1 & W3 < WO{ V4(i+1) = V4(i);V3(i+1) = [3xV4(i) + V3(i)] / 4;V2(i+1) = [V4(i) + V3(i)] / 2;V1(i+1) = [3xV3(i) + V4(i)] / 4;V0(i+1) = V3(i); } else if W2 < W3 & W2 < W1 & W2 < WO{ V4(i+1) = V3(i);V3(i+1) = [3xV3(i) + V2(i)] / 4;V2(i+1) = [V3(i) + V2(i)] / 2;V1 (i+1 ) = [3 x V2(i) + V3(i)] / 4;V0(i+1 ) = V2(i); } else if W1 < W3 & W1< W2 & W1 < WO{ V4(i+1 ) = V2(i);V3(i+1 ) = [3 x V2(i) + V1(i)] / 4;V2(i+1 ) = [V2(i) + V1(i)] / 2;V1 (i+1 ) = [3 x V1 (i) + V2(i)] / 4;VO(i+1 ) = V1 (i); } else{ V4(i+1 ) = V1 (i);V3(i+1 ) = [3 x V1 (i) + V0(i)] / 4;V2(i+1 ) = [V1 (i) + V0(i)] / 2;V1 (i+1 ) = [3 x V0(i) + V1(i)] / 4;VO(i+1 ) = VO(i); }

[0089] FIGs. 25 and 26 are block diagrams illustrating a further embodiment of the system wherein an alternative methodology trades spatial resolution for depth resolution in full range mode and peak search mode, respectively. The system supports trading spatial resolution for depth resolution by clustering or “binning” SPADs and in-pixel circuitry of multiple adjacent pixels into super-pixels for an increased number of SPADs per pixel, while the clustered or “binned” in-pixel circuitry jointly generates a distributed histogram with denser bins.

[0090] FIG. 25 represents a super-pixel with a configuration for full range mode comprising an S-stage R-nary tree combined with in-pixel circuitry of Rsadjacent pixels, where the R-nary tree indicates one of a binary, ternary, or quaternary tree, and a B,T,Q UNIT represents one of a bisection, trisection, or quadrisection circuit. The S- stage R-nary tree receives an aggregated TBS at the first stage unit. The aggregated TBS-A signal is a signal aggregated from output signals of coincidence detection modules 340 in each of the multiple combined pixels. In full range mode, this “superpixel” TBS-A signal input is divided by the S-stage R-nary tree and sent to Rspixels each has an N-stage R-nary tree to track R<N+S) histogram bins jointly. In some applications, the LiDAR sensor hardware is already present in the system for certain applications with pre-defined depth resolution, while other applications require higher depth resolution thereby requiring more bins in a histogram. FIG. 25 represents a treestructure with multiple output values in each stage which , when combined with output values from Rsadjacent pixels, supports a higher depth resolution. Post processing module reads all the output values from the S-stage R-nary tree and the Rscombined pixels and performs a simple calculation to obtain R(N+S)- 1 histogram boundaries for photon arrival time of the corresponding super-pixel.

[0091] FIG. 26 represents a super-pixel with a configuration for peak search mode combing in-pixel circuitry of R(N+S-1)adjacent pixels sharing an aggregated TBS-A signal which is a signal aggregated from output signals of coincidence detection modules 340 in each of the multiple combined pixels. FIG. 26 illustrates multiple (N+S)-iteration R-nary units from each of the R<N+S-1) adjacent pixels, each R-nary units comprising a recursive structure configured to search by N+S iterations, as illustrated in FIG. 24, where the TBS-A input is shared by all the R-nary units. In peak search mode, this “super-pixel” TBS-A signal input is sent to R(N+S-1 )pixels each has an (N+S)- iteration R-nary unit to track R<N+S) histogram bins jointly. Each (N+S)-iteration R-nary unit is searched by N+S iterations according to a respective search pattern assigned from one of the R(N+S-1)predefined search patterns to traverse the histogram and support a higher depth resolution. Post processing module reads all the output values from the last (N+S)thiteration of the R(N+S-1)combined pixels and performs a simple calculation to obtain R<N+S) - 1 histogram boundaries for photon arrival time of the corresponding super-pixel.

[0092] FIG. 27 illustrates one application for a single photon dToF system 300 such as that illustrated in FIG. 3. FIG. 27 illustrates a single photon dToF system 300a which may be used for supervised training of a depth estimation network 2710. Deeplearning based end-to-end depth estimation networks can be trained without explicitly solving the per-pixel depth map values. Raw quantile values from the system 300a can be used directly as inputs to the networks. The output of the system 300a may be provided to a depth estimation network 2710 which produces estimated depth maps 2715. A loss function calculation 2725 is based on ground truth depth maps 2720. The loss function quantifies how well or poorly the network 2710 is performing by comparing the model's estimated depth map values to the ground truth depth map values. The ground truth 2720 can be derived from RGB-D training datasets 2730 and / or single photon training datasets 2735. The photon timestamp data 2705 may be provided from a photon timestamp emulator 2740 or from the single photon trainingdatasets 2735 which are comprised of actual photon timestamp data and corresponding ground truth depth maps. The emulator 2740 emulates photon timestamps according to input RGB-D images from the RGB-D training datasets 2730 which are comprised of actual RGB-D images and corresponding ground truth depth maps.

[0093] FIG. 28 illustrates another application for a single photon dToF system 300 such as that illustrated in FIG. 3. FIG. 28 illustrates a single photon dToF system 300a which may be used for supervised training of a 3D object detection network 2810. Deep-learning based end-to-end 3D object detection networks can be trained without explicitly solving the per-pixel depth map values. Raw quantile values from the system 300a can be used directly as inputs to the networks. The output of the system 300a may be provided to a 3D object detection network 2810 which produces detected object classes, bounding boxes, and scores 2815. A loss function calculation 2825 is based on ground truth object classes and bounding boxes 2820. The loss function quantifies how well or poorly the network 2810 is performing by comparing the model's detected results to the ground truth results. The ground truth 2820 can be derived from RGB-D training datasets 2830 and / or single photon training datasets 2835. The photon timestamp data 2805 may be provided from a photon timestamp emulator 2740 or from the single photon training datasets 2835 which are comprised of actual photon timestamp data and corresponding ground truth object classes and bounding boxes. The emulator 2740 emulates photon timestamps according to input RGB-D images from the RGB-D training datasets 2830 which are comprised of actual RGB-D images and corresponding ground truth object classes and bounding boxes.

[0094] FIG. 29 illustrates a network processing device 2900 which may be used to implement certain post processing operations in the embodiments discussed herein. In embodiments, the network processing device 2900 may contain multiple instances of a component, such as multiple processing units, processors, memories, interfaces, etc.

[0095] The device 2900 may comprise a central processing unit (CPU) 2915, a graphics processing unit (GPU) 2910, a memory 2920, a mass storage device 2930, and an I / O interface 2960 connected to a bus 2970. The I / O interface 2960 may be connected to the timing and sequence controller 375 and the single photon dToFsystem 2300. The bus 2970 may be one or more of any type of several bus architectures including a memory bus or memory controller, a peripheral bus, or the like. A network interface 2950 enables the network processing device 2900 to communicate over a network 50 with other processing devices (not shown in FIG. 29).

[0096] The mass storage device 2930 may comprise any type of storage device configured to store data, programs, and other information and to make the data, programs, and other information accessible via the bus 2970. The mass storage device 2930 may comprise, for example, one or more of a solid-state drive, hard disk drive, a magnetic disk drive, an optical disk drive, or the like. The mass storage device 2930 includes distance, pixel, and post processing instructions 2935 which when executed by the CPU 2915 (or GPU 2910) cause the processor to perform the methods described herein. The mass storage 2930 may include code in the form of application modules and data stored thereon, comprising the instructions 2935 for causing the CPU 2915 (or GPU 2910), when executing the instructions 2935 loaded to the memory 2920, to implement the components of a distance estimation engine 2922, a post processing module 2926 for per-pixel photon detection data, and an iterative search module 2924.

[0097] The CPU 2915 (or GPU 2910) may comprise any type of electronic data processor. The memory 2920 may comprise any type of system memory such as static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), a combination thereof, or the like. In an embodiment, the memory 2920 may include ROM for use at boot-up, and DRAM for program and data storage for use while executing programs. In embodiments, the memory 2920 is non-transitory. In one embodiment, the memory 2920 includes the computer-readable instructions 2935 that are executed by the processor(s) 2910 and 2915 to implement examples of the disclosed embodiments.

[0098] In one embodiment, the mass storage 2930 code for implementing instances of the distance estimation engine 2922, the post processing module 2926 for per-pixel photon detection data, and the iterative search module 2924, all of which are configured to cause the processor(s) 2910 and 2915, when executing the instructions 2935 thereof, to perform the functions described for such elements herein. In embodiments, the distance estimation engine 2922 performs the functions of thedepth estimation network 2710 illustrated in FIG. 27, the post processing module 2926 performs the functions of the post processing module 370 illustrated in FIG. 3, and the iterative search module 2924 performs the functions of the iteration control circuit 360 illustrated in FIG. 3.

[0099] The disclosed embodiments support robust photon arrival time histogram analysis against strong ambient and background lighting by employing photon coincidence detection and photon arrival time quantile tracking. By employing quantile tracking of the photon arrival time instead of histogram storage and updating, the system and methods can achieve smaller in-pixel circuitry, lower per-pixel power dissipation, and lower per-pixel data throughput. By successively refining the tracking range for quantile tracking of the photon arrival time, detector time gating centered around the histogram peak can be achieved to avoid background-induced saturation without increasing the measurement time. By utilizing an automatically adjustable coincidence threshold at the pixel-level determined by event counting information, the coincident event rate can be maintained within an optimal operative zone with no signal events loss while filtering out background. By integrating photon coincidence detection, photon arrival time analysis, and automatically coincidence thresholding, efficient single-photon depth sensing and 3D imaging can be achieved with better performance, lower cost, data throughput, and power. By employing the disclosed inpixel processing circuitry, performance, and cost of direct ToF sensors can be substantially improved.

[0100] For the purposes of this document, it should be noted that the dimensions of the various features depicted in the FIGs. may not necessarily be drawn to scale.

[0101] For purposes of this document, reference in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “another embodiment” may be used to describe different embodiments or the same embodiment.

[0102] For the purposes of this document, a connection may be a direct connection or an indirect connection (e.g., via one or more other parts). In some cases, when an element is referred to as being connected or coupled to another element, the element may be directly connected to the other element or indirectly connected to the other element via intervening elements. When an element is referred to as being directly connected to another element, then there are no intervening elements between theelement and the other element. Two devices are “in communication” if they are directly or indirectly connected so that they can communicate electronic signals between them.

[0103] It is understood that the present subject matter may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this subject matter will be thorough and complete and will fully convey the disclosure to those skilled in the art. Indeed, the subject matter is intended to cover alternatives, modifications, and equivalents of these embodiments, which are included within the scope and spirit of the subject matter as defined by the appended claims. Furthermore, in the preceding detailed description of the present subject matter, numerous specific details are set forth in order to provide a thorough understanding of the present subject matter. However, it will be clear to those of ordinary skill in the art that the present subject matter may be practiced without such specific details.

[0104] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses (systems) and computer program products according to embodiments of the disclosure. It will be understood that each step or block of the flowchart illustrations and / or block diagrams, and combinations of steps or blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable instruction execution apparatus, create a mechanism for implementing the functions / acts specified in the flowchart step or steps, and / or block diagram block or blocks.

[0105] The description of the present disclosure has been presented for purposes of illustration and description but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The aspects of the disclosure herein were chosen and described in order to best explain the principles of the disclosure and the practical application, and toenable others of ordinary skill in the art to understand the disclosure with various modifications as are suited to the particular use contemplated.

[0106] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

CLAIMSWhat is claimed is:1 . A distance measurement system, comprising: a plurality of single photon detection sensors each outputting photon detection signals, the photon detection signals at least partially based on photons reflected from a pulse emitted by a laser toward an object; coincidence detection circuitry including a configurable coincidence threshold coupled to each of the plurality of single photon detection sensors in an array, the coincidence detection circuitry identifies simultaneous photon detection events within a defined spatial correlation window across the plurality of SPADs in the array during a temporal coincidence window and over the configurable coincidence threshold; photon detection processing circuitry coupled to the coincidence detection circuitry, including: tracking circuitry detecting photon arrival events, the tracking circuitry including at least one of: a quantile tracking circuit and a photon counting circuit; and iteration control circuitry updating the tracking circuitry for one mode of: throughout a transient distribution and around a transient distribution peak; and pixel data output circuitry outputting per-pixel photon detection data.

2. The distance measurement system of claim 1 wherein the photon detection processing circuitry comprises photon counting circuitry.

3. The distance measurement system of claim 1 wherein the photon detection processing circuitry comprises photon arrival timing circuitry.

4. The distance measurement system of any of claims 1 or 3 wherein the tracking circuitry comprises quantile value calculation circuitry.

5. The distance measurement system of claim 4 wherein the quantile tracking circuitry comprises a trisection circuitry outputting two quantile values.

6. The distance measurement system of any of claims 4 through 5 wherein the quantile tracking circuitry comprises a quadrisection circuitry outputting three quantile values.

7. The distance measurement system of any of claims 1 through 2 wherein the tracking circuitry comprises a trisection circuitry tracking photon counts over two gating windows and outputting two photon count values.

8. The distance measurement system of any of claims 1 through 2 wherein the tracking circuitry comprises a quadrisection circuitry tracking photon counts over three gating windows and outputting three photon count values.

9. The distance measurement system of any of claims 1 through 8 wherein the tracking circuitry comprises a plurality of stages, each stage comprising one or more of a bisection tracking circuit, a trisection tracking circuit or a quadrisection tracking circuit.10 The distance measurement system of claim 9 wherein the tracking circuitry includes a feedforward full range mode to traverse all bin boundaries of a histogram in a tree structure.11 . The distance measurement system of claim 9 wherein the system includes an iterative peak search mode to successively track histogram bin boundaries around a distribution peak.

12. The distance measurement system of claim 1 wherein the tracking circuitry is responsive to a gating window which tracks photon arrival timing or photon counting during the gating window.

13. The distance measurement system of any of claims 1 through 12 wherein the tracking circuitry from adjacent pixels may be combined.

14. The distance measurement system of any of claims 1 through 13 wherein the single photon detection sensors comprise at least one single-photon avalanche diode (SPAD).

15. A distance measurement device, comprising: a laser configured to emit laser pulses toward an object; pulse detection circuitry, comprising: a plurality of single photon detection sensors each outputting photon detection signals, the photon detection signals at least partially based on photons reflected from a pulse emitted by the laser toward the object; coincidence detection circuitry including a configurable coincidence threshold coupled to each of the plurality of single photon detection sensors in an array, the coincidence detection circuitry identifies simultaneous photon detection events within a defined spatial correlation window across the plurality of SPADs in the array during a temporal coincidence window and over the configurable coincidence threshold; photon detection processing circuitry coupled to the coincidence detection circuitry, including: tracking circuitry detecting photon arrival events, the tracking circuitry including at least one of: a quantile tracking circuit and a photon counting circuit; anditeration control circuitry updating the tracking circuitry for one mode of: throughout a transient distribution and around a transient distribution peak; pixel data output circuitry outputting per-pixel photon detection data; a non-transitory memory storage comprising instructions; and one or more processors in communication with the memory, wherein the one or more processors execute the instructions to: control the laser output pulses; and determine a distance to the object.

16. The distance measurement device of claim 15 wherein the photon detection processing circuitry comprises photon counting circuitry.

17. The distance measurement device of claim 15 wherein the photon detection processing circuitry comprises photon arrival timing circuitry.

18. The distance measurement device of any of claims 15 or 17 wherein the tracking circuitry comprises quantile value calculation circuitry.

19. The distance measurement device of claim 18 wherein the quantile tracking circuitry comprises a trisection circuitry outputting two quantile values.

20. The distance measurement device of any of claims 18 through 19 wherein the quantile tracking circuitry comprises a quadrisection circuitry outputting three quantile values.

21. The distance measurement device of any of claims 15 through 16 wherein the tracking circuitry comprises a trisection circuitry tracking photon counts over two gating windows and outputting two photon count values.

22. The distance measurement device of any of claims 15 through 16 wherein the tracking circuitry comprises a quadrisection circuitry tracking photon counts over three gating windows and outputting three photon count values.

23. The distance measurement device of any of claims 15 through 22 wherein the tracking circuitry comprises a plurality of stages, each stage comprising one or more of a bisection tracking circuit, a trisection tracking circuit or a quadrisection tracking circuit.

24. The distance measurement device of claim 23 wherein the tracking circuitry includes a feedforward full range mode to traverse all bin boundaries of a histogram in a tree structure.

25. The distance measurement device of claim 23 wherein the device includes an iterative peak search mode to successively track histogram bin boundaries around a distribution peak.

26. The distance measurement device of claim 15 wherein the tracking circuitry is responsive to a gating window which tracks photon arrival timing or photon counting during the gating window.

27. The distance measurement device of any of claims 15 through 26 wherein the tracking circuitry from adjacent pixels may be combined.

28. The distance measurement device of any of claims 15 through 27 wherein the single photon detection sensors comprise at least one single-photon avalanche diode (SPAD).

Images