Systems and methods for three-dimensional sensing with single-photon cameras for resource-constrained applications

Abstract

A method and system are provided for rapidly determining equi-depth histogram (ED histogram) boundaries for single photon arrival times without storing complete photon timestamp histories. A plurality of control values (CVs) is initialized for a plurality of binners that classify photon events as early or late relative to the respective CVs. Upon detecting photon arrivals, early and late arrival counts are used to update the CVs using an improved stepping strategy. The disclosed stepping strategy includes scaling a base step size, smoothing with exponential smoothing, temporally decaying the smoothed step size, and further smoothing to produce the step size for updating the CV. The disclosed stepping approach enables the CVs to rapidly converge to the desired quantile locations with low bias and variance compared to previous methods. The optimized stepping strategy is robust to noise and background light, providing accurate distance estimates while reducing bandwidth requirements.

Description

SYSTEMS AND METHODS FOR THREE-DIMENSIONAL SENSING WITH SINGLE-PHOTONCAMERAS FOR RESOURCE-CONSTRAINED APPLICATIONSCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 660,420, entitled “SYSTEMS AND METHODS FOR THREE-DIMENSIONAL SENSING WITH SINGLE-PHOTON CAMERAS FOR RESOURCE-CONSTRAINED APPLICATIONS,” and filed June 14, 2024, the entire contents of which are hereby incorporated by reference for all purposes.GOVERNMENT SUPPORT

[0002] This invention was made with government support under Grant No. 2138471 awarded by the National Science Foundation. The U.S. Government has certain rights in the invention.FIELD

[0003] The disclosure relates to equi-depth histograms (ED histograms) (also referred to as equiheight histograms (EH histograms)), and more particularly to ED histograms captured by single-photon sensing 3D cameras.BACKGROUND

[0004] Single-photon cameras (SPCs) based on single-photon avalanche diode (SPAD) technology are increasingly popular for dense three-dimensional (3D) sensing and imaging applications. SPCs combine extreme sensitivity and the time-of-flight principle to capture scene distance maps with high resolution. Each pixel of an SPC measures the round-trip time taken for a laser pulse to travel from the camera to the scene and back by sampling many photon arrival timestamps over repeated laser cycles. This allows reconstruction of the true "transient" laser pulse shape to estimate the time-of-flight and thus the scene depth.

[0005] Conventional SPC pixels construct equi-width histograms (EW histograms) of the photon arrival times, with the location of the histogram peak used to estimate depth. However, forming these ED histograms requires high in-pixel memory and power consumption, as well as high bandwidth to transfer large amounts of data off the sensor chip. The volume of ED histogram data can reach several gigabytes per second for megapixel resolution SPCs operating at video rates, exceeding the capabilities of current interfaces and hardware.

[0006] An alternative approach is to use equi-depth histograms (ED histograms) (or equi-height histograms (EH histograms), which can represent peaky transient distributions more efficiently by using variable-width bins containing equal photon counts. ED histograms can significantly reduce bandwidth requirements compared to EW histograms. However, current techniques for determining the EDhistogram bin boundaries tend to converge slowly, especially in low signal-to-noise conditions. Existing approaches often rely on tracking the full history of photon timestamps, which is memory, and power, intensive and not amenable to efficient hardware implementation.

[0007] Existing approaches for computing ED histogram boundaries use recursive filtering or sorting-based techniques that can exhibit biases, and slow convergence in the presence of noise and background light. The bin boundaries determined by these methods tend to oscillate around the true quantile locations rather than settling at precise values. This can lead to quantization errors when using ED histograms to estimate depth from the transient peak location. Further, existing approaches for determining ED histogram boundaries employ recursive, tree-based boundary estimation techniques which constrain the number of quantiles which may be determined to powers of two. e.g., 2, 4. 8. 16. etc.

[0008] What is needed is a hardware-efficient approach for rapidly determining ED histogram boundaries in an online manner without storing complete photon timestamp histories. The bin boundary estimation should be robust to noise and background light with low bias and variance. Such a solution would enable the benefits of ED histograms for bandwidth reduction while providing accurate distance estimates for 3D imaging applications at reduced in-pixel memory and power consumption.SUMMARY

[0009] The current disclosure provides systems and methods which at least partially address the above identified issues. In one embodiment, a method is provided that comprises initializing a plurality of control values (CV's) for a plurality of binners, where each binner is configured to classify photon return events from a pixel of an imaging detector as either early events or late events based on a reference signal controlled by a respective CV. The method involves transmitting a light pulse from a pulsed light source and detecting one or more photon return events following the light pulse. Upon detecting the one or more photon return events, the method simultaneously updates an early arrival count and a late arrival count for each of the plurality of CV's across the plurality of binners. The early arrival count tracks the number of early events for the pulse, while the late arrival count tracks the number of late events relative to the respective CV. The plurality of CV's are then updated based on the updated early arrival count, the updated late arrival count, and a CV index associated with each respective CV. The method determines whether the plurality of CV's satisfy a convergence criterion and. upon satisfying the convergence criterion, reads out the plurality of CV's. The plurality of CV's correspond to boundaries of an equi-depth histogram (ED histogram), usable for computer vision tasks, such as determining a distance of an object in a scene, semantic segmentation, object detection, 3D reconstruction, simultaneous localization and mapping (SLAM), and camera tracking.

[0010] This approach addresses the above identified issues by providing a hardware-efficient method for rapidly determining ED histogram boundaries in an online manner without storing completephoton timestamp histories. The bin boundary estimation is robust to noise and background light with low bias and variance, enabling the benefits of ED histograms for bandwidth reduction while providing accurate distance estimates for 3D imaging applications at reduced in-pixel memory and power consumption. Further, the current approach enables all CV values to be independently determined (i.e., the CV values are not dependent on one another, unlike in prior approaches that organize multiple binners in a tree structure), allowing for CV values to be determined fully in parallel, partially in parallel, or fully in series, depending on the desired tradeoff between convergence speed and the hardware constraints associated with the particular use case.

[0011] In a second embodiment, a method is provided for determining quantile boundaries of an ED histogram of photon arrival times. The method comprises initializing a plurality of CVs and CV stepping parameters. For each CV of the plurality of CVs, a CV index (j) corresponding to the CV is selected. A light pulse is transmitted from a pulsed light source, and photon arrivals following the transmitted light pulse are detected. At the nAth laser pulse, an early arrival count (En) and a late arrival count (Ln) are updated for the selected CV index (j) based on the detected photon arrivals, wherein the early arrival count (En) tracks a number of photon arrivals from the pulse occurring before a time delay represented by the CV, and the late arrival count (Ln) tracks a number of photon arrivals occurring after the time delay represented by the CV. The CV corresponding to the selected CV index (j) is updated based on the updated early arrival count (En), the updated late arrival count (Ln), and the CV index (j). A determination is made as to whether a convergence criterion is satisfied for the updated CV corresponding to the selected CV index (j). Upon determining the convergence criterion is satisfied for die updated CV corresponding to the selected CV index (j), the method proceeds to update a next CV of the plurality' of CVs. The plurality’ of updated CVs are read out when a predetermined number of the plurality' of CVs have converged, wherein the plurality’ of updated CVs represent time delays corresponding to quantile boundaries.

[0012] The above approach allows for an arbitrary number of quantile boundaries to be determined, rather than being constrained to powers of two as in previous recursive tree-based approaches. This flexibility is advantageous as it pennits greater freedom in the number of quantile boundaries determined, enabling design of photon binning circuits which are better tailored to the particular use case in which they are employed.

[0013] It should be understood that the summary above is provided to introduce in simplified form a selection of concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is defined uniquely by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The disclosure may be beter understood from reading the following description of nonlimiting embodiments, with reference to the atached drawings, wherein below:

[0015] FIG. 1 schematically shows an example imaging environment including a time-of-flight 3D camera according to an embodiment;

[0016] FIG. 2A illustrates an example equi-width histogram (EW histogram) representation of photon arrival times;

[0017] FIG. 2B illustrates an example equi-depth histogram (ED histogram) representation of photon arrival times;

[0018] FIG. 3 illustrates a transient distribution of photon arrival times and control values used for dynamically updating histogram bin boundaries;

[0019] FIG. 4 is a flowchart of a method for dynamically updating histogram bin boundaries based on detected photon arrivals;

[0020] FIG. 5 is a flowchart of another method for dynamically updating histogram bin boundaries based on detected photon arrivals;

[0021] FIG. 6 is a flowchart illustrating a method for updating a control value used to dynamically adjust histogram bin boundaries;

[0022] FIG. 7 shows example plots of a transient distribution, control value stepping behavior using proportional stepping, and control value stepping behavior using smoothed and time-decayed stepping;

[0023] FIG. 8 is a flowchart of a method for estimating scene depth from an ED histogram using bin width analysis;

[0024] FIG. 9 is a flowchart of a method for estimating scene depth from an ED histogram using local photon density estimation; and

[0025] FIG. 10 is a flowchart of a method for mapping local photon density estimates to scene distance estimates using a deep neural network.DETAILED DESCRIPTION

[0026] The present disclosure provides systems and methods for efficient three-dimensional (3D) imaging using single-photon cameras (SPCs). Single-photon sensing has emerged as a promising technology for high-resolution 3D imaging and distance mapping. SPCs combine extreme sensitivity and the time-of-flight principle to capture scene distance information with high spatial and temporal resolution. However, the high data rates generated by SPCs present significant challenges for processing, storage, and transfer of the captured data, limiting the applicability of SPCs in resource- constrained applications such as mobile devices, augmented / virtual reality systems, and autonomous robotics.

[0027] Conventional SPC pixels construct equi-width histograms (EW histograms) of photon arrival times to estimate scene depth. The location of the peak in this histogram corresponds to the time- of-flight, and hence the distance, of the scene point being imaged. Forming these EW histograms requires high bandwidth to transfer large amounts of data off the sensor chip and significant on-chip memory to store the histogram bins. For example, a megapixel resolution SPC operating at video rates can generate several gigabytes of EW histogram data per second, far exceeding the capabilities of current interfaces and hardware.

[0028] While it is possible to have EWH SPCs with very few bins to allow better spatial resolution, this affects the accuracy (that is. the temporal resolution). The disclosed techniques concerning the constructing of equi-depth histograms (ED histograms) alleviates this tradeoff, thereby allowing for improved spatial resolution and higher accuracy.

[0029] To address this issue, the present disclosure introduces techniques for constructing and processing ED histograms (or equi-height histograms (EH histograms)), which provide a more compact representation of the photon data compared to conventional EW histograms. Instead of fixed-width equi-depth bins, ED histograms use variable -width bins containing equal photon cormts. This representation is particularly efficient for capturing "peaky" transient distributions typical of laser pulses reflected from surfaces. However, existing approaches for determining ED histogram bin boundaries tend exhibit slow convergence, especially in low signal-to-noise conditions, and can suffer from biases and high variance in the estimated bin boundary locations.

[0030] The present disclosure overcomes these limitations by providing a hardware-efficient approach for rapidly estimating ED histogram bin boundaries in an online manner without storing full photon timestamp data. The disclosed techniques employ a plurality of "binner" circuits, with each binner tracking the estimated location of an arbitrary quantile boundary of the photon transient distribution. The binners classify incoming photon events as early or late relative to their current quantile estimate and iteratively update this estimate based on the ratio of early / late events observed. The current disclosure provides a more stable and effective update mechanism to the binner control values, which is designed to ensure rapid convergence to the true quantile locations, even in the presence of noise and background light. This is achieved through an optimized stepping strategy that combines proportional stepping, exponential smoothing, and temporal decay to refine the quantile estimates over time. The convergence behavior is highly robust, exhibiting low bias and variance compared to prior techniques.

[0031] Further, the herein disclosed CV update approach enables multiple binners to work fully in parallel for a single pixel, and can therefore simultaneously update an arbitrary number of quantile boundaries based on a same number of recorded single photon arrival events, enabling rapid and computationally efficient generation of multi-bin ED histograms. This allows the number of quantile boundaries (and hence histogram bins) to be tailored to the specific application needs and hardwareconstraints. For example, resource-limited devices can employ an ED histogram with fewer bins and / or less parallelization to minimize data transfer and storage, while more capable systems can leverage higher-resolution histograms and / or greater parallelization for improved distance precision and speed.

[0032] To further enhance distance estimation accuracy, the disclosed techniques incorporate a deep learning approach that maps the compact ED histogram representation to high-quality distance maps. In certain embodiments, a deep neural network (DNN) can be trained to exploit spatial and temporal correlations in the ED histogram outputs across neighboring pixels and exposures, thereby enabling improved resolution and denoising of the distance maps. The synergistic combination of optimized ED histogram hardware and deep learning post-processing improves the performance of single-photon imaging for a wide range of computer vision tasks in resource-constrained settings. Example applications enabled by the disclosed techniques include visual odometry for mobile robots, 3D reconstruction for augmented / virtual reality, and real-time semantic segmentation or object detection leveraging both photon depth and conventional image data.

[0033] In one embodiment, FIG. 1 schematically shows an example imaging environment 100 including a time-of-flight 3D camera for capturing scene depth information. The environment includes a light source 102 for illuminating an object 104 with a transmitted light pulse, a detector 106 with an array of pixels for detecting return photon events reflected off the scene object 104, and a computing device 110 coupled to the detector 106.

[0034] FIGS. 2A and 2B illustrate different approaches for representing the distribution of detected photon arrival times at each pixel of the detector 106. FIG. 2A shows an EW histogram 200A representation, where photon counts are accumulated into fixed-width equally spaced bins spanning the entire time range. In contrast, FIG. 2B depicts an ED histogram 200B representation, which uses variable-width bins such that each bin contains approximately equal photon counts, allowing for more efficient encoding of "peaky" transient distributions typical of laser pulses reflected off surfaces.

[0035] FIG. 3 illustrates the concept of a transient distribution 302 formed by the photon timestamps 304 detected at a pixel, and the use of a control value (CV) 306 to dynamically update the ED histogram bin boundaries by comparing the CV 306 to the true median 308 (or other arbitrary quantile) of the transient. The CV 306 is adjusted through a step size Snbased on the relative counts of early photon arrivals Enand late photon arrivals Lnwith respect to the current CV value C„.

[0036] FIGS. 4 and 5 present flowcharts outlining methods 400 and 500 for dynamically updating the ED histogram bin boundaries by iteratively adjusting the CV values based on detected photon arrivals. FIG. 6 provides a more detailed flowchart 600 describing the method for computing the step size Snused to update each CV value in a given iteration.

[0037] FIG. 7 shows an example plot 700 comparing the convergence behavior of the CV determined using a basic proportional stepping strategy 706 versus an optimized smoothed and time-decayed stepping strategy 708 for adjusting the CV toward the true median 704 of an example transient photon distribution 702.

[0038] FIGS. 8 and 9 outline methods 800 and 900 for estimating scene distance from the dynamically computed ED histogram bin boundaries, using either bin width analysis (FIG. 8) or local photon density estimation through linear interpolation (FIG. 9). FIG. 10 describes a method 1000 that uses a deep neural network to map the local photon density estimates to scene distance estimates while exploiting spatial and temporal correlations in the ED histogram data.

[0039] FIG. 1 schematically shows an example imaging environment 100 for imaging a scene with a single-photon-sensing 3D imaging system according to an embodiment. The environment 100 includes a light source 102, an object 104, and a detector 106. The environment further includes a computing device 110. While the light source 102 and the detector 106 are shown as separate devices in FIG. 1, it is to be appreciated that the light source 102 and the detector 106 may be integrated in a single device (such as integrated within the computing device 110). The light source 102 and detector 106 may form a single-photon-sensing 3D camera (SPC) that captures distance (e.g., depth) information using the time-of-flight principle — akin to echolocation, but with light instead of sound. Consider a single scene point where distance needs to be estimated as shown in FIG. 1. The light source 102 (which may be a laser) illuminates a scene point (which may be part of object 104) with a short light pulse (e.g., tire transmitted pulse shown in FIG. 1). Detector 106 may include an array of detector elements, with each detector element configured to (separately) detect photons. In some examples, each detector element may be a diode, such as a single-photon avalanche diode or an avalanche photodiode. As used herein, the detector elements may be referred to as pixels and thus the detector 106 may include a plurality of pixels. A pixel of detector 106 may capture a stream of return events (e.g., the return photon events of FIG. 1) as photons arrive at different time delays with respect to the time the original light pulse was transmitted. (The detector 106 in general captures more than one return event in response to each laser pulse sent into the scene.) Moreover, this return stream may also contain spurious photon events not due to the light pulse (signal), but rather due to ambient background light and other sources of noise in the image-sensing hardware. An EW histogram is constructed by accumulating photon counts at different delays over many light pulse (e.g., laser pulse) cycles. The location along a time axis of the peak location of this histogram gives an estimate of the true distance of the scene point. As will be explained in more detail below, power consumption, memory requirements, and processing power needed to determine object distances may be reduced if the true distance of the scene point is estimated using an ED histogram instead.

[0040] As mentioned above, the light source 102 may be a laser or another suitable light source capable of transmitting light in pulses at a frequency dictated by a clock 108. The detector 106 may be a single-photon avalanche diode (SP AD) sensor with single-photon sampling, an avalanche photodiode (APD) sensor, or another suitable image sensor capable of capturing single photons with high timeresolution. The detector 106 may receive pulse frequency information from the clock 108 (e.g., indicating the start and end of each pulse of the light source 102).

[0041] The computing device 110 (which may include the light source 102 and / or of the detector 106 in some examples) may be a smartphone camera, a light detection and ranging (LiDAR) sensor (e.g., for autonomous robotics), a camera for scientific imaging, a virtual reality device, an augmented reality device, a desktop computer, a laptop, a mobile device (e.g., smartphone or tablet), or another suitable device.

[0042] Due to their compatibility with CMOS fabrication technology, there is increasing availability of high (kilo-to-megapixel) resolution arrays of single-photon-detecting (e.g., SPAD) pixels with additional data processing embedded in the hardware chip that includes the single-photondetecting pixels. Unfortunately, the high sensitivity' and high speed is a double-edged sword: the amount of raw data generated by these detectors is orders of magnitude higher than can be reasonably processed or transferred in real-time. This aspect limits their applicability' in many real-world applications, especially those that are power and bandwidth constrained.

[0043] As mentioned above, the detector 106 (and at least in some embodiments, also the light source 102) may be integrated in or operably coupled to the computing device 110. While not shown in FIG. 1, it is to be appreciated that computing device 110 may include a logic subsystem such as a processor and a data-holding subsy stem such as a memory.

[0044] The processor comprises one or more physical devices configured to execute one or more instructions. For example, the processor may execute one or more instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more devices, or otherwise arrive at a desired result.

[0045] The processor may thus include one or more processors configured to execute software instructions. Additionally or alternatively, the processor may comprise one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. As illustrative and non-limiting examples, the processor may comprise one or more central processing units (CPU), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs). Neural Processing Units (NPUs), and so on. The processor may be single or multi-core, and the programs executed thereon may be configured for parallel or distributed processing. The processor may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and / or configured for coordinated processing. Such devices may be connected via a netw ork.

[0046] The memory of the computing device 110 may comprise one or more physical, non- transitoiy devices configured to hold data and / or instructions executable by the processor to implementdie methods and processes described herein. When such methods and processes are implemented, the state of the memory may be transformed (for example, to hold different data).

[0047] The memory' may include removable media and / or built-in devices. The memory' may include optical memory' (for example, CD, DVD, HD-DVD, Blu-Ray Disc, and so on), and / or magnetic memory devices (for example, hard drive disk, floppy disk drive, tape drive, MRAM. and so on), and the like. The memory may include devices with one or more of the following characteristics: volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and content addressable. In some embodiments, the processor and the memory may be integrated into one or more common devices, such as an application-specific integrated circuit or a system on a chip.

[0048] The computing device 110 may optionally include a display subsystem, a communication subsystem, a user interface subsystem, and other components. The communication subsystem may enable the computing device 110 to wirelessly transmit or stream the ED histogram data generated by the detector 106 to other devices or systems. This wireless transmission capability can be advantageous in scenarios where the computing device 110 has limited processing power, memory, or battery capacity.

[0049] The computing device 110 may be communicatively coupled to a display device. As illustrative and non-limiting examples, the display device may display an image captured by detector 106, the display device may display an image that is sized / positioned based on a distance map determined from the output of die detector 106, etc. The display device may include one or more display devices utilizing virtually any type of display technology such as, but not limited to, cathode ray tube (CRT), liquid crystal display (LCD), light-emitting diode (LED), organic LED (OLED), electroluminescent display (ELD), active-matrix OLED (AMOLED), quantum dot (QD) displays, and so on. As another example, the display device may comprise a display projector device such as a digital light processing (DLP) projector, a liquid-cry stal-on-silicon (LCoS) projector, a laser projector, an LED projector, and so on. As yet another example, the display device may comprise an augmented reality (AR) display system, a virtual reality (VR) display system, or a mixed reality (MR) display system.

[0050] Referring to FIGS. 2A and 2B, tw o types of histograms for representing the distribution of photon events over time are shown: an EW histogram 200A and an ED histogram 200B. Both histograms aim to capture the transient distribution 202 of photon timestamps 204 generated by a pulsed light source reflecting off an object in a scene. The photon timestamps 204 may be generated by detector elements (e.g., single-photon avalanche diodes) in an image sensor, with each detector element corresponding to a pixel in the captured image. The transient distribution 202 represents the expected distribution of photon arrival times for a single laser pulse reflecting off an object at a particular distance from the camera. By accurately estimating the peak location of the transient distribution 202 from the ED histogram 200B, the distance to the object can be determined based on the time-of-flight principle.

[0051] The EW histogram 200A is a conventional approach where the time axis is divided into equal-width bins, and the number of photon events falling within each bin is counted. This results in a histogram with uniform bin widths along the discretized time axis. While simple to construct, the EW histogram 200A can be an inefficient representation when the underlying transient distribution 202 is "peaky," such as a narrow laser pulse reflected from a surface. In this case, most of the EW histogram bins will contain few or no photon counts, wasting resources to represent the background noise or ambient light level.

[0052] One embodiment of the ED histogram 200B addresses this inefficiency by using variablewidth bins along the time axis, with each bin designed to contain an approximately equal number of photon events. This allows the ED histogram to allocate more narrow bins in regions of high photon density (e.g.. around the peak of the laser pulse) and fewer wider bins in regions of low photon density (e.g., background noise or ambient light). The ED histogram 200B can represent peaky transient distributions 202 more accurately compared to the EW histogram 200A. reducing bandwidth requirements for transferring histogram data off the image sensor. In general, an ED histogram can achieve the same accuracy as an EW histogram, but using fewer bins than the EW histogram. Or, if the two histograms have the same number of bins, the ED histogram will capture the transient distribution more accurately.

[0053] Referring to FIG. 3, a graphical depiction 300 of a process for updating a control value (CV) 306 based on photon counts derived from an underlying transient photon distribution 302 is shown. FIG. 3 illustrates the concept of counting early and late photons 304 relative to a current CV 306 for a given transient distribution 302. The transient distribution 302 represents the true underlying distribution of photon arrival times, which exhibits a peak corresponding to the laser pulse reflection from the scene. The true median 308 of this distribution is the target quantile that the CV 306 aims to track, although it will be appreciated that a particular advantage of the current disclosure is the ability to track arbitrary quantiles simultaneously, and that the median is shown as the target quantile in FIG. 3 for simplicity, and should not be construed to limit the scope of the current disclosure.

[0054] The CV 306 determines a reference signal at a certain delay, dividing the incoming photons into two categories: early photons (E„) that arrive before the time delay represented by the current CV 306. and late photons (Ln) that arrive after this time delay. By comparing the relative counts of early (En) and late (L„) photons, it may be determined whether to increase or decrease the CV 306 for the next cycle. Specifically, in the example shown in FIG. 3, if En / (En+ Ln) > 1 / 2, indicating more early photons than expected for the target quantile, the step size (Sn) is negative, causing CV 306 to decrease for the next cycle (Cn+i = Cn+ Sn).

[0055] Referring to FIG. 4, a flowchart of a method 400 for determining an arbitrary number of control values (CV's) for a single pixel in parallel is shown. In one embodiment, method 400 may be executed to determine a subset of CVs in parallel, rather than fully parallelizing the updates across allburners. In this way, the degree of parallelization may be tailored based on hardware constraints and performance requirements.

[0056] The method 400 begins at operation 402, where a plurality of control values (CV's) is initialized. In one embodiment, the plurality of CV's may be initialized to predetermined values proportional to their respective CV indices (j). where j is an integer between 1 and Q, and Q is the total number of CV's to be determined for the pixel (note, Q is one less than the number of bins). For example, if Q = 15 (meaning a total number of equi-depth bins is equal to 16), the 15 CV's defining the 16 equidepth bins may be initialized to values proportional to 1 / 16. 2 / 16, 3 / 16, ..., 15 / 16 of a maximum possible time delay between a transmitted light pulse and a photon arrival at the pixel.

[0057] However, in another embodiment, at operation 402 the CV's may be initialized based on CV values from neighboring pixels or past CV values for the same pixel from previous frames or exposures. For example, if CV values have already converged for neighboring pixels, the CV's for the current pixel may be initialized as a weighted combination of the converged CV values from the neighboring pixels. Similarly, if the current pixel has converged CV values from a previous exposure or video frame, the new CV's may be initialized as the previous converged values plus an offset based on expected motion between frames. This initialization approach leverages spatial and temporal correlations that may provide a better initial estimate of the CV values, enabling faster convergence compared to arbitrary predetermined initialization values.

[0058] At operation 404, CV stepping parameters are initialized. These stepping parameters control how the CV's are updated in subsequent operations based on detected photon arrivals. In one embodiment, the stepping parameters may include a scaling factor, a temporal decay factor, and smoothing parameters for exponential smoothing of step sizes used to update the CV's.

[0059] The method 400 then enters a CV update cycle 406 that encompasses operations 408-414. At operation 408, a light pulse is transmitted from a pulsed light source, such as a laser. The light pulse may be directed towards a scene or object whose distance is to be detennined.

[0060] At operation 410, one or more photon arrivals are detected at the pixel following the transmitted light pulse. The detected photon arrivals may comprise photons reflected from the scene or object and captured by the pixel's single-photon detector, or may comprise an ambient photon. The early arrival count (En) and late arrival count (Ln) are computed for the current laser cycle or may be accumulated over a few (e g.. 5-10) laser cycles (cycle group). Enand L„ are not cumulative; they are initialized to 0 before each cycle or cycle group. More recent single-photon detectors are capable of detecting multiple photon arrivals per laser cycle, enabled by shorter detector dead times (e.g.. tens of nanoseconds) and multi-event time-to-digital converters that can timestamp more than just the first arriving photon. This allows the early and late photon counts to more accurately capture the full transient distribution within each laser cycle.

[0061] At operation 412, the early arrival count (En) and the late arrival count (Ln) are updated for each CV based on the detected photon arrivals for the current laser cycle or cycle group. In the nthlaser cycle (or cycle group), the early arrival count (En) tracks the number of photon arrivals occurring before the time delay represented by the respective CV, while the late arrival count (Ln) tracks the number of photon arrivals occurring after the time delay represented by the respective CV. After the CV is updated at operation 414, Enand Lnare reset to 0 for the next cycle or cycle group. By accumulating the early and late photon counts over multiple laser cycles before updating the C V values, the disclosed technique can improve the photon statistics and enable more rapid convergence. This multi-cycle accumulation method is also compatible with modern single-photon sensors that can capture multiple photons per laser cycle.

[0062] At operation 414. the CV's are updated based on their respective updated early arrival count (En). late arrival count (Ln), and CV index (j). In one embodiment, updating the CV's may involve determining a base step size for adjusting each CV based on the updated early arrival count (En) and late arrival count (Ln) for that CV. The base step size may be scaled by a scaling factor to produce a scaled step size. The scaled step size may then be smoothed, temporally, by using exponential smoothing, and spatially, by computing a weighted average of the scaled step size of 3x3 neighboring pixels, to produce a first smoothed step size. The first smoothed step size may be temporally decayed by multiplying it with a temporal decay factor raised to a power equal to the number of cycles over which the respective CV has been updated. The temporally decayed step size may then be smoothed again using exponential smoothing to produce a second smoothed step size, which is added to the respective CV to produce the updated CV for the subsequent cycle.

[0063] In another embodiment, the base step size for adjusting each CV may be determined as the difference between a target fraction corresponding to the CV index (j) and the ratio of the updated early arrival count (En) to the sum of the updated early arrival count (En) and the updated late arrival count (Ln). For example, if the CV index (j) is 5 and the total number of CV's (Q) is 15 the target fraction would be 5 / 16 = 0.3125. If the updated early arrival count (En) is 100 and the updated late arrival count (Ln) is 200, the ratio of En / (En+Ln) would be 100 / (100+200) = 0.3333. The base step size would then be 0.3125 - 0.3333 = -0.0208, indicating that the CV should be decreased to bring the ratio En / (En+Ln) closer to the target fraction of 0.3125.

[0064] At operation 416, a check is performed to determine if the CV convergence criteria are satisfied for each of the updated CV's. In one embodiment, a convergence criterion may be based on the change in the respective CV over a predetermined number of cycles being less than a predetermined threshold. In another embodiment, a convergence criterion may comprise a pre-determined number of laser cycles having elapsed. If the convergence criteria are not satisfied for one or more CV's, the method 400 returns to the CV update cycle 406 to transmit another light pulse (operation 408), detect the subsequent photon arrivals (operation 410), update the early and late arrival counts (operation 412),and update the CV's again (operation 414). This cycle continues until the convergence criteria are satisfied for all CV's. In some embodiments it may not alway s be necessary or desirable for all CV's to fully converge. In some instances, it may be advantageous to terminate the iterative update process after a pre-determined number of cycles, even if some CV's have not yet converged. This approach can prevent excessive computational effort from being expended on CV's that correspond to background regions with low photon counts, while still allowing the CV's near the peak of the transient distribution to converge accurately.

[0065] If the convergence criteria are satisfied at operation 416, the method 400 proceeds to operation 418. where the converged CV's are read out. The plurality of converged CV's represent time delays corresponding to quantile boundaries of the photon arrival time distribution for the pixel. In one embodiment, the converged CV's may be read out and used to determine distances in a scene based on the time-of-flight of photons from the pulsed light source to objects in the scene and back to the pixel.

[0066] The method 400 provides a hardware-efficient approach for rapidly determining an arbitrary number of quantile boundaries (represented by the converged CV's) of an ED histogram of photon arrival times for a single pixel in parallel. The ED histogram representation can significantly reduce bandwidth requirements compared to conventional EW histograms, while still enabling accurate distance estimation for 3D imaging applications. The optimized stepping strategy used to update the CV's ensures fast convergence and low variance, even in the presence of noise and background light. By parallelizing the determination of multiple CV's for a single pixel, the method 400 enables rapid CV convergence while reducing the need for extensive in-pixcl memory to store photon timestamps.

[0067] Referring to FIG. 5, a flowchart of a method 500 for determining an arbitrary' number of control values (CV's) for a pixel in series is shown. The method 500 begins at operation 502, where a plurality of control values (CV's) is initialized. In one embodiment, the plurality of CV's may be initialized to predetermined values proportional to their respective CV indices (j), where j is an integer between 1 and Q, and Q is the total number of CV's to be determined for the pixel. For example, if Q = 15 (meaning there are a total of 16 equi-depth bins), the 15 CV's may be initialized to values proportional to 1 / 16. 2 / 16, 3 / 16, ..., 15 / 16 of a maximum possible time delay between a transmitted light pulse and a photon arrival at the pixel.

[0068] In another embodiment, at operation 502 the plurality of CV's may be initialized using a smart initialization approach that leverages information from neighboring pixels or past CV values. For instance, the CV's for a given pixel may be initialized based on the converged CV values from one or more neighboring pixels in a current or previous frame or exposure. This approach can provide a more accurate initial estimate of the CV values, potentially leading to faster convergence during the iterative update process. Alternatively, the CV's for a given pixel may be initialized based on the CV values from the same pixel in a previous frame or exposure, taking into account any temporal variations in the scene or imaging conditions. The smart initialization approach can be particularly beneficial in scenarioswhere the scene or imaging conditions exhibit spatial or temporal correlations, enabling the system to leverage these correlations to improve the efficiency and accuracy of the CV estimation process. The smart initialization approach may also incorporate information from other sensors or data sources, such as distance maps or motion estimates, to further refine the initial CV values.

[0069] At operation 504, CV stepping parameters are initialized. These stepping parameters control how the CV's are updated in subsequent operations based on detected photon arrivals. In one embodiment, the stepping parameters may include a scaling factor, a temporal decay factor, and smoothing parameters for exponential smoothing of step sizes used to update the CV's.

[0070] The method 500 then proceeds to operation 506, where a CV index (j) is selected from the plurality of CV indices corresponding to the plurality of CV's. The selected CV index (j) determines which CV will be updated during the subsequent CV update cycle.

[0071] At operation 508, a CV update cycle is initiated, which encompasses operations 510-516. At operation 510, a light pulse is transmitted from a pulsed light source, such as a laser. The light pulse may be directed towards a scene or object whose distance is to be determined.

[0072] At operation 512, one or more photon arrivals are detected at the pixel following the transmitted light pulse. The detected photon arrivals may comprise photons reflected from the scene or object and captured by the pixel's single-photon detector. The early arrival count (En) and late arrival count (Ln) are computed for the current laser cycle or may be accumulated over a few (e g., 5-10) laser cycles. Enand Lnare not cumulative; they are initialized to 0 before each cycle or cycle group. More recent single-photon detectors are capable of detecting multiple photon arrivals per laser cycle, enabled by shorter detector dead times (e.g., tens of nanoseconds) and multi-event time-to-digital converters that can timestamp more than just the first arriving photon. This allows the early and late photon counts to more accurately capture the full transient distribution within each laser cycle.

[0073] At operation 514, the early arrival count (En) and the late arrival count (Ln) are updated for the selected CV index (j) based on the detected photon arrivals for the current laser cycle or cycle group. The early arrival count (En) tracks the number of photon arrivals occurring before the time delay represented by the CV corresponding to the selected CV index (j), while the late arrival count (Ln) tracks the number of photon arrivals occurring after the time delay represented by the CV corresponding to the selected CV index (j). In one embodiment, the early arrival count (En) and late arrival count (Ln) may be updated by incrementing the respective count based on whether the detected photon arrival occurred before or after the time delay represented by the CV corresponding to the selected CV index (j). Accumulating the photon counts over multiple laser cycles can be advantageous even with multiple detections per cycle; it can also work for detectors that detect at most one photon per cycle.

[0074] At operation 516, the CV corresponding to the selected CV index (j) is updated based on die updated early arrival count (En), the updated late arrival count (Ln), and the CV index (j). In one embodiment, updating the CV may involve determining a base step size for adjusting the CV based ondie updated early arrival count (En) and late arrival count (L„) for that CV. The base step size may be scaled by a scaling factor to produce a scaled step size. The scaled step size may then be smoothed, temporally, by using exponential smoothing, and spatially, by computing a weighted average of the scaled step size of 3x3 neighboring pixels, to produce a first smoothed step size. The first smoothed step size may be temporally decayed by multiplying it widi a temporal decay factor raised to a power equal to the number of cycles over which the respective CV has been updated. The temporally decayed step size may then be smoothed again using exponential smoothing to produce a second smoothed step size, which is added to the respective CV to produce the updated CV for the subsequent cycle.

[0075] At operation 518, a check is performed to determine if the convergence criteria are satisfied for the updated CV corresponding to the selected CV index (j). In one embodiment, a convergence criterion may be based on the change in the CV over a predetermined number of cycles being less than a predetermined threshold. In another embodiment, a convergence criterion may comprise a predetermined number of laser cycles having elapsed. If the convergence criteria are not satisfied for the updated CV corresponding to the selected CV index (j), the method 500 returns to the CV update cycle 508 to transmit another light pulse (operation 510), detect the next photon arrival (operation 512), update the early and late arrival counts (operation 514), and update the CV again (operation 516). This cycle continues until the convergence criteria are satisfied for the CV corresponding to the selected CV index (j).

[0076] If the convergence criteria are satisfied for the updated CV corresponding to the selected CV index (j) at operation 518, the method 500 proceeds to operation 520, where a determination is made as to whether a predetermined number (q) of the plurality of CV's have converged. If the predetermined number (q) of CV's have not converged, the method 500 proceeds to operation 522, where a next CV index (j) is selected from the remaining unconverged CV's. After selecting the next CV index (j) at operation 522, tire method 500 returns to the CV update cycle 508. and operations 510- 516 are repeated for the newly selected CV index (j). This process continues, with the CV update cycle 508 being repeated for each unconverged CV index (j) until the predetennined number (q) of CV's have converged.

[0077] Once the predetermined number (q) of CV's have converged at operation 520, the method 500 proceeds to operation 524. The plurality of converged CV's represents time delays corresponding to quantile boundaries of the photon arrival time distribution for the pixel. In one embodiment, the converged CV's may be read out and used to determine distances in a scene based on the time-of-flight of photons from the pulsed light source to objects in the scene and back to the pixel.

[0078] The method 500 provides a computationally efficient approach for determining an arbitrary number of control values (CV's) in series, enabling minimal hardware requirements for on-sensor implementation. By updating and converging each CV sequentially based on the detected photon arrivals, the method 500 eliminates the need for extensive in-pixel memory to store photon timestampsor histograms. Only a single CV needs to be tracked at a time, significantly reducing the memory footprint compared to conventional techniques that require storing complete photon data cubes or histograms. Furthermore, the optimized stepping strategy used to update the CV's ensures rapid convergence, even in the presence of noise and background light, minimizing the computational overhead. This computational efficiency translates to lower power consumption and simpler hardware designs, making the method 500 well-suited for resource-constrained applications such as mobile devices, augmented reality systems, and autonomous robotics, where single-photon imaging capabilities are desired without sacrificing power and area budgets. While the method 500 iterates through each CV until convergence, it may not always be necessary for all CV's to fully converge before terminating the process. In some embodiments, the method 500 may terminate after a pre-determined number of cycles, regardless of whether all CV's have converged. This approach can prevent excessive computational cycles from being dedicated to CV's corresponding to background regions, while still allowing accurate convergence for the CV's proximal to the peak of the transient distribution.

[0079] Referring to FIG. 6, a method 600 for updating a CV to converge on a j111quantile of a photon distribution is shown. The method 600 may be performed by a proportional binner circuit or module within a single-photon camera (SPC) pixel or sensor. The method 600 provides an optimized stepping strategy for iteratively adjusting the CV of a proportional binner to track a desired quantile of a transient photon distribution.

[0080] At operation 602, the proportional binner receives Enand Lnfor the j* CV, where Enrepresents the number of photons arriving earlier than the current CV, and Lnrepresents the number of photons arriving later than the current CV during the current laser cycle. The proportional binner may include circuitry or logic to count or accumulate the early and late photon events relative to the current CV.

[0081] At operation 604, the proportional binner determines a base step size for the jthCV based on Enand Ln. In one embodiment, the base step size may be determined by calculating a target, a fraction j / Q. where j is the desired quantile and Q is the total number of quantiles being tracked, and comparing this to the actual ratio of photon arrivals before and after a time delay proportional to the current CV. The target fraction represents the fraction of photons that should ideally arrive earlier than the C V when it has converged to the j111quantile location. The target fraction may then be compared to the actual fraction En / (En+L„) to determine the base step size for the current CV. This may be expressed symbolically as:where Anis die base step size.

[0082] At operation 606, the proportional bimier scales the base step size to produce a scaled step size. In one embodiment, the scaling may involve multiplying the base step size by a scaling factor K,where K is a percentage of the total number of time inter als B used to discretize the transient photon distribution (note B is unrelated to Q). The scaling factor K allows for larger step sizes to be taken early on in a convergence process, potentially speeding up the convergence of the CV towards the desired quantile location. In one embodiment, the scaling factor K used in operation 606 may be chosen as a percentage of the total number of time intervals B. such as 1%, 3%, or 5%. A higher value of K allows for larger step sizes, potentially speeding up convergence, but may also increase the risk of overshooting or oscillating around the true quantile location. Symbolically the scaled step size may be expressed as:

[0083] At operation 608, the proportional binner smooths the scaled step size, spatially and temporally, to produce a first smoothed step size. In one embodiment, the spatial smoothing may be performed by computing a weighted average of the scaled step sizes of 3x3 neighboring pixels, and the temporal smoothing may be performed using an exponential smoothing technique, where the scaled step size is combined with a weighted sum of previous step sizes. The spatial and temporal smoothing may reduce the variance and noise in the step size adjustments, leading to a more stable convergence behavior. In one embodiment, the temporal smoothing may be performed using an exponential smoothing technique, where the scaled step size is combined with a weighted sum of previous step sizes. In one embodiment, the first smoothed step size may be expressed as:where Anis the first smoothed step size for the current cycle, ft is a first exponential smoothing parameter, and An-1is the first smoothed step size for an immediately preceding cycle. The inventors herein have experimentally determined use of a first smoothing parameter ft close to 0.95 works well for a wide range of transient distributions over various combinations of signal and background strengths.

[0084] At operation 610. the proportional binner applies temporal decay to the first smoothed step size to produce a temporally decayed step size. In one embodiment, the temporal decay may involve multiplying the first smoothed step size by a decay factor y. where 0 < y < 1. The temporal decay factor y is chosen such that it limits the wandering of the CV around the true quantile location after convergence, while still allowing the CV to approach its final position quickly during the initial stages of the iterative update process. The temporally decayed step size may be expressed as:^ An, decayed — i rznA where n is the index of the current cycle andn, decayed isthe temporally decayed step size. The inventors herein have experimentally determined use of a temporal decay factor y close to 1, such as 0.99902, works well for a wide range of transient distributions over various combinations of signal and background strengths, and allows for rapid initial convergence while still limiting the wandering of theCV around the true quantile location after convergence. In some embodiments, the temporal decay may be applied only for a pre -determined number of initial cycles, after which die step size is held constant to maintain the final CV position.

[0085] At operation 612, the proportional binner smooths the temporally decayed step size to produce a second smoothed step size. In one embodiment, the smoothing operation may be similar to the smoothing performed at operation 608, using a different set of exponential smoothing parameters to further reduce the variance in the step size adjustments. The second exponentially smoothed step size may be expressed as:where Snis the smoothed and temporally decayed step size for the current cycle. / ?2is the second smoothing parameter, and Sn - 1is the smoothed and temporally decayed step size for the immediately preceding cycle. The inventors herein have experimentally determined use of a first smoothing parameter / ?2close to 0.80 works well for a wide range of transient distributions over various combinations of signal and background strengths. In some embodiments, the exponential smoothing parameters used in operations 608 and 612 may be timed to balance the trade-off between responsiveness to new photon events and stability of the step size adjustments. For example, a higher smoothing factor may result in smoother step size adjustments but slower response to changes in the photon distribution.

[0086] At operation 614, the proportional bimrer adds the second smoothed step size to the jthCV to produce an updated jthCV for the current cycle. The updated CV represents the new estimate of the j11quantile location based on the photon events observed during the current laser cycle and the stepping strategy applied in operations 604-612. The updated jthCV may be expressed as:

[0087] Following operation 614, method 600 may end. In some embodiments, the proportional binner may include additional logic or circuitry to detect convergence of the CV to the desired quantile location. For example, the binner may monitor the magnitude of the step size adjustments or the variation in the CV over a certain number of cycles. If the CV satisfies one or more convergence criteria the binner may terminate the iterative update process or switch to a different mode of operation, such as tracking a different quantile or performing additional post-processing on the converged CV value.

[0088] The method 600 provides an improved stepping strategy7for updating the CV of a proportional binner, enabling efficient and accurate tracking of arbitrary7quantiles of a transient photon distribution. The various operations, including scaling, smoothing, and temporal decay, contribute to faster convergence, reduced variance, and improved stability of the CV estimates. The improved stepping strategy7may be particularly beneficial in resource-constrained settings, such as single-photon cameras or other time-of-flight imaging systems, where efficient and accurate quantile estimation isparticularly advantageous for downstream computer vision tasks like distance estimation, 3D reconstruction, semantic segmentation, or visual odometry.

[0089] Referring to FIG. 7, a plot 700 illustrating the improved convergence performance of the CV's using the stepping strategy of method 600 is shown. The plot 700 depicts a comparative plot of the CV convergence trajectories for different stepping strategies applied to a proportional binner tracking a specific quantile of a transient photon distribution 702.

[0090] The transient photon distribution 702 represents the underlying distribution of photon arrival times at a single-photon detector pixel, which may be generated by a pulsed light source reflecting off an object in a scene. The true median 704 of the transient photon distribution 702 is indicated by a horizontal line, representing the target quantile location that the proportional binner aims to converge to.

[0091] One curve, labeled "Proportional stepping" 706. shows the convergence trajectory of the CV when using a basic proportional stepping strategy as described in operation 604 of method 600. In this embodiment, the CV is adjusted by a base step size determined as the difference between the target quantile fraction (j / Q) and the actual fraction of photons arriving before the current CV. While this basic proportional stepping strategy enables the CV to converge towards the true median 704, the convergence can be slow, and the CV may exhibit significant oscillations or wandering around the true median location, as illustrated by the fluctuations in curve 706.

[0092] To address these limitations, the method 600 employs an optimized stepping strategy that incorporates scaling, smoothing, and temporal decay operations on the base step size. The curve labeled "Smoothed and time decayed stepping" 708 illustrates the improved convergence performance achieved by applying these operations. As shown, the CV converges rapidly towards the true median 704 during the initial cycles, benefiting from the larger step sizes and smoothing operations. As the CV approaches the true median location, the temporal decay and further smoothing operations limit the wandering or oscillations around the true median, resulting in a stable and accurate estimate of the target quantile location.

[0093] Referring to FIG. 8, a flowchart of a method 800 for estimating the scene distance for a pixel based on time-of-flight measurements is shown. The method 800 utilizes the ED histogram bin boundaries determined by the techniques described earlier to estimate the scene distance for a pixel. A bank of proportional binners used to estimate the ED histogram is referred to herein as a proportional equi-depth histogrammer (PEDH).

[0094] At operation 802, the method 800 receives the ED histogram bin boundaries {to, ti, ..., tq+i} for a pixel, where Q is the number of quantile boundaries tracked by the PED histogram, and to and tq+i arc untrackcd boundaries at zero delay, and a prc-dctcrmincd maximum delay, respectively. These bin boundaries represent the time delays corresponding to the quantile locations of the photon transient distribution captured at the pixel. The bin boundaries are determined by the PED histogram using theoptimized stepping strategy described previously, which enables rapid convergence to the true quantile locations with low bias and variance.

[0095] In one embodiment, the bin boundaries {to, ti, ..., tq+i } are read out directly from the CV’s of the parallel bank of proportional binners that constitute the PED histogrammer. Each binner tracks a distinct quantile boundary, with the CV representing the time delay corresponding to that quantile. The PED histogrammer may be implemented as an on-sensor circuit, with the bin boundary values transferred off-chip for further processing.

[0096] At operation 804, the method 800 computes the bin widths w[i] for each bin i, where w[i] = t - ti-i and i = 1, 2, .... Q+l. The bin widths represent the time intervals between successive quantile boundaries and provide an estimate of the local photon density within the transient distribution. Narrower bin widths correspond to regions of higher photon density, while wider bin widths indicate regions of lower photon density. In one embodiment, the bin width computation is performed by a dedicated hardware unit or a software module within the computing device 110. The bin width values may be stored in a memory buffer or passed directly to subsequent processing stages.

[0097] At operation 806, the method 800 determines the narrowest bin among the set of bin widths {w[l], w[2], ..., w[Q+l]}. The narrowest bin is expected to correspond to the region of highest photon density, which typically coincides with the peak of the laser pulse transient distribution. In one embodiment, the narrowest bin is identified by comparing the bin width values in a sequential or parallel fashion using dedicated comparator circuits or softw are routines. The index i* of the narrowest bin is determined as i* = arg min{w[i]}.

[0098] At operation 808, the method 800 computes the midpoint of the narrowest bin identified in operation 806. The midpoint is calculated as (ti* + ti»-i) / 2, where i* is the index of the narrowest bin. In an embodiment, the midpoint calculation is performed by a dedicated arithmetic unit or a software routine within the computing device 110. The midpoint value represents an estimate of the time-of- flight for the laser pulse reflected from the scene point corresponding to the pixel under consideration.

[0099] At operation 810, the method 800 estimates the time-of-flight based on the midpoint of the narrowest bin computed in operation 808. In one embodiment, the time-of-flight estimate is simply set equal to the midpoint value, under the assumption that the narrowest bin is centered around the true peak of the transient distribution. In an alternative embodiment, the time-of-flight estimate may be refined by applying a correction factor or a calibration curve to account for systematic biases or nonlinearities in the PED histogram bin boundary7estimates.

[0100] At operation 812, the method 800 determines the scene distance for the pixel from the estimated time-of-flight. The scene distance is calculated as z = (c * t) / 2. where z is the distance, c is the speed of light, and t is the estimated timc-of-flight from operation 810. In one embodiment, the distance calculation is performed by a dedicated arithmetic unit or a software routine within the computing device 110. The computed distance value may be stored in a memory buffer or used directlyfor subsequent processing tasks, such as constructing a distance map, performing 3D reconstruction, and semantic segmentation.

[0101] The method 800 provides a computationally efficient approach for estimating scene distances from the compact ED histogram representation generated by the PED histogram. By leveraging the narrowest bin as an indicator of the transient peak location, the method avoids the need for complex peak detection algorithms or histogram deconvolution techniques. However, note that the narrowest bin midpoint estimator may be susceptible to errors in scenarios where the transient distribution exhibits multiple peaks or significant asymmetry. In such cases, the narrowest bin may not accurately represent the true peak location, leading to biased distance estimates. To mitigate these limitations, an alternative embodiment of the method 800 may employ more sophisticated techniques for estimating the time-of-flight from the ED histogram bin boundaries. One such approach, described in the following sections, involves computing a linearly interpolated local photon density estimate from the bin boundaries and using the location of the peak in this density estimate as the time-of-flight estimate.

[0102] Referring to FIG. 9, a flowchart of a method 900 for estimating scene distance for a pixel from time-of-flight data is shown. The method 900 utilizes the ED histogram bin boundaries determined by the techniques described earlier to estimate the scene distance for a pixel.

[0103] At operation 902, the method 900 receives the ED histogram bm boundaries [to, ti, ..., tQ+i J for a pixel, where Q is the number of quantile boundaries tracked by the PED histogram, and to and tQ+i are untracked boundaries at zero delay, and a pre-determined maximum delay, respectively. These bin boundaries represent the time delays corresponding to the quantile locations of the photon transient distribution captured at the pixel. The bin boundaries are determined by the PED histogram using the optimized stepping strategy described previously, which enables rapid convergence to the true quantile locations with low bias and variance. In one embodiment, the bin boundaries {to. ti, .... tq+i } are read out directly from the CV’s of the parallel bank of proportional binners that constitute the PED histogrammer. Each binner tracks a distinct quantile boundary, with the CV representing the time delay corresponding to that quantile. The PED histogram may be implemented as an on-sensor circuit, with the bin boundary values transferred off-chip for further processing.

[0104] At operation 904. the method 900 computes a sequence of pairs ((tj.i + tj) / 2, l / (tj - tj.i)) for j = 1 to Q+l. Each pair represents the midpoint between successive bin boundaries and the reciprocal of the bin width, respectively. The reciprocal of the bin width serves as an estimate of the local photon density within the corresponding time interval, under the assumption that narrower bins correspond to regions of higher photon density. Said another way. for each bin i from 1 to Q, a point is computed comprising a first coordinate equal to the midpoint value of bin i, given by (t,.i + b) / 2, and a second coordinate equal to the inverse bin width value of bin i, given by l / (t, - tyi). The midpoint value of bin i represents the average of the preceding bin boundary value tyi and the succeeding bin boundary valuetbwhile the inverse bin width value is the inverse of the difference between the succeeding bin boundary value fe and the preceding bin boundary value ti-i . In one embodiment, the computation of the sequence of pairs is performed by a dedicated hardware unit or a software module within the computing device 110. The sequence of pairs may be stored in a memory buffer or passed directly to subsequent processing stages. In some embodiments, the values U and Ui may be stored as floating-point values, reducing a probability of a difference between t, and ti-i equaling zero. In another embodiment, a predetermined value, epsilon, may be added to the above term ti - ti-i. to prevent division by zero, where epsilon is greater than 0 but substantially smaller than one.

[0105] At operation 906, the method 900 interpolates the sequence of pairs computed in operation 904 onto a predetermined number of uniformly spaced discrete time locations between 0 and B, where B represents the maximum time delay or the duration of the photon transient distribution. This interpolation step produces a linearly interpolated local photon density estimate p, (t). which provides a continuous representation of the underlying photon density as a function of time delay. In one embodiment, the linear interpolation is performed using a dedicated hardware interpolator or a software routine that implements a suitable interpolation algorithm, such as linear or piecewise linear interpolation. The number of uniformly spaced time locations used for the interpolation may be chosen based on the desired resolution and computational complexity constraints.

[0106] At operation 908, the method 900 estimates the time-of-flight based on the time delay t at which the linearly interpolated local photon density pi (t) is maximized. The location of the maximum of pi (t) is expected to correspond to the peak of the true photon transient distribution, which represents the most likely time-of-flight for the laser pulse reflected from the scene point corresponding to the pixel under consideration. In one embodiment, the time delay t at which pi(t) is maximized is determined by a dedicated hardware unit or a softw are routine that performs a peak detection algorithm on the interpolated local photon density estimate. This may involve techniques such as finding the maximum value in the discretized pi(t) sequence, fitting a continuous function to the discrete samples and analytically determining the maximum, or using a gradient-based optimization method to locate the peak.

[0107] At operation 910, the method 900 determines the scene distance for the pixel from the estimated time-of-flight obtained in operation 908. The scene distance is related to the time-of-flight through the simple relationship: z = (c * t) / 2, where z is the distance, c is the speed of light, and t is the estimated time-of-flight from operation 908. In one embodiment, the scene distance calculation is performed by a dedicated arithmetic unit or a software routine that applies the above formula using the estimated time-of-flight and the known speed of light constant. The calculated distance value may be stored in a memory buffer or used directly for subsequent processing tasks, such as constructing a distance map or performing 3D reconstruction. Alternatively, in another embodiment, the scene distance determination may involve applying a calibration cun e or correction factors to account forsystematic biases or non-linearities in the time-of-flight estimation process. These calibration parameters may be pre-computed through controlled experiments or learned from training data using machine learning techniques.

[0108] The method 900 provides a way to estimate scene distance for a pixel by leveraging the compact representation of the photon transient distribution provided by the ED histogram bin boundaries. By interpolating the bin boundary information to obtain a continuous local photon density estimate, the method can accurately locate the peak of the transient distribution and translate it into a distance measurement. This approach enables efficient use of the limited bandwidth and memory resources available in resource-constrained applications while maintaining high spatial and temporal resolution for 3D imaging tasks.

[0109] Referring to FIG. 10, a flowchart of a method 1000 for estimating distances from ED histogram data using a deep neural network is shown. The method 1000 leverages the compact representation of photon arrival times provided by ED histograms, and employs a deep learning approach to exploit spatial and temporal correlations in the ED histogram data, enabling accurate and high-resolution distance estimation.

[0110] At 1002. the method 1000 includes receiving ED histogram bin boundaries for a plurality of pixels of a single-photon detector. In one embodiment, the ED histogram bin boundaries arc determined using the improved CV stepping technique described earlier, which employs a plurality of optimized bimrer circuits to rapidly converge on the desired quantile boundaries of the photon transient distribution at each pixel.[OHl] At 1004, the method 1000 determines linearly interpolated local photon density estimates pi from the received ED histogram bin boundaries for each pixel. In one embodiment, the linearly interpolated photon density' estimate pi (t) is obtained by taking the sequence of non-uniformly spaced ED histogram bin boundary locations {t, J and interpolating them onto a grid of uniformly spaced discrete time locations between 0 and the maximum time-of-flight range B. The interpolated photon density estimate pi(t) can be expressed as a piecewise linear function, with the slope between adjacent bin boundaries inversely proportional to the local photon density. In an alternative embodiment, a piecewise constant photon density estimate po(t) may be used, where the photon density within each ED histogram bin is assumed to be constant and equal to the reciprocal of the bin width.

[0112] At operation 1006, the method 1000 provides the linearly interpolated local photon density estimates pi or the quantile boundary estimates obtained directly from the PED histogram for multiple pixels simultaneously, as input to a deep neural network. In one embodiment, the deep neural network is a convolutional neural network (CNN) architecture designed to process the spatiotemporal data represented by the pi estimates or quantile boundary estimates across multiple pixels. The CNN takes tire pi estimates or quantile boundary' estimates from multiple pixels as a spatial input, akin to an image, and processes them in parallel. Alternatively, the deep neural network may be trained to directly takedie quantile boundary estimates obtained from the PED histogram as input, without requiring interpolation to produce pi estimates. This direct use of the quantile boundary estimates can further reduce computational overhead and provide a more compact representation of the photon data for the deep neural network.

[0113] In another embodiment, the deep neural network may be a recurrent neural network (RNN) or a combination of a CNN and an RNN, which can process the sequence of pi estimates across multiple exposures or laser cycles for each pixel, capturing temporal correlations in the data, while also processing the spatial distribution of pi estimates across pixels within each exposure or cycle.

[0114] At 1008, the method 1000 maps the plurality of p, estimates to a corresponding plurality of distance estimates using the deep neural network. In one embodiment, the deep neural network is trained on a large dataset of simulated or real-world photon data, with the ground truth distance maps serving as the target output. The training process involves minimizing a loss function that captures the discrepancy between the predicted distance maps and the ground truth, such as a combination of Kullback-Leibler divergence between the pi estimates and total variation distance between the distance maps. In some embodiments, the loss may also include a per-pixel RMSE loss. During inference, the trained deep neural network takes the pi estimates as input and produces a denoised and high-resolution distance map as output, leveraging the learned spatial and temporal correlations in the data. In some embodiments, the deep neural network may be trained to predict not only distance estimates but also other scene properties such as surface normals, material properties, or semantic segmentation labels, directly from the input pi estimates or quantile boundary estimates.

[0115] In an alternative embodiment, the deep neural network may be trained using transfer learning, where a pre-trained network is fine-tuned on the specific task of mapping pi estimates to distance maps. The pre-trained network may have been trained on a large, general-purpose dataset and learned to extract useful features from spatiotemporal data. By fine-tuning this pre-trained network on the specific task of distance estimation, the method can leverage the feature extraction capabilities of the pre-trained network while adapting it to the unique characteristics of the ED histogram data.

[0116] The deep learning approach employed in method 1000 provides several advantages over traditional rule-based or optimization-based techniques for distance estimation from photon data. First, the deep neural network can effectively capture complex, non-linear relationships between the input pi estimates and the desired distance maps, enabling accurate depth reconstruction even in challenging scenarios with noise, ambient light, and other sources of interference. Second, by exploiting spatial and temporal correlations in the data, the deep neural network can produce high-resolution and denoised distance maps, overcoming the limitations of per-pixel distance estimation techniques. Finally, the deep learning approach is highly scalable and can be adapted to different hardware platforms and resource constraints by adjusting the network architecture and complexity.

[0117] Method 1000 represents an advantageous combination of optimized hardware for efficient ED histogram generation and advanced deep learning techniques for accurate and high-resolution distance estimation. This approach paves the way for practical deployment of single-photon imaging technology in resource-constrained applications, enabling a wide range of computer vision tasks such as visual odometry, 3D reconstruction, and semantic segmentation to be performed on mobile devices, augmented / virtual reality systems, and autonomous robots.

[0118] The disclosure also provides support for a method comprising: initializing a plurality of control values (CV's) for a plurality of binners, each binner configured to classify photon return events from a pixel of an imaging detector as either early events or late events based on a reference signal controlled by a respective CV, transmitting a light pulse from a pulsed light source, detecting one or more photon return events following the light pulse, simultaneously updating an early arrival count and a late arrival count for each of the plurality of CV’s based on the detected one or more photon return events across the plurality of binners. wherein the early arrival count tracks a number of early events and the late arrival count tracks a number of late events relative to a respective CV of the plurality of CV’s, updating the plurality of CV’s based on the updated early arrival count, the updated late arrival count, and a CV index associated with each respective CV, determining whether the plurality of CV’s satisfy a convergence criterion, and responsive to the plurality of CV’s satisfying the convergence criterion, reading out the plurality of CV's, wherein the plurality of CV's correspond to boundaries of an ED histogram usable for computer vision tasks. In a first example of the method, the computer vision tasks include: determining a distance of an object in a scene, semantic segmentation, object detection, three-dimensional (3D) reconstruction, simultaneous localization and mapping (SLAM), and camera tracking. In a second example of the method, optionally including the first example, the method further comprises: estimating a location of a peak in a distribution of the photon return events based on the ED histogram, wherein the peak corresponds to a time-of-flight of photons from the pulsed light source to the object and back to the pixel, wherein estimating the location of the peak comprises one of: identifying a midpoint of a narrowest bin of the ED histogram as the location of the peak, and fitting a curve to a plurality of points corresponding to multiple bins of the ED histogram to estimate the location of the peak, and determining the distance of the object based on the estimated location of the peak. In a third example of the method, optionally including one or both of the first and second examples, determining the plurality of points corresponding to multiple bins of the ED histogram comprises: for each bin i from 1 to Q. computing a point comprising a first coordinate equal to a midpoint value of bin i. and a second coordinate equal to an inverse bin width value of bin i. wherein the midpoint value of bin i is an average of a preceding bin boundary value and a succeeding bin boundary value, and the inverse bin width value is an inverse of a difference between the succeeding bin boundary value and the preceding bin boundary value. In a fourth example of the method, optionally including one or more or each of the first through third examples, updating the respective CV for each binner of the plurality of binners comprises: determining a base step size for adjusting the respective CV based on the updatedearly arrival count, the updated late arrival count, and the CV index, for the respective CV, scaling the base step size to produce a scaled step size for the respective CV, smoothing the scaled step size to produce a first smoothed step size for the respective CV, temporally decaying the first smoothed step size to produce a temporally decayed step size for the respective CV, and smoothing the temporally decayed step size to produce a second smoothed step size for the respective CV. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, scaling the base step size comprises multiplying the base step size by a scaling factor that is a percentage of a total number of equi-depth bins. In a sixth example of the method, optionally including one or more or each of the first through fifth examples, temporally decaying the first smoothed step size comprises multiplying the first smoothed step size by a decay factor less than 1. In a seventh example of the method, optionally including one or more or each of the first through sixth examples, initializing the plurality of CV's comprises: setting each CV of the plurality of CV's to an initial value proportional to a CV index (j) associated with the respective CV, wherein the CV index is an integer between 1 and Q. where Q is a total number of CV's for the pixel.

[0119] The disclosure also provides support for a method comprising: initializing a plurality of control values (CV's) and CV stepping parameters, for each CV of the plurality of CV's: selecting a CV index (j) corresponding to the CV, transmitting a light pulse from a pulsed light source, detecting one or more photon arrivals following the transmitted light pulse, updating an early arrival count (En) and a late arrival count (Ln) for the selected CV index (j) based on the detected one or more photon arrivals, wherein the early arrival count (En) tracks a number of photon arrivals occurring before a time delay represented by the CV, and the late arrival count (Ln) tracks a number of photon arrivals occurring after the time delay represented by the CV, updating the C V corresponding to the selected CV index (j) based on the updated early arrival count (En), the updated late arrival count (Ln), and the CV index (j), determining if a convergence criterion is satisfied for the updated CV corresponding to the selected CV index (j), upon determining the convergence criterion is satisfied for the updated CV corresponding to the selected CV index (j), proceeding to update a next CV of the plurality of CV's, reading out the plurality of updated CV's when a predetermined number of the plurality of CV's have converged, wherein the plurality of updated CV's represent time delays corresponding to quantile boundaries. In a first example of the method, updating the CV corresponding to the selected CV index (j) comprises: determining a base step size for the CV based on the updated early arrival count (En) and the updated late arrival count (Ln), scaling the base step size to produce a scaled step size, smoothing the scaled step size to produce a first smoothed step size, temporally decaying the first smoothed step size to produce a temporally decayed step size, smoothing the temporally decayed step size to produce a second smoothed step size, and adding the second smoothed step size to the C V to produce the updated C V for a current cycle. In a second example of the method, optionally including the first example, the base step size is determined as a difference between a target fraction corresponding to the CV index (j) and a ratio of the updated early arrival count (En) to a sum of the updated early arrival count (En) and the updatedlate arrival count (Ln). In a third example of the method, optionally including one or both of the first and second examples, temporally decaying the first smoothed step size comprises multiplying the first smoothed step size by a temporal decay factor raised to a power equal to a number of cycles over which the CV has been updated. In a fourth example of the method, optionally including one or more or each of the first through third examples, the convergence criterion is satisfied when a change in the updated CV corresponding to the selected CV index (j) is less than a predetennined threshold over a predetermined number of cycles. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, the method further comprises: determining distances in a scene based on the plurality of updated CV's representing the quantile boundaries, wherein determining the distances comprises: receiving ED histogram bin boundaries corresponding to the plurality of updated CV's for a plurality of pixels, determining linearly interpolated local photon density estimates from the ED histogram bin boundaries for each pixel of the plurality' of pixels, providing the linearly interpolated local photon density estimates as input to a deep neural network, and mapping the linearly interpolated local photon density estimates to a corresponding plurality of distance estimates using the deep neural network. In a sixth example of the method, optionally including one or more or each of the first through fifth examples, the plurality of CV's corresponds to quantile boundaries spanning a range of time delays between the transmitted light pulse and photon arrivals at a pixel detector.

[0120] The disclosure also provides support for a system for determining boundaries of an ED histogram of single photon arrival times, comprising: a memory, wherein the memory stores instructions, an ED histogram module comprising a plurality of binners, a pulsed light source, an imaging detector comprising a single photon detecting pixel, and a processor communicably coupled to die memory , the pulsed light source, the ED histogram module, and the imaging detector, wherein, when executing the instructions, the processor is configured to: initialize a plurality of control values (CV's) for the plurality of binners, each binner configured to classify photon return events from the pixel of the imaging detector as either early events or late events based on a reference signal controlled by a respective CV, transmit a light pulse via the pulsed light source, detect one or more photon return events at the imaging detector following the light pulse, simultaneously update an early arrival count and a late arrival count for each of the plurality of CV's based on the detected one or more photon return events across the plurality of binners, wherein the early arrival count tracks a number of early events and the late arrival count tracks a number of late events relative to a respective CV of the plurality of CV's, update the plurality of CV's based on the updated early arrival count, the updated late arrival count, and a CV index associated with each respective CV. detennine whether the plurality of CV's satisfy a convergence criterion, and responsive to the plurality of CV's satisfying the convergence criterion, read out the plurality of CV's, wherein the plurality of CV's correspond to boundaries of an ED histogram usable to determine a distance of an object in a scene. In a first example of the system when updating the respective CV for each binner of the plurality of binners. the processor is further configured to: determine a base step size for adjusting the respective CV based on the updated earlyarrival count, the updated late arrival count, and a CV index, for the respective CV, scale the base step size to produce a scaled step size for the respective CV, smooth the scaled step size to produce a first smoothed step size for the respective CV, temporally decay the first smoothed step size to produce a temporally decayed step size for the respective CV, and smooth the temporally decayed step size to produce a second smoothed step size for the respective CV, wherein the respective CV is updated by adding the second smoothed step size. In a second example of the system, optionally including the first example, temporally decaying the first smoothed step size comprises multiplying the first smoothed step size by a decay factor less than 1 raised to a power equal to a number of cycles over which the respective CV has been updated. In a third example of the system, optionally including one or both of the first and second examples, the processor is further configured to: estimate a location of a peak in a distribution of the photon return events based on the ED histogram, wherein the peak corresponds to a time-of-flight of photons from the pulsed light source to the object and back to the pixel, and determine the distance of the object based on the estimated time-of-flight. In a fourth example of the system, optionally including one or more or each of the first through third examples, to estimate the location of the peak, the processor is configured to perform one of: identify a midpoint of a narrowest bin of the ED histogram as the location of the peak, and fit a curve to a plurality of points corresponding to multiple bins of the ED histogram to estimate the location of the peak. In a fifth example of the system, optionally including one or more or each of the first through fourth examples, the processor is further configured to: receive ED histogram bin boundaries corresponding to the plurality of CV's for a plurality of pixels, determine linearly interpolated local photon density estimates from the ED histogram bin boundaries for each pixel of the plurality of pixels, provide the linearly interpolated local photon density estimates as input to a deep neural network, and map the linearly interpolated local photon density estimates to a corresponding plurality of distance estimates using the deep neural network.

[0121] As used herein, the terms “system’' or “module" or “modulator” may include a hardware and / or software system that operates to perform one or more functions. For example, a module or system may include a computer processor, controller, or other logic-based device that performs operations based on instructions stored on a tangible and non-transitory computer readable storage medium, such as a computer memory. Alternatively, a module or system may include a hard-wired device that performs operations based on hard-wired logic of the device. Various modules or units shown in the attached figures may represent the hardware that operates based on software or hardwired instructions, the software that directs hardware to perform the operations, or a combination thereof.

[0122] The foregoing described aspects depict different components contained within, or connected with different other components. It is to be understood that such depicted architectures are merely exemplary, and that in fact many other architectures can be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively associated such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality7can be seen as "associated with" eachother such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being "operably connected," or "operably coupled," to each other to achieve the desired functionality.

[0123] As used in this application, an element or step recited in the singular and proceeded with the word “a” or “an” should be understood as not excluding plural of said elements or steps, unless such exclusion is stated. Furthermore, references to “one embodiment” or “one example” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. The terms “first,” "second,” “third,” and so on are used merely as labels, and are not intended to impose numerical requirements or a particular positional order on their objects. The following claims particularly point out subject matter from the above disclosure that is regarded as novel and non-obvious.

Claims

CLAIMS:

1. A method comprising : initializing a plurality of control values (CV's) for a plurality' of binners, each bimier configured to classify photon return events from a pixel of an imaging detector as either early events or late events based on a reference signal controlled by a respective CV; transmitting a light pulse from a pulsed light source; detecting one or more photon return events following the light pulse; simultaneously updating an early arrival count and a late arrival count for each of the plurality of CV’s based on the detected one or more photon return events across the plurality of binners, wherein the early arrival count tracks a number of early events and the late arrival count tracks a number of late events relative to a respective CV of the plurality of CV’s; updating the plurality' of CV’s based on the updated early arrival count, the updated late arrival count, and a CV index associated with each respective CV; determining whether the plurality of CV’s satisfy a convergence criterion; and responsive to the plurality' of CV’s satisfying the convergence criterion, reading out the plurality of CV's. wherein the plurality7of CV's corresponds to boundaries of an equi-depth histogram (ED histogram) usable for computer vision tasks.

2. The method of claim 1, wherein the computer vision tasks include: determining a distance of an object in a scene; semantic segmentation; object detection; surface normal estimates; determining material properties; three-dimensional (3D) reconstruction; simultaneous localization and mapping (SLAM); and camera tracking.

3. The method of claim 2, further comprising: estimating a location of a peak in a distribution of the photon return events based on the ED histogram, wherein the peak corresponds to a time-of-flight of photons from the pulsed light source to the object and back to the pixel; wherein estimating the location of the peak comprises one of: identifying a midpoint of a narrowest bin of the ED histogram as the location of the peak; andfitting a curve to a plurality of points corresponding to multiple bins of the ED histogram to estimate the location of the peak; and determining the distance of the object based on the estimated location of the peak.

4. The method of claim 3, wherein determining the plurality of points corresponding to multiple bins of the ED histogram comprises: for each bin i from 1 to Q+ 1, computing a point comprising a first coordinate equal to a midpoint value of bin i, and a second coordinate equal to an inverse bin width value of bin i, wherein the midpoint value of bin i is an average of a preceding bin boundary value and a succeeding bin boundary value, and the inverse bin width value is an inverse of a difference between the succeeding bin boundary value and the preceding bin boundary value.

5. The method of claim 1, wherein updating the respective CV for each binner of the plurality of binners comprises one or more of the following: determining a base step size for adjusting the respective CV based on the updated early arrival count, the updated late arrival count, and the CV index, for the respective CV; scaling the base step size to produce a scaled step size for the respective CV; smoothing the scaled step size to produce a first smoothed step size for the respective CV; temporally decaying the first smoothed step size to produce a temporally decayed step size for the respective CV; and smoothing the temporally decayed step size to produce a second smoothed step size for the respective CV.

6. The method of claim 5. wherein scaling the base step size comprises multiplying the base step size by a scaling factor that is a percentage of a total number of time intervals.

7. The method of claim 5, wherein temporally decaying the first smoothed step size comprises multiplying the first smoothed step size by a decay factor less than 1.

8. The method of claim 1, wherein initializing the plurality of CV's comprises: setting each CV of the plurality of CV's to an initial value proportional to a C V index (j) associated with the respective CV. wherein the CV index is an integer between 1 and Q, where Q is a total number of CV's for the pixel.

9. A method comprising: initializing a plurality of control values (CV's) and CV stepping parameters; for each CV of the plurality of CV's:selecting a CV index (j) corresponding to the CV; transmitting a light pulse from a pulsed light source; detecting one or more photon arrivals following the transmitted light pulse; updating an early arrival count (En) and a late arrival count (Ln) for the selected CV index (j) based on the detected one or more photon arrivals, wherein the early arrival count (E„) tracks a number of photon arrivals occurring before a time delay represented by the CV, and the late arrival count (Ln) tracks a number of photon arrivals occurring after the time delay represented by the CV; updating the CV corresponding to the selected CV index (j) based on the updated early arrival count (En), the updated late arrival count (Ln), and the CV index (j); determining if a convergence criterion is satisfied for the updated CV corresponding to the selected CV index (j); upon determining the convergence criterion is satisfied for the updated CV corresponding to the selected CV index (j), proceeding to update a next CV of the plurality of CV's; reading out the plurality of updated CV's when a predetermined number of the plurality of CV's have converged, wherein the plurality of updated CV's represents time delays corresponding to quantile boundaries.

10. The method of claim 9, wherein updating the CV corresponding to the selected CV index (j) comprises one or more of the following: determining a base step size for the CV based on the updated early arrival count (En) and the updated late arrival count (Ln); scaling the base step size to produce a scaled step size; smoothing the scaled step size to produce a first smoothed step size; temporally decaying the first smoothed step size to produce a temporally decayed step size; smoothing the temporally decayed step size to produce a second smoothed step size; and adding the second smoothed step size to the CV to produce the updated CV for a current cycle.

11. The method of claim 10. wherein tire base step size is determined as a difference betw een a target fraction corresponding to the CV index (j) and a ratio of the updated early arrival count (En) to a sum of the updated early arrival count (En) and the updated late arrival count (L„).

12. The method of claim 10, wherein temporally decaying the first smoothed step size comprises multiplying the first smoothed step size by a temporal decay factor raised to a power equal to a number of cycles over which the CV has been updated.

13. The method of claim 9, wherein the convergence criterion is satisfied when a change in the updated CV corresponding to the selected CV index (j) is less than a predetermined threshold over a predetermined number of cycles.

14. The method of claim 9, further comprising: determining distances in a scene based on the plurality of updated CV's representing the quantile boundaries, wherein determining the distances comprises: receiving ED histogram bin boundaries corresponding to the plurality of updated CV's for a plurality of pixels: determining linearly interpolated local photon density' estimates from the ED histogram bin boundaries for each pixel of the plurality of pixels; providing the linearly interpolated local photon density estimates as input to a deep neural network; and mapping the linearly interpolated local photon density estimates to a corresponding plurality of distance estimates using the deep neural network.

15. The method of claim 9, wherein the plurality of CV's corresponds to quantile boundaries spanning a range of time delays between the transmitted light pulse and photon arrivals at a pixel detector.

16. A system for determining boundaries of an equi-depth histogram (ED histogram) of single photon arrival times, comprising: a memory', wherein the memory' stores instructions; an ED histogrammer module comprising a plurality of binners; a pulsed light source; an imaging detector comprising a single photon detecting pixel; and a processor communicably coupled to the memory, the pulsed light source, the ED histogrammer module, and the imaging detector, wherein, when executing the instructions, the processor is configured to: initialize a plurality of control values (CVs) for the plurality of binners, each binner configured to classify photon return events from the pixel of the imaging detector as either early events or late events based on a reference signal controlled by a respective CV; transmit a light pulse via the pulsed light source: detect one or more photon return events at the imaging detector following the light pulse: simultaneously update an early arrival count and a late arrival count for each of the plurality' of CVs based on the detected one or more photon return events across the plurality of binners.wherein the early arrival count tracks a number of early events and the late arrival count tracks a number of late events relative to a respective CV of the plurality of CV's; update the plurality of CV's based on the updated early arrival count, the updated late arrival count, and a CV index associated with each respective CV; determine whether the plurality of CV's satisfy a convergence criterion; and responsive to the plurality of CV's satisfying the convergence criterion, read out the plurality of CV's, wherein the plurality of CV's corresponds to boundaries of an ED histogram usable to determine a distance of an object in a scene.

17. The system of claim 16. wherein, when updating the respective CV for each binner of the plurality' of binners, the processor is further configured to perform at least one of the following: determine a base step size for adjusting the respective CV based on the updated early arrival count, the updated late arrival count, and a CV index, for the respective CV; scale the base step size to produce a scaled step size for the respective CV; smooth the scaled step size to produce a first smoothed step size for the respective CV; temporally decay the first smoothed step size to produce a temporally decayed step size for the respective CV; and smooth the temporally decayed step size to produce a second smoothed step size for the respective CV. wherein the respective CV is updated by adding the second smoothed step size.

18. The system of claim 17, wherein temporally decaying the first smoothed step size comprises multiplying the first smoothed step size by a decay factor less than 1 raised to a power equal to a number of cycles over which the respective CV has been updated.

19. The system of claim 16, wherein the processor is further configured to: estimate a location of a peak in a distribution of the photon return events based on the ED histogram, wherein the peak corresponds to a time-of-flight of photons from the pulsed light source to the object and back to the pixel; and determine the distance of the object based on the estimated time-of-flight.

20. The system of claim 19, wherein to estimate the location of the peak, the processor is configured to perform one of: identify a midpoint of a narrowest bin of the ED histogram as the location of the peak; and fit a curve to a plurality of points corresponding to multiple bins of the ED histogram to estimate the location of the peak.

21. The system of claim 16, wherein the processor is further configured to: receive ED histogram bin boundaries corresponding to the plurality of CV's for a plurality of pixels; determine linearly interpolated local photon density estimates from the ED histogram bin boundaries for each pixel of the plurality of pixels; directly provide the linearly interpolated local photon density estimates or quantile boundary estimates as input to a deep neural network; and using the deep neural network to map the linearly interpolated local photon density estimates to a corresponding plurality of at least one of the following: distance estimates, surface normals, material properties, or semantic segmentation labels, directly from the input pi estimates or quantile boundary estimates.

Images