Systems and methods for count-free histograms in 3D imaging

Abstract

A system and method are provided for count-free iso-depth histograms that can be used in 3D imaging. In one embodiment, the method includes receiving, at a bin classifier, a stream of photon return events from pixels of an imaging detector, the stream of photon return events being generated by photons transmitted from a pulsed light source and reflected from objects in a scene, classifying each photon return event as either an early event or a late event based on a reference signal controlled by a control value, the control value being configured to vary based on a relative proportion of early events to late events, and outputting, on demand, the control value from the bin classifier that can be used to determine the distance of an object in the scene.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 375,045, filed September 8, 2022, entitled "System and Method for Count-Free Histograms in 3D Imaging," the entire contents of which are incorporated herein by reference for all purposes.

[0002] Government support This invention was made with government support under Grant No. 2138471 awarded by the National Science Foundation. The United States Government has certain rights in this invention.

[0003] Field The present disclosure relates to iso-depth histograms (also referred to as iso-height histograms), and more particularly to iso-depth histograms captured by single-photon sensitive 3D cameras. [Background technology]

[0004] background Single-photon sensing has recently emerged as a promising new technology for high-resolution 3D imaging. Single-photon 3D cameras capture the round-trip time of a laser pulse by precisely time-tagging the arrival of individual photons at each camera pixel. Capturing a photon timestamp histogram is a fundamental operation in single-photon 3D imaging. However, forming a standard histogram at each camera pixel is computationally intensive, power-consuming, and requires a large amount of memory within each pixel. Summary of the Invention

[0005] overview As discussed further below, various systems and methods are provided that significantly improve the computation of depth maps (e.g., distance maps) of imaged scenes using single-photon sensing 3D cameras. In one embodiment, the method includes receiving, at a bin classifier, a stream of photon return events from pixels of an imaging detector, the stream of photon return events being generated by photons transmitted from a pulsed light source and reflected from objects in the scene, classifying each photon return event using the bin classifier as either an early or late event based on a reference signal controlled by a control value, the control value being configured to vary based on the relative proportion of early events to late events, and outputting, on demand, from the bin classifier, a control value that can be used to determine the distance of an object in the scene.

[0006] In this way, the distance of objects in an imaged scene may be determined without explicitly forming a histogram at each pixel by using an adaptive approach for distance estimation in resource-constrained settings with limited bandwidth, limited memory, and limited computation. The approach described herein constructs an equal-depth histogram, as opposed to the equal-width histograms of other methods. An equal-depth histogram is a more concise representation of a "peaked" distribution, such as that obtained by a single-photon detector from a laser pulse reflected by a surface. This approach utilizes a bin classifier element that adaptively converges to the median (or more generally, any other k-quantile) of the distribution. In some embodiments, multiple bin classifiers may be combined to form an equal-depth histogram generator (EDH) capable of generating multi-bin histograms.

[0007] It should be understood that the foregoing brief description is provided to introduce a selection of concepts in a simplified form that are further described in the detailed description. It is not intended to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims that follow the detailed description. Moreover, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or anywhere in this disclosure. [Brief explanation of the drawings]

[0008] BRIEF DESCRIPTION OF THE DRAWINGS The present disclosure may be better understood by reading the following description of non-limiting embodiments with reference to the accompanying drawings, in which:

[0009] [Figure 1] 1 illustrates a schematic diagram of an exemplary imaging environment including a time-of-flight 3D camera, according to one embodiment.

[0010] [Figure 2A-2B] FIG. 1 is a circuit diagram for a bin classifier, according to one embodiment.

[0011] [Figure 3A-3B] 10 illustrates the boundaries of a bin classifier at a relatively early cycle in a run, according to one embodiment.

[0012] [Figure 4] 10 illustrates the convergence of a bin classifier over multiple cycles of a run, according to one embodiment.

[0013] [Figures 5A-5D] 10 shows the convergence of three different bin classifier configurations according to one embodiment at various combinations of signal strength and background light, including high signal, low background conditions.

[0014] [Figures 6A-6B] 10 illustrates the bin boundaries output by the bin classifier under low background light, according to one embodiment.

[0015] [Figure 7] FIG. 1 is a circuit diagram for a multi-stage equal-depth histogram generator (EDH), according to one embodiment.

[0016] [Figures 8A-8G] 10 illustrates the boundaries output by the EDH over multiple cycles of a run, according to one embodiment.

[0017] [Figures 9A-9D] The output of EDH (with 16 bins) with low background is shown.

[0018] [Figure 10] 10 shows example distance maps of two scenes using 8-bin equal-width and equal-depth histograms.

[0019] [Figure 11] 10 shows an example distance map of a scene using a 16-bin equal-width histogram and an equal-depth histogram.

[0020] [Figure 12] 1 is a flowchart illustrating an example method for identifying bin boundaries of a transient distribution using a bin classifier.

[0021] [Figure 13] 10 is a flowchart illustrating an example method for identifying multiple bin boundaries of a transient distribution using an equal-depth histogram generator.

[0022] [Figure 14] 1 illustrates a schematic diagram of an exemplary histogram generator coupled to multiple pixels of a detector.

[0023] [Figure 15] FIG. 10 is an exemplary circuit diagram illustrating an analog implementation of a control value.

[0024] [Figure 16] 1 shows an exemplary circuit diagram illustrating a digital implementation of a control value. DETAILED DESCRIPTION OF THE INVENTION

[0025] Detailed Description The following description relates to various embodiments that employ iso-depth histograms to calculate the distance (e.g., depth) of one or more objects in a scene imaged with a single-photon sensing 3D camera. For example, as depicted in FIG. 1 , a single-photon sensing 3D imaging system may include a pulsed light source and a detector positioned to receive photons transmitted from the pulsed light source and reflected from one or more objects in the scene. In particular, the detector may include a bin classifier, such as the bin classifier depicted in FIG. 2A , for each pixel (or group of pixels, as depicted in FIG. 14 ), that can adaptively identify bin boundaries from ranges in the transient distribution (e.g., ranges of delay times of photon return events). The bin classifier may separate the stream of photon return events into a late stream and an early stream based on a reference signal controlled by a control value (which may be implemented in analog, as depicted in FIG. 15 , or digitally, as depicted in FIG. 16 , although other implementations are possible without departing from the scope of this disclosure), which varies based on the number of detected early and late events, as depicted by the bin classifier output in FIG. 2B . The bin classifier may ultimately generate bin boundaries that converge at or near the median of the return events or another target k-quantile of the distribution of return events, as shown in Figures 3A-3B and 4. The accuracy of identifying the actual median of the return events may be affected by the signal strength and the amount of background light in the scene, as shown in Figures 5A-5D and 6A-6B.

[0026] The bin classifier may have a single stage, such as the bin classifier of FIG. 2A, or multiple bin classifiers may be implemented with multiple stages to achieve an equal-depth histogram generator (EDH), as shown by the multi-stage histogram generator of FIG. 7. The multi-stage histogram generator may identify / converge on multiple bin boundaries, such as 7 or 15 bin boundaries (and thus 8 or 16 bins), depending on the number of stages in the multi-stage histogram generator. As can be seen from FIGS. 8A-8G, which show the convergence of 15 bin boundaries over multiple cycles of a run, the bin boundaries may be centered around the true peak location of the transient distribution. Furthermore, multiple bins may allow background light to be "absorbed" by a portion of the bins, causing the remaining bins to cluster around distribution features, such as the distribution's peak(s). Such a histogram generator may be less susceptible to background light, as shown by FIGS. 9A and 9B. Exemplary distance maps, equal-width and equal-depth histograms generated using 8 and 16 bins, respectively, are shown in Figures 10 and 11. Thus, a bin classifier may be used to identify bin boundaries in a transient distribution (e.g., of photon events) according to the exemplary method of Figure 12, and a histogram generator may be used to identify multiple bin boundaries in a transient distribution according to the exemplary method of Figure 13.

[0027] Referring now to the figures, FIG. 1 schematically illustrates an exemplary imaging environment 100 for imaging a scene using a single-photon sensing 3D imaging system, according to one embodiment. The environment 100 includes a light source 102, an object 104, and a detector 106. The environment further includes a computing device 110. While the light source 102 and the detector 106 are shown as separate devices in FIG. 1, it should be understood that the light source 102 and the detector 106 may be integrated into a single device (e.g., integrated into the computing device 110). The light source 102 and the detector 106 may form a single-photon sensing 3D camera (SPC) that captures distance (e.g., depth) information using a time-of-flight principle similar to echolocation, but using light instead of sound. As shown in FIG. 1, consider a single scene point whose distance needs to be estimated. The light source 102 (which may be a laser) illuminates the scene point (which may be part of the object 104) with a short light pulse (e.g., the transmit pulse shown in FIG. 1). The detector 106 may include an array of detector elements, each configured to (separately) detect a photon. In some embodiments, each detector element may be a diode, such as a single-photon avalanche diode or an avalanche photodiode. As used herein, a detector element may be referred to as a pixel, and thus the detector 106 may include multiple pixels. The pixels of the detector 106 may capture a stream of return events (e.g., the return photon events of FIG. 1 ) as the photons arrive at different time delays relative to the time the original light pulse was transmitted. (The detector 106 typically captures two or more return events in response to each laser pulse sent into the scene.) Furthermore, this return stream may also include spurious photon events that are not due to the light pulse (signal) but rather due to ambient background light and other noise sources in the image sensing hardware. Traditionally, a histogram is constructed by accumulating photon counts at different delays over many light (e.g., laser) cycles.The location of the "arg max" peak in this histogram gives an estimate of the true distance of the scene point (relying on the simple relationship that the speed of light multiplied by the time delay is equal to twice the distance to the scene point). As discussed in more detail below, if the true distance of a scene point is instead estimated using an iso-depth histogram, the power consumption, memory requirements, and processing power needed to determine object distance can be reduced.

[0028] As mentioned above, light source 102 may be a laser or another suitable light source capable of transmitting pulses of light at a frequency dictated by clock 108. Detector 106 may be a single-photon avalanche diode (SPAD) sensor using single-photon sampling, an avalanche photodiode (APD) sensor, or another suitable image sensor capable of capturing single photons. Detector 106 may receive pulse frequency information from clock 108 (e.g., indicating the start and end of each pulse of light source 102).

[0029] Computing device 110 (which in some embodiments may include one of light source 102 and / or detector 106) may be a smartphone camera, a light detection and ranging (LiDAR) sensor (e.g., for autonomous robotics), a camera for scientific imaging, a virtual reality device, an augmented reality device, a desktop computer, a laptop, a mobile device (e.g., a smartphone or tablet), or another suitable device.

[0030] Due to their compatibility with CMOS fabrication techniques, there is an increasing availability of high (kilo- to mega-) resolution arrays of single-photon detection pixels (e.g., SPADs) with additional data processing integrated into the hardware chip containing the single-photon detection pixels. Unfortunately, high sensitivity and high speed are a double-edged sword: the amount of raw data generated by these detectors is several orders of magnitude greater than can reasonably be processed or transferred in real time. This aspect limits their applicability in many real-world applications, especially those where power and bandwidth are limited.

[0031] Therefore, as discussed in more detail below, embodiments are provided herein that offer a different approach to direct time-of-flight imaging that is compatible with a variety of detectors and illumination schemes. Capturing and transferring the entire received waveform (either single-photon sampling using SPAD pixels or high-speed analog-to-digital conversion of APDs) is resource-intensive. Instead of attempting to capture the complete waveform in the digital domain (which often consumes a large portion of the total power), the embodiments disclosed herein perform as much processing as possible in the analog domain. To this end, aspects of the field of race logic are applied, where information is encoded in the signal's precise arrival time rather than its voltage level. This approach is naturally well-suited for single-photon time-of-flight 3D sensing, since the arrival times of photon return events naturally convey useful scene information (scene distance and reflectivity). Additionally, the embodiments disclosed herein utilize an equal-depth (ED) histogram to represent the transient distribution of photon return events, rather than the equal-width (EW) histogram shown in Figure 11 employed by other methods. The power and bandwidth limitations of single-photon cameras significantly limit the wider applicability of high-resolution SPC arrays. Creating a complete EW histogram on the sensor is infeasible due to severe memory constraints, while moving the photon timestamp data off-sensor is undesirable because it introduces latency and consumes power. The use of ED histograms and photon arrival times described herein addresses these issues by reducing or eliminating the need for extensive storage on-sensor and utilizing very little power by transferring very little data off-sensor.

[0032] As mentioned above, the detector 106 (and in at least some embodiments, the light source 102) may be integrated within or operably coupled to the computing device 110. Although not shown in FIG. 1 , it should be understood that the computing device 110 may include a logic subsystem, such as a processor, and a data retention subsystem, such as a memory. The computing device 110 may optionally include a display subsystem, a communication subsystem, a user interface subsystem, and other components. A processor may comprise one or more physical devices configured to execute one or more instructions. For example, a processor may execute one or more instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more devices, or otherwise achieve a desired result.

[0033] Thus, a processor may include one or more processors configured to execute software instructions. Additionally or alternatively, a processor may comprise one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. As illustrative and non-limiting examples, a processor may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc. A processor may be single-core or multi-core, and programs executed thereon may be configured for parallel or distributed processing. A processor may optionally include individual components distributed across two or more devices, which may be remotely located and / or configured for cooperative processing. Such devices may be connected via a network.

[0034] The memory of computing device 110 may comprise one or more physical, non-transitory devices configured to hold data and / or instructions executable by the processor to implement the methods and processes described herein. The state of the memory may be changed (e.g., to hold different data) when such methods and processes are implemented.

[0035] The memory may include removable media and / or embedded devices. The memory may include optical memory (e.g., CDs, DVDs, HD-DVDs, Blu-ray discs, etc.) and / or magnetic memory devices (e.g., hard disk drives, floppy disk drives, tape drives, MRAM, etc.). The memory may include devices having one or more of the following characteristics: volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and content addressable. In some embodiments, the processor and memory may be integrated into one or more common devices, such as an application specific integrated circuit or a system on a chip.

[0036] The computing device 110 may be communicatively coupled to a display device. As an illustrative and non-limiting example, the display device may display an image captured by the detector 106, or the display device may display an image sized / positioned based on a distance map determined from the output of the detector 106, or the like. The display device may include one or more display devices utilizing virtually any type of display technology, such as, but not limited to, a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), an electroluminescent display (ELD), an active matrix OLED (AMOLED), a quantum dot (QD) display, or the like. As another example, the display device may comprise a display projector device, such as a digital light processing (DLP) projector, a liquid crystal on silicon (LCoS) projector, a laser projector, an LED projector, or the like. As yet another example, the display device may comprise an augmented reality (AR) display system, a virtual reality (VR) display system, or a mixed reality (MR) display system.

[0037] FIG. 2A shows a circuit diagram for a single-stage bin classifier according to one embodiment of the present disclosure, and FIG. 2B shows an example result of the bin classifier. As used herein, the term bin classifier refers to a circuit that generates control values ​​that adaptively converge to the k-quantile of a transient distribution over the course of many laser pulses. In particular, FIG. 2A includes a circuit diagram of a bin classifier 200 configured to adaptively find and adjust bin boundaries within a transient distribution (e.g., photon return events) using the proportion of return events arriving earlier and later than the current boundaries. The range over which return delay information is accumulated is referred to herein as a window, with the scale expressed in the smallest discernible units. For the example presented herein, the window may be 1024 units long, which is discretized into steps of 128 picoseconds. This corresponds to a bin resolution of approximately 3 cm. However, other window lengths (e.g., a 2000 unit window, a 1024-2000 unit window, or a window greater than 2000 units) and step lengths are possible without departing from the scope of this disclosure. In some embodiments, the maximum window length may be limited by the laser pulse cycle time, which determines the maximum distance range of the 3D camera. The aggregated photon return event information over the window may be referred to as a transient distribution.

[0038] In its simplest form, the bin classifier 200 adjusts the bin boundaries so that an equal portion of the returned events fall on either side of the bin boundary, causing the bin classifier to seek the median value. An important aspect of the bin classifier 200 is that at least the early stages of the processing of the returned events can be implemented using low-power race logic. The current value of the bin boundary may be represented via a reference signal that starts high at the beginning of each cycle and goes low at a point within a window corresponding to the delay represented by the bin boundary. By leaving the returned events as delayed, very few device switches are required to classify the returned events relative to the reference signal, rather than converting the returned events to timestamps. The early and late event streams may be used to adjust the duration of the reference signal and, therefore, the current location of the bin boundary.

[0039] The bin classifier 200 includes a reference signal generator 202 that outputs a reference signal RS having a waveform that starts high at the beginning of each cycle (when a laser pulse is sent) and goes low after a delay corresponding to the bin boundary position. The bin classifier 200 combines RS with a stream of returning events (SR) to produce two output streams. The SRs may be voltage pulses generated by detector pixels, each indicating that a photon has struck a detector pixel. The two output streams include an early stream (SE) of events arriving earlier than the current boundary and a late stream (SL) of events arriving later than the current boundary. In addition to the possibility of feeding the SE and SL streams to additional bin classifier stages (discussed in more detail below with respect to FIG. 7), the bin classifier 200 uses the SE and SL streams to adjust a control value (CV) 204 that dictates the length of the high portion of the reference signal RS. Events in the SE decrease the CV, while events in the SL increase the CV. The reference signal generator 202 may be a monostable multivibrator (e.g., a one-shot circuit) with a waveform (e.g., the duration of the high portion of the waveform) that depends on a control value. Because the CV controls / adjusts the RS, the CV may also be referred to herein as a reference signal modulator. The bin sorter 200 further includes two AND gates, namely, AND gate G1 206 and AND gate G2 208 (G2 has one input inverted), that generate streams SE and SL from the input stream SR. In addition, the bin sorter 200 provides a readout of the current control value at any point.

[0040] The control value CV may be an integer in a suitable range determined based on the configuration of the single-photon camera (e.g., light source intensity, pulse frequency). For example, CV may be an integer in the range of 1 to 1024, with each number corresponding to a specific number of time units of delay. As a specific, non-limiting example, each time unit may be 128 picoseconds, representing a round-trip distance of 1.9 cm or 3.8 cm. The bin classifier may be configured to find bin boundaries at the 50th percentile (i.e., median) of the transient distribution; therefore, CV may be initialized to 512. SE and SL return events may decrement and increment CV by one unit, respectively.

[0041] The bin classifier 200 may be implemented using a field programmable gate array (FPGA), or the bin classifier may be implemented within a pixel using an application-specific integrated circuit (ASIC). The ASIC may have a mixed-signal configuration in which some parts are implemented in the analog domain and other parts are implemented in the digital domain after A / D conversion. Additional details of the bin classifier configuration are presented below with respect to FIGS. 15 and 16. The bin classifier 200 may be configured to find the median (or other k-quantile, such as the 75th percentile) of the transient distribution of return events detected by a single pixel. In such an embodiment, each pixel of the detector may be coupled to a respective bin classifier. In other embodiments, the bin classifier 200 may be capable of coupling to two or more pixels and may sequentially determine the bin boundaries of the transient distribution of return events for each pixel that the bin classifier is configured to couple to.

[0042] FIG. 2B schematically illustrates an exemplary output 250 from the bin classifier 200 for one window of return events (e.g., a window beginning at the window start and ending at the window end) obtained over the course of one cycle (e.g., in response to one output light pulse). Return events within the return event stream (SR) over the window are shown along the top plot, with each peak / square wave representing a detected photon. The reference signal (RS) over the window is shown along the first plot from the top. The RS has a waveform with a first portion having a high value and a second portion having a low value. The first portion, in this example, has a relatively long duration, e.g., greater than 50% of the window length, and switches to the second portion at the dashed vertical line. SE events over the window are displayed from the top plot to the second plot, and SL events over the window are displayed in the bottom plot.

[0043] In the illustrated window, six return events are detected. The first four return events are classified as early events (e.g., the events are detected while RS is high) due to each return event having a delay time (e.g., from the light pulse to the detection of a photon) shorter than the time specified by the reference signal. At the time indicated by the dashed line, RS switches to a low value, and the subsequent two return events are classified as late events (e.g., have a delay time greater than the time specified by the reference signal). In one example, an event within SE decrements the control value by one unit, and an event within SL increments the control value by one unit, thus decrementing the control value by two units at the end of the window. However, the bin classifier may be configured so that events within SE decrement the control value by different amounts and / or events within SL increment the control value by different amounts.

[0044] The bin classifier may operate over multiple cycles (e.g., multiple light pulses). In a given cycle, if the current bin boundary (as represented by the reference signal) is below the median of the distribution, more return events within SL are expected than return events within SE, resulting in a larger increment in CV, thus moving the bin boundary toward the median. For example, assume the true median is 820 and the CV is currently at 612. In a given cycle, three return events may be classified as SE and five return events may be classified as SL, thus moving the CV to 614. Over a sufficiently long series of cycles, called a run, the bin boundary will approximate the true median of the transient distribution, and a control value CV may be read out for further image processing. Thus, the bin classifier disclosed herein does not store a history of photon counts over cycles to form a histogram. Rather, the bin classifier instantly and locally updates its CV at each laser cycle based solely on the photons received in that cycle. Each pixel only needs to maintain its CV, providing a significant reduction in data requirements.

[0045] The movement of the bin boundaries is probabilistic. Movement in the desired direction at any particular cycle is not guaranteed, but on average, there should be convergence toward the median value. Figures 3A and 3B illustrate the movement of the bin classifier boundaries over multiple cycles in a run. Figure 3A includes a first plot 300 showing the bin boundaries output by a bin classifier (such as bin classifier 200) at a relatively early cycle in the run (specifically, at cycle 160 of 5000 cycles). The transient distribution, shown by curve 302, is a Gaussian pulse with a constant offset, peaking at approximately 300. The x-axis of first plot 300 indicates window units ranging from 0 to 1023, which corresponds to the CV range discussed above (e.g., each unit of CV may equal 128 picoseconds). The bin boundary reached by the bin classifier at cycle 160, shown by line 304, is approximately 450.

[0046] As the 5000 cycles run, the bin boundaries may stabilize closer to the actual peak. Figure 3B is a second plot 350 showing the bin boundaries reached by the bin classifier at a relatively late cycle in the run (specifically, at cycle 4800 of the 5000 cycles). The bin boundary output at cycle 4800, indicated by line 352, is close to the peak (e.g., 330, with the peak at 300). The bin boundaries may stabilize slightly off the peak due to bias from background light.

[0047] FIG. 4 shows a plot 400 of the mean absolute deviation from the true peak location as a function of cycle number for a Monte Carlo simulation of a single-bin classifier (e.g., bin classifier 200) simulated 100 times over 5000 laser cycles. At each cycle, the bin classifier's estimate of the bin boundaries is incremented or decremented by a step size of + / - 1. Thus, plot 400 shows the convergence pattern over 5000 cycles for a strong signal pulse (Nsig=0.5 intensity, which controls the number of expected return events within a cycle resulting from an output laser pulse) with low ambient light (Nbkg=0.005, which is the background, expressed in terms of the expected number of events per window unit) over 100 runs selected from parameter space, each run using a different true peak location over a window of 1024 time units. As the CV approaches the median, it will "wander" around the median. Thus, the bin classifier may return slightly different values ​​for the bin boundaries depending on when the control value is read, which can limit its accuracy, although accuracy improves when multiple reads are combined.

[0048] Plot 400 in Figure 4 clearly shows that a bin classifier can, at least in some cases, take 1000 cycles to converge on the median / true peak location. Note that each cycle is on the order of nanoseconds to microseconds, so the total elapsed time is at most a few milliseconds, which may seem insignificant. However, from an energy consumption perspective, speeding up convergence can have significant advantages, as long as it does not involve extensive processing on the detector.

[0049] In one aspect, convergence speed may be increased by using a better starting estimate for the bin boundaries. When multiple runs are performed on a slowly changing scene, the median value from the previous run may be used as the initial control value / bin boundary. When readings for multiple points in a scene are obtained using one pixel via scanning (or using the same bin classifier successively on different nearby pixels), the estimated median value of neighboring pixels may be used as the initial value. In another aspect, convergence speed may be increased by using a large increment / decrement step size for the control value early in the run and decreasing the step size as the run progresses. However, with a coarser step size, the CV may not reach the exact median value. If the median value (or near the median value) is reached too quickly, the CV will oscillate further around the median value until the step size is smaller. Therefore, using a variable CV step size may speed up convergence.

[0050] In some embodiments, an adaptive method may be applied to speed up convergence. In an adaptive method, instead of a fixed adjustment to the CV per cycle, the CV may be adjusted based on the number of returning events. Intuitively, if the CV is far from the median, more events will fall on the median side of the current bin boundary rather than on the opposite side. Thus, at the beginning of the run, there will be more movement per cycle, but as the boundary approaches the median, that movement will decrease. Note that essentially the same effect is observed by adjusting the CV once per cycle in steps equal to the difference between early and late returning events for the cycle.

[0051] A bin classifier may require different numbers of cycles to converge under different conditions. Therefore, it may be possible to terminate a run early if the bin boundaries have converged. Attempting to determine convergence on a pixel-by-pixel basis may have only marginal benefit, as it may require significant additional circuitry and lasers to maintain the pulse until all pixels have converged. However, in some embodiments, overall convergence may be determined by periodically reading (a subset of) the pixel control values ​​and determining whether there is a sufficiently large change in any of the control values ​​(by comparing them to some predetermined threshold). However, reading from the detector is one of the more energy-intensive activities in bin classifier operation. For example, reading can be energy-intensive, especially for detectors containing millions of pixels, as moving data consumes energy, and therefore such an approach may not result in an overall net power savings.

[0052] In other aspects, convergence time may be predicted based on current conditions, primarily signal strength (e.g., of the light source) and background level. The background level is essentially ambient light, which may be measured using a co-located photodetector, or simply by measuring the total number of return events recorded by the detector's pixels over a fixed exposure time with the laser turned off. Signal strength may be determined by running a bin classifier for a selection of pixels over a short period (e.g., 100 cycles) and seeing how much shift in boundaries is observed over that time. In some aspects, the method accounts for variations in reflectance across the scene, which affects the number of return events. A rough estimate of reflectance may also be inferred from images acquired by a co-located RGB camera.

[0053] As previously described, in some embodiments, each return event on the SE or SL adjusts the CV by one unit. However, other options for adjusting the CV are contemplated without departing from the scope of this disclosure. For example, the CV may be adjusted once per cycle based on the relative number of return pulses on the SE and SL. In some aspects, determining the relative number of pulses on each output stream does not necessarily mean counting the number of pulses. Instead, the pulses can be used to charge two capacitors whose difference (e.g., as determined by an op amp) indicates the direction of the adjustment, or the SL and SE streams can be fed directly into an op amp configured as an integrator. Additionally, the step size for the adjustment need not be one unit. A larger step size may be advantageous, especially near the beginning of a run, and the step size may be decreased over time to more accurately seek the median value.

[0054] In the initial description provided above, the adjustments corresponding to the SE and SL are equal, thus moving the bin classifier boundaries toward the median value. However, by using different values ​​for the decrement and increment, different percentiles for the bin boundaries may be targeted. For example, a decrement-to-increment step size ratio of 3:1 tends the bin classifier boundaries toward the 25th percentile. That is, if the control values ​​support fractional amounts, a decrement of 1 and an increment of 1 / 3 may be applied. If the control values ​​do not support fractional adjustments, or if larger step sizes are desired to accelerate convergence, other combinations of decrement / increment step sizes may be used, such as 3 and 1, or 6 and 2, both of which will converge toward the 25th percentile. Yet another alternative is to make equal-size adjustments, selecting all decrements but only one-third of the increments, either deterministically or probabilistically.

[0055] The term "bin classifier" itself, as used above, may refer to a bin classifier that finds the median, and the term "proportional bin classifier" may be used to describe a bin classifier that targets a particular kth quantile or percentile of the return distribution. Another variation is to initialize the CV to something other than the midpoint of the window. That capability may be useful in EDHs built from bin classifiers, which are described in more detail below.

[0056] Another important aspect of the bin classifier design space is the representation of the control values ​​and how the bin classifier generates the reference signal from the control values. The main options are to represent the CV as a digital number or as an analog quantity such as charge or voltage (one example is shown in Figure 15). The digital case (one example is shown in Figure 16) requires circuitry to generate a reference signal whose length is proportional to the CV, which likely requires a significant number of circuit elements that consume power. The advantage of a digital representation is that the CV is easy to read and initialize. An analog representation of the CV may consume less power (but may be more difficult to initialize). However, analog CVs may require conversion to digital form at some point for further image processing. Even so, that conversion can be done (partially) continuously at the end of a run. For an array of pixels, the charge or voltage from various CVs of a row of pixels may be shifted to one edge where an A / D converter exists for each row. Another approach to CV readout is to actually route the bin classifier's reference signal to a time-to-digital converter (TDC). In either the variable step size or proportional binning cases, scaling of the increment and decrement values ​​may be easier with an analog representation of the CV, since the step size or the proportionality itself can be an analog value.

[0057] Figures 5A-5D illustrate the effect of different factors on convergence. Specifically, Figures 5A-5D show the results of three different stepping schemes: a naive method that takes small, constant steps of size 1 (shown as solid curves in the plots of Figures 5A-5D and labeled "Step Size 1"); a weighted-step method that takes steps equal to the difference in the number of return events in a cycle (shown as dashed curves in the plots of Figures 5A-5D and labeled "lr dif step"); and finally, an ad hoc variable step-size schedule that takes large steps first and then reduces the step size for subsequent laser pulses (shown as dotted curves in the plots of Figures 5A-5D and labeled "Step 8, 4, 2, 1"). In the example shown in Figures 5A-5D, the variable step-size schedule provides the best convergence speed. Plot 502 in Figure 5A shows that in a high-signal, low-background imaging regime, the variable step-size method converges quickly to a median value. Plot 504 in Figure 5B shows that in the high-signal, high-background regime, rapid convergence is observed, but the final estimate is still very noisy. Plot 506 in Figure 5C shows that in the low-signal, low-background regime, convergence takes longer than in the high-intensity regime, but the variable step-size schedule achieves a 10-fold improvement over other schemes. Plot 508 in Figure 5D shows that in the low-signal, high-background regime, the final estimate exhibits significant deviation from the true median location. These plots also suggest that averaging multiple measurements after convergence can be beneficial, especially in high-background situations.

[0058] The plots in Figures 5A-5D were generated by performing a Monte Carlo simulation study of a single-bin classifier with three different stepping schemes: a constant small step size of 1, a photon-number-weighted step size, and a four-stage coarse-to-fine step size schedule (8 → 4 → 2 → 1) where each cycle is one-quarter of the total cycle, over different operating conditions (low and high signal power in the presence of low and high background light levels).

[0059] The coarse-to-fine stepping scheme provides the fastest convergence under all operating conditions. In most cases, it is at least 10 times faster than the constant step size method. In the high-signal-intensity regime, the scheme settles to an optimum very quickly and then becomes step-size limited. This effect can be seen in plots 502 and 506 in Figures 5A and 5C, respectively. The dotted plots initially converge very rapidly, but then further improve with finer step sizes, as seen from discrete jumps of 1250 and 2500 cycles. This result suggests that signal-dependent step size optimization can further accelerate convergence. In the high-intensity regime, it helps to rapidly decay the step size to the optimal level, while in the low-intensity regime, larger steps should be maintained for a larger fraction of the total exposure time budget. In practice, the heuristic step size schedule informed by this simulation study may be used.

[0060] In some embodiments, an initial "calibration" scan may be performed to evaluate signal-to-background ratio (SBR) conditions at different scene points (SBR is the ratio of the expected number of signal photons to background photons, aggregated over the entire window range). High-SBR pixels may decay quickly to small step sizes, while low-SBR pixels use large step sizes for longer durations. The difference between the constant small step size and weighting schemes is less pronounced, with the weighting scheme (dashed line) converging faster than the constant-size scheme (solid line). Figures 5A-5D show only a slight advantage for the low-intensity regime, where the weighting scheme has a convergence rate similar to the constant-step scheme. This is because in the low-signal and background regimes, very few photon events are generated each cycle, and therefore the boundary rarely moves more than one unit per cycle. In the high-intensity regime, the weighting scheme performs significantly better than the constant-step scheme. However, at strong signals, convergence is faster for all schemes.

[0061] While low power and fast convergence are important, it is also desirable for the bin classifier to converge to the correct location (the true median). Figures 6A and 6B show the bin boundaries of the bin classifier at low background levels (Nbkg = 0.0001, Figure 6A) and high background levels (Nbkg = 0.005, Figure 6B) after 1000 cycles of the same composite peak (Nsig = 1.0, pulse width 5 ns). In the high-background case shown in Figure 6B, the final boundaries are farther away from the peak. The reason for the shift may be that the bin classifier accurately reflects the median of the complete transient distribution consisting of the composite peak plus background. The median of the background, if truly uniform, would be the midpoint of the window. Thus, the midpoint of the combined distribution is "pulled" toward the center of the range by background events. The stronger the background, the greater the bias toward the midpoint.

[0062] In some embodiments, background bias may be compensated for by measuring the background level without an output laser pulse. If there are enough bins in an iso-depth histogram, such as those generated by the iso-depth histogram generator described in more detail below, multiple bins may "absorb" the background light, leaving the structure around the peak unshifted. While the bins that actually capture the peak may be different in low-background and high-background cases, locally around the peak, the relative bin widths will be very similar.

[0063] A single-bin classifier finds a single boundary, and therefore produces only a two-bin ED histogram. While a single-bin classifier may be effective in identifying a single peak, as discussed above, it suffers from bias from background light levels. To obtain an EDH with three or more bins, multiple bin classifiers may be combined.

[0064] One approach to obtaining an EDH with B bins is to use an array of B proportional bin classifiers, each targeting a different percentile. For example, for 10 bins, nine proportional bin classifiers set to the 10th, 20th, 90th percentiles may be used. In this approach, all of the bin classifiers in the array of bin classifiers may receive photon return events directly from the detector / pixel. Alternatively, if adjustable proportional bin classifiers are used, nine runs may be performed with the adjustable proportional bin classifiers sequentially set to different percentile values. In this way, a single proportional bin classifier may be operated for multiple runs with differently adjusted control values ​​in different runs (e.g., targeting different percentiles) to generate different percentiles corresponding to bin boundaries in the histogram. The latter approach may consume more energy from the laser but may use simpler pixel circuitry. Because the level of gradation that can be obtained using proportional bin classifiers may be limited, in other embodiments, median bin classifiers may be combined in a recursive style to provide a power-of-2 number of bins for the ED histogram. In this arrangement, each bin classifier BN at stage i feeds its output streams SE and SL to two bin classifiers, BN1 and BN2, at stage i+1, respectively.

[0065] For illustrative purposes, FIG. 7 shows a three-stage EDH 700 that generates eight bins. In FIG. 7, the cycle start lines to the bin classifiers have been omitted to avoid cluttering the figure. Each bin classifier may be configured similarly to the bin classifier 200 of FIG. 2A. The EDH 700 includes a first stage with a first bin classifier BN1 that receives a stream of return events (SR) and may separate the stream into an early stream (SE1) and a late stream (SL1) according to a control value, as previously described. The EDH 700 includes a second stage with two bin classifiers (BN1.1 and BN1.2), where the early stream SE1 is fed to BN1.1 and the late stream SL1 is fed to BN1.2. BN1.1 may separate SE1 into two streams based on the corresponding CVs, the early stream SE1.1 and the late stream SL1.1. Similarly, BN1.2 may split SL1 into two streams (SE1.2 and SL1.2) based on the corresponding CV. The EDH 700 further includes a third stage with four bin classifiers (BN1.1.1, BN1.1.2, BN1.2.1, and BN1.2.2). Each stream may be fed to a respective subsequent bin classifier (e.g., SE1.1 may be fed to BN1.1.1, SL1.1 may be fed to BN1.1.2, and SE1.2 and SL1.2 may be fed to BN1.2.1 and BN1.2.2, respectively). Seven bin boundaries are read from each CV to generate D1, D2, ..., D7, corresponding to an ordered traversal of the bin classifier tree. Figure 7 shows a three-stage EDH generating eight bins. Adding a fourth stage containing eight additional bin classifiers (e.g., such that SE1.1.1, SL1.1.1, SE1.1.2, SL1.1.2, SE1.2.1, SL1.2.1, SE1.2.2, and SL1.2.2 are each fed into a respective additional bin classifier) ​​results in 16 bin identifications.

[0066] Figures 8A-8G show the movement of bin boundaries using a four-stage (16-bin) EDH. Each stage is given 500 cycles to converge. The boundaries are then "frozen" at each stage, and the bin classifier is activated at the next stage. The line patterns indicate the boundaries at different stages: solid line = 1st, dense dotted line = 2nd, dashed line = 3rd, dense dotted line = 4th. The transient distribution in this case is a Gaussian signal pulse with a low background level.

[0067] Specifically, FIG. 8A shows the bin boundaries (solid lines 802) determined by the first stage (e.g., the first bin classifier) ​​of the EDH at cycle 483 of the run. The bin boundaries are near the true median (the bin boundaries are 642, and the true median is 643.86). At cycle 500, as shown in FIG. 8B, the first stage is frozen (so that the bin boundaries determined by the first stage remain unchanged after they are identified) and the second stage is activated (solid lines 804 indicate the bin boundaries of the first stage, while dark dashed lines, such as line 806, indicate the initial locations of the bin boundaries of the second stage). FIG. 8C shows the bin boundaries of the second stage at cycle 985, near cycle 1000 (when the second stage is frozen); therefore, the bin boundaries shown in FIG. 8C are approximately the identified bin boundaries of the first and second stages.

[0068] At cycle 1000, as shown in Figure 8D, the second stage is frozen (so that the bin boundaries determined by the second stage remain unchanged after they are identified) and the third stage is activated (solid lines 804 indicate the bin boundaries of the first stage, dark dotted lines such as line 808 indicate the final locations of the bin boundaries of the second stage, and dashed lines such as line 810 indicate the initial locations of the bin boundaries of the third stage). Figure 8E shows the bin boundaries of the third stage at cycle 1496, which is near cycle 1500 (when the third stage is frozen); therefore, the bin boundaries shown in Figure 8E are approximately the identified bin boundaries of the first, second, and third stages.

[0069] In cycle 1500, as shown in FIG. 8F, the third stage is frozen (so that the bin boundaries determined by the third stage remain unchanged after they are identified) and the fourth stage is activated. (The solid lines 804 indicate the first bin boundaries; the dark dotted lines, such as line 808, indicate the final locations of the second stage bin boundaries; the dashed lines, such as line 812, indicate the final locations of the third stage bin boundaries; and the sparse dotted lines, such as line 814, indicate the initial locations of the fourth stage bin boundaries.) FIG. 8G shows the bin boundaries in cycle 1983, near the end of the run; therefore, the bin boundaries shown in FIG. 8G are approximately the identified bin boundaries of the first, second, third, and fourth stages. Thus, the signal strength is high enough that EDH uses only 4 of the 16 bins to absorb background events, while the remaining 12 EDH bins stabilize around the true peak location.

[0070] In the operation of a multi-stage EDH, the convergence of later stages is influenced by earlier stages. First, later stages receive fewer return events than earlier stages. Consider bin classifiers BN1 and BN1.1 in Figure 7. By the time BN1 converges to the median of input stream SR, bin classifier BN1.1 will receive only half of SE1's return events. Therefore, bin classifier BN1.1 will tend to converge more slowly, but the initialization of the bin classifier (see below) and the smaller window ranges on the bin classifiers of later stages compensate for the reduced number of events. Also, if the boundaries of BN1 are not frozen before starting the next stage (or if all bin classifiers are launched at once), the subrange of the transient distribution that BN1.1 handles will shift, and therefore the median for BN1.1 is a moving target.

[0071] In general, the more stages in a multi-stage EDH, the more cycles the EDH will take before convergence. In some embodiments, this effect may be improved by initializing the boundaries of the different bin classifiers. For example, random events may be fed into the SR before beginning a run, which will tend to move the bin boundaries D1, D2, ..., D7 to equally spaced positions in the window range. (However, if the transient distribution contains strong peaks near the beginning or end of the window, some boundaries may still need to be moved across a large portion of the range of the transient distribution.) In additional embodiments, stages may be frozen sequentially, starting with the bin classifier BN1 and working down the tree. Thus, the distribution subranges that the bin classifiers address are fixed after a certain point during the run.

[0072] In other embodiments, the bin classifiers in stage i+1 are activated only after the bin classifiers in stage i are frozen. To maximize the benefit from this approach, the bin boundaries from stage i may be used to initialize them in stage i+1. For example, if bin classifier BN1 has a boundary value v when frozen, BN1.1 and BN1.2 may be initialized to v / 2 and (w+v) / 2, respectively, where w is the window size. The bin classifiers in later stages may be initialized to the midpoints of their ranges when activated. (FIGS. 8A-8G depict this strategy.) The bin classifiers may be initialized by adding circuitry to set control values; in other embodiments, uniformly distributed events may be fed to the RS, which will move the boundaries of unfrozen bin classifiers to the midpoints of their ranges. (Even without initialization, sequential activation of bin classifiers in the EDH can save power even in the absence of convergence time.)

[0073] Additionally or alternatively, the bin boundaries from one run of the EDH may serve as starting values ​​for subsequent runs. This situation arises when multiple runs are tracking a scene over time. ED histograms from consecutive scenes are likely to have similar bin boundaries, and therefore the time it takes for the EDH to adjust from one scene to the next should be short.

[0074] Another case where boundaries from runs are likely to be similar is adjacent pixels in the array. In some embodiments, the boundary for pixel P may serve as the starting point for adjacent pixel Q. Exploiting this correlation implies sequential runs for P and Q. However, the run for Q can be shorter. P and Q may then use the same EDH circuit, which avoids the problem of transferring boundary values ​​from one EDH to another. More generally, pixels may be grouped into small regions (e.g., 2x2) that use the same EDH, with a "full" run for one pixel in the region followed by shorter runs for the remaining pixels in the region, each utilizing the bin boundaries of the previous run. Using a single EDH for a group of pixels accelerates convergence by increasing the number of events; a photon stream fed into an EDH processing a 2x2 pixel block will have four times the number of events (on average) than any individual pixel in that block. One example of a single EDH for a group of pixels is shown in FIG. 14.

[0075] Another nuance of the operation of a multi-bin classifier EDH is the comparability of control values ​​for different bin classifiers. Manufacturing differences may mean that the same CV for two bin classifiers results in slightly different reference signal delays. To address this issue, various strategies may be applied. In some embodiments, a first strategy may involve an initial calibration in which a random stream of events (either generated by an ambient light source such as an incandescent light illuminating the detector or a periodic stream of events generated artificially using a clock source) is fed to the EDH via the SR, allowing the bin boundaries to converge to equally spaced positions. These values ​​may be stored for use during post-processing to adjust later readouts. Another strategy is direct readout. While the CV values ​​for the bin classifiers in the EDH may not be directly comparable, the reference signal delays are comparable. Thus, in other embodiments, these signals may be read out and sent to a "time-to-digital" converter (either sequentially or in parallel) to generate comparable histogram boundaries. Yet another strategy is joint readout, which is a variant of direct readout that takes all or exclusive reference signals of all bin classifiers in the EDH. The resulting joint signal has a transition at each bin boundary. A transition detector in parallel with the high-speed counter may then process the joint signal and extract all bin boundaries in one pass.

[0076] Background bias is less of a problem in multi-stage EDHs than in isolated bin classifiers. As mentioned previously, high background levels in the transient distribution can bias the boundaries of the bin classifier toward the center of its range (where the boundaries converge at uniform background levels). This bias can be problematic when using single-bin classifiers in applications such as "peak trackers." However, background bias is less of a problem for EDHs. The reason is that "side bins" away from the peak "absorb" many of the background events in the distribution, and clusters of bins around the peak(s) of the transient distribution still align well with those peaks. As an example of this effect, consider Figures 9A-9D, which show the output of a four-stage EDH (with 16 bins) along with an actual transient distribution. Figure 9A shows the low-background case with only a few side bins, where the bins are clustered around the distribution peak at 615. Figure 9C shows the high-background case. In this case, there are more side bins, but there are still enough bins to resolve the range around the peak. The predicted peak locations for the two histograms (using a curve fitting method described in more detail below) are 611.2 for the low background case and 628.4 for the high background case, respectively, both close to the true peaks.

[0077] Thus, Figures 9A-9D show the effect of background light level on EDH bin width. The 16-bin EDH correctly converges toward the true peak location for both low (Figure 9A) and high (Figure 9C) background light levels in this simulated example. In the low-background case, approximately 10 of the 16 EDH bins are clustered around the peak. In the high-background case, EDH requires additional bins to accommodate background events, and approximately 7 of the 16 EDH bins are clustered around the peak. Enlarged views of the plots in Figures 9A and 9C are shown in Figures 9B and 9D, respectively, highlighting the location of the narrowest bins. Figure 9B shows a tighter distribution of bin boundaries around the main peak than Figure 9D. The narrowest bins (vertical speckled bands) are in roughly the same location; the only difference is that in the low-background case (Figure 9B) the narrowest bins are narrower than in the high-background case (Figure 9D).

[0078] 10 and 11 show example distance maps (e.g., depth maps) that may be generated for an imaged scene using the EDH described herein. FIG. 10 shows single-photon 3D imaging using an 8-bin ED histogram (EDH). FIG. 10 includes two images: a first image 1002 that is a grayscale version of an RGB image of a rendered "kitchen" scene, and a second image 1004 that is a grayscale version of an RGB image of a rendered "dining" scene. FIG. 10 includes multiple distance maps for each imaged scene. Distance maps 1006 and 1008 are ground truth distance maps (e.g., distance map 1006 is a ground truth distance map for the kitchen scene imaged in first image 1002, and distance map 1008 is a ground truth distance map for the dining scene imaged in second image 1004). The rendered kitchen and dining scenes may be generated using 3D models that include information about material textures, reflectance, and transparency, along with existing lighting sources. Ground truth distance maps generated from these 3D models may represent the "true" distances of objects in the rendered scenes. In some aspects, the ground truth distance maps may have a resolution of 0.25 cm (2000 units over a maximum range of 5 m).

[0079] Distance maps 1010 and 1012 show distances for the kitchen and dining scenes, respectively, reconstructed using a simple coarse histogram method with eight EW bins. Distance maps 1014 and 1016 are distance maps for the kitchen and dining scenes, respectively, obtained with eight-bin EDH, as described herein, demonstrating that reliable distance reconstruction can be performed with as few as eight ED bins. These distance maps (distance maps 1014 and 1016) were simulated in low background light, using the center point of the narrowest bin to estimate peak locations, and no averaging or post-processing was performed. Post-processing further improves the quality of the distance maps. (Distance maps 1006, 1008, 1010, 1012, 1014, and 1016 have the same scale of 0 to 5 meters.)

[0080] As can be seen from FIG. 10 , the correspondence between the distance maps generated using EDH (distance maps 1014 and 1016) and the ground truth is very good. In some aspects, any granularity may be reduced through temporal or spatial averaging. In contrast, the distance maps generated using conventional 8-bin EWH per pixel (distance maps 1010 and 1012) capture less detail in the scene and introduce some edge artifacts (such as windows in distance map 1012). In addition, EWH distance map 1012 has visible discrete steps due to its low histogram resolution.

[0081] Figure 11 shows the same rendered image of the kitchen scene (first image 1002) and the same ground truth distance map (distance map 1006) as shown in Figure 10. However, in Figure 11, the EWH distance map (distance map 1102 in Figure 11) and the EDH distance map (distance map 1104 in Figure 11) were generated using 16 bins instead of the 8 bins of Figure 10.

[0082] 10 and 11 show that both the 8-bin EDH and the 16-bin EDH generate distance maps that are more accurate than distance maps generated using an EWH with the same number of bins. The determination of the number of bins to generate using an EDH may be based on various factors, including the scene being imaged. For example, as the number of stages (and therefore bins) increases, the width of the bins around peaks becomes narrower, which can make their boundaries difficult to resolve. On the other hand, with too few bins, there may not be enough bins to capture all peaks, especially in the presence of high background levels. In at least some embodiments, 16 bins (e.g., as generated by a 4-stage EDH) may be applied to distance mapping of indoor scenes, and 8 bins (e.g., as generated by a 3-stage EDH) may be applied to distance mapping of close-up scenes.

[0083] Additionally, there may be situations where higher resolution is desired, such as close peaks (as can occur on transparent surfaces such as window glass) or highly reflective surfaces that produce multiple peaks from different return paths. Such cases may be addressed by adding further stages to the EDH, each doubling the number of bin sorters.

[0084] 12 is a flowchart illustrating an example method 1200 for identifying bin boundaries of a transient distribution using a bin classifier. Method 1200 will be described with reference to the systems and components of FIGS. 1 and 2A and / or 7, although it will be understood that method 1200 may be implemented using other systems and components without departing from the scope of this disclosure.

[0085] At 1202, method 1200 includes transmitting pulsed light at a commanded frequency. The pulsed light may be transmitted from a suitable light source, such as a laser (e.g., light source 102 of FIG. 1). The transmitted light may reflect off objects in the scene being imaged. At 1204, a stream of return events (SR) is received at a bin classifier (e.g., bin classifier 200 of FIG. 2). As discussed above, the bin classifier is coupled to at least one pixel of a single-photon detector. The detector pixel generates a voltage pulse each time a photon strikes the pixel, and the voltage pulse is received at the bin classifier as an SR.

[0086] The light source may be pulsed at a commanded frequency for multiple cycles (e.g., each cycle corresponds to one pulse). The bin classifier may receive the SR for multiple cycles. For the first cycle of the multiple cycles, the bin classifier may generate a reference signal (RS) via a reference signal generator, as shown at 1206, having a waveform with a high value extending for an initial duration. The initial duration may be a default duration, such as the midpoint of a window, or another suitable duration, for example, using a CV from a previous run, although additional durations are contemplated. At 1208, the SR is separated into two output streams, SE and SL, by two AND gates in the bin classifier based on the RS. SE is a stream of early events, and SL is a stream of late events. Early events are defined as those occurring during the high value of the RS waveform, and late events are defined as those occurring during the low value of the RS waveform (where the high value is defined as being higher than the low value). At 1210, a control value (CV) is adjusted based on the number of events classified as early events (in the first cycle) (e.g., the number of events in SE) relative to the number of events classified as late events (e.g., the number of events in SL). As a non-limiting example, the CV is initialized at an initial value, then decremented by one unit for each early event and incremented by one unit for each late event. The CV may be an integer within the range of time units of the cycle. For example, a cycle may correspond to a window having multiple time units, where the time unit may be 128 picoseconds, and the CV may be an integer in the range of 1 to 1024 (so that the cycle has a total duration equal to 1024 time units). At 1212, the RS is adjusted based on the CV. For example, if the CV is decreased from its initial value by a given number of time units, the duration of the high value of the RS waveform is decreased by the same number of time units.

[0087] The same process is then repeated for each additional cycle of the multiple cycles. At the beginning of the additional cycle (corresponding to the beginning of the next light pulse), a new RS with an adjusted duration is generated, as shown at 1214 (e.g., if the duration is adjusted based on the CV of the previous cycle). The SR is divided into SE and SL based on the RS, as shown at 1216, and the CV is again adjusted based on the number of events in the SE relative to the SL, as shown at 1218. The RS duration for the next cycle is adjusted based on the adjusted CV, as shown at 1220. In this manner, the CV and therefore the RS are adjusted with each cycle until the CV (and therefore the RS) converge to or near a target quantile (e.g., the median for a median-bin classifier or another quantile for a proportional-bin classifier) ​​of the distribution of time delays (from the beginning of each light pulse) of photons impinging on the detector pixel.

[0088] At 1222, the value of CV is read when requested. CV may be read at any point during the run, for example, at the end of the run (e.g., after the last cycle), or one or more times during the run. If CV is read multiple times during the run, the different reads may be averaged to determine the final CV. The lower the signal strength and the higher the background light level, the greater and longer the deviation will be from the median value once reached. Therefore, when taking multiple reads for averaging, the number and / or spacing of reads may be adjusted based on the signal level and / or background level. CV may be used to determine the distance of an object being imaged by the pixel. Because CV represents a number of time units, CV may be multiplied by the time unit (e.g., 128 picoseconds), and this value may be used with the speed of light to determine the distance it took for the photon to reach the pixel, i.e., the object distance value. Method 1200 then ends.

[0089] It should be understood that method 1200 may be performed for each pixel of the detector, such that a distance value is determined for each pixel or group of pixels using a respective bin classifier associated with each pixel or group of pixels. These distance values ​​may be used to construct a distance map that can be used for various tasks. A single-bin classifier, such as that described above with respect to FIG. 12, may be affected by background light and therefore may not identify the peak of the distribution as accurately as an iso-depth histogram generator. Therefore, when using a single-stage bin classifier to identify distance values ​​and generate a distance map, the final CV output by the bin classifier may be adjusted based on background light, which may be measured by the detector when no light pulses are occurring from the light source, as discussed above. However, to improve the accuracy of the distance determination, an iso-depth histogram generator may be used, as described below. In some embodiments, method 1200 may be applied to the bin classifier of the iso-depth histogram generator, and the SE and SL separated by the bin classifier as described above may be provided to the respective bin classifiers of the second stage of the iso-depth bin classifier.

[0090] 13 is a flowchart illustrating an example method 1300 for identifying multiple bin boundaries of a transient distribution using a histogram generator having a multiple bin classifier (e.g., EDH). Method 1300 will be described with reference to the systems and components of FIGS. 1 and 2A and / or 7, although it will be understood that method 1300 may be implemented using other systems and components without departing from the scope of this disclosure.

[0091] At 1302, method 1300 includes transmitting pulsed light at a commanded frequency. The pulsed light may be transmitted from a suitable light source, such as a laser (e.g., light source 102 of FIG. 1). The transmitted light may reflect off objects in the scene being imaged. At 1304, a stream of return events (SR) is received by a first-stage bin classifier of a multi-stage histogram generator (e.g., bin classifier BN1 of FIG. 7). As discussed above, the multi-stage histogram generator is coupled to at least one pixel of a single-photon-sensitive detector. The detector pixel generates a voltage pulse each time a photon strikes the pixel, and the voltage pulse is received by the first-stage bin classifier as the SR. The light source may be pulsed at the commanded frequency for multiple cycles (e.g., each cycle corresponds to one pulse). The first-stage bin classifier may receive the SR for multiple cycles.

[0092] At 1306, method 1300 includes setting bin boundaries for the first stage by adjusting the CV and RS over a first set of cycles. The first set of cycles may be a suitable number of cycles, such as the first 500 cycles. To set the first bin boundaries, the CV may be adjusted as described above with respect to FIG. 12; for example, the return events may be separated into two streams, SE and SL, and the CV may be adjusted at each cycle based on the number of events in the SE and SL. The CV may modulate the waveform duration of the RS at each cycle.

[0093] At 1308, the first stage is frozen after the first set of cycles. When the first stage is frozen, the bin boundary identified by the bin classifier of the first stage (e.g., the CV at the end of the first set of cycles) is set as the first bin boundary, and the first bin boundary is no longer adjusted. The SR is still received by the bin classifier in the first stage and separated into SE and SL, which are then provided to respective bin classifiers of a second stage of the multi-stage histogram generator, as shown at 1310. Each bin classifier of the second stage may be, for example, bin classifiers BN1.1 and BN1.2 of FIG. 7.

[0094] At 1312, the bin boundaries of the second stage are established by adjusting the CV and RS for each bin classifier of the second stage over a second set of cycles. The second set of cycles may include the same number of cycles as the first set of cycles (e.g., 500) or may occur immediately after the first set of cycles (e.g., the second set may include cycles 501 through 1000). The CV and RS may be adjusted as described above (e.g., each bin classifier separates the input stream into SE and SL and adjusts the CV, and therefore RS, based on the number of events in the SE and SL). At 1314, the bin boundaries identified by the second stage bin classifiers are frozen after the second set of cycles is completed (e.g., at cycle 1000). At 1316, the process is repeated for each additional stage (if any) of the multi-stage histogram generator. For example, if the multi-stage histogram generator includes three stages, the SE and SL streams from each bin classifier in the second stage may be fed to a respective bin classifier in the third stage (which may include four bin classifiers), thereby identifying four additional bin boundaries (seven bin boundaries total). If the multi-stage histogram generator includes four stages, the SE and SL streams from each bin classifier in the third stage may be fed to a respective bin classifier in the fourth stage (which may include eight bin classifiers), thereby identifying eight additional bin boundaries (fifteen bin boundaries total). At 1318, the CVs (one from each bin classifier) ​​may be read out when required, such as at the end of the run (e.g., after the final stage is frozen / completed). Each CV may represent a bin boundary, which may be used to identify one or more peaks in the distribution, as shown at 1320. For example, the distribution may include one peak indicating the distance of the imaged object / point, and the peak may be identified from the bin boundary. The output of the EDH may therefore be used to estimate the position(s) / location(s) of the peak(s) in the transient distribution of the pixel. In a single-bin classifier, the median boundary itself provides that estimate.However, in multi-bin EDH, a "locally narrow" bin generally marks the peak, and the locally narrow bin is narrower than its neighboring bin(s). Because the narrowest bin still has width, further calculations may be performed to generate an estimated peak location. Two methods may be applied to generate such an estimate: bin midpoint and curve fitting. The bin midpoint method simply uses the midpoint of the locally narrow bin as the peak estimate. For example, if EDH is 2. K -1 bin boundaries D[1, 2, 3, . . . , 2 K -1]. The first and last bin edges are, by definition, located at the extreme ends of the time window, i.e., D[0]=0, D[2 K ]=T.

number

number

[0095] where c is the speed of light. This method of distance estimation may be extended to handle multiple peaks (e.g., due to multiple reflections or the presence of semi-transparent materials along the viewing direction) by replacing the argmax operation with a more general peak-finding routine that may return the locations of all "locally narrow" bins (those narrower than neighboring bins).

[0096] Although simple, the bin midpoint method has the limitation that actual peaks tend not to be symmetrical, which can pull the narrow bin to one side of the peak. Curve fitting methods fit a curve through bins neighboring the narrow bin (including the narrow bin), and then use the location of the maximum of that curve as the peak estimate. To fit the curve, the bins may be converted to (x,y) points by taking the bin midpoint as the x value and the bin "density" as the y value; this bin density may be calculated as p / w, where p is the bin population and w is the bin width. In some embodiments, all bins have the same population, and only the maximum location is searched, so bin B i is (m i , 1 / w i ), where m i is Bin B i Polynomial curve fitting may be used for the fitting, although other functions that may be able to handle the asymmetry of the peaks are contemplated for the fitting.

[0097] Thus, the peak location may be estimated by fitting a curve to points corresponding to multiple bins. Fitting a curve may include fitting a quadratic curve to points corresponding to three bins, or fitting a Gaussian pulse shape to points corresponding to three or more bins. In some embodiments, the peak location may be estimated using template matching techniques such as cross-correlation, normalized cross-correlation, or a matched filter with a known laser pulse shape used as a template. Quadratic curve fitting methods may provide a finer estimate of the peak location. i = (D[i]-D[i-1]) / 2, and y i = 1 / D[i]-D[i-1], the quadratic function y=αx 2 +βx+γ is the ED histogram bin index (x i ,y i) pairs. The number of bins on either side of the narrowest bin is adaptively chosen to be within one standard deviation of all ED bin widths. In some embodiments, this means that the bins {i * -2,i * -1,i * ,i * +1,i * + 2} is selected for curve fitting.

number

[0098] In some examples, when using a single EDH readout to estimate distance, the quadratic curve fitting method can result in a relatively wide spread in the mean absolute error values, which may be due to variations in the bin classifier output (e.g., if the bin classifier output does not line up exactly around the peak location, the curve fitting may further "amplify" the error). This effect can be avoided by averaging multiple measurements.

[0099] The energy associated with peaks in the transient distribution (and associated with the overall distribution) may indicate parameters in addition to object distance, such as surface composition and pose. Current methods employing single-photon cameras to measure reflectance typically involve counting. However, the EDH approach described herein specifically avoids counting return events; therefore, only relative peak intensities in the histogram are known, and peak intensities between pixels are not comparable. While counting circuitry may be included to perform an overall count of return events per pixel, for example, counting circuitry may add extra complexity and power usage. Therefore, simple extensions to the EDH that can provide a comparable "measure" per pixel may be applied. For example, a bin classifier circuit may be utilized for intensity estimation by using the bin classifier's register memory as a passive photon counter. As another example, an artificial event may be injected into the EDH at "pseudotime" 0 each cycle, thus creating a peak of known intensity. Then, since bin 1 will have equivalent "energy" across pixels, the width of bin 1 may be used to calibrate the bin width of the actual peak in the transient distribution. Such an approach may consume bins and therefore provide lower resolution relative to the actual transient distribution. Furthermore, if there is a true peak near 0, some of the peak's energy may "bleed" into bin 1. However, despite these potential drawbacks, the advantage of this technique lies in its simplicity, as there is already a signal marking the start of every cycle going to each bin classifier (possibly with a small delay) that can be used as a simulated return event.

[0100] The number of bins the EDH is configured to identify may be calibratable (e.g., 4 bins, 8 bins, 16 bins, etc.). As the number of bins in the EDH increases, the bin widths around peaks become narrower, making boundary resolution more difficult. On the other hand, with too few bins, there may not be enough bins to capture all peaks, especially in the presence of high background levels. In some embodiments, 16 bins (as produced by a four-stage EDH) may be used for distance mapping of indoor scenes, and 8 bins (as produced by a three-stage EDH) may be used for distance mapping of close-up scenes. However, there may be situations where higher resolution is required, such as close peaks (as can occur with transparent surfaces such as window glass) or highly reflective surfaces that generate multiple peaks from different return paths. In some embodiments, to address the complexity issues caused by adding extra bins while still allowing for capture of all peaks, a “zoom” process may be applied to zoom in on a subrange of the return distribution, which is described in detail below.

[0101] Some of the examples provided herein utilize window sizes of 128 picoseconds, each 1024 units. However, the window size is somewhat arbitrary (although there are physical limits on the smallest time unit in delay that can be achieved). The maximum distance to be captured may determine the length of the window (and the power of the laser used). The default in the previous examples is generally a 128 ps window, each 1024 units, which corresponds to a maximum distance of approximately 20 meters. If the window size were instead set to 1500 units within the window, the maximum distance would be approximately 30 meters.

[0102] Using a 1000-unit window as an example, with a strong reflective surface at 24 meters (which would be the peak at 1200 units in a 1500-unit window), assuming one cycle begins immediately after the previous cycle, the pulse reflected from that surface will appear at approximately the 200-unit mark in the subsequent window. To the EDH, that peak will appear identical to a peak of similar intensity at a distance of 200 units. This example illustrates the problem of aliasing: a peak outside the window at N units will appear as an "in-window" peak at Nk·(window length) units for some integer k. In this example, k is 1, and the 1200-unit peak will appear as a peak at 1200-1·1000=200 units within the window. This simple formula provides an approach to "de-aliasing." That is, as the window length is increased, the true peak will appear in the same place, but the "alias" peak will be shifted forward by a number of units that is a multiple of the difference in the window lengths. If the aliased peak's previous position is Nk·(window length), its new position will be Nk·(new window length), thus shifted forward by k·((new window length)-(old window length)). Thus, the aliased peak will shift when the window duration changes slightly, while the true peak will stay in the same place.

[0103] For a given number of bins, there may be return distributions or portions thereof that EDH cannot adequately resolve. For example, ED histograms of some distributions may present limited bins that converge on each peak, which makes peak estimation and detecting asymmetry of the weakest peak problematic. In such cases, one may zoom in on a subrange of the distribution by filtering return events outside that subrange. For example, EDH of a transient distribution from 0 to 2000 may have actual peak locations of two strong peaks at 985 and 1219.3. The original ED histogram may yield estimates of 989.5 and 1214.5 via curve fitting. On the other hand, a zoomed histogram generated using filtering (e.g., a time-gated signal implemented using race logic) may accept only returning photons over a zoomed-in subrange covering the first two peaks of interest (in this case, the range 750 to 1400) and yield estimates of 988.2 and 1216, providing improvements in both cases. Thus, when zooming, a first unfiltered run may be performed to identify a subrange of the distribution. One or more subsequent runs may be performed with filtered input as described herein (e.g., to filter events outside the subrange). In some examples, the first run with unfiltered input and the subsequent run(s) may use different numbers of bin classifier stages in the multi-stage histogram generator; for example, the first run may use all bin classifier stages and the subsequent run(s) may use less than all bin classifier stages.

[0104] When zooming, the subrange zoomed in may be adjustable based on the desired results or imaging goals. If it is desired to more accurately determine a peak or separate two nearby peaks, the subrange may be set based on a fixed number of bins or a specific interval in either direction in addition to the locally narrow bin (or bins). For example, assume it is desired to determine the first two strong peaks in the multi-peak transient distribution of the example presented above. Two locally narrow bins may be identified at 980 and 1220. A subrange covering these two peaks may be zoomed in. In this case, for example, approximately 1.5 bins on either side of the two locally narrow bins may be zoomed in, and subsequent runs of 16-bin EDH may be used to provide a refined shape estimate. If it is desired to resolve a faint peak, a "wider" locally narrow bin range, perhaps with a small additional interval on either side of the bin, may be used.

[0105] Performing a zoom may involve determining which subrange to zoom into for each pixel "off detector" due to the amount of computation involved. For example, simply determining what is a locally narrow bin may involve arithmetic and comparison operations on each bin. Thus, a value indicating the subrange for the zoom may be sent back to the detector, which may require additional circuitry and inbound bandwidth. Once the subrange is determined at a pixel, the stream of returning events SR may be filtered down to the specified subrange. In some aspects, when the EDH is built from an adjustable proportional bin classifier, the proportional bin classifier may target a range of interest. For example, assume that the component bin classifiers initially target the 10th, 20th, ..., 90th percentiles, and it is desired to resolve the distribution within the 30th to 50th percentile subrange. The bin classifier may then retarget the 32nd, 34th, ..., 48th percentiles to divide that subrange into 10 bins. (Note that the locations of the 30th and 50th percentile boundaries may already be known.) In another embodiment, a signal may be generated for each cycle that goes high during a sub-range for a pixel. Such a signal may be generated using a gating circuit and trace logic to AND this signal with SR. Additionally, a series of gating signals may be generated for a sequence of window sub-ranges, such as 0 to 100, 50 to 150, 100 to 200, ..., 900 to 1000, etc. Each pixel may be provided with a single numeric value (or a bit vector if multiple sub-ranges are to be searched) indicating which sub-range in the series it should operate on. The sub-ranges may be swept linearly or adaptively.

[0106] The EDH will eventually converge with the zoom operation, even if it is started with the initial boundaries used for normal runs. The initial bin boundaries do not need to be within the zoom subrange; they will tend to be on that side of the subrange as all received return pulses are on that side. However, convergence may be accelerated if the bin boundaries are initialized more intelligently. Spreading them evenly throughout the zoom subrange may be closer to ideal, and this may be performed with an initial phase of randomly or regularly spaced synthetic background events.

[0107] Zooming around a strong peak may reach convergence in a time comparable to a normal run. The majority of the return pulses are in the subrange, so the bin boundaries do not need to be moved far. Resolving a faint peak may take longer. The proportion of return pulses will necessarily be smaller because the subrange contains the faint peak. Depending on how the subrange is chosen, it may be substantially wider than for a strong peak, possibly also contributing to the convergence time of the zoom operation.

[0108] In some embodiments, the ED histograms from the original run and one or more zoomed runs may be combined. If the task is simply to find peaks, the zoomed runs may replace the original runs. However, for tasks requiring a representation of the complete transient distribution, the situation is more complicated because the bin depths are different, and the zoomed histograms cannot simply be pasted onto the original histograms.

[0109] FIG. 14 schematically illustrates a detector 1400 comprising multiple pixels arranged in rows / columns. While only a few pixels are shown, it should be understood that the detector 1400 may include millions of pixels or more. In the embodiment shown in FIG. 14, the pixels are arranged into groups of pixels called macropixels (in the embodiment shown, each macropixel has four pixels, but the number may vary). Each macropixel is coupled to a respective equal-depth histogram generator (e.g., the multi-stage histogram generator of FIG. 7, a single-bin classifier as shown in FIG. 2A, or a histogram generator with a different number of bin classifiers). For example, pixels 0-3 are coupled via a multiplexer to a B-bin histogram generator (e.g., with any number of bin classifiers). The pixels may be controlled by row and column drivers, as shown. The detector 1400 may receive signals from various devices. For example, the detector 1400 may receive an input indicating the timing of laser pulses. In some embodiments, the inputs may include target percentile(s) for the proportional bin classifier or other inputs that may be used to control the bin classifier. The histogram generator may determine bin boundaries for each pixel in the macro-pixel sequentially, or the histogram generator may determine bin boundaries for the entire macro-pixel.

[0110] 15 and 16 show example circuit diagrams of a bin classifier (e.g., the bin classifier of FIG. 2A or the bin classifier of the multi-stage histogram generator of FIG. 7) that may be included in a histogram generator, with the control values ​​implemented in analog (as shown in FIG. 15) or digital (as shown in FIG. 16). Referring initially to FIG. 15, a first circuit diagram 1500 for a bin classifier is shown. The bin classifier includes a monostable multivibrator that generates RS, which is fed along with SR to a set of AND gates. Each AND gate is coupled to a respective capacitor, which is coupled to a difference amplifier that provides an input to an integrator. As discussed above, the AND gates may split SR into SE and SL based on RS. Each time an event passes through an AND gate, the capacitor is charged and then discharged to a difference amplifier that outputs the difference between the capacitor charges to determine the proportionality of SE and SL events (e.g., CV), which provides an input to the integrator. The capacitor may be reset at a desired time, such as at the end of each cycle (e.g., determined based on a clock signal input to the RS generator). Therefore, CV may be the output of an integrator. CV is fed back to adjust RS (e.g., to set the duty cycle of the RS waveform). SE and SL may also be fed to a buffer amplifier and output to the bin classifier of the next stage when the bin classifier is within a multi-stage histogram generator. In this manner, CV (also called a reference signal modulator) is the output of an integrator coupled to a differential amplifier, which is coupled to a pair of capacitors, each coupled to a respective AND gate that receives RS and SR. It should be understood that when a bin classifier is downstream of a previous bin classifier, SR may be the SE or SL of the previous bin classifier.

[0111] FIG. 16 shows a second circuit diagram 1600 of the bin classifier, which includes some elements similar to the first circuit diagram 1500 (such as a reference signal generator and AND gates). Instead of capacitors and amplifiers, the CV may be implemented digitally using a 12-bit register. Each AND gate feeds into a register that is incremented for each SE event and decremented for each SL event. The value of the register is the CV, which is output after each event detection or after multiple event detections (e.g., at the end of a window). As discussed above, the CV is fed back to adjust the RS (e.g., to set the duty cycle of the RS waveform), and the SE and SL may also be output to the next stage of the bin classifier.

[0112] The technical effect of the bin classifier and equal-depth histogram generator described herein is that scene distance information can be captured in a bandwidth- and energy-efficient manner. Using a bin classifier that adaptively finds a given quantile as a basic building block, an equal-depth histogram generator can be generated that determines multiple bin boundaries without explicitly storing the photon count history. The bin classifier can be adapted for implementation using race logic, a technology that operates in the delay domain, which is suitable for processing return events at single-photon pixels. This approach can reduce bandwidth while maintaining similar distance accuracy to existing resource-intensive methods that rely on storing and processing equal-width histograms using thousands of bins. EDH-based SPC can achieve energy savings of approximately 10-100 times, depending on various factors such as the number of laser pulses required for convergence and the energy consumption of each readout.

[0113] The present disclosure also provides support for a method of receiving, with a bin classifier, a stream of photon return events from pixels of an imaging detector, the stream of photon return events being generated by photons transmitted from a pulsed light source and reflected from objects in a scene, classifying each photon return event as either an early event or a late event based on a reference signal controlled by a control value, the control value being configured to vary based on the relative proportion of early events to late events, and outputting, upon request, a control value from the bin classifier that can be used to determine the distance of an object in the scene. In a first embodiment of the method, each photon return event includes a time delay of a voltage pulse generated by the pixel relative to a pulse time of the pulsed light source, the control value being represented as an analog quantity or a numeric register. In a second embodiment of the method, optionally including the first embodiment, classifying each photon return event as either an early event or a late event based on the reference signal includes classifying each photon return event having a time delay less than the duration of the reference signal as an early event and classifying each photon return event having a time delay greater than the duration of the reference signal as a late event. A third embodiment of the method optionally includes one or both of the first and second embodiments, wherein the method further includes increasing the control value for each detected late event and decreasing the control value for each detected early event. A fourth embodiment of the method optionally includes one or more or each of the first through third embodiments, wherein the method further includes increasing the control value based on a number of detected late events over one or more cycles of the pulsed light source and decreasing the control value based on a number of detected early events over one or more cycles of the pulsed light source.A fifth embodiment of the method optionally includes one or more of the first through fourth embodiments, further including: receiving a stream of photon return events over a first cycle; adjusting a duration of the reference signal based on the current control value at the end of the first cycle; receiving a second stream of photon return events from the pixels over a second cycle; and classifying each photon return event in the second stream as either an early event or a late event based on the adjusted duration of the reference signal. A sixth embodiment of the method optionally includes one or more of the first through fifth embodiments, wherein the control value is output after multiple streams of photon return events over multiple cycles are received, the multiple cycles defining a run. A seventh embodiment of the method optionally includes one or more of the first through sixth embodiments, wherein the control value is adjusted by a fixed amount for each early and late event over each cycle of the run. In an eighth embodiment of the method, optionally including one or more or each of the first through seventh embodiments, the control value is adjusted by an amount that varies over two or more cycles of the run. In a ninth embodiment of the method, optionally including one or more or each of the first through eighth embodiments, the control value is adjusted so that the control value converges toward a median value of the distribution of all photon return events in the run. In a tenth embodiment of the method, optionally including one or more or each of the first through ninth embodiments, adjusting the control value includes, for a first cycle of the plurality of cycles, adjusting the control value from an initial value, the initial value being one or more of: a midpoint of a range of an expected distribution of photon return events; identified by starting a bin classifier on a random event stream; based on a previous run of the bin classifier; and based on a second run of a second bin classifier on another pixel of the imaging detector.An eleventh embodiment of the method optionally includes one or more or each of the first to tenth embodiments, and further includes: the bin classifier is a first bin classifier, and outputs a first stream of early events to a second bin classifier and a second stream of late events to a third bin classifier; and reading respective additional control values ​​from the first bin classifier, the second bin classifier, and the third bin classifier to derive values ​​corresponding to boundaries of a histogram, wherein the histogram is used to determine the distance of the object.A twelfth embodiment of the method optionally includes one or more or each of the first to eleventh embodiments, and further includes estimating a location of a peak in a distribution of photon return events over a run based on the histogram, wherein the run includes a plurality of cycles, each cycle including receiving a respective stream of photon return events generated by photons transmitted from a respective pulse of the pulsed light source, wherein the peak is used to determine the distance of the object. A thirteenth embodiment of the method optionally includes one or more or each of the first through twelfth embodiments, wherein estimating the location of the peak includes estimating the location of the peak from the midpoint of a locally narrow bin of the histogram, or estimating the location of the peak by fitting a curve to a plurality of points corresponding to a plurality of bins of the histogram.

[0114] The present disclosure also provides support for a system comprising: a single-photon counting detector including a plurality of pixels, the plurality of pixels including a first pixel; and a bin classifier coupled to the first pixel and including a reference signal generator, a reference signal modulator, a first gate, and a second gate, wherein the stream of photon return events generated by the first pixel and a reference signal generated by the reference signal generator are provided to the first gate and the second gate, respectively, to generate a fast stream of photon return events and a slow stream of photon return events, and the reference signal modulator is configured to adjust a duration of the reference signal generated by the reference signal generator based on a relative proportion of the fast stream of photon return events to the slow stream of photon return events. In a first embodiment of the system, the bin classifier is a first bin classifier in a first stage of a multi-stage histogram generator, and the multi-stage histogram generator further includes a second stage including a second bin classifier and a third bin classifier, the second bin classifier configured to receive an early stream of photon return events and the third bin classifier configured to receive a late stream of photon return events. A second embodiment of the system optionally includes the first embodiment, wherein the early stream of photon return events is a first early stream, the late stream of photon return events is a first late stream, the reference signal modulator is a first reference signal modulator, the reference signal is a first reference signal, the second bin classifier is configured to generate the second early stream and the second late stream and includes a second reference signal modulator configured to adjust a second duration of the second reference signal generated by the second bin classifier based on a relative proportion of the second early stream to the second late stream, and the third bin classifier is configured to generate a third early stream and a third late stream and includes a third reference signal modulator configured to adjust a third duration of the third reference signal generated by the third bin classifier based on a relative proportion of the third early stream to the third late stream.A third embodiment of the system optionally includes one or both of the first and second embodiments, wherein the first reference signal modulator adjusts a duration of the first reference signal based on a first control value, the second reference signal modulator adjusts a duration of the second reference signal based on a second control value, and the third reference signal modulator adjusts a duration of the third reference signal based on a third control value, and the first, second, and third control values ​​are read to determine boundaries of a histogram, the histogram being usable to determine distances of points within the scene.A fourth embodiment of the system optionally includes one or more or each of the first through third embodiments, and wherein the bin classifier is coupled to a second pixel of the plurality of pixels.

[0115] The present disclosure also provides support for a method for a single-photon sensitive imaging system, the method including generating a distance map of a scene based on a plurality of iso-depth histograms, each iso-depth histogram being generated for a respective pixel of a plurality of pixels of a detector of the single-photon detection imaging system, each iso-depth histogram including a plurality of bins having the same number of photon return events, each photon return event being partitioned into the plurality of bins based on the delay time of that photon return event; and displaying the distance map and / or using the distance map to generate and / or arrange one or more images for display.

[0116] The description of the embodiments has been presented for purposes of illustration and description. Suitable modifications and variations to the embodiments can be implemented in light of the above description or acquired from practicing the methods. For example, unless otherwise indicated, one or more of the described methods may be performed by any suitable device and / or combination of devices, such as the system described above with respect to FIG. 1. The methods may be performed by executing stored instructions using one or more logic devices (e.g., processors) in combination with one or more hardware elements, such as storage devices, memories, hardware network interfaces / antennas, switches, actuators, clock circuits, etc. The described methods and related operations may also be performed in various orders, in parallel, and / or simultaneously in addition to the order described herein. The described systems are exemplary in nature and may include additional elements and / or omit elements. The subject matter of the present disclosure includes all novel and non-obvious combinations and subcombinations of the various systems and configurations, and other features, functions, and / or properties disclosed.

[0117] As used herein, the terms "system," "module," or "modulator" may include hardware and / or software systems that operate to perform one or more functions. For example, a module or system may include a computer processor, controller, or other logic-based device that performs operations based on instructions stored in a tangible, non-transitory, computer-readable storage medium, such as a computer memory. Alternatively, a module or system may include a hardwired device that performs operations based on the device's hardwired logic. The various modules or units shown in the accompanying figures may represent hardware that operates based on software or hardwired instructions, software that causes hardware to perform operations, or a combination thereof.

[0118] The foregoing aspects depict different components contained within or connected to different other components. It should be understood that such depicted architectures are merely exemplary, and that in fact many other architectures may be implemented that achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Thus, any two components herein that are combined to achieve a particular functionality can be viewed as being "associated" with each other such that the desired functionality is achieved, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be viewed as being "operably connected" or "operably coupled" with each other to achieve the desired functionality.

[0119] As used in this application, elements or steps described in the singular followed by the word "a" or "an" are to be understood as not excluding those elements or steps unless the plural of such elements or steps is also expressly stated as excluding them. Furthermore, references to "one embodiment" or "one example" of the present disclosure are not to be construed as excluding the existence of additional embodiments incorporating the recited features. The terms "first," "second," "third," etc. are used merely as labels and are not intended to impose numerical requirements or a particular positional order on their objects. The following claims particularly point out the subject matter of the above disclosure which is believed to be new and unobvious.

Claims

1. receiving, at a bin classifier, a stream of photon return events from pixels of an imaging detector, said stream of photon return events being generated by photons transmitted from a pulsed light source and reflected from objects in a scene; using the bin classifier to classify each photon return event as either an early or late event based on a reference signal controlled by a control value, the control value being configured to vary based on the relative proportion of early events to late events; outputting, on demand, from the bin classifier, the control value that can be used to determine the distance of the object in the scene. method.

2. The method of claim 1 , wherein each photon return event comprises a time delay of a voltage pulse generated by the pixel relative to a pulse time of the pulsed light source, and the control value is represented as an analog quantity or a numeric register.

3. 3. The method of claim 2, wherein classifying each photon return event as either an early event or a late event based on the reference signal comprises classifying each photon return event having a time delay less than a duration of the reference signal as an early event and classifying each photon return event having a time delay greater than the duration of the reference signal as a late event.

4. The method of claim 3 , further comprising increasing the control value for each detected late event and decreasing the control value for each detected early event.

5. 4. The method of claim 3, further comprising increasing the control value based on a number of late events detected over one or more cycles of the pulsed light source and decreasing the control value based on a number of early events detected over the one or more cycles of the pulsed light source.

6. 4. The method of claim 3, wherein the stream of photon return events is received over a first cycle, and further comprising adjusting the duration of the reference signal based on a current control value at the end of the first cycle, receiving a second stream of photon return events from the pixel over a second cycle, and classifying each photon return event in the second stream as either an early event or a late event based on the adjusted duration of the reference signal.

7. The method of claim 6 , wherein the control value is output after a plurality of streams of photon return events are received over a plurality of cycles, the plurality of cycles defining a run.

8. The method of claim 7 , wherein the control value is adjusted by a fixed amount for each early and late event over each cycle of the run.

9. The method of claim 7 , wherein the control value is adjusted by an amount that varies over two or more cycles of the run.

10. The method of claim 7 , wherein the control value is adjusted such that the control value converges towards the median of the distribution of all photon return events in the run.

11. Adjusting the control value includes adjusting the control value from an initial value for the first cycle of the plurality of cycles, the initial value being: the midpoint of a range of expected distribution of said photon return events; identified by starting the bin classifier with a random event stream; based on a previous run of the bin classifier; based on a second run of a second bin classifier for another pixel of the imaging detector; The method of claim 7, wherein the method is one or more of:

12. 2. The method of claim 1 , wherein the bin classifier is a first bin classifier and further outputs a first stream of early events to a second bin classifier and a second stream of late events to a third bin classifier, and reads respective additional control values ​​from the first bin classifier, the second bin classifier, and the third bin classifier to derive values ​​corresponding to boundaries of a histogram, and the histogram is used to determine the distance of the object.

13. 13. The method of claim 12, further comprising estimating, based on the histogram, a location of a peak in a distribution of photon return events over a run, the run including a plurality of cycles, each cycle including receiving a respective stream of photon return events generated by photons transmitted from a respective pulse of the pulsed light source, and the peak being used to determine the distance of the object.

14. 14. The method of claim 13, wherein estimating the location of the peak comprises estimating the location of the peak from the midpoint of a locally narrow bin of the histogram, or estimating the location of the peak by fitting a curve to a plurality of points corresponding to a plurality of bins of the histogram.

15. a single-photon counting detector including a plurality of pixels, the plurality of pixels including a first pixel; a bin classifier coupled to the first pixel and including a reference signal generator, a reference signal modulator, a first gate, and a second gate, wherein the stream of photon return events generated by the first pixel and a reference signal generated by the reference signal generator are provided to the first gate and the second gate, respectively, to generate an early stream of photon return events and a late stream of photon return events, and the reference signal modulator is configured to adjust a duration of the reference signal generated by the reference signal generator based on a relative proportion of the early stream of photon return events to the late stream of photon return events; Equipped with system.

16. 16. The system of claim 15, wherein the bin classifier is a first bin classifier in a first stage of a multi-stage histogram generator, the multi-stage histogram generator further comprising a second stage including a second bin classifier and a third bin classifier, the second bin classifier configured to receive the early stream of photon return events and the third bin classifier configured to receive the late stream of photon return events.

17. the fast stream of photon return events is a first fast stream, the slow stream of photon return events is a first slow stream, the reference signal modulator is a first reference signal modulator, and the reference signal is a first reference signal; the second bin classifier is configured to generate a second early stream and a second late stream, and includes a second reference signal modulator configured to adjust a second duration of a second reference signal generated by the second bin classifier based on a relative proportion of the second early stream to the second late stream; 17. The system of claim 16, wherein the third bin classifier is configured to generate a third early stream and a third late stream, and further comprising a third reference signal modulator configured to adjust a third duration of a third reference signal generated by the third bin classifier based on a relative proportion of the third early stream to the third late stream.

18. 18. The system of claim 17, wherein the first reference signal modulator adjusts the duration of the first reference signal based on a first control value, the second reference signal modulator adjusts the duration of the second reference signal based on a second control value, and the third reference signal modulator adjusts the duration of the third reference signal based on a third control value, and the first control value, the second control value, and the third control value are read to determine boundaries of a histogram, and the histogram can be used to determine distances of points in a scene.

19. The system of claim 15 , wherein the bin classifier is coupled to a second pixel of the plurality of pixels.

20. 1. A method for a single-photon sensitive imaging system, comprising: generating a distance map of the scene based on a plurality of iso-depth histograms, each iso-depth histogram being generated for a respective pixel of a plurality of pixels of a detector of the single-photon detection imaging system, each iso-depth histogram including a plurality of bins having the same number of photon return events, each photon return event being partitioned into a bin of the plurality of bins based on a delay time of the photon return event; A method of displaying the distance map and / or using the distance map to generate and / or arrange one or more images for display.

Images