Super-reduced time-of-flight histogram processing

Abstract

The invention relates to super-reduced time-of-flight histogram processing. According to an embodiment, a time-of-flight sensor system includes a light emitter, a detector array, and histogram processing circuitry. The histogram processing circuitry processes the time-of-flight measurement data using continuous bin-by-bin histogram processing. The method enables efficient processing with minimal memory requirements, suitable for low power applications. The circuitry applies immediate operations during processing, including crosstalk removal and peak detection. A correlator circuit within the system uses a multiply accumulate (MAC) circuit to compute ambient light contributions, perform primary correlation, and compute crosstalk contributions in a bin serial fashion. Additionally, the phase / box arithmetic circuit applies filter coefficients to the pre-processed histogram data, and detects a positive zero crossing to calculate a median phase.

Description

Technical Field

[0001] This disclosure generally relates to time-of-flight sensor technology, and in certain embodiments to an ultra-lean histogram processing architecture for time-of-flight sensors. Background Technology

[0002] Time-of-flight (ToF) technology is becoming increasingly prevalent in various applications, including robotics, automotive systems, and consumer electronics. ToF systems measure the time it takes for light to travel from an emitter to an object and back to the sensor, enabling distance measurement and 3D environment mapping.

[0003] Time-of-Flight (ToF) sensors are gaining popularity in mobile devices for applications such as facial recognition, augmented reality, and camera autofocus systems. These sensors can employ either direct or indirect ToF methods. Direct ToF systems measure the actual time it takes for light to travel, while indirect ToF systems measure the phase shift in a modulated light signal.

[0004] Direct Time-of-Flight (ToF) systems typically use single-photon avalanche diodes (SPADs) as detectors due to their high sensitivity and ability to detect individual photons. These systems generate a histogram of photon arrival times, which is processed to determine distance information. The histogram data contains information about the reflected light pulses, including their shape, intensity, and timing.

[0005] With advancements in Time-of-Flight (ToF) technology, there is growing interest in integrating it into a wider range of devices, including those with power and size constraints. This has led to efforts to develop ToF systems that can operate within the power budget of battery-powered devices while maintaining accuracy and performance.

[0006] Histogram processing is a component of direct Time-of-Flight (ToF) systems. It involves analyzing collected photon timing data to extract distance information. This processing can include noise reduction, peak detection, and phase calculation. The complexity of these operations can vary depending on application requirements and environmental conditions, such as the presence of multiple reflective surfaces or ambient light interference.

[0007] Histogram processing in ToF systems may rely on general-purpose microcontrollers or digital signal processors to perform the calculations. These methods offer flexibility, but may also present considerations regarding power consumption, processing speed, and integration within the form factor of compact devices.

[0008] A Time-of-Flight (ToF) system can include a vertical cavity surface-emitting laser (VCSEL) as the light source, emitting short light pulses. Typically, a SPAD array detects the reflected light. The time difference between the emission and detection of the light pulse is used to calculate the distance to the reflecting object.

[0009] Crosstalk can occur in Time-of-Flight (ToF) systems when light from the transmitter undergoes internal reflection within the sensor package or telephone housing before reaching the detector. This can lead to inaccurate distance measurements. Crosstalk mitigation techniques can be employed to reduce these effects and improve measurement accuracy.

[0010] Multi-object detection is another consideration in Time-of-Flight (ToF) systems. In some scenarios, emitted light pulses may be reflected by multiple objects at different distances. Processing techniques can be used to distinguish these multiple reflections and determine the distances to different objects in the scene.

[0011] Power management is critical for ToF sensor design, especially for battery-powered devices. This can involve optimizing the light emission, detection, and processing stages to minimize overall power consumption while maintaining desired performance characteristics.

[0012] Integrating ToF sensors with other system components, such as application processors or image signal processors, can offer opportunities for enhanced functionality and efficiency. Integration can involve considerations of data transmission, processing distribution, and overall system architecture.

[0013] Furthermore, conventional time-of-flight systems rely on substantial memory resources to store and process histogram data. These systems typically use general-purpose microcontrollers or digital signal processors with ample RAM to maintain multiple copies of the histogram data for each processing stage. However, in applications where power consumption, cost, and size are critical factors, such as in battery-powered devices or compact mobile electronics, the use of large memory banks becomes prohibitive. Summary of the Invention

[0014] The embodiments of this disclosure typically offer technical advantages, describing an ultra-precise simple graph processing architecture for time-of-flight sensors.

[0015] The first aspect relates to a time-of-flight sensor system, comprising: a light emitter configured to emit light pulses toward a target; a detector array configured to detect light pulses reflected from the target; and a histogram processing circuit coupled to the detector array and configured to: receive time-of-flight measurement data from the detector array, process the measurement data from the detector array using continuous bin-by-bin histogram processing, and apply one or more on-the-fly operations during the continuous bin-by-bin histogram processing.

[0016] The second aspect relates to a method for processing time-of-flight sensor data, the method comprising: emitting a light pulse toward a target; detecting a light pulse reflected from the target; receiving time-of-flight measurement data based on the detected reflected light pulse; processing the time-of-flight measurement data using sequential bin-by-bin histogram processing, wherein sequential bin-by-bin histogram processing includes sequentially processing individual bins of a histogram, maintaining the state of a predetermined number of recently processed bins, and generating output data for each bin before proceeding to subsequent bins; and applying one or more real-time operations during sequential bin-by-bin histogram processing.

[0017] The third aspect relates to a correlator circuit for a time-of-flight sensor system configured to process input data in a box-serial manner, the correlator circuit comprising: an ambient multiply-accumulate (MAC) circuit configured to calculate the ambient light contribution; a filter MAC circuit configured to perform a main correlation operation on the input histogram data; a crosstalk MAC circuit configured to calculate the crosstalk contribution; a nearest target circuit coupled to the filter MAC circuit and configured to determine the nearest detected target; and a maximum value circuit coupled to the filter MAC circuit and configured to identify the peak with the highest signal-to-noise ratio (SNR).

[0018] The fourth aspect relates to a method for processing input histogram data from a time-of-flight sensor, the method comprising processing the input histogram data in a bin-serial manner, the processing including: calculating the ambient light contribution; performing a master correlation operation on the input histogram data; calculating the crosstalk contribution; determining the nearest detection target based on the result of the master correlation operation; and identifying the peak with the highest signal-to-noise ratio (SNR) based on the result of the master correlation operation.

[0019] The fifth aspect relates to a phase / box operation circuit for a time-of-flight sensor system, the phase / box operation circuit comprising: a filter multiply-accumulate (MAC) circuit configured to apply filter coefficients to preprocessed histogram data within a defined search and pulse window; and a positive zero-crossing detection circuit coupled to the filter MAC circuit and configured to: detect positive zero-crossings in the filtered data output by the filter MAC circuit, store current and previous filter values ​​upon detection of a positive zero-crossing, and output a detected median flag and median box position, wherein the phase / box operation circuit is configured to: process the input data in a box-serial manner, and calculate the median phase by interpolating the current filter value and the previous filter value.

[0020] The sixth aspect relates to a method for processing time-of-flight sensor data, the method comprising: receiving preprocessed histogram data; applying filter coefficients to the preprocessed histogram data within defined search and pulse windows to generate filtered histogram data; processing the filtered histogram data in a bin-serial manner; detecting positive zero crossings in the filtered histogram data; storing current and previous filter values ​​in response to detecting positive zero crossings; outputting a detected median flag and median bin position; and calculating the median phase by interpolating the current and previous filter values.

[0021] The implementation can be carried out in hardware, software, or a combination thereof. Attached Figure Description

[0022] To gain a more complete understanding of this disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which:

[0023] Figure 1 This is a block diagram of the Time-of-Flight (ToF) sensor system of an embodiment;

[0024] Figure 2 This is a block diagram of the histogram processing circuit in the embodiment;

[0025] Figure 3 This is a flowchart of an embodiment method for generating crosstalk calibration data;

[0026] Figure 4 This is a block diagram of the correlator circuit in an embodiment;

[0027] Figure 5 This is a flowchart of an embodiment method for processing ToF sensor input data in a box-serial manner;

[0028] Figure 6 This is a block diagram of the phase / box operational circuit of the embodiment; and

[0029] Figure 7 This is a flowchart of an embodiment of a method for instantaneous median estimation of a potential target using, for example, a simplified impulse weighted histogram method. Detailed Implementation

[0030] This disclosure provides numerous applicable inventive concepts that can be practiced in a variety of specific contexts. Specific embodiments are merely illustrative of particular configurations and do not limit the scope of the claimed embodiments. Features from different embodiments may be combined to form other embodiments, unless otherwise mentioned. Various embodiments are illustrated in the accompanying drawings, wherein the same reference numerals identify the same components and elements, and repeated descriptions are omitted for brevity.

[0031] Variations or modifications described in one embodiment may also be applied to other embodiments. Furthermore, various changes, substitutions, and alterations may be made herein without departing from the spirit and scope of this disclosure as defined by the appended claims.

[0032] While these aspects of the invention are described primarily in the context of time-of-flight (ToF) sensors for mobile devices, it should be recognized that these inventive aspects are also applicable to other types of distance measurement systems and applications. In particular, various aspects of this disclosure can be similarly applied to LiDAR systems, structured light 3D scanners, automotive driver assistance systems, industrial automation systems, and gesture recognition devices.

[0033] Furthermore, the disclosed technology may be particularly well-suited for battery-powered applications, such as automatic faucets or water dispensers in sanitation settings, mobile phone cameras for autofocus and facial recognition, and other autonomous ranging devices that require long-term battery-powered operation. The system's efficiency and compact design also make it suitable for a variety of consumer electronics products where size and power constraints are key factors.

[0034] Embodiments of this disclosure relate to a hardware architecture for histogram processing in time-of-flight (ToF) sensors. This architecture includes a modular hardware processing unit comprising separate hardware processing circuitry capable of performing ultra-low-power full histogram processing at the sensor level. The proposed architecture allows for on-chip processing in direct ToF sensors while maintaining a low-power target.

[0035] In this embodiment, the hardware processing unit combines a custom peak finder circuit with a multi-purpose multiply-accumulate (MAC) circuit. While current implementations of the custom peak finder output the strongest and closest target from the complete histogram data, this architecture is scalable to support the detection of additional targets for each histogram. The MAC circuitry can be shared across various hardware circuits to enable complete histogram processing. The architecture includes a correlator-based matched filter approach, compatible with multiple pulse width and ranging mode options, adaptable to short-, medium-, or long-range applications, while maintaining consistent memory requirements.

[0036] The correlator, together with a custom peak finder, implements a nearest-edge detection algorithm that operates on the rising edge of the histogram data. This implementation leverages stacking mitigation to achieve robust ranging of the nearest target. Effective crosstalk mitigation can be applied by preserving full histogram processing capabilities. Optionally, a full histogram bypass mode is available, allowing off-chip processing on the host device if needed.

[0037] This architecture can incorporate highly efficient crosstalk shape approximation and removal processes into histogram data, including adaptive crosstalk shape updates. Additionally, the architecture can extend histogram computation with robust backoff options to achieve consistent ranging in challenging scenarios, such as providing median range outputs when merging target clusters.

[0038] The embodiments of this disclosure aim to reduce power consumption, achieve compact designs with lower area requirements, increase integration potential, reduce overall cost, and target extremely low gate / logic counts. These goals can be achieved through custom hardware circuitry, shared MAC circuitry, specific lookup table (LUT) memory circuitry, and minimal memory usage throughout the processing pipeline.

[0039] In the embodiments, the proposed architecture adapts to and evolves existing methods for minimizing logic footprint and incorporates discrete hardware circuitry to achieve robust performance with the closest and strongest peak detection options. This approach allows for efficient implementation of full histogram processing capabilities within the sensor, reducing the need for external processing and minimizing power consumption.

[0040] The modular approach to hardware processing circuitry allows for flexibility in handling various ranging scenarios. For example, a correlator-based matched filter approach can be configured for three pulse widths and three different ranging mode options, enabling it to adapt to short-, medium-, or long-range applications while maintaining consistent memory requirements. This versatility allows ToF sensors to perform effectively across a wide range of use cases without compromising efficiency or accuracy.

[0041] The embodiments disclosed herein address the challenge of achieving high-performance time-of-flight sensing under extreme memory constraints. The disclosed system operates without dedicated RAM, relying on a small number of select registers and buffers for temporary data storage.

[0042] Histogram processing circuitry can be configured to operate efficiently with minimal memory resources. Instead of storing multiple copies of the complete histogram or processed data, the system employs a bin-serial processing approach. This method allows for streaming processing of histogram data, significantly reducing memory requirements.

[0043] In this embodiment, the system's microcontroller unit (MCU) primarily functions as a sequencer, with limited local storage in the form of registers. This choice eliminates the need for a separate memory system, reducing power consumption and chip area. The architecture can be optimized to minimize memory accesses, as each memory fetch operation consumes multiple clock cycles and power.

[0044] By implementing these memory-efficient techniques, the system achieves full histogram processing capabilities comparable to more resource-intensive solutions, while maintaining ultra-low power consumption suitable for integration into battery-powered devices. These and additional details are further elaborated below.

[0045] Figure 1 The illustration shows a block diagram of an embodiment of a time-of-flight (ToF) sensor system 100. The ToF sensor system 100 includes a vertical cavity surface-emitting laser (VCSEL) 102, a VCSEL driver 104, a returning single-photon avalanche diode (SPAD) array 106, a reference SPAD array 108, a first OR tree 110, a second OR tree 112, histogram processing circuitry 114, a microcontroller unit (MCU) 116, an input / output (I / O) interface 118, power management circuitry 120, and a one-time programmable (OTP) memory 122, which may (or may not) be arranged as shown.

[0046] The ToF sensor system 100 may include additional components, not shown, such as a cover glass positioned above the VCSEL 102, the return SPAD array 106, and the reference SPAD array 108. The cover glass can protect the optical components from environmental factors and reduce unwanted light scattering within the ToF sensor system 100.

[0047] VCSEL 102 emits light pulses directed at the target object. VCSEL driver 104 is coupled to VCSEL 102 and controls the timing and intensity of the emitted light pulses. Return SPAD array 106 detects the reflected light pulses from the target object, while reference SPAD array 108 detects a portion of the emitted light pulses directly from VCSEL 102.

[0048] The first OR tree 110 is coupled to the return SPAD array 106 and combines signals from multiple SPADs in the return array. Similarly, the second OR tree 112 is coupled to the reference SPAD array 108 and combines signals from multiple SPADs in the reference array. The first OR tree 110 and the second OR tree 112 can help reduce the number of signal lines and simplify subsequent processing.

[0049] Histogram processing circuitry 114 is coupled to the first OR tree 110 and the second OR tree 112. It processes combined signals from the return SPAD array 106 and the reference SPAD array 108 to generate histogram data representing time-of-flight measurements. Histogram processing circuitry 114 may include various sub-circuits, such as correlator circuitry, histogram circuitry, range calculator circuitry, and rate calculation circuitry, as detailed below.

[0050] The MCU 116 is coupled to the VCSEL driver 104, the return SPAD array 106, the reference SPAD array 108, and the histogram processing circuitry 114. The MCU 116 can perform additional processing on the histogram data, control various system components, manage the overall system operation, and provide timing and control signals for the detection process.

[0051] I / O interface 118 is coupled to MCU 116 and communicates with external devices or systems. Power management circuitry 120 is coupled to various system components and manages power distribution and consumption.

[0052] The OTP memory 122 can store calibration data, system parameters, and other non-volatile information used by the ToF sensor system 100. It can be programmed during manufacturing or initial setup and retains data even in the event of a power outage.

[0053] In one embodiment, the ToF sensor system 100 includes a histogram bypass mode (i.e., bypassing the histogram processing circuitry 114), which allows raw histogram data to be sent off-chip for processing on a host device, providing flexibility for applications that require more complex or custom processing algorithms.

[0054] Conventional techniques for histogram processing of time-of-flight (ToF) sensors typically involve using a powerful off-chip microcontroller unit (MCU) for complete histogram processing. Off-chip processing usually involves several steps, including crosstalk estimation and compensation, pulse segmentation (i.e., an iterative process for each histogram), and phase estimation for non-ideal pulse shapes.

[0055] On-chip processing presents several drawbacks, such as the need for large MCUs in the sensor package, which consumes significant power and increases readout complexity. Even when idle, such MCUs require minimal power, leading to increased standby power consumption. Additionally, the more complex algorithms employed in these techniques require longer runtimes at the same clock frequency, further increasing power consumption.

[0056] Furthermore, using larger MCUs and more complex processing algorithms results in a larger overall system silicon area. This increased area has cascading effects, including higher module costs, larger module size, and increased power requirements. These factors make integrating such ToF sensors into compact, power-constrained devices, such as mobile phones or wearable computing devices, challenging.

[0057] Furthermore, the complexity of conventional technical solutions may limit their scalability and adaptability to different ranging scenarios. The iterative processes involved in pulse segmentation and phase estimation may be optimal for some use cases, especially in environments with rapidly changing conditions or multiple reflective surfaces.

[0058] These drawbacks highlight the need for more efficient, compact, and energy-saving methods for histogram processing of ToF sensors. Embodiments of this disclosure aim to address these limitations by providing an architecture that reduces power consumption, reduces silicon area, and maintains or improves performance across a variety of ranging scenarios.

[0059] Advantageously, the ToF sensor system 100 can detect and process multiple targets, adapting to various environmental conditions, including ambient light interference. It can operate in different ranging modes to suit short-, medium-, or long-range applications, while maintaining consistent memory requirements.

[0060] The ToF sensor system 100 addresses typical constraints in mobile and battery-powered applications, including power, size, cost, resources, I / O bandwidth, and limited power budgets. Its compact and efficient design with integrated on-chip processing helps overcome the package size limitations often encountered in 3D imagers.

[0061] The histogram processing circuit 114 minimizes the logic, memory usage, and runtime (operation cycles) required for histogram processing while maintaining high ranging performance within strict power constraints. This approach allows the system to perform accurate range estimation using histogram data while significantly reducing overall power consumption, effectively addressing the dual challenges of performance and energy efficiency. By performing sophisticated histogram processing on-chip and implementing efficient power management strategies, minimal power consumption is achieved while maintaining functionality.

[0062] In one embodiment, the histogram processing circuit 114 includes multiple circuits capable of performing on-chip ultra-low power histogram processing. These circuits may include correlator-based matched filter circuitry compatible with multiple pulse widths, closest peak detection circuitry, strongest peak detection circuitry, and crosstalk mitigation circuitry. The histogram processing circuit 114 may also include a shared MAC circuitry supporting various processing tasks across different sub-circuits.

[0063] In this embodiment, the histogram processing circuit 114 employs techniques to optimize performance and efficiency. For example, using a lookup table (LUT) allows for efficient data storage and histogram processing, effectively mitigating crosstalk. The crosstalk mitigation process can be adaptive, capable of updating the crosstalk shape during operation.

[0064] In one embodiment, the histogram processing circuit 114 provides the strongest and closest target outputs through histogram data, thereby achieving robust ranging with accumulation mitigation.

[0065] In one embodiment, the histogram processing circuit 114 includes histogram calculation circuitry with a robust backoff option for consistent ranging in challenging scenarios.

[0066] Figure 2 A block diagram of an embodiment of the histogram processing circuit 200 is shown. In this embodiment, the histogram processing circuit 200 is implemented as follows: Figure 1 The histogram processing circuit 114. The histogram processing circuit 200 includes a correlator circuit 202, a phase / box operation circuit, a range calculation circuit 206, a rate calculation circuit 208, and a crosstalk histogram generator circuit 218, which may (or may not) be arranged as shown.

[0067] The histogram processing circuit 200 includes discretized hardware circuitry responsible for processing histogram data. This configuration allows for a simpler and smaller microcontroller, enabling full integration on a single substrate. The histogram processing circuit 200 is configured to process inputs, including reference / return histograms and crosstalk histograms, and calculate rate and range values ​​based on measured ToF sensor data.

[0068] The histogram processing circuit 200 enables efficient, modular, on-chip processing of time-of-flight data, reducing the need for external processing (i.e., eliminating the need for a powerful microcontroller) and minimizing power consumption. By integrating multiple processing steps and utilizing shared resources, the histogram processing circuit 200 can achieve high performance under tight power and area constraints.

[0069] Instead of processing the complete reference / return histogram input as an entire array in memory, the histogram processing circuit 200 is advantageously configured to iterate the histogram by serially operating on the relevant outputs in a single pass. Each filter operation is applied in real time to a small block of memory. Relevant processing information is stored for the desired output without modifying the histogram.

[0070] In one embodiment, the crosstalk histogram generator circuit 218 generates a crosstalk histogram by performing linear interpolation using crosstalk calibration data and silicon temperature as input. The crosstalk calibration data is a compact description (e.g., 19 bytes) relative to... Figure 3 Further details.

[0071] In this embodiment, the crosstalk histogram is generated based on an adaptive crosstalk estimator technique used to correct crosstalk under varying conditions, as disclosed in U.S. Patent Application 18 / 348,600 (Attorney General's File No. ST-23-EDI-0386US01), which is assigned to the same entity as this disclosure and is incorporated herein by reference in its entirety. U.S. Patent Application 18 / 348,600 discloses a low-cost crosstalk shape storage and approximation technique for efficient and adaptive crosstalk removal.

[0072] Histogram processing circuitry 200 includes correlator circuitry 202 that implements a nearest-edge detection algorithm. This algorithm operates on the rising edge of the histogram data to achieve robust ranging of the nearest target. In embodiments, the algorithm can perform stacking mitigation, as disclosed in U.S. Patent Application 18 / 342,965 (Attorney General's File No. ST-22-EDI-1129US01), which is assigned to the same entity as this disclosure and is incorporated herein by reference in its entirety.

[0073] Typically, when multiple reflected pulses overlap or arrive rapidly, they accumulate, leading to measurement errors in conventional ToF systems. The correlator circuit 202 can distinguish overlapping signals, allowing for accurate distance measurements even in complex environments with multiple reflective surfaces or partial obstacles.

[0074] Stacking mitigation capabilities enhance the robustness and reliability of ToF sensors, especially in scenarios where multiple objects are at different distances.

[0075] In this embodiment, correlator circuit 202 receives a reference / return histogram and a crosstalk histogram as inputs. It processes these inputs based on the start and end points of a window to determine two main outputs: the closest and the strongest maximum values. These outputs can represent the nearest and strongest detected targets provided as inputs to phase / box operation circuit 204.

[0076] In this embodiment, the phase / box operation circuit 204 receives a reference / return histogram, a crosstalk histogram, and output data from the correlator circuit 202. The output of the phase / box operation circuit 204 is further processed by firmware, which calculates and outputs the median phase.

[0077] In this embodiment, the range calculation circuit 206 uses the median phase, reference phase, range gain correction factor (configuration / calibration data), and calibrated range offset (calibration data) to calculate the median range. The median range can represent the estimated distance to the detected target. In this embodiment, the median phase is calculated by the phase / box calculation circuit 204.

[0078] In this embodiment, the rate calculation circuit 208 processes the reference / return histogram and crosstalk histogram inputs to determine pulse events. Pulse events and rate gain correction factors (configuration / calibration data) are used to calculate the ambient and peak rates for each SPAD. These rates can provide information about the optical conditions and signal strength of the detected pulses.

[0079] In this embodiment, the histogram processing circuit 200 implements a bin-serial processing method. This method can efficiently process time-of-flight data without requiring a large amount of memory. Instead of accumulating and storing the complete histogram before processing, the circuit processes segments of the histogram as it traverses it, thus effectively achieving a real-time approach.

[0080] Histogram data is processed one bin at a time using a bin-sequential processing method. For example, firmware or a sequencer continuously presents each histogram bin to the processing hardware. The hardware circuitry processes the bin, and the firmware presents the next bin when it is finished. This method creates a state machine that maintains a minimal internal buffer, significantly reducing memory requirements compared to traditional methods that process the entire histogram simultaneously.

[0081] The bin-serial architecture can be applied across various processing stages, including the correlator circuit 202 and the phase / bin operation circuit. By processing one bin at a time while maintaining the relationship between the histogram, environment, and crosstalk data for each bin, this circuit achieves efficient processing with minimal memory requirements. This approach can introduce minimal latency, as several bins may be needed before the first result is output. Nevertheless, this trade-off allows for a lower memory footprint and enables the processing of multiple bins in a single region even with limited resources.

[0082] The bin-serial approach minimizes memory usage and reduces the number of memory fetches required. In traditional implementations, these operations typically involve numerous memory fetches, which can be power-intensive and time-consuming. By implementing bin-serial processing, the system avoids repetitive memory fetches, making the entire system more efficient in terms of both power consumption and processing speed.

[0083] Figure 3 This is a flowchart of an embodiment method 300 for generating crosstalk calibration data. It should be noted that all steps outlined in the flowchart of method 300 are not necessarily required and may be optional. Furthermore, it is similarly conceivable to change the arrangement of steps, remove one or more steps and path connections, and add steps and path connections.

[0084] In step 302, N amplitude coefficients (e.g., eight amplitude coefficients) and M leakage filter coefficients (e.g., four leakage filter coefficients) are used to describe the crosstalk pulse shape. One feature of using a leakage integrator to describe the pulse tail is that variable-size crosstalk shapes spanning from 16 to 128 bins can be generated from the same crosstalk lookup table description, thus enabling flexible adaptation to different system requirements.

[0085] This method allows for a compact and accurate representation of complex crosstalk pulse shapes. N amplitude coefficients capture the main characteristics of the pulse, particularly its peak value and surrounding environment, while M leakage filter coefficients efficiently model the pulse tail. To effectively handle high crosstalk amplitudes and long integration times, a dynamic range of 10000:1 can be used for full-pulse tail subtraction. The crosstalk leakage integral (LUT) allows for a wide dynamic range without increasing the size of the crosstalk shape description. Using combinations of coefficients, Method 300 can represent a wide range of pulse shapes with minimal data, which may be beneficial for efficient storage and processing in resource-constrained time-of-flight systems. The choice of N and M coefficients balances the trade-off between representational accuracy and computational efficiency.

[0086] In step 304, N coefficients are used to describe the main approximate region of the pulse. In this embodiment, the coefficients are stored in a first lookup table. The main region of the pulse typically includes the rising edge, peak, and initial falling edge of the crosstalk pulse shape. Method 300 captures the most significant and energy-dense portion of the pulse by focusing on this region. Using a lookup table to store the coefficients allows for fast access and efficient processing during runtime.

[0087] This method enables the system to quickly reconstruct the main characteristics of crosstalk pulses without requiring complex calculations for each box. The number of coefficients can be selected to balance approximate accuracy with memory requirements and processing speed.

[0088] It should be noted that the number of lookup table (LUT) coefficients used is programmable (i.e., ≤N).

[0089] In step 306, the M leakage filter coefficients describe a composite leakage filter. The leakage filter approximates a pulse tail after the main approximation region. In this embodiment, the M coefficients are stored in a second lookup table.

[0090] Leakage filters can be particularly effective for modeling the gradual decay of pulse tails, which often exhibit exponential-like behavior. Using compositions with leakage filters, method 300 can more accurately represent complex tail shapes that may not follow simple exponential decay.

[0091] This method allows for efficient computation of tailing regions without storing separate values ​​for each bin, significantly reducing memory requirements. Using separate lookup tables for these coefficients enables the system to quickly switch between different tailing models, providing flexibility for various operating conditions or impulse characteristics.

[0092] In step 308, a crosstalk pulse shape is constructed, where N represents the maximum number of lookup table entries (e.g., 8), and P represents the number of active lookup table entries (where P ≤ N, e.g., 6). This construction can use up to N values ​​from the first lookup table centered on a programmable median bin, where P controls the crossover point processed by the leakage integrator. For bins outside this region, a leakage filter defined by a second lookup table can be used to compute the values.

[0093] The hybrid approach can accurately represent the main region of the pulse while efficiently approximating the extended tail. The programmable median box provides the flexibility to align the pulse shape with observed crosstalk patterns, which may vary due to temperature or manufacturing variations.

[0094] Using a first lookup table for the central region ensures an accurate representation of the pulse peak and its surroundings, where accuracy is advantageous. The transition to a leakage filter-based operation for the tail region balances accuracy with computational efficiency, allowing the system to model extended pulse shapes without requiring extensive memory or complex calculations for each box.

[0095] The processing follows specific rules based on bin location, where N represents the maximum number of lookup table entries (e.g., 8), and P represents the number of active lookup table entries (e.g., 6, where P ≤ N). The system can use lookup table elements in bins with fewer than the number of active lookup table entries.

[0096] At the P-th bin, the system initializes two leakage integrators (e.g., leaky(0) and leaky(1)). For bins other than P, the system updates these leakage integrators by multiplying them by their corresponding coefficients (e.g., coeff(0) and coeff(1)). The leakage filter output at position P can be computed as the sum of the previous values ​​of the two leakage integrators: M coefficients define the characteristics of leaky1 and leaky2. If P is less than N, the output of the leakage filter will overwrite the corresponding lookup table value of the bin after P.

[0097] In step 310, the crosstalk pulse shape generated from step 308 is scaled using, for example, a crosstalk amplitude scalar. The scaler can be a constant or adaptive value, allowing for flexibility under different operating conditions. The scaling step facilitates adjusting the amplitude of the crosstalk correction to match the crosstalk intensity observed in actual time-of-flight measurements.

[0098] Constant scalers may be suitable for systems with stable crosstalk characteristics, providing a simple and computationally efficient approach. Adaptive scalers, on the other hand, can dynamically adjust crosstalk correction based on real-time measurements or environmental factors. Adaptive methods can account for changes in crosstalk intensity due to variations in temperature, voltage, or other operating conditions, potentially improving the accuracy of crosstalk mitigation across various scenarios. The choice between constant and adaptive scaling allows the system to balance simplicity and robustness based on the specific requirements of the application.

[0099] In step 312, the crosstalk pulse shape generated and scaled in step 310 is subtracted from the first region of the histogram. The "first region" typically refers to the early bins of the histogram, where crosstalk effects may be more pronounced due to internal reflections and scattering within the sensor package. Method 300 can isolate the true signal reflected from the target object by subtracting the modeled crosstalk shape (serially per bin).

[0100] This process improves the accuracy of distance measurements, particularly for close-range targets where crosstalk effects can be particularly destructive. Subtraction can be performed box-by-box, with the correction amplitude at each box determined by the constructed and scaled crosstalk pulse shape. This method allows for precise correction, which can interpret the specific temporal characteristics of crosstalk in the system. By applying correction early in the signal processing chain, subsequent peak detection and distance calculation algorithms can operate on a clearer and more accurate representation of the reflected signal.

[0101] Figure 3 The crosstalk pulse shape described in the paper describes how to achieve a highly compact representation of crosstalk data. Conventional methods typically require storing 128 or 144 16-bit data bins, resulting in a memory footprint of approximately 256 bytes. However, the proposed implementation reduces the required storage to, for example, 19 bytes.

[0102] Compact representations can be achieved by combining a lookup table approach with multiple (e.g., two) leakage integrators to accurately model complex pulse shapes. Variable-length (e.g., 16 to 128 bins) crosstalk pulse shapes can be generated from the same compact description using N amplitude coefficients (e.g., eight) and M leakage filter coefficients (e.g., four), while reducing memory requirements. The leakage integrators are sized to achieve, for example, a dynamic range of 10000:1 (14 bits) for faithful reproduction of pulse tails. This method saves storage space and enables efficient real-time generation of crosstalk histograms because the compact description can be rapidly decompressed and scaled on the fly during processing.

[0103] A significant reduction in memory usage, for example, from 256 bytes to 19 bytes, represents a major optimization for memory-constrained systems, allowing for effective mitigation of crosstalk without compromising the overall system's low memory footprint.

[0104] Figure 4 A block diagram of the correlator circuit 400 of an embodiment is shown. In this embodiment, the correlator circuit 400 is implemented as follows: Figure 2 The correlator circuit 202. The correlator circuit 400 includes an ambient MAC (multiply-accumulate-add) circuit 402, a filter MAC circuit 404, a crosstalk MAC circuit 406, a closest-to-target circuit 408, and a maximum-value circuit 410, which may (or may not) be arranged as shown.

[0105] The correlator circuit 400 receives inputs including ToF sensor data (i.e., reference / return histograms and crosstalk event histograms) and configuration / calibration data (i.e., filter shape and the start and end points of the histogram window). The correlator circuit 400 processes the inputs to generate an output, which includes the closest and strongest maximum values.

[0106] Advantageously, the correlator circuit 400 enables efficient processing of time-of-flight data, detecting the closest and strongest targets while taking into account ambient light and crosstalk effects. Its configuration facilitates low-power operation through shared MAC circuitry, box-serial processing, and optimized data representation.

[0107] In this embodiment, the correlator circuit 400 is an accelerated matched filter correlator with a tunable filter shape, configured to detect the strongest and most recent peaks from the ToF data measurements. The correlator circuit 400 is compatible with multiple pulse / box widths (i.e., timing options) to enable short-range (2.4 m), medium-range (e.g., 4.8 m), and long-range (e.g., 9.6 m) operation.

[0108] The filter shape defines the correlation pattern and multiple pulse widths for signal detection. In an embodiment, the filter shape input includes a set of filter coefficients (e.g., 5) representing the expected return pulse shape in the time-of-flight measurement. Each filter coefficient can be represented with a predetermined bit width (e.g., 11 bits). The coefficients can be in a signed fixed-point format (e.g., signed 1.10 format), allowing for both positive and negative values ​​with fractional precision. In an embodiment, the filter shape is tunable.

[0109] The filter coefficient set defines the filter shape used in the MAC circuit. The coefficients can be programmable, allowing the filter response to be tuned to optimize performance for different operating conditions or pulse characteristics.

[0110] Filter coefficients can be represented using a fixed-point format that balances precision and computational efficiency. For example, the system can use a 10-bit representation, where a value of 1024 corresponds to 1.0. This method allows for the representation of fractional values ​​with sufficient precision, such as 1.9 or 0.5, while maintaining computational efficiency.

[0111] For example, the MAC circuit processes histogram data and multiplies each 24-bit histogram bin value by a 10-bit filter coefficient. To maintain the correct scaling and prevent overflow, the result can be divided by 1024 (equivalent to right-shifting by 10 bits). This division effectively implements the fractional part of the filter coefficients.

[0112] Fixed-point representation enables the system to achieve high precision in filter coefficients without requiring larger storage or complex floating-point operations. This approach can be particularly advantageous in memory-constrained environments, allowing complex filtering operations to be implemented efficiently with minimal computational resources.

[0113] In this embodiment, the MAC circuit maintains a depth buffer (e.g., a five-element buffer) for the output of the most recent filter. This buffer enables zero-crossing detection without requiring the storage or access of the full histogram, thereby further contributing to the system's memory efficiency.

[0114] The reference / return histogram input represents time-of-flight data measurements collected by a ToF sensor. Each bin in the reference / return histogram corresponds to a specific time delay, and the value in each bin represents the number of photons detected at that delay. The reference histogram can be generated from a known reference signal, while the return histogram can be generated from actual reflected signals from the scene. The reference / return histogram input can consist of a variable number of bins (e.g., 32 or 128) representing multi-bit unsigned data (e.g., 24 bits) of time-of-flight measurements. The variable number of bins (e.g., 32 or 128) allows for flexibility in the temporal resolution and range of the measurements.

[0115] The crosstalk event histogram input models the effect of internal optical crosstalk, which can occur when unwanted signals are generated by adding a cover glass on top of the device. Internal optical crosstalk can be different from electro-optic detector crosstalk between adjacent SPADs. While both types of crosstalk have optical components, internal optical crosstalk refers to the signal from the optical path between the emitter and the cover glass within the assembly. In contrast, electro-optic crosstalk is a detector-level effect that can be triggered by any incoming light from internal (e.g., assembly-level) and external optical paths. The crosstalk event histogram represents the expected internal optical crosstalk pattern, and its shape is primarily determined by the VCSEL emission and SPAD response characteristics. By processing the internal optical crosstalk data separately, the correlator circuit 400 can more accurately distinguish between the real signal and noise caused by crosstalk. The crosstalk event histogram input can include several bins (e.g., 64) of multi-bit unsigned data (e.g., 24 bits) to model the crosstalk effect.

[0116] The correlator circuit 400 can process these two histograms in a bin-serial manner, allowing for efficient data processing with minimal memory requirements. A wide dynamic range is provided for the multi-bit unsigned data format (e.g., 24 bits) per bin to accurately capture both weak and strong signals. The high bit depth can be particularly advantageous for maintaining accuracy under low-light conditions and when processing reflections from objects of varying distances and reflectivities.

[0117] The correlator circuit 400 takes the start and end points of the histogram window as input parameters. These parameters define the specific region of interest (ROI) within the reference / return histogram for processing. The start point indicates the box number where the correlation analysis should begin, while the end point specifies the box number where the correlation analysis should end. This windowing approach allows the correlator circuit 400 to focus on the most relevant portion of the histogram, which can significantly reduce processing time and power consumption. This can be particularly useful in scenarios where the approximate range of the target is known, or when processing a specific range of distances to avoid processing irrelevant data. The ability to adjust the window parameters provides flexibility to adapt the correlator's operation to different measurement scenarios or to focus on specific ROIs within the time-of-flight data.

[0118] In an embodiment, the correlation circuit 400 includes an ambient fill circuit 432 and a zero-fill circuit 434 to handle edge cases during its processing. The ambient fill circuit 432 can be used at the start of the correlation process or when the operation extends beyond the limits of the available data array or window. In these cases, the circuit can fill with ambient light values ​​instead of zero. This method helps maintain the integrity of the correlation, especially at the start of processing when historical data may be unavailable. The zero-fill circuit 434 can be used for crosstalk data, filling any out-of-range bins with zeros. The filling technique enables the correlator circuit 400 to perform its operation consistently across the entire range of the input data, including at the edges of the histogram.

[0119] In one embodiment, between the ambient filling circuit 432 and the filter MAC circuit 404, the correlator circuit 400 includes subtraction and clipping operations. The subtraction operation performs an immediate subtraction of crosstalk data from the histogram data. This can be beneficial for isolating the signal of interest from crosstalk effects.

[0120] After subtraction, the pruning operation ensures that the resulting value remains non-negative. Pruning can be advantageous because subtraction can occasionally result in negative values, which are invalid in this context. The subtraction operation is performed if the histogram event is greater than the crosstalk event; otherwise, the output is zero. This method ensures that the data fed into the MAC circuit 404 of the filter is positive, thus maintaining the integrity of subsequent calculations.

[0121] The environmental MAC circuit 402, the filter MAC circuit 404, and the crosstalk MAC circuit 406 are configured to perform multiply-accumulate operations on the input data. These circuits can use, for example, multi-bit histogram box values ​​(e.g., 24 bits), multi-bit unsigned coefficients (e.g., 11 bits), and several bits for the number of coefficients (e.g., 3 bits) to produce multi-bit outputs (e.g., 38 bits).

[0122] In an embodiment, a division operation (e.g., division by 1024) is performed after each MAC circuit. This division may be part of a fixed-point arithmetic method that maintains accuracy when operating in a 32-bit domain. The filter shape coefficients may be expressed such that the divider value (e.g., 1024) corresponds to 1.0, thereby allowing for accurate representation of fractional values ​​(e.g., 1.9 or 0.5).

[0123] For example, when performing a MAC operation, a 24-bit histogram value can be multiplied by a 10-bit (1024 resolution) coefficient. In this example, applying division by 1024 (equivalent to a right shift of 10 bits) to rescale the result effectively achieves the fractional part of the filter coefficients. This method allows the correlation circuitry 400 to maintain high resolution of the filter coefficients without requiring larger storage or floating-point operations, thereby optimizing accuracy and efficiency.

[0124] In this embodiment, the correlator circuit 400 implements a bin serial processing method, processing data from one bin at a time. This method minimizes memory usage and reduces memory fetches, allowing for efficient processing with limited hardware resources.

[0125] The ambient light MAC circuit 402 calculates the ambient light contribution, the filter MAC circuit 404 performs the main correlation operation, and the crosstalk MAC circuit 406 calculates the crosstalk contribution. Each MAC circuit can use a multi-element buffer (e.g., a five-element deep buffer) to maintain a short history of processed data without requiring a large amount of memory. This enables efficient operations such as zero-crossing detection.

[0126] The correlator circuit 400 processes data along three main paths, each serving a specific purpose in the signal processing chain. The first path processes ambient light data. Starting with the ambient MAC circuit 402, it calculates the ambient light contribution (…). This value is used in signal-to-noise ratio (SNR) calculations, particularly in the molecule, from histogram correlation (). Subtract this value from the background ambient light to isolate the signal of interest.

[0127] In this embodiment, the operation is performed serially, processing one filter coefficient at a time. For each filter coefficient, the ambient MAC retrieves the current filter coefficient. It multiplies the filter coefficient by the corresponding ambient light value for the current bin (i.e., the ambient event for each bin). The result is added to an accumulator. After processing all coefficients, the final value in the accumulator represents the ambient light contribution of the current bin, taking into account the shape of the expected signal defined by the filter coefficients. This method allows ambient light calculation to be sensitive to the same signal shape sought by the principal correlation, potentially improving the accuracy of ambient light estimation and subsequent SNR calculation.

[0128] The second path processes the reference / return histogram data. It begins at the filter MAC circuit 404, where the histogram data is related to the filter shape. Histogram correlation ( The output of the histogram correlation is fed to hardware accelerator 412, where the SNR value is calculated. The output of ) is provided to the closest target circuit 408 to help identify the earliest valid signal return, and is provided to the strongest maximum value circuit 410 to identify the peak with the highest SNR.

[0129] In an embodiment, the correlator circuit 400 includes a hardware accelerator 412 for operations such as square root and division used in SNR calculation. For example, the process may involve multi-bit addition and subtraction (e.g., 28-bit), left shift of a certain number of bits (e.g., 4-bit), multi-bit square root operation (e.g., 32-bit) that produces a multi-bit output (e.g., 16-bit), and multi-bit division (e.g., 32-bit).

[0130] In this embodiment, the hardware accelerator 412 is a dedicated circuit designed to efficiently perform the complex mathematical operations required for SNR calculation. It consists of multiple sub-components, including subtraction circuits, addition circuits, square root circuits, and division circuits.

[0131] SNR calculation is performed for each bin in the histogram (defined by the start and end windows), allowing the correlator circuit 400 to evaluate signal quality across the entire time-of-flight measurement range. Bin-by-bin SNR calculation is beneficial for accurately identifying the closest and strongest targets in the scenario.

[0132] The third path processes crosstalk data. Starting from the crosstalk MAC circuit 406, it calculates the crosstalk contribution ( This value is used in the denominator of the SNR calculation and added to the histogram correlation (). ), to take all noise sources into account before the square root operation.

[0133] Hardware accelerator 412 uses the output from the MAC circuitry to calculate the SNR for each bin, creating an SNR distribution map across the histogram. The SNR calculation can be performed using the following equation: ,in This represents the relevant values ​​of the histogram data. This indicates the correlation value of ambient light, and This represents the correlation value for crosstalk.

[0134] In an embodiment, to achieve this equation, the hardware accelerator 412 obtains information from the histogram correlation (…). Subtract environmental relevance from ) This step isolates the signal of interest from the background ambient light. Next, it performs histogram correlation analysis (…). Add to crosstalk correlation ( ( ), to take into account all noise sources.

[0135] Hardware accelerator 412 performs a square root operation to scale the photon shot noise (Poisson standard deviation) term in the SNR calculation. The square root operation can be implemented using a dedicated hardware accelerator to optimize speed and power efficiency.

[0136] Finally, the hardware accelerator 412 will move the molecules ( Divide by the denominator ( To obtain the SNR value.

[0137] The closest target circuit 408 determines the closest detection target. It uses the current correlation ( Results and previous correlations ( The earliest valid signal return is identified by the value of the closest target. The output of the closest target circuit (i.e., the closest maximum value) can include the closest maximum value, which provides information such as the start bin, end bin, peak position, and SNR of the closest detected target. For closest target detection, the closest target circuit 408 uses the output from the third path (i.e., the histogram correlation value) to identify the earliest valid signal return. It analyzes the data box by box, looking for the first occurrence of a valid signal exceeding a predetermined threshold. The circuit also considers the SNR value calculated for each box to ensure that the detected target is not just noise.

[0138] In an embodiment, the closest detection target determination is performed using the method disclosed in U.S. Patent Application 18 / 176,163 (Attorney’s File No. ST-22-EDI-0847US01), which is assigned to the same entity as this disclosure and is incorporated herein by reference in its entirety.

[0139] The strongest maximum value circuit 410 identifies the peak with the highest SNR and stores its location and SNR value. This provides information about the strongest detected target. The output of the strongest maximum value circuit 410 (i.e., the strongest maximum value) can provide information such as the peak location and SNR of the strongest detected target. The output can be represented using various bit widths suitable for the application.

[0140] In this embodiment, the strongest maximum circuit 410 identifies and stores information about the strongest signal detected in the histogram. As the correlator circuit 400 processes the histogram data box by box, the hardware accelerator 412 calculates the SNR for each box. The strongest maximum circuit 410 tracks the highest SNR value seen so far and its corresponding box location.

[0141] The newly calculated SNR is compared to the stored maximum SNR for each new bin being processed. If the new SNR is higher than the stored maximum, it becomes the new maximum. In this case, the new maximum SNR value and the bin location where that maximum occurred (the peak location) replace the existing data. However, if the new SNR is not higher than the stored maximum, the previously stored maximum SNR and its corresponding location are retained. This process continues for all bins in the histogram. At the end of the processing, the strongest maximum circuit 410 identifies the bin with the highest SNR across the entire histogram.

[0142] The output of the maximum value circuit 410 consists of a peak location and a calculated SNR value. The peak location corresponds to the bin number with the highest SNR. In an embodiment, it can be represented as an 8-bit value, allowing up to 256 different bin locations. The calculated SNR corresponds to the actual SNR value at the peak location. In an embodiment, it can be represented in a fixed-point format (e.g., 12.4 format) to balance range and precision when representing SNR.

[0143] The strongest signal information may be advantageous for identifying the most prominent object in a scene, which may not always be the closest object.

[0144] Correlator circuit 400 maintains several pieces of information for storage and output. For the closest target, it stores the start bin, end bin, peak position, and SNR. These can be derived from the closest target circuit 408 and the corresponding SNR calculation. For the strongest target, it stores the peak position and SNR. These can be determined by the strongest maximum value circuit 410, which identifies the bin with the highest SNR value.

[0145] The bin serial processing method can be applied across all three paths. It allows the circuit to process one bin at a time while maintaining the relationship between the histogram, environment, and crosstalk data for each bin. This method achieves efficient processing with minimal memory requirements.

[0146] The correlator circuit 400 terminates by outputting the closest maximum value (from the closest target circuit 408) and the strongest maximum value (from the maximum value circuit 410), providing key information about the detected target, including its location and signal strength.

[0147] In this embodiment, the input to the correlator circuit includes reference / return histogram data, representing time-of-flight measurements collected by the sensor. Each bin in this histogram corresponds to a specific time delay, and the value in each bin represents the number of photons detected at that delay. The correlator circuit uses a crosstalk event histogram to model the effects of optical and electrical crosstalk in the ToF sensor.

[0148] Additional inputs include filter shapes and histogram window parameters (start and end points) that define the correlation pattern of signal detection. These parameters allow Method 500 to focus on the most relevant parts of the histogram, thereby reducing processing time and power consumption.

[0149] Figure 5 This is a flowchart of an embodiment method 500 for processing Time-of-Flight (ToF) sensor input data in a box-serial manner. In this embodiment, a correlator circuit 400 is used to implement method 500. Unlike previous methods that process the histogram as an entire array in memory, method 500 performs histogram operations serially in a single pass. The correlator circuit traverses the histogram, calculating values ​​along the way. This method significantly reduces memory requirements and enables real-time processing of time-of-flight data.

[0150] In step 502, the correlator circuit 400 receives and prepares its inputs. The inputs include a filter shape that defines the correlation pattern for signal detection; an ambient event for each bin representing the background light level; a reference / return histogram containing time-of-flight measurement data; and a crosstalk histogram that models internal reflection effects in the sensor.

[0151] The ambient MAC circuit 402, filter MAC circuit 404, and crosstalk MAC circuit 406 each receive their respective inputs. The ambient fill circuit 432 and zero fill circuit 434 prepare the reference / return histogram input and crosstalk histogram input for processing. In this embodiment, subtraction and clipping operations refine the data before it enters the filter MAC circuit 404.

[0152] The preparation steps ensure that the input data is adequately adjusted in subsequent processing stages, laying the foundation for accurate time-of-flight measurement and target detection.

[0153] In step 504, the correlator circuit 400 uses MAC circuitry to perform parallel or serial processing: an ambient MAC circuit 402, a filter MAC circuit 404, and a crosstalk MAC circuit 406. Each MAC circuit operates on its input data to produce an instantaneous correlation value. The MAC circuitry implements serialized histogram processing, allowing bin-by-bin operation without storing the entire histogram in memory. This circuitry can compute correlations and apply corrections in real time by processing data in this way as it traverses the histogram bins.

[0154] The ambient MAC circuit 402 processes the filter shape and ambient events for each box. It multiplies each ambient event value by the corresponding filter coefficient and sums the results. This operation quantifies the contribution of ambient light to the overall signal, thereby generating... Output.

[0155] The filter MAC circuit 404 operates on the preprocessed reference / return histogram input. It applies filter coefficients to the reference / return histogram input, effectively correlating the measured signal with the expected pulse shape. This process helps isolate the actual reflected signal from noise, thereby producing... Output.

[0156] The crosstalk MAC circuit 406 processes the zero-filled crosstalk histogram. The filter coefficients are applied to the pre-processed crosstalk histogram input to quantify the effect of internal reflections within the sensor, thereby generating... Output.

[0157] In this embodiment, each MAC circuit is followed by a divider operation. This division reduces the accumulated result to prevent overflow in fixed-point arithmetic systems. Furthermore, it effectively implements the fractional part of the filter coefficients, allowing for precise representation of coefficient values ​​between -1 and 1.

[0158] Fixed-point arithmetic facilitated by divider operations enables the correlator circuit to maintain high precision when operating within the constraints of the 32-bit operational domain. This method achieves a balance between accuracy and computational efficiency for real-time processing in compact, low-power devices.

[0159] The output of the MAC circuit (i.e., , and This forms the basis for subsequent processing steps, including SNR calculation and target detection. In this embodiment, by performing correlation in parallel, the circuitry efficiently processes time-of-flight data, which is used for accurate distance measurement and target identification.

[0160] The filter operations used to enhance the signal and suppress noise are applied in real time as the circuit processes each bin. Instead of storing the entire filtered histogram, the circuit maintains a small buffer of the most recent filter output. The number of filter coefficients determines the buffer size, which is typically much smaller than the full histogram, thus reducing the circuit's memory footprint. In the pulse-weighted histogram implementation, the buffer achieves maximum detection through first-order derivative analysis, using ±1 filters to approximate the gradient by calculating the inter-bin differences.

[0161] In step 506, hardware accelerator 412 performs signal-to-noise ratio (SNR) calculation using the output from the MAC circuitry. The calculation is performed on-the-fly for each box, eliminating the need to store the SNR values ​​for the entire histogram, further demonstrating the efficiency of the serialization process. The SNR value provides a measure of signal quality for each box, enabling the system to distinguish between true reflections and noise, and accurately identify the strongest and closest targets.

[0162] It receives three main inputs: from the ambient MAC circuit 402. , from the filter MAC circuit 404 and from the crosstalk MAC circuit 406 .

[0163] Hardware accelerator 412 processes the input to calculate the SNR for each bin in the histogram. It implements the formula: . Input the signal of interest, subtract To remove the contribution of ambient light, and to Add to the denominator to account for noise from crosstalk. The square root operation in the denominator scales the noise term.

[0164] In step 508, the strongest maxima circuit 410 operates to identify the peak with the highest signal-to-noise ratio (SNR) in the time-of-flight measurement. It receives input from hardware accelerator 412, which provides the SNR value for each bin in the histogram. This circuit also accesses input from the filter MAC circuit 404. The value allows it to correlate the peak SNR with its corresponding histogram position.

[0165] As the highest SNR circuit 410 processes these inputs, it continuously compares the current SNR value with the highest SNR encountered so far. When a new maximum value is found, the circuit updates its stored value, tracking the bin location (peak location) where the maximum value occurred and the corresponding SNR value.

[0166] The output of the strongest maximum value circuit 410 is the strongest maximum value, which typically includes the peak location and the calculated SNR value of the strongest detected signal. This information is useful for applications that identify the most prominent object in a scene, which may not always be the closest object. The strongest maximum value data is also fed back to the nearest target circuit 408, providing context for detecting the nearest target.

[0167] In step 510, the closest-to-target circuit 408 processes the data to identify the nearest significant reflection in the time-of-flight measurement. In an embodiment, it receives several inputs to perform this task: the current... Value, previous related values ​​( The SNR output from the hardware accelerator 412 and the maximum value information from the maximum value circuit 410.

[0168] The closest-to-target circuit 408 analyzes the input on a box-by-box basis, searching for the earliest occurrence of a valid signal exceeding a predetermined threshold. It will then... Value and previous related values ​​( The circuit compares the data to detect rising edges in the relevant data, which indicate the start of a potential target reflection. The circuit also considers the SNR value to ensure that the detected signal is real and not just noise. The strongest maximum value information from step 508 provides additional context, allowing the closest-to-target circuit 408 to refine its detection criteria.

[0169] When processing data, the closest-to-target circuit 408 tracks key parameters related to the most recently detected target. These can include the start box (where the signal first exceeds a threshold), the end box (where the signal drops below a threshold), the peak position (the box with the highest correlation within the target), and the corresponding SNR value.

[0170] The output of the closest target circuit 408 is the closest maximum value, which encapsulates information about the most recently detected target. This output provides data for applications requiring precise measurement of the distance to the nearest object, such as collision avoidance systems or short-range gesture recognition. When measuring complex objects with varying depths (such as sloping surfaces), this circuit can identify the distance to the nearest portion of the object, making it particularly effective in collision avoidance applications. By efficiently identifying the closest target, this circuit enables time-of-flight sensor systems to respond quickly to nearby objects, enhancing their effectiveness in real-time sensing applications.

[0171] Method 500 achieves several advantages by processing histogram data serially, bin by bin. It significantly reduces memory requirements by eliminating the need to store complete histograms or intermediate results. This method enables real-time, on-demand processing of time-of-flight data, including complex operations such as correlation, crosstalk correction, and SNR calculation. The serialization method allows for efficient implementation in resource-constrained hardware, making it particularly suitable for low-power, compact ToF sensor systems.

[0172] It should be noted that not all steps outlined in the flowchart of Method 500 are necessarily required and may be optional. Furthermore, it is similarly conceivable to change the arrangement of steps, remove one or more steps and path connections, and add steps and path connections.

[0173] Figure 6 A block diagram of an embodiment of the phase / box operation circuit 600 is shown. In this embodiment, the phase / box operation circuit 600 is implemented as follows: Figure 2 The phase / box operation circuit 204. The phase / box operation circuit 600 includes an environment fill circuit 602, a zero fill circuit 604, a filter MAC circuit 606, and a positive zero crossing detection circuit 608, which may (or may not) be arranged as shown.

[0174] In one embodiment, combining the phase / box operation circuit 600 with the correlator circuit 400 can produce a single circuit. Integration can reduce overall circuit complexity and improve efficiency by sharing common components and minimizing data transfer between individual blocks.

[0175] The inputs to the phase / box operation circuit 600 include ToF sensor data (i.e., reference / return histogram and crosstalk event histogram) and configuration / calibration data (i.e., filter coefficients, peak positions, and offset parameters). The reference / return histogram inputs and crosstalk event histogram inputs are similar to those discussed as inputs to the correlator circuit 400, and will not be repeated for the sake of brevity.

[0176] The phase / bin operation circuit 600 processes data in a bin-serial manner, traversing each bin within a defined search and pulse window. This method minimizes memory requirements and allows for efficient processing with limited hardware resources.

[0177] In this embodiment, firmware 610 implemented within the phase / box operation circuit 600 receives several input parameters to define the processing window for the pulse width histogram operation. The inputs include peak position inputs provided by the correlator circuit 400 and configuration parameters, including search start offset, search end offset, window start offset, and window end offset.

[0178] Firmware 610 interfaces with the hardware components in the phase / box operation circuit 600 and the advanced control system. For example, implemented as embedded software stored in non-volatile memory, firmware 610 can initialize and configure the hardware components of the phase / box operation circuit, including setting filter coefficients for the filter MAC circuit 606. It receives and processes peak position and offset parameters to calculate the boundaries of the search and pulse windows.

[0179] Firmware 610 can also utilize the output of the positive zero-crossing detection circuit 608 to implement an interpolation algorithm for accurate median phase calculation. In the absence of a positive zero-crossing detection, firmware 610 can execute a backoff option to implement an alternative phase estimation method.

[0180] In this embodiment, the median phase calculation can be implemented in hardware or firmware.

[0181] Search start offset, search end offset, window start offset, and window end offset are configuration parameters used to define the processing window for pulse width histogram operations. The offsets, in conjunction with the peak position, determine the specific region of the reference / return histogram input to be analyzed.

[0182] The search start offset and search end offset define the boundaries of the search window relative to the peak position. The search start offset can be negative (e.g., -2), indicating how many bins before the peak position the search begins. The search end offset can be positive (e.g., +2), specifying how many bins after the peak position the search ends. The search window is used to locate the median bins within a narrow range around the detected peak.

[0183] The window start and end offsets define a wider pulse window relative to the peak position. The window start offset can be a larger negative value (e.g., -3) compared to the search start offset, while the window end offset can be a larger positive value (e.g., +4). The wider window covers most of the pulse shape, ensuring the full range of the pulse is captured for accurate phase calculations.

[0184] The offset parameters allow the system to adapt to different pulse widths and shapes. By adjusting the offset, the phase / box operation circuit 600 can focus on the most relevant portion of the histogram, optimizing processing efficiency while maintaining accuracy. Configuring these offsets provides flexibility to adapt to various operating conditions and target detection scenarios.

[0185] Firmware 610 uses these inputs to calculate the boundaries of the search and pulse windows. The search window can be defined as:

[0186] Search start = Search start offset + Peak position

[0187] Search end = Search end offset + Peak position.

[0188] Similarly, a pulse window can be defined as:

[0189] Pulse start = Window start offset + Peak position

[0190] Pulse end = window end offset + peak position.

[0191] The calculated search start, search end, pulse start, and pulse end values ​​are provided as inputs to the phase / bin operation circuit 600. These values ​​define the range of bins that the hardware components will process.

[0192] Similar to environment fill circuit 432, environment fill circuit 602 receives environment event data for each bin and fills the reference / return histogram input data when processing exceeds the available data. Similar to zero fill circuit 434, zero fill circuit 604 uses a zero-fill crosstalk histogram input for bins that exceed the range.

[0193] In this embodiment, similar to correlator circuit 400, the phase / box operation circuit 600 includes subtraction and clipping operations between environment fill circuit 602 and filter MAC circuit 606. The subtraction operation performs an immediate subtraction of crosstalk from the reference / return histogram input. This can be beneficial for isolating the signal of interest from crosstalk effects. After subtraction, the clipping operation ensures that the resulting value remains non-negative. Clipping can be advantageous because subtraction may occasionally result in negative values, which are invalid in this context. The subtraction operation is performed if the reference / return histogram input event is greater than the crosstalk event; otherwise, the output is zero. This method ensures that the data fed into filter MAC circuit 606 is positive, thereby maintaining the integrity of subsequent calculations.

[0194] The filter MAC circuit 606 applies filter coefficients box-by-box within a search and pulse window defined, for example, by firmware, to the preprocessed reference / return histogram input. The filtering operation helps isolate and enhance relevant pulse information from the overall histogram data, thereby facilitating subsequent detection of the median phase of the reflected light pulse.

[0195] Filter coefficients define the shape of the filter applied to histogram data to enhance certain features and suppress others. In an embodiment, the filter MAC circuit 606 uses the filter coefficients in a convolution-like operation with the preprocessed reference / return histogram input. For each bin in the processing window, the filter MAC circuit 606 multiplies the histogram bin value by each signed filter coefficient. The results of the multiplication are accumulated over the coefficient values. This operation effectively applies a weighted average or filtering effect to the preprocessed reference / return histogram input.

[0196] In this embodiment, the filter shape created by the filter coefficients is designed to have a specific response that facilitates the detection of the median phase of the pulse. Typically, the filter coefficients are arranged to create a shape that transitions from positive to negative values, thereby allowing the detection of zero-crossings corresponding to the median phase of the pulse.

[0197] In this embodiment, the pulse width half-width parameter is used to generate filter coefficients. This parameter affects the characteristics of the filter applied to the reference / return histogram input. In pulse-weighted histogram processing, "half-width" refers to approximately half the expected width of the pulse in the histogram. The pulse-weighted half-width parameter helps to customize the filter MAC circuit 606 to match the expected pulse shape and width in the reference / return histogram input.

[0198] When generating filter coefficients, the half-width at half-maximum (WWHM) parameter determines how many positive and negative coefficients and their relative values ​​are used. For example, a larger WWHM value produces a wider filter with more coefficients, suitable for handling wider pulses. Conversely, a smaller WWHM value may result in a narrower filter, better suited for sharper pulses.

[0199] By adjusting the half-width parameter, the system can optimize the filter response to better match the characteristics of the expected pulse, potentially improving the accuracy of median phase detection. This adaptability allows the phase / box operation circuit 600 to be tuned for different operating conditions or pulse shapes encountered in various time-of-flight measurement scenarios.

[0200] Using a filtering method, the phase / box operation circuit 600 can efficiently process the reference / return histogram input to extract timing information for accurate distance measurement, while minimizing computational resources. The combination of the box serial processing method and the compact representation of the filter coefficients allows for efficient implementation in hardware with limited memory resources.

[0201] The output of the filter MAC circuit 606 is coupled to the positive zero-crossing detection circuit 608. Based on the analysis of the results from the filter MAC circuit 606, the positive zero-crossing detection circuit 608 generates several outputs. These include the previous filter value, the current filter value, the detected median flag, and the median bin.

[0202] The positive zero-crossing detection circuit 608 compares the current output value from the filter MAC circuit 606 with the previous output value to detect a zero crossing. When a positive zero crossing is detected, the current and previous filter values ​​are stored.

[0203] In this embodiment, the detected median flag indicates whether a valid median has been detected. When the positive zero-crossing detection circuit 608 detects a positive zero-crossing in the filtered data, it sets a binary output to indicate that the midpoint of the pulse has been detected. The detected median flag confirms that a valid pulse median has been identified, and median phase calculation can be performed.

[0204] The median bin output indicates the bin number where a positive zero-crossing occurred. This value corresponds to the approximate location of the pulse median in the histogram. The median bin can be an integer value that identifies a specific histogram bin where the filtered data transitions from negative to positive, indicating the center of the pulse.

[0205] The firmware 610 of the phase / box operation circuit 600 combines the discovered median flag and median box output with the previous and current filter values ​​to perform interpolation to determine the median phase.

[0206] In an embodiment, firmware 610 uses a first interpolation formula to calculate the median phase, which can be expressed as: In this embodiment, the median phase output is represented as a 19-bit unsigned value in 8.11 format, providing a balance between range and accuracy for phase measurements.

[0207] In an embodiment, the phase calculation focuses on the rising edge of the detected pulse, as disclosed in U.S. Patent Application 18 / 304,589 (Attorney’s File No. ST-22-EDI-1099US01), which is assigned to the same entity as this disclosure and is incorporated herein by reference in its entirety.

[0208] This method can be effective in cases with multiple reflected or overlapping pulses. The filter MAC circuit 606 applies carefully designed filter coefficients to the preprocessed histogram data, emphasizing the rising edge of the pulse. The positive zero-crossing detection circuit 608 identifies the point where the filtered data transitions from negative to positive, corresponding to the median phase of the rising edge.

[0209] In an embodiment, if no positive zero crossing is detected (i.e., zero crossing search fails), which indicates a challenging scenario such as merging targets, the fallback option (i.e., expansion) can achieve consistent ranging even when the standard method fails.

[0210] In the rollback option, firmware 610 uses the offset maximum condition as an approximation of the first derivative (i.e., The median phase is derived using the [interpolation method]. The backoff option provides a suitable estimate of the median phase. The second interpolation equation represents the filtering operation applied to histogram data in the histogram circuit. It describes a discrete approximation of the first derivative of the histogram data.

[0211] The convolution formula operation can be represented as ,in This represents the derivative (rate of change) of the histogram data with respect to time. This represents the convolution operation. h This indicates the input histogram data, and [ ] represents the convolution kernel or filter coefficients.

[0212] In practice, the median phase is interpolated in the boundary region of the maximum value derived from the maximum estimated offset using the second interpolation formula. Maximum bin ( max_bin The value represents the bin with the highest value after filtering. The offset is a predefined value (usually +1) used to select the bin adjacent to the maximum value for interpolation.

[0213] Two time points were calculated:

[0214] T1 = max_bin + offset

[0215] T2 = max_bin + offset + 1.

[0216] The time point represents the bin after the detected maximum value, where the pulse shape is expected to change rapidly. The phase is interpolated within the boundary region of the maximum value.

[0217] In this embodiment, the second interpolation may follow the formula ,in and It is the time value corresponding to the selected box, and and These are the filtered histogram values ​​at these bins.

[0218] This method allows for a reasonable estimation of the median range, even when standard zero-crossing methods fail, such as in the case of merged target clusters. Finally, the phase / box operation circuit outputs the calculated median phase.

[0219] Figure 7 A flowchart illustrating an embodiment of method 700 for instantaneous median estimation of a potential target using, for example, a simplified pulse-weighted histogram method. In this embodiment, method 700 is implemented using a phase / box operation circuit 600. Method 700 represents median phase calculation, which enables efficient time-of-flight data processing without requiring a full microcontroller and its associated cost and size constraints.

[0220] In step 702, the phase / box operation circuit 600 receives input data, including a reference / return histogram, a crosstalk event histogram, filter coefficients, peak positions, and offset parameters. The filter coefficients define the shape of the filter to be applied to the reference / return histogram input. In this embodiment, the peak position is provided by the correlator circuit 202 and serves as the starting point for analysis. The offset parameters include search start offset, search end offset, window start offset, and window end offset, which define the processing window. The reference / return histogram input and the crosstalk event histogram input are similar to those previously discussed and will not be repeated for the sake of brevity.

[0221] When the processing exceeds the available data, the environment fill circuit 602 applies environment fill to the reference / return histogram input using per-box environment event information. The zero-fill circuit 604 applies zero fill to the out-of-range box-to-crosstalk histogram input.

[0222] Instantaneous crosstalk subtraction can be performed on the reference / return histogram data. Subtraction is performed if the reference / return histogram event is greater than the crosstalk event; otherwise, the output is set to zero. A pruning operation can be followed to ensure all values ​​remain non-negative. Preprocessing ensures consistent data processing at the defined window edges and isolates the signal of interest from crosstalk effects.

[0223] In step 704, firmware 610 defines a search and pulse window based on the peak position and offset parameter inputs. A narrower search window is typically used to locate the median bin within the focusing range around the detected peak. A wider pulse window typically ensures the capture of the full range of pulses for accurate phase calculations. The calculated values ​​define the range of bins that the hardware components will process.

[0224] In step 706, filter coefficients are applied to the preprocessed reference / return histogram input by performing convolution-like operations to enhance the relevant impulse information, and the filter MAC circuit 606 performs the filtering operation.

[0225] When convolved with histogram data, the filter effectively computes the difference between the sum of four consecutive bins and the sum of the next four consecutive bins. This operation highlights rapidly changing regions in the histogram data, corresponding to the rising and falling edges of the pulses. The zero-crossing points of the filtered data (transitioning from positive to negative) correspond to the median of the pulses in the original histogram.

[0226] Using a filtering method, this circuit can efficiently detect the pulse median without processing the entire histogram simultaneously, thus allowing for box-by-box processing and reducing memory requirements.

[0227] The filter MAC circuit 606 can apply filter coefficients to preprocessed histogram data bin-by-bin within a defined search and pulse window. The filtering operation helps isolate and enhance relevant pulse information from the overall histogram data. The filter coefficients define the filter shape transitioning from positive to negative values, allowing detection of zero-crossings corresponding to the pulse median phase. The half-width parameter of the pulse width can be used to generate the filter coefficients, customizing the filter to match the desired pulse shape and width.

[0228] In this embodiment, the filter MAC circuit is a 30-bit circuit that multiplies 24-bit histogram bin data by 2-bit signed filter coefficients and sums the results across four coefficient values. This method efficiently processes histogram data to extract timing information for accurate distance measurement while minimizing computational resources.

[0229] In step 708, the positive zero-crossing detection circuit 608 searches for positive zero-crossings in the filtered data and processes the results. This step effectively combines zero-crossing detection and result storage operations, simplifying the process of identifying the pulse median.

[0230] The positive zero-crossing detection circuit compares the current output value from the filter MAC circuit 606 with the previous output value. In an embodiment, when a positive zero-crossing is detected, the median of the indication pulse is stored, the current and previous filter values ​​are stored, a detected median flag is set, and the median bin position is recorded.

[0231] The detected median flag confirms that a valid pulse median has been identified, and the median phase calculation can proceed to step 710. The median bin output indicates the bin number where a positive zero-crossing occurred, corresponding to the approximate location of the pulse median in the histogram. However, if the positive zero-crossing detection circuit 608 does not detect a positive zero-crossing, the method moves to step 712.

[0232] In step 710, firmware 610 calculates the median phase by interpolating the current and previous filter values. This calculation provides a finer resolution than individual box granularity.

[0233] In step 712, if no positive zero-crossing is found (indicating a challenging scenario such as merging targets), a fallback option (i.e., expansion) is used to calculate the median phase even if the standard method fails. This method allows for a reasonable estimate of the median phase, even when the standard zero-crossing method fails, such as in the case of merging target clusters.

[0234] It should be noted that not all steps outlined in the flowchart of Method 700 are required and may be optional. Furthermore, it is similarly conceivable to change the arrangement of steps, remove one or more steps and path connections, and add steps and path connections.

[0235] The first aspect relates to a time-of-flight sensor system, comprising: a light emitter configured to emit light pulses toward a target; a detector array configured to detect light pulses reflected from the target; and a histogram processing circuit coupled to the detector array and configured to: receive time-of-flight measurement data from the detector array, process the measurement data from the detector array using continuous bin-by-bin histogram processing, and apply one or more instantaneous operations during continuous bin-by-bin histogram processing.

[0236] In a first implementation of the time-of-flight sensor system, according to the first aspect itself, the histogram processing circuit includes a correlator circuit comprising: an ambient multiply-accumulate (MAC) circuit configured to calculate the ambient light contribution; a filter MAC circuit configured to perform a main correlation operation on the time-of-flight measurement data; and a crosstalk MAC circuit configured to calculate the crosstalk contribution, wherein the correlator circuit is configured to process the time-of-flight measurement data in a box-serial manner.

[0237] In a second implementation of the time-of-flight sensor system, according to the first aspect itself or any prior implementation of the first aspect, the correlator circuitry further includes: a closest target circuit coupled to the filter MAC circuitry and configured to determine the closest detection target; and a maximum value circuit coupled to the filter MAC circuitry and configured to identify the peak value with the highest signal-to-noise ratio (SNR).

[0238] In a third implementation of the time-of-flight sensor system, according to the first aspect itself or any prior implementation of the first aspect, the histogram processing circuitry includes a phase / box operation circuitry comprising: an ambient fill circuitry configured to fill input data with ambient light values ​​in response to processing extended beyond available time-of-flight measurement data; a zero-fill circuitry configured to zero-fill crosstalk data for out-of-range boxes; a filter multiply-accumulate (MAC) circuitry configured to apply filter coefficients to preprocessed histogram data within a defined search and pulse window; and a positive zero-crossing detection circuitry coupled to the filter MAC circuitry and configured to detect positive zero-crossings in the filtered data output by the filter MAC circuitry.

[0239] In a fourth implementation of the time-of-flight sensor system, depending on the first aspect itself or any previous implementation of the first aspect, the phase / box operation circuit is further configured to calculate the median phase by interpolating the current and previous filter values ​​when a positive zero crossing is detected.

[0240] In a fifth implementation of the time-of-flight sensor system, the histogram processing circuitry is further configured, according to the first aspect itself or any prior implementation of the first aspect, to: generate a crosstalk histogram using linear interpolation based on crosstalk calibration data and silicon temperature data; and to adaptively update the crosstalk correction during operation.

[0241] In a sixth implementation of the time-of-flight sensor system, depending on the first aspect itself or any prior implementation of the first aspect, the histogram processing circuit is configured to process the time-of-flight measurement data using a sliding window of histogram data.

[0242] In a seventh implementation of the time-of-flight sensor system, the histogram processing circuitry is configured to perform full histogram processing capabilities, depending on the first aspect itself or any prior implementation of the first aspect, while maintaining a low-power target for integration into a battery-powered device.

[0243] In an eighth implementation of the time-of-flight sensor system, according to the first aspect itself or any prior implementation of the first aspect, the time-of-flight measurement data includes histogram data comprising: a plurality of bins, each bin corresponding to a time delay; and a value in each bin representing the number of photons detected at the corresponding time delay, wherein the histogram data represents the distribution of photon arrival times from the reflected light pulse.

[0244] In a ninth implementation of the time-of-flight sensor system, depending on the first aspect itself or any prior implementation of the first aspect, applying one or more instantaneous operations includes: removing crosstalk in a predefined region of the histogram data; tracking and detecting a first peak in the histogram data; and tracking and detecting the largest peak in the histogram data.

[0245] The second aspect relates to a method for processing time-of-flight sensor data, the method comprising: emitting a light pulse toward a target; detecting a light pulse reflected from the target; receiving time-of-flight measurement data based on the detected reflected light pulse; processing the time-of-flight measurement data using sequential bin-by-bin histogram processing, wherein sequential bin-by-bin histogram processing includes sequentially processing individual bins of a histogram, maintaining the state of a predetermined number of recently processed bins, and generating output data for each bin before proceeding to subsequent bins; and applying one or more real-time operations during sequential bin-by-bin histogram processing.

[0246] In a first implementation of the method, according to the second aspect itself, processing the time-of-flight measurement data includes: calculating the ambient light contribution by an ambient multiply-accumulate (MAC) circuit; performing a master correlation operation on the time-of-flight measurement data by a filter MAC circuit; and calculating the crosstalk contribution by a crosstalk MAC circuit, wherein the time-of-flight measurement data is processed in a box-serial manner.

[0247] In a second implementation of the method, depending on the second aspect itself or any prior implementation of the second aspect, the method further includes: using the closest target circuit coupled to the filter MAC circuit to determine the closest detection target; and using the strongest maximum value circuit coupled to the filter MAC circuit to identify the peak with the highest signal-to-noise ratio (SNR).

[0248] In a third implementation of the method, processing the time-of-flight measurement data, according to the second aspect itself or any prior implementation of the second aspect, includes: filling the input data with ambient light values ​​in response to processing extended beyond the available time-of-flight measurement data; zero-filling crosstalk data for out-of-range bins; applying filter coefficients to preprocessed histogram data within a defined search and pulse window; and detecting positive zero crossings in the filtered data output by applying the filter coefficients.

[0249] In a fourth implementation of the method, depending on the second aspect itself or any previous implementation of the second aspect, the method further includes calculating the median phase by interpolating the current and previous filter values ​​in response to the detection of a positive zero crossing.

[0250] In a fifth implementation of the method, depending on the second aspect itself or any prior implementation of the second aspect, the method further includes: generating a crosstalk histogram using linear interpolation based on crosstalk calibration data and silicon temperature data; and adaptively updating the crosstalk correction during operation.

[0251] In a sixth implementation of the method, processing time-of-flight measurement data includes using a sliding window with histogram data, depending on the second aspect itself or any previous implementation of the second aspect.

[0252] In a seventh implementation of the method, depending on the second aspect itself or any previous implementation of the second aspect, the method performs full histogram processing capabilities while maintaining a low-power target for integration into battery-powered devices.

[0253] In an eighth implementation of the method, according to the second aspect itself or any previous implementation of the second aspect, the time-of-flight measurement data includes histogram data comprising: a plurality of bins, each bin corresponding to a time delay; and a value in each bin representing the number of photons detected at the corresponding time delay, wherein the histogram data represents the distribution of photon arrival times from the reflected light pulse.

[0254] In a ninth implementation of the method, applying one or more immediate operations, depending on the second aspect itself or any prior implementation of the second aspect, includes: removing crosstalk in a predefined region of the histogram data; tracking and finding a first peak in the histogram data; and tracking and finding the largest peak in the histogram data.

[0255] The third aspect relates to a correlator circuit for a time-of-flight sensor system configured to process input data in a box-serial manner, the correlator circuit comprising: an ambient multiply-accumulate (MAC) circuit configured to calculate the ambient light contribution; a filter MAC circuit configured to perform a main correlation operation on the input histogram data; a crosstalk MAC circuit configured to calculate the crosstalk contribution; a nearest target circuit coupled to the filter MAC circuit and configured to determine the nearest detected target; and a maximum value circuit coupled to the filter MAC circuit and configured to identify the peak with the highest signal-to-noise ratio (SNR).

[0256] In a first implementation of the correlator circuit, according to the third aspect itself, the input histogram data includes: a plurality of bins, each bin corresponding to a time delay; a value in each bin representing the number of photons detected at the corresponding time delay; and wherein the input histogram data represents the distribution of photon arrival times from the reflected light pulse.

[0257] In a second implementation of the correlator circuit, depending on the third aspect itself or any previous implementation of the third aspect, the environment MAC circuit, the filter MAC circuit, and the crosstalk MAC circuit are configured to perform multiply-accumulate operations on the multi-bit histogram box values ​​using multi-bit unsigned coefficients.

[0258] In a third implementation of the correlator circuit, depending on the third aspect itself or any previous implementation of the third aspect, the correlator circuit also includes a division circuit configured to perform a division operation after each MAC circuit to perform fixed-point arithmetic within a predetermined bit field.

[0259] In a fourth implementation of the correlator circuit, depending on the third aspect itself or any previous implementation of the third aspect, the filter MAC circuit is configured to apply filter coefficients to the input histogram data, which define the filter shape transitioning from positive to negative values ​​to detect zero crossings corresponding to the median phase of the pulse.

[0260] In a fifth implementation of the correlator circuit, depending on the third aspect itself or any previous implementation of the third aspect, the correlator circuit also includes a signal-to-noise ratio (SNR) calculation circuit configured to calculate the SNR of each bin in the input histogram data.

[0261] In the sixth implementation of the correlator circuit, the SNR calculation circuit is configured based on the equation, depending on the third aspect itself or any previous implementation of the third aspect. Calculate SNR, where This indicates the relevant values ​​of the input histogram data. Indicates the contribution of ambient light, and This indicates the crosstalk contribution.

[0262] In the seventh implementation of the correlator circuit, depending on the third aspect itself or any previous implementation of the third aspect, the closest target circuit is configured to analyze the output of the box-by-box filter MAC circuit to identify the first occurrence of a valid signal exceeding a predetermined threshold.

[0263] In the eighth implementation of the correlator circuit, depending on the third aspect itself or any previous implementation of the third aspect, the strongest maximum value circuit is configured to track the highest SNR value and the corresponding bin position when the correlator circuit processes the input histogram data.

[0264] In a ninth implementation of the correlator circuit, according to the third aspect itself or any prior implementation of the third aspect, the correlator circuit further includes: an ambient fill circuit configured to fill the input histogram data with ambient light values ​​when the processing extends beyond the available data; and a zero fill circuit configured to zero fill crosstalk data for bins that are out of range.

[0265] The fourth aspect relates to a method for processing input histogram data from a time-of-flight sensor, the method comprising processing the input histogram data in a bin-serial manner, the processing including: calculating the ambient light contribution; performing a master correlation operation on the input histogram data; calculating the crosstalk contribution; determining the nearest detection target based on the result of the master correlation operation; and identifying the peak with the highest signal-to-noise ratio (SNR) based on the result of the master correlation operation.

[0266] In a first implementation of the method, according to the fourth aspect itself, the input histogram data includes: multiple bins, each bin corresponding to a time delay; and a value in each bin representing the number of photons detected at the corresponding time delay, wherein the input histogram data represents the distribution of photon arrival times from the reflected light pulse.

[0267] In a second implementation of the method, the ambient light contribution is calculated, the main correlation operation is performed, and the crosstalk contribution is computed based on the fourth aspect itself or any previous implementation of the fourth aspect, including performing a multiply-accumulate operation on the multi-bit histogram box values ​​using multi-bit unsigned coefficients.

[0268] In a third implementation of the method, depending on the fourth aspect itself or any prior implementation of the fourth aspect, the method further includes performing a division operation after each of the calculation, execution, and arithmetic steps to implement fixed-point arithmetic within a predetermined domain.

[0269] In a fourth implementation of the method, depending on the fourth aspect itself or any previous implementation of the fourth aspect, performing the main correlation operation includes applying filter coefficients to the input histogram data, which define a filter shape that transitions from positive to negative values ​​to enable the detection of zero crossings corresponding to the median phase of the pulse.

[0270] In a fifth implementation of the method, depending on the fourth aspect itself or any previous implementation of the fourth aspect, the method further includes calculating the signal-to-noise ratio (SNR) of each bin in the input histogram data.

[0271] In the sixth implementation of this method, calculating the SNR, based on the fourth aspect itself or any previous implementation of the fourth aspect, includes using the equation... ,in This indicates the relevant values ​​of the input histogram data. Indicates the contribution of ambient light, and This indicates the crosstalk contribution.

[0272] In a seventh implementation of the method, determining the closest detection target, based on the fourth aspect itself or any previous implementation of the fourth aspect, includes analyzing the results of the master correlation operation box by box to identify the first occurrence of a valid signal exceeding a predetermined threshold.

[0273] In the eighth implementation of the method, identifying the peak with the highest SNR, depending on the fourth aspect itself or any previous implementation of the fourth aspect, includes tracking the highest SNR value and the corresponding bin position when processing the input histogram data.

[0274] In a ninth implementation of the method, depending on the fourth aspect itself or any prior implementation of the fourth aspect, the method further includes: filling the input histogram data with ambient light values ​​when the processing extends beyond the available data; and zero-padding crosstalk data for bins that are out of range.

[0275] The fifth aspect relates to a phase / box operation circuit for a time-of-flight sensor system, the phase / box operation circuit comprising: a filter multiply-accumulate (MAC) circuit configured to apply filter coefficients to preprocessed histogram data within a defined search and pulse window; and a positive zero-crossing detection circuit coupled to the filter MAC circuit and configured to: detect positive zero-crossings in the filtered data output by the filter MAC circuit, store current and previous filter values ​​upon detection of a positive zero-crossing, and output a detected median flag and median box position, wherein the phase / box operation circuit is configured to: process the input data in a box-serial manner, and calculate the median phase by interpolating the current filter value and the previous filter value.

[0276] In a first implementation of the phase / box operation circuit, according to the fifth aspect itself, the phase / box operation circuit further includes: an ambient filling circuit configured to fill the input data with an ambient light value when the processing extends beyond the available data; and a zero-filling circuit configured to zero-fill crosstalk data for out-of-range boxes.

[0277] In a second implementation of the phase / box operation circuit, the filter coefficients are generated based on the half-width parameter of the expected pulse width, depending on the fifth aspect itself or any previous implementation of the fifth aspect.

[0278] In a third implementation of the phase / box operation circuit, depending on the fifth aspect itself or any previous implementation of the fifth aspect, the filter MAC circuit is configured to perform a convolution operation on the preprocessed histogram data to enhance the relevant pulse information.

[0279] In a fourth implementation of the phase / bin operation circuit, according to the fifth aspect itself or any previous implementation of the fifth aspect, the phase / bin operation circuit is further configured to compute an alternative phase estimate in response to the failure to detect a positive zero crossing.

[0280] In a fifth implementation of the phase / bin operation circuit, the alternative phase estimation, according to the fifth aspect itself or any previous implementation of the fifth aspect, includes: identifying the largest bin in the filtered data; selecting a bin adjacent to the largest bin for interpolation; and using the selected bin to calculate an approximate median phase.

[0281] In a sixth implementation of the phase / box operation circuit, according to the fifth aspect itself or any previous implementation of the fifth aspect, the phase / box operation circuit is configured to receive input parameters including: peak position; search start offset; search end offset; window start offset; and window end offset.

[0282] In a seventh implementation of the phase / box operation circuit, the phase / box operation circuit is configured to use the received input parameters to calculate the boundaries of the search and pulse windows, depending on the fifth aspect itself or any previous implementation of the fifth aspect.

[0283] In the eighth implementation of the phase / box operation circuit, the phase / box operation circuit is configured based on the equation, according to the fifth aspect itself or any previous implementation of the fifth aspect. To calculate the median phase, the median bin is the number of bins that detected a positive zero crossing, the previous filter value is the filter output value of the bin before the positive zero crossing, and the current filter value is the filter output value of the bin when a positive zero crossing is detected.

[0284] In a ninth implementation of the phase / box operation circuit, the phase / box operation circuit is further configured, according to the fifth aspect itself or any previous implementation of the fifth aspect, to: perform instantaneous subtraction of crosstalk from the preprocessed histogram data; and to prune the resulting values ​​to ensure that they remain non-negative.

[0285] The sixth aspect relates to a method for processing time-of-flight sensor data, the method comprising: receiving preprocessed histogram data; applying filter coefficients to the preprocessed histogram data within defined search and pulse windows to generate filtered histogram data; processing the filtered histogram data in a bin-serial manner; detecting positive zero crossings in the filtered histogram data; storing current and previous filter values ​​in response to detecting positive zero crossings; outputting a detected median flag and median bin position; and calculating the median phase by interpolating the current and previous filter values.

[0286] In a first implementation of the method, according to the sixth aspect itself, the method further includes: filling the input data with ambient light values ​​when the processing extends beyond the available data; and filling crosstalk data with zeros for bins that are out of range.

[0287] In a second implementation of the method, the filter coefficients are generated based on the half-width parameter of the expected pulse width, depending on the sixth aspect itself or any previous implementation of the sixth aspect.

[0288] In a third implementation of the method, depending on the sixth aspect itself or any previous implementation of the sixth aspect, the applied filter coefficients include performing a convolution operation on the preprocessed histogram data to enhance the relevant impulse information.

[0289] In a fourth implementation of the method, depending on the sixth aspect itself or any previous implementation of the sixth aspect, the method also includes implementing an alternative phase estimation when no positive zero crossing is detected.

[0290] In a fifth implementation of the method, alternative phase estimation methods, based on the sixth aspect itself or any previous implementation of the sixth aspect, include: identifying the largest bin in the filtered histogram data; selecting bins adjacent to the largest bin for interpolation; and using the selected bins to compute an approximate median phase.

[0291] In a sixth implementation of the method, depending on the sixth aspect itself or any previous implementation of the sixth aspect, the method further includes receiving input parameters including: peak position; search start offset; search end offset; window start offset; and window end offset.

[0292] In a seventh implementation of the method, depending on the sixth aspect itself or any prior implementation of the sixth aspect, the method further includes calculating the boundaries of the search and pulse windows using the received input parameters.

[0293] In the eighth implementation of this method, the calculation of the median phase, based on the sixth aspect itself or any previous implementation of the sixth aspect, includes calculation based on the equation The median box is the number of boxes that detected a positive zero crossing, the previous filter value is the filter output value of the box before the positive zero crossing, and the current filter value is the filter output value of the box when a positive zero crossing is detected.

[0294] In a ninth implementation of the method, depending on the sixth aspect itself or any previous implementation of the sixth aspect, the method further includes: performing crosstalk subtraction from the preprocessed histogram data on an instantaneous basis; and pruning the resulting values ​​to ensure that they remain non-negative.

[0295] While this description has been described in detail, it should be understood that various changes, substitutions, and alterations may be made without departing from the spirit and scope of this disclosure as defined by the appended claims. In the various drawings, the same elements are designated by the same reference numerals. Moreover, the scope of this disclosure is not intended to be limited to the specific embodiments described herein, as it will be readily apparent from this disclosure to those skilled in the art that existing or later-developed processes, machines, manufactures, compositions of matter, components, methods, or steps can perform substantially the same functions or achieve substantially the same results as the corresponding embodiments described herein. Therefore, the appended claims are intended to include such processes, machines, manufactures, compositions of matter, components, methods, or steps within their scope.

[0296] Therefore, the specification and drawings are simply regarded as illustrations of the present disclosure as defined by the appended claims, and are contemplated to cover any and all modifications, variations, combinations or equivalents falling within the scope of the present disclosure.

Claims

1. A time-of-flight sensor system, comprising: A light emitter is configured to emit light pulses toward a target; A detector array is configured to detect light pulses reflected from the target; as well as Histogram processing circuitry is coupled to the detector array and configured as follows: Receive time-of-flight measurement data from the detector array. Measurement data from the detector array are processed using continuous bin-by-bin histogram processing, and During the continuous bin-by-bin histogram processing, one or more instantaneous operations are applied.

2. The time-of-flight sensor system according to claim 1, wherein the histogram processing circuit includes a correlator circuit, the correlator circuit comprising: The ambient light multiplier-accumulator MAC circuit is configured to calculate the ambient light contribution. The filter MAC circuit is configured to perform a master correlation operation on the time-of-flight measurement data; as well as The crosstalk MAC circuit is configured to calculate the crosstalk contribution. The correlator circuit is configured to process the time-of-flight measurement data in a box-serial manner.

3. The time-of-flight sensor system according to claim 2, wherein the correlator circuit further comprises: The circuit closest to the target is coupled to the filter MAC circuit and configured to determine the closest detection target; as well as The strongest maximum value circuit is coupled to the filter MAC circuit and is configured to identify the peak with the highest signal-to-noise ratio (SNR).

4. The time-of-flight sensor system according to claim 1, wherein the histogram processing circuit includes a phase / box operation circuit, the phase / box operation circuit comprising: An ambient light filling circuit is configured to fill the input data with ambient light values ​​in response to the processing extending beyond the available time-of-flight measurement data. Zero-fill circuitry is configured to handle out-of-range box-type zero-fill crosstalk data. The filter multiply-accumulate MAC circuit is configured to apply the filter coefficients to preprocessed histogram data within a defined search and pulse window. as well as A positive zero-crossing detection circuit is coupled to the filter MAC circuit and configured to detect positive zero-crossings in the filtered data output by the filter MAC circuit.

5. The time-of-flight sensor system of claim 4, wherein the phase / box operation circuit is further configured to: calculate the median phase by interpolating the current filter value and the previous filter value when the positive zero crossing is detected.

6. The time-of-flight sensor system according to claim 1, wherein the histogram processing circuit is further configured to: Based on crosstalk calibration data and silicon temperature data, a crosstalk histogram is generated using linear interpolation; and Crosstalk correction is adaptively updated during operation.

7. The time-of-flight sensor system of claim 1, wherein the histogram processing circuit is configured to process the time-of-flight measurement data using a sliding window of histogram data.

8. The time-of-flight sensor system of claim 1, wherein the histogram processing circuitry is configured to perform full histogram processing capabilities while maintaining a low-power target for integration into a battery-powered device.

9. The time-of-flight sensor system according to claim 1, wherein the time-of-flight measurement data includes histogram data, the histogram data comprising: Multiple boxes, each corresponding to a time delay; as well as The value in each bin represents the number of photons detected at the corresponding time delay. The histogram data represents the distribution of photon arrival times from the reflected light pulse.

10. The time-of-flight sensor system of claim 9, wherein applying the one or more real-time operations comprises: Remove crosstalk from the predefined regions of the histogram data; Track and detect the first peak in the histogram data, and Track and discover the maximum peak value in the histogram data.

11. A method for processing time-of-flight sensor data, the method comprising: Emit light pulses toward the target; Detect the light pulse reflected from the target; Time-of-flight measurement data is received based on the detected reflected light pulses; The flight time measurement data is processed using continuous bin-by-bin histogram processing, wherein the continuous bin-by-bin histogram processing includes: continuously processing each bin of the histogram, maintaining the state of a predetermined number of recently processed bins, and generating output data for each bin before proceeding to subsequent bins; as well as During the continuous bin-by-bin histogram processing, one or more instantaneous operations are applied.

12. The method of claim 11, wherein processing the time-of-flight measurement data comprises: The ambient light contribution is calculated by multiplying and accumulating the ambient light using a MAC circuit. The time-of-flight measurement data is subjected to a master correlation operation by the filter MAC circuit; as well as The crosstalk contribution is calculated by the crosstalk MAC circuit, wherein the time-of-flight measurement data is processed in a box-serial manner.

13. The method of claim 12, further comprising: The closest detection target is determined by the closest target circuit coupled to the filter MAC circuit. as well as The peak value with the highest signal-to-noise ratio (SNR) is identified by the strongest maximum value circuit coupled to the filter's MAC circuit.

14. The method of claim 11, wherein processing the time-of-flight measurement data comprises: In response to the processing extending beyond the available time-of-flight measurement data, the input data is populated with ambient light values; Zero-padded crosstalk data is used for bins that are out of range; Apply the filter coefficients to the preprocessed histogram data within the defined search and pulse window; as well as Detect positive zero crossings in the filtered data output by applying the filter coefficients.

15. The method of claim 14, further comprising, in response to detecting a positive zero crossing, calculating the median phase by interpolating the current filter value and a previous filter value.

16. The method of claim 11, further comprising: Based on crosstalk calibration data and silicon temperature data, a crosstalk histogram is generated using linear interpolation. as well as Crosstalk correction is adaptively updated during operation.

17. The method of claim 11, wherein processing the time-of-flight measurement data includes using a sliding window of histogram data.

18. The method of claim 11, wherein applying the one or more instantaneous operations comprises: Remove crosstalk from the predefined regions of the histogram data; Track and detect the first peak in the histogram data, and Track and discover the maximum peak value in the histogram data.

19. The method of claim 11, wherein the time-of-flight measurement data includes histogram data, the histogram data comprising: Multiple boxes, each corresponding to a time delay; as well as The value in each bin represents the number of photons detected at the corresponding time delay, wherein the histogram data represents the distribution of photon arrival times from the reflected light pulse.

20. A method for processing time-of-flight sensor data, the method comprising: Receive time-of-flight measurement data from the detector array, the measurement data representing light pulses reflected from the target; as well as The time-of-flight measurement data is processed using a histogram processing circuit, wherein the processing includes: The filter coefficients are applied to the time-of-flight measurement data within the defined search and pulse window. Detect positive zero-crossings in the filtered data output by applying the filter coefficients. Upon detecting the positive zero crossing, store the current filter value and the previous filter value, and The median phase is calculated by interpolating the current filter value and the previous filter value. The processing is performed in a bin-sequential manner, processing each bin of the histogram consecutively while maintaining the state of a predetermined number of recently processed bins.

Images