Dynamic gating of range sensor array

Abstract

A method, system, and computer program for dynamically strobing a pixel array in a ranging sensor using signal transmission and reflection. At least one histogram of reflection arrival time counts for at least one pixel of a frame is analyzed. Based on the analysis of the at least one histogram, at least one metric for the at least one pixel is determined. Based on the at least one metric and a collection policy, reflection collection for the at least one pixel is selectively disabled.

Description

Background Technology

[0001] Some embodiments described in this invention relate to distance detection, and more specifically, but not exclusively, to dynamic gating of pixel arrays in a ranging sensor.

[0002] Ranging sensors and / or ranging imaging systems are used in a variety of applications, including autonomous vehicles and driver assistance systems. Ranging sensors can detect the distance between themselves and objects without physical contact. Several types of ranging sensors exist, including acoustic sensors, visible or infrared-based sensors, and time-of-flight (ToF) sensors. Acoustic sensors use sound waves to detect obstacles, while visible or infrared-based sensors use light to measure distance. Ranging imaging systems typically use a combination of ranging sensors and cameras to create 3D images of the environment.

[0003] Light direction and range (LiDAR) is a technique used to generate three-dimensional (3D) depth maps. It is similar to radio detection and ranging (RADAR), but uses visible light instead of radio waves. Direct time-of-flight (dToF) is a type of LiDAR that works by precisely measuring the time difference between the emitted and received light pulses. Using appropriate optics, the distance between scene objects and the ranging sensor can be measured at each point in a two-dimensional (2D) image (i.e., an array of pixels), thus obtaining a 3D depth map. Laser devices are typically used to generate brief, intense light pulses, with durations potentially on the order of nanoseconds.

[0004] To detect reflected photons, a light sensor with excellent sensitivity must be used. This is typically a single-photon avalanching diode (SPAD). SPAD arrays (usually 2D arrays) can be created to form depth sensors. After a photon hits the SPAD and thus triggers an avalanche, there is a dead time (the period during which the SPAD is no longer sensitive to photons before it is recharged).

[0005] The reflected photons strike the sensor with a fixed delay after the laser pulse, the time delay being determined by the distance to the object. SPAD can be time-gated, meaning it is activated only for a brief period when the expected reflection arrives. In addition to the received photons, there is usually some background light (noise) arriving at random times. To correctly distinguish between true reflections and background noise, measurements are repeated multiple times, and the arrival times are plotted in a histogram. x The axis (horizontal direction) represents the time delay. y The axis (vertical direction) represents the number of photons. The peak in the histogram corresponds to the desired signal, i.e., the time it takes for the image to be reflected from the object.

[0006] Depth sensors can include SPAD arrays fabricated on silicon substrates. Photolithography refers to the process used to etch detailed, multi-layered integrated designs onto a silicon wafer.

[0007] The intensity of light illuminating a surface is usually measured in lux, which is defined as one lumen per square meter. A lumen is the unit of luminous flux, that is, the amount of light emitted by a light source in all directions.

[0008] The reflected light is received as individual photons. When photons are received continuously at a low rate, this can be described as a Poisson process with Poisson statistics, meaning that the number of photons within a given time window may fluctuate due to the randomness of the process. This effect adds some uncertainty when attempting to determine the intensity of the received light and is described as shot noise.

[0009] Autonomous mobile devices typically need to track their location. Simultaneous localization and mapping (SLAM) is a technique that builds a 3D digital map of an environment while keeping track of the current location within that environment. Summary of the Invention

[0010] One object of the present invention is to describe a system and method for a dynamically gated ranging sensor array.

[0011] The foregoing and other objectives are achieved through the features of the independent claims. Other implementations are apparent from the dependent claims, the specification, and the drawings.

[0012] According to one aspect of some embodiments of the disclosed subject matter, a method is provided for dynamically gating a pixel array in a ranging sensor using signal transmission and reflection, the method comprising: analyzing at least one histogram of reflection arrival time counts of at least one pixel of a frame; determining at least one metric for at least one pixel based on the analysis of the at least one histogram; and selectively disabling reflection collection of at least one pixel according to the at least one metric and a collection strategy.

[0013] According to another aspect of some embodiments of the disclosed subject matter, a processing circuit is provided for dynamically gating a pixel array in a ranging sensor using signal transmission and reflection. The processing circuit includes a memory and is configured to: analyze at least one histogram of reflection arrival time counts of at least one pixel of a frame; determine at least one metric of at least one pixel based on the analysis of the at least one histogram; and selectively disable reflection collection of at least one pixel according to the at least one metric and a collection strategy.

[0014] According to another aspect of some embodiments of the disclosed subject matter, a computer program is provided for dynamically gating a pixel array in a ranging sensor using signal transmission and reflection. The computer program includes program instructions that, when executed by at least one processor, cause at least one processor to: analyze at least one histogram of reflection arrival time counts of at least one pixel of a frame; determine at least one metric for at least one pixel based on the analysis of the at least one histogram; and selectively disable reflection collection of at least one pixel according to the at least one metric and a collection strategy.

[0015] Optionally, the analysis of at least one histogram includes calculating at least one statistic based on the counts of at least one histogram.

[0016] Optionally, calculating at least one statistic includes calculating at least one of the following: at least one peak of the counts of at least one histogram; the cumulative sum of the values ​​of the counts of at least one histogram.

[0017] Optionally, at least one metric is calculated based on at least one statistic, which includes at least one of energy consumption, confidence level, accuracy estimate, and detection duration estimate.

[0018] Optionally, for each pixel of a frame, the number of transmissions corresponding to the activation timing of the signal source of the ranging sensor are counted, and at least one metric is calculated based on the number of transmissions and at least one statistic.

[0019] Optionally, at least one histogram includes multiple histograms of adjacent pixels.

[0020] Optionally, the collection strategy varies based on at least one region of interest.

[0021] Optionally, at least one region of interest is determined by analyzing at least one of the depth map and intensity image of at least one other frame preceding the frame.

[0022] Optionally, at least one region of interest is determined by analyzing scene information from an intensity image generated from the histogram of at least one subset of pixels.

[0023] Optionally, the analysis of scene information in the intensity image is performed using a simultaneous localization and mapping (SLAM) process.

[0024] Optionally, for each pixel of the frame, the number of transmissions corresponding to the activation timing of the signal source of the ranging sensor are counted; the count value obtained from the count in the pixel's histogram is adjusted using the number of transmissions.

[0025] Optionally, the count value is adjusted based on the ratio between the number of transmissions for pixel counting and the total number of transmissions made by the signal source for the frame.

[0026] Optionally, the signal source of the ranging sensor is directional, and the corresponding beam portion of the signal source in the direction corresponding to at least one pixel is selectively disabled according to at least one measurement and collection strategy.

[0027] Other systems, methods, features, and advantages of the present invention will be apparent to those skilled in the art upon studying the following figures and detailed description. It is intended that all such other systems, methods, features, and advantages be included in this specification, within the scope of the invention, and protected by the appended claims.

[0028] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as those known to one of ordinary skill in the art to which the embodiments pertain. While similar or equivalent methods and materials to those described herein may be used in the practice or testing of the embodiments, exemplary methods and / or materials are described below. In case of any conflict, this specification (including the definitions) shall prevail. Furthermore, these materials, methods, and examples are illustrative only and are not necessarily restrictive. Attached Figure Description

[0029] This document describes some embodiments by way of example and in conjunction with the accompanying drawings. The detailed description below, with specific reference to the accompanying drawings, emphasizes that the details shown are merely illustrative and for the purpose of discussing embodiments. Thus, it will be apparent to those skilled in the art, based on the accompanying drawings, how to practice the embodiments.

[0030] In the attached diagram: Figure 1 This is a schematic diagram of an exemplary direct time-of-flight (dToF) light direction and range (LiDAR) imaging system according to some embodiments; Figure 2This is an exemplary histogram of photon event times for a pixel in a dToF LiDAR imaging system according to some embodiments; Figure 3 This is a schematic diagram illustrating an exemplary disadvantage of high power consumption in a dToF LiDAR imaging system according to some embodiments; Figure 4 This is another exemplary histogram of the photon event time of a pixel in a dToF LiDAR imaging system according to some embodiments, as obtained in an exemplary adverse scenario of a dark pixel; Figure 5 This is a schematic diagram of an exemplary use case of a depth sensor in a car for parking assistance according to some embodiments; Figure 6 These are schematic diagrams of corresponding exemplary depth maps and intensity images generated by a depth sensor according to some embodiments; Figure 7 This is a block diagram of an exemplary module for dynamic gating of a pixel array in a dToF LiDAR imaging system according to some embodiments; Figure 8 This is a block diagram of another exemplary module for dynamically gating a pixel array in a dToF LiDAR imaging system using pixel block data analysis, according to some embodiments. Figure 9 This is a schematic diagram of a combination of histogram data from multiple pixels of a sensor array, according to some embodiments; Figure 10 This is a schematic diagram of pixel block grouping in a sensor array according to some embodiments; Figure 11 This is a block diagram of yet another exemplary module for dynamically gating a pixel array in a dToF LiDAR imaging system using analysis of previous frame data, according to some embodiments. Detailed Implementation

[0031] Some embodiments described in this invention relate to distance detection, and more specifically, but not exclusively, to dynamic gating of pixel arrays in a ranging sensor.

[0032] Three-dimensional (3D) direct time-of-flight (dToF) light direction and range (LiDAR) imaging systems, as well as other similar ranging sensors and / or ranging imaging systems, use laser pulses to measure distance. Short laser pulses (on the order of 1 nanosecond (or several nanoseconds) in duration) are reflected by the object, and these reflections return to the sensor at different times. Optics are used to focus the image onto a sensor array, such as a two-dimensional (2D) array of single-photon avalanching diodes (SPADs).

[0033] Now for reference Figure 1 , Figure 1 This is a schematic diagram of an exemplary direct time-of-flight (dToF) light direction and range (LiDAR) imaging system according to some embodiments.

[0034] like Figure 1 As shown, a 3D dToF LiDAR imaging system can use a laser emitter and / or a similar device suitable for generating short pulses of electromagnetic radiation (e.g., flashes) as a signal transmission source. A two-dimensional (2D) pixel array of a photosensitive sensor (e.g., a SPAD pixel array) can be used to detect photons arriving at the system, and the system can also record their arrival times with reference to the time when the laser source transmits the laser pulses. After the laser pulse is transmitted, reflections of the transmitted laser pulse from objects in the scene and / or the environment to be imaged can then be returned to the SPAD pixel array. The reflections can be collected by appropriate optics and oriented toward the corresponding pixels of the SPAD pixel array. Depending on the travel length of the laser pulse from the laser source to the corresponding object and back to the SPAD pixel array, reflections from objects in the scene may arrive at the SPAD pixel array at different times. For example, the travel distance of the laser pulse to and from a distant object is longer than that to a nearby object, such as... Figure 1 As shown by the conical and spherical objects, since the speed of light is constant, reflections from closer objects arrive at the SPAD pixel array earlier than reflections from farther objects.

[0035] When a returning photon illuminates the SPAD array in the sensor, the SPAD array is triggered. Precise timestamps of these events are recorded. This process is repeated multiple times, and the timestamps are used to construct a histogram of photon event times for each pixel. This gives the 3D volume (height, width, and time) of the reflected photon. One or more peaks in each histogram indicate the distance (depth) of the reflective surface seen by the corresponding pixel.

[0036] Now for reference Figure 2 , Figure 2 This is an exemplary histogram of photon event times for a pixel in a dToF LiDAR imaging system according to some embodiments.

[0037] like Figure 2 As shown, photon event times (i.e., the recorded arrival times of photon reflections detected by individual elements in a SPAD pixel array) can be categorized into bins, each bin having a predetermined interval (e.g., 50 picoseconds) in width, collectively forming a domain of continuous arrival times for photon reflections. All instances falling into the same bin can be counted, and the total number of each bin in the corresponding bin can be recorded, thus obtaining a histogram of photon event times (reflection arrival times). The peak of the histogram, i.e., the maximum count of photon event instances across all bins, can be located, and its position (i.e., the arrival time represented by the bin with the highest event count) can be used to determine the depth value at the corresponding pixel.

[0038] One technical challenge of dToF LiDAR imaging systems and / or other similar ranging sensors is that storing a complete histogram for each pixel may require a large amount of memory within the sensor, with memory requirements increasing proportionally to depth resolution (i.e., the width of each bin) and maximum depth range (i.e., the total number of bins used). Since only the “correct” depth (i.e., the location of the peak in the histogram) is needed, the histogram stores a large amount of unnecessary data.

[0039] Therefore, many systems offer a choice between coarse depth measurement (for cases with poor depth resolution) or step / windowed measurement. In step / windowed measurement, a histogram is used to record only a subset of timestamps (e.g., between 0 nanoseconds and 10 nanoseconds). Measurements are repeated multiple times with different offsets to progressively build a complete high-resolution histogram.

[0040] Another technical challenge for dToF LiDAR imaging systems and / or similar ranging sensors is the significant variation in scene dynamic ranging and signal-to-background noise ratio (SBR).

[0041] High dynamic range and / or variable SBR may be caused by one or more of the following reasons: - Varying background light (unwanted noise): 3D sensors can be used indoors and outdoors, where there may be a huge range of background light intensity (e.g., up to 100,000 lux in direct sunlight on the one hand, and as low as less than 1 lux in the dark on the other). - Variable echo signal: The intensity of the laser echo can be very strong (for nearby, reflecting objects) or very weak (for distant, small, non-reflecting objects). This is expressed in the radar ranging equation (see equation (1) in this paper), which also applies to LiDAR detection. Very weak echoes may contain only a few photons, which follow Poisson statistics (shot noise).

[0042] Equation (1) in this paper describes the radar ranging equation, where the received power is related to 1 / R 4 Proportional (for objects smaller than one pixel): (1)

[0043] in It's the receiving power. It is the power output of the radar transmitter. It refers to antenna gain, which is the efficiency with which a radar antenna focuses and transmits energy in a specific direction. It is the effective aperture area of ​​the receiving antenna. It is the radar cross section (RCS), a measure of a target's ability to reflect radar signals back to the receiver. It is a pattern propagation factor.

[0044] Poisson statistics are used to model systems with a large number of possible events, each of which is rare. The parameters of the Poisson distribution are... This represents the constant average rate at which such events occur within a fixed time interval, regardless of the duration between events. Equation (2) here describes the probability that the total number of events occurring within the same time interval is equal to... As shown below: (2)

[0045] For low-reflectivity and / or distant parts of the imaging scene, the photon arrival time histogram of the pixel has high shot noise because each cell samples the Poisson distribution using a very small λ.

[0046] Recent advances in photolithography have enabled smaller SPADs, which cover a smaller solid angle in an image and therefore receive fewer photons, thus exacerbating shot noise in the signal and background.

[0047] The high dynamic range in scene data presents significant challenges for dToF LiDAR imaging systems and similar ranging sensors, especially in outdoor automotive applications.

[0048] One of the technical challenges is the enormous power consumption under direct sunlight, which comes from the SPADs that repeatedly avalanche due to exposure to very strong light.

[0049] Now for reference Figure 3 , Figure 3 This is a schematic diagram illustrating an exemplary disadvantageous scenario of high power consumption in a direct time-of-flight (dToF) light direction and range (LiDAR) imaging system according to some embodiments.

[0050] exist Figure 3 In the exemplary adverse scenario shown, a considerable area of ​​the SPAD pixel array of the dToF LiDAR imaging system is exposed to direct sunlight, resulting in excessive energy consumption due to repeated photon events.

[0051] Each avalanche (photon event) may require energy on the order of approximately 1 to 100 picojoules (pJ), and they may occur consecutively (with intervals on the order of approximately 1 nanosecond per pixel), resulting in an energy consumption on the order of approximately 1 milliwatt (mW) per pixel.

[0052] It should be noted that SPADs are known to be "time-gated" so that they can be activated only during specific time periods, such as when reflected photons are expected to arrive. This may be done to save power or to ensure that the SPAD is sensitive to peaks (i.e., to avoid it being in the "dead time" after an avalanche, when it cannot detect photons).

[0053] However, time gating may not completely eliminate this problem of excessive and / or high power consumption, for example, in cases of extremely bright pixels and / or oversaturation, such as in exemplary adverse scenarios such as exposure to direct sunlight as discussed and illustrated herein.

[0054] Another technical challenge brought about by high dynamic range is the difficulty in histogram peak detection in dark pixels (such as pixels on non-reflective and / or distant surfaces) because there are very few photons (making it difficult to determine where the peak is).

[0055] Now for reference Figure 4 , Figure 4 This is an exemplary histogram of photon event times for a pixel in a dToF LiDAR imaging system, as obtained in this exemplary adverse scenario for a dark pixel.

[0056] like Figure 4 As shown, for dark and / or underexposed pixels, such as for non-reflective and / or distant surfaces, the total number of photon arrival events at the SPAD pixel array may be very small. Furthermore, due to the high variance of background noise as discussed herein, the arrival times of various records of such sporadic photon events can be scattered across different and highly varied time bins of the histogram, where even those histogram bins filled with photon events may contain only a few such photons (e.g., at most one or two), and several candidate peaks may appear at different bin locations. Thus, it is difficult (if not impossible) to distinguish which is the “correct” candidate peak representing the actual distance to the surface.

[0057] Existing methods for dToF LiDAR imaging tools and / or technologies offer some of the features described below: A system offers users a configurable choice between operating modes: low-resolution or high-resolution. This refers to the sensor's 2D spatial resolution. Low-resolution mode performs better at long distances, while high-resolution mode captures more detail at close range.

[0058] Some systems offer a “stepped” or windowed histogram capture mode, providing high resolution by repeating depth measurements multiple times across different depth windows. For example, the first measurement might cover a range from 0 centimeters (cm) to 10 cm, the second measurement might cover a range from 10 cm to 20 cm, and so on. This approach allows such systems to provide high spatial resolution with lower memory requirements.

[0059] Some systems offer “coarse” and “fine” depth measurement modes that can operate in a two-stage manner. First, a coarse (low depth resolution) measurement is performed to detect an approximate depth (i.e., histogram peaks), and then a second fine (high depth resolution) measurement is performed to overwrite the approximate depth indicated by the first measurement. For example, the first (coarse) mode might have a depth resolution of 10 cm, while the second (fine) mode might have a depth resolution of 1 millimeter (mm).

[0060] However, this existing technology and / or tool has several drawbacks as described in this article.

[0061] A system offering selectable “fine” and “coarse” 2D spatial resolution modes is ill-suited to current usage scenarios. Users must configure it for distant or high 2D resolution modes. This means that in many environments, either spatial resolution is sacrificed, or depth results are noisy and unreliable. Since typical automotive scenarios involve both near and far objects, it is impossible to configure this system to achieve optimal performance on both.

[0062] Systems that offer "step-through" or windowed histogram capture modes have very slow frame rates because they must repeat depth measurements multiple times to collect different portions of the complete histogram. This also means they face the problem of high power consumption per frame.

[0063] When the SBR is low, the system performance for providing two-stage depth measurement (i.e., coarse-to-fine) is poor. Both the coarse and fine stages must successfully identify the correct peaks to generate the correct depth measurement; if the correct peaks are not identified in the coarse stage, the fine stage will operate at the wrong depth.

[0064] All existing systems and / or methods also face power consumption issues under high background illumination (e.g., when the sun is visible). The SPADs in this part of the image are repeatedly reactivated, leading to high power consumption.

[0065] Existing systems and / or methods also perform the same number of depth measurements across all parts of the image, thus keeping all SPADs active for the same total time (during the same number of laser flashes). This makes it difficult to optimize the number of laser flashes, as fewer flashes will result in the inability to detect distant objects (due to shot noise from weak reflections), while more flashes will waste power when measuring nearby objects (which produce strong reflections).

[0066] The disclosed subject matter aims to improve existing tools and / or techniques for distance detection and / or imaging, provide advantages over these existing tools and / or techniques, and / or overcome many of the disadvantages of these existing tools and / or techniques.

[0067] First, the disclosed subject matter aims to address or mitigate the adverse effects of high power consumption in the presence of excessively bright visible light sources (such as the sun or car headlights). This undesirable outcome arises because the SPAD is constantly re-triggered when new photons illuminate it, and each SPAD trigger consumes energy on the order of several picojoules (pJ). Given the large number of photons under bright conditions (and the sheer number of pixels in large sensor arrays), this results in excessive power consumption.

[0068] Secondly, the disclosed topic aims to address the difficulty of peak detection in poorly reflective regions of an image, which leads to missed object detection. This problem arises because there are very few photons in these regions of the image, resulting in high shot noise in the photon arrival time histogram for each pixel (because each cell samples the Poisson distribution using a very small λ).

[0069] Third, the disclosed topic addresses the problem of inconsistent depth accuracy across images. This problem arises because nearby reflective objects reflect more photons back to the sensor than distant non-reflective objects; this means that histogram peaks are more clearly defined in nearby objects and can therefore be measured more accurately than in distant objects.

[0070] As used herein, the term “gating” refers to an operating scheme in which a sensing element or detector is activated or enabled only for a portion of a measurement or measurement cycle and is selectively deactivated for the remainder of the measurement, such as “time gating”, in which a sensing device is activated only at a specified time when a signal is expected to arrive.

[0071] As used herein, the term “dynamic gating” refers to the selective disabling and / or deactivation of sensing elements, such as one or more pixels in a range sensor or range imaging device, based on information collected and / or updated at least in part while the measurement is still in progress, during a measurement or measurement cycle (e.g., the capture or reconstruction of a single frame), for example, after each emission of a sensing signal from a source (e.g., a laser flash in a dToF LiDAR imager).

[0072] According to some embodiments, as the photon arrival time histograms for each pixel are progressively constructed, they are monitored throughout the measurement process, and their contents are analyzed to determine one or more pixel-related metrics, including but not limited to: (i) energy consumption metrics for each pixel, (ii) confidence levels of correctly identified peaks, (iii) accuracy estimates of identified peaks, etc. Based on one or more metrics determined for a given pixel and / or group of pixels, a decision can be made regarding whether to disable that pixel and / or group of pixels, i.e., deactivate one or more corresponding SPADs in the sensor array. Optionally, this selective disabling and / or deactivation (i.e., dynamic gating) of one or more pixels can then be performed, for example, for the remainder of the measurement and / or at least a portion thereof, to save power and / or comply with other specifications and / or requirements of a predefined measurement strategy, such as applying one or more thresholds to one or more determined metrics, etc.

[0073] In some embodiments, histograms from multiple pixels can be grouped together, and the data within can be analyzed collectively. Due to the small number of photons, the combined, accumulated histogram can reveal peaks that cannot be identified in the individual pixel histograms. This can be used to estimate (i) the peak location (i.e., depth estimation) and related statistics of this pixel group, and (ii) how many laser flashes are needed to produce strong peaks independently in individual pixels within the group. This histogram grouping operation can be performed iteratively in a hierarchical manner, merging increasingly larger regions until a peak is obtained. For example, pixels observing a vehicle at a mid-range distance might show that using twice the number of laser flashes can resolve the object at a higher resolution, while pixels observing the sky will never give a measurable peak.

[0074] Optionally, real-time information about the photon event time recorded in the histogram of each pixel and / or group of pixels can be evaluated at each laser flash, and the photon histogram can be updated when the reflection reaches the SPAD pixel array and triggers a photon avalanche therein.

[0075] In some embodiments, depth measurement and / or imaging can be performed over multiple frames, for example, at a rate of approximately 30 frames per second (fps). Monitoring and analysis of the photon histogram for each pixel and / or group of pixels used for pixel activation / deactivation decisions can be performed from the start of the measurement, i.e., when the capture and reconstruction of a new frame begins, and continue as it proceeds during the time interval in which the measurement is performed, i.e., a duration of 1 / 30 of a second, after which all histogram data can be discarded and new data for subsequent frames can be collected.

[0076] Optionally, it is possible to control when and / or at one or more specific points in time to selectively disable individual pixels (or groups of pixels) in order to save power for that frame when it is captured.

[0077] Additionally or alternatively, a configuration strategy can be followed to determine when a pixel is disabled based on histogram statistics for each pixel and the corresponding metrics determined therefrom, such as power consumption, estimated time required before a useful peak is available for that pixel, peak accuracy, and confidence level.

[0078] In some embodiments, scene analysis may be performed and / or used to identify regions of interest (ROIs) in an image, where higher confidence and / or accuracy levels may be required, and different strategies may be followed in these regions. For example, objects identified as moving vehicles or pedestrians may require higher resolution and distance accuracy, while areas identified as the sky or sun may not require accurate depth measurements.

[0079] Alternatively, scene information can be obtained by analyzing an intensity image generated based on the total amount of light falling on each pixel, such as that which can be determined from individual pixel histograms, for example, by summing over all bins.

[0080] Optionally, scene analysis may use data from one or more previous frames (if applicable). Data from a single previous frame may include at least one of a depth image and an intensity image (e.g., a conventional grayscale image).

[0081] Optionally, scene analysis may employ Simultaneous Localization and Mapping (SLAM) algorithms and / or similar processes used to simultaneously determine self-motion and scene reconstruction.

[0082] In some embodiments, a per-pixel laser flash count can be tracked and stored, representing the number of laser flashes corresponding to the time or event that activated the respective pixel. Each pixel can be associated with its own dedicated counter, which can be reset to zero at the start of the measurement (e.g., at the beginning of each new frame to be captured). Whenever a laser flash occurs (typically hundreds or thousands per frame), a timing unit controlling the laser may signal all flash counters, and the counter for enabled pixels may increment its count. The counter for disabled pixels remains unchanged. At the end of the measurement (or the corresponding frame capture duration), the counter value displays the total number of laser flashes during which the associated pixel was active. In other words, this is the duration for which the pixel remained active, measured in laser flashes.

[0083] Optionally, the per-pixel flash count during pixel activation can be used to calculate a detection duration estimate (i.e., an estimate of how many more flashes are needed for depth detection at the desired accuracy and / or confidence level). For example, if 100 flashes have been conducted so far when a pixel is enabled, and pixel metric analysis suggests that more than 50% of flashes are needed, then it can be determined that the pixel needs 50 more flashes. For example, the percentage or ratio of additional flashes required relative to the total number of flashes emitted so far can be estimated from the pixel block's hierarchical level and / or group size, where individual pixel histograms are merged until a peak can be detected at the desired accuracy and / or confidence level, as discussed herein. Additionally or alternatively, scene information from analysis of one or more current and / or previous frames can be used to assess the relationship or proportion of the required additional flashes, for example, based on accuracy and / or confidence levels associated with the identified region of interest.

[0084] Optionally, pixel activation flash counts can be used to reconstruct grayscale images captured by a sensor, i.e., by adjusting the per-pixel photon event count in a histogram obtained from measurements based on the recorded per-pixel flash count. As discussed herein, a histogram calculated by a 3D depth sensor (e.g., a dToF LiDAR imager) can be converted into a 2D image by summing the total number of photons in each histogram. Typically, the per-pixel photon count determines the brightness of that pixel in a photograph (intensity image). However, if a pixel is deactivated early, its photon count will be lower. For example, if a pixel is deactivated after 10% of the laser flashes, the true brightness is 10 times the brightness obtained solely from the photon count in the histogram. To obtain the true brightness of a pixel, the photon count in the histogram must be divided by the laser flash counter value for that pixel, and the result multiplied by the total number of laser flashes in the measurements (e.g., per frame) to obtain the correct intensity value for the current pixel. Optionally, the 2D photographic image can be used for object detection and identification, as well as for selecting regions of interest for the next frame, as discussed herein.

[0085] In addition, pixel activation flash counts can also be used in other situations and / or applications, such as when correcting histograms to eliminate the effects of SPAD dead time, as is known in the art.

[0086] In some embodiments, the laser itself can also be adjusted to save power using pixel metric analysis as discussed herein. For example, contrary to scenarios where the only way to save power is to turn off pixels, other laser types can be provided due to the use of non-directional omnidirectional lasers, which can be focused on specific areas. In these systems, the dynamic gating methods described herein can be used on both pixels and lasers. In other words, if it is decided to turn off a pixel, a portion of the laser beam illuminating that pixel can also be turned off simultaneously with that pixel.

[0087] One technical effect of utilizing the disclosed subject matter is to provide favorable power consumption for relatively close objects. As discussed in this paper, nearby objects produce strong reflections (for objects covering multiple pixels, due to the 1 / R ratio between intensity and distance). 2 (Relationship). This provides a strong, clear histogram peak after a few laser flashes. Therefore, by monitoring and analyzing the histogram content as it is updated throughout the measurement process according to the disclosed subject, it is possible to determine the accurate peak detected after a few laser flashes and to turn off the corresponding pixels to save power.

[0088] Another technical advantage of utilizing the disclosed subject is achieving favorable power consumption under bright lighting conditions. As discussed herein, in pixels focused on the sun, direct sunlight generates very high photon flux, which repeatedly triggers SPADs in the sensor. Therefore, by using histogram monitoring and analysis based on the disclosed subject, it is possible to identify those pixels with high power consumption that do not form detectable histogram peaks even when combined. Thus, these SPADs can be disabled to save power.

[0089] Another technical effect of utilizing the disclosed subject matter is to provide favorable power consumption under diffuse scattering (e.g., fog) conditions. Diffuse scattering media (e.g., fog, etc.) produce strong reflections, which continuously excite SPADs. This typically leads to high power consumption because the SPADs will be repeatedly triggered multiple times for each laser flash. However, by utilizing the disclosed subject matter, measurements can be stopped early for affected pixels. As discussed herein, measurement metrics (and optional grouping / segmentation) can be used to assess whether depth measurement peaks will occur before the energy consumption threshold is breached. Affected pixels / SPADs will be shut down, thus saving power.

[0090] Other and / or additional technical issues involved, the solutions and / or effects of the disclosed subject matter will become apparent from further discussion herein.

[0091] It should be understood that while the disclosed topics are described and illustrated primarily in the context, implementation and / or deployment within dToF LiDAR sensors and / or imaging systems for the sake of better understanding and / or ease of discussion, this is not intended to limit the scope in such a manner, and they can be used and / or applied in a similar way in combination with other similar ranging sensors, for example, in automotive applications and / or other technical fields, real-world scenarios, etc.

[0092] Specifically, the disclosed subject matter can be used with any LiDAR sensor that has a pixel array. For example, LiDAR based on other methods, such as Frequency Modulated Continuous Wave (FMCW) and other similar techniques. In these sensors, some pixels can be turned off early in each frame to save power. Additionally, for example, in FMCW sensors, a significant amount of per-pixel computation is required to determine the depth peak. This can be avoided for areas that are clearly less interesting and / or too distant, such as portions of the sky, by means of analysis of previous frames or by hierarchical pixel grouping.

[0093] Similarly, the disclosed subject matter can also be applied to other types of ranging sensors, including, for example, acoustic sensors, such as ultrasonic systems, which have pixel arrays and / or signal transmission sources (with pixel-level orientation and / or focusing capabilities). For example, in an ultrasonic system with a sensor array, ultrasonic transducer pixels can use a lower duty cycle depending on the intensity of the echo signal. Additionally or alternatively, hierarchical grouping can be used to identify areas that will not produce useful echoes, so the measurement frequency in these areas may be lower, or calculations can be performed at lower resolution in these areas.

[0094] Before detailing at least one embodiment, it should be understood that the embodiments are not necessarily limited to the detailed descriptions and / or drawings and / or examples illustrating the construction and / or setup of the components and / or methods. The implementations described herein support other embodiments, or can be practiced or performed in various ways.

[0095] An embodiment may be a system, method, and / or computer program product. A computer program product may include one or more computer-readable storage media having computer-readable program instructions that cause a processor to perform various aspects of the embodiment.

[0096] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, but are not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, and any suitable combination of the foregoing. The term "computer-readable storage media" as used herein should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0097] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded to an external computer or external storage device via a network such as the Internet, local area network, wide area network, and / or wireless network. The network may include copper transmission cables, fiber optic cables, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them for storage in a computer-readable storage medium within the respective computing / processing device.

[0098] Computer-readable program instructions used to perform the operations in the embodiments may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages. These programming languages ​​include object-oriented programming languages, such as Smalltalk, C++, etc., and traditional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet provided by an Internet service provider). In some embodiments, electronic circuits including programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) can execute computer-readable program instructions to customize the electronic circuits by using state information of computer-readable program instructions, thereby performing various aspects of the embodiments.

[0099] This document describes various aspects of the embodiments in conjunction with flowchart illustrations and / or block diagrams of the methods, apparatus (systems), and computer program products provided in the embodiments. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a manner for implementing the functions / actions detailed in the blocks of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium having the instructions stored therein comprises an article of manufacture containing the instructions for implementing aspects of the functions / actions detailed in one or more blocks of the flowchart and / or block diagram.

[0101] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, implement the functions / actions detailed in one or more blocks of a flowchart and / or block diagram.

[0102] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products provided in various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions described in the blocks may not be implemented in the order shown in the figures. For example, in fact, two blocks shown consecutively may be executed almost simultaneously, or sometimes, the blocks may be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be executed by a system based on dedicated hardware that performs a specific function or action, or a combination of dedicated hardware and computer instructions.

[0103] Now for reference Figure 5 , Figure 5 This is a schematic diagram of an exemplary use case of a depth sensor in a car for parking assistance, according to some embodiments.

[0104] Also refer to Figure 6 , Figure 6 These are schematic diagrams of corresponding exemplary depth maps and intensity images generated by a depth sensor according to some embodiments.

[0105] like Figure 5 As shown, in an exemplary use case scenario, a depth sensor can be used for prompting and / or assistance functions in a car, such as as a parking assistance sensor.

[0106] Depth sensors can be mounted on vehicles. A single depth sensor can be used, or alternatively, multiple depth sensors can be used to achieve greater coverage, accuracy, and reliability. The depth sensor can be a LiDAR sensor, such as a dToF LiDAR imaging system. The laser attached to the LiDAR sensor emits precisely timed light pulses, and the LiDAR sensor receives reflections from surfaces within its field of view.

[0107] The sensor can measure the distance to each pixel within its field of view. It can also measure the amount of light falling on each pixel, just like a traditional camera. This will generate two images that are perfectly aligned with each other. - Depth map (i.e., depth estimation per pixel); - Grayscale regular images (intensity or brightness images).

[0108] Figure 6 The left side shows the depth sensor. Figure 5 The illustration shows a depth image generated for the scene shown, with the grayscale image generated by the depth sensor for the scene on the right.

[0109] In some embodiments, the scene can be analyzed to decompose the depth map into surfaces, obstacles, etc. Object detection and identification can be performed. For example, this information can be used to perform one or more of the following operations: - Avoid collisions between the car and obstacles; - Avoid collisions with pedestrians or other road users; - Assess whether the car is suitable for parking space; - Track the car's location as it navigates to the parking space; -etc.

[0110] This article will discuss in further detail the internal structure and design of sensors based on some embodiments of the disclosed subject matter.

[0111] Now for reference Figure 7 , Figure 7 This is a block diagram of an exemplary module for dynamic gating of a pixel array in a dToF LiDAR imaging system according to some embodiments.

[0112] like Figure 7As shown, in some embodiments, a histogram monitoring module (or simply monitoring module) may be provided for performing one or more functions or actions of dynamic gating of the sensor array and / or executing related instructions. The histogram monitoring module can analyze the histogram as photon arrival times are collected and may have the ability to control and configure the dToF LiDAR sensor. The histogram monitoring module can enable and / or disable SPAD groups based on the results obtained from such analysis of the collected histograms.

[0113] The histogram monitoring module may include and / or be coupled to a histogram data module (not shown). The histogram data module may collect raw data of the histogram of photon arrival times and / or derive statistics from its contents, and provide the data to the histogram monitoring module and / or one or more of its sub-components for further processing, storage, etc. The statistical information derived from the histogram data may include the corresponding peak values ​​of photon events counted at individual bins in each histogram, the corresponding cumulative sum of photon event counts across all bins in each histogram, and other information.

[0114] The histogram monitoring module may include a pixel metric estimator that can use statistics derived from the histogram (and peaks, etc.), such as those available from the histogram data module, and can calculate metrics such as: (a) energy consumption metric for each pixel, (b) confidence that the histogram peaks have been correctly identified, (c) accuracy estimate of the identified peaks, etc.

[0115] A pixel metric estimator can calculate the energy consumption of a pixel based on the sum of its histogram values, as this represents the total number of photon events (i.e., avalanches) at the pixel SPAD. Confidence can be calculated by the pixel metric estimator based on the signal-to-background ratio (SBR), which is obtained, for example, by comparing the peak value (height / number of events) with the level of underlying background noise, represented by the height (mean, median, maximum, etc.) of the event counts in the remaining bins of the histogram. Accuracy estimation can be determined by the pixel metric estimator based on known parameters and / or characteristics of the sensor and its physical composition, thereby allowing for the prediction of accuracy under specified conditions. These characteristics can include, for example, factors and / or mechanisms that may cause distortion in the histogram (i.e., peak shift, shot noise, etc.), such as the duration of the dead time after photon irradiation, during which affected SPADs in the pixel array require recharging, the shape and / or precision of the laser pulse (e.g., whether it is short and reaches its peak quickly, or is more dispersed in time, rises more slowly, and / or has afterpulses, etc.), and so on. When examining the contents of a histogram, such as the intensity and / or location of peaks (i.e., the distance at which laser flash reflections return from the object to the sensor), SBR, background noise diffusion and / or level, given a known performance benchmark for the sensor, an assessment of the measurement accuracy can be made. For example, depth measurements may be accurate to 1 cm for objects at certain specified distances, while for objects at greater distances and / or under different conditions, depth measurements may be only 10 cm or even less precise.

[0116] The histogram monitoring module may include a continue / stop decision module (or simply continue / stop decision module) that can use pixel metrics (peak confidence and / or accuracy, power consumption, etc.) from a pixel metric estimator to determine whether to continue collecting photons in each pixel (SPAD remains enabled) or to stop photon collection (SPAD is disabled). In some embodiments, the continue / stop module can make this determination based on a configuration strategy for accuracy requirements, peak confidence, and / or power consumption. Optionally, scene information, such as that obtained by analyzing the histogram and the generated intensity image, can be used; these requirements may differ for different parts of the scene, as discussed herein.

[0117] In some embodiments, the histogram monitoring module may include data storage of per-pixel laser flash counts for recording and / or updating the number of flashes emitted by a laser source whose SPAD pixels have been activated or enabled. The histogram monitoring module may acquire the total number of laser flashes emitted throughout the measurement process, as represented by a laser flash counter, which, together with the stored per-pixel laser flash counts, can be used by the continue / stop module to adjust pixel brightness values ​​in the intensity image (i.e., the histogram), for example, as a preprocessing step before performing analysis to obtain scene information. Additionally or alternatively, for example, the continue / stop module may use the stored per-pixel laser flash counts and / or laser flash counters to compensate for histogram distortion (e.g., due to SPAD dead time, etc.) and check for compliance with a configuration strategy for modified pixel accuracy and / or confidence metrics.

[0118] Now for reference Figure 8 , Figure 8 This is a block diagram of another exemplary module for dynamically gating a pixel array in a dToF LiDAR imaging system using pixel block data analysis, according to some embodiments.

[0119] like Figure 8 As shown, in some embodiments, the same as described herein may be provided. Figure 7 The histogram monitoring module described and illustrated is similar to other histogram monitoring modules, and may further include a hierarchical histogram grouping and compartmentalization module (or simply a grouping and compartmentalization module) that can combine histograms from neighboring pixels. If each histogram in a pixel's histogram contains too few photons, peaks cannot be reliably identified due to shot noise in Poisson statistics, as discussed herein. Peaks in the combined histograms can be determined more reliably by accumulating a set of histograms from neighboring pixels. Depending on the quality of the detected peaks, this grouping and summing may need to be performed multiple times at different scales until a peak reliably appears. This information can be used to estimate the length of time a pixel needs to remain enabled in order to identify strong peaks at smaller scales (e.g., down to a single pixel).

[0120] For example, suppose that block merging of pixels and / or groups of pixels must be performed 5 times to provide a combined histogram with peak identifiers at sufficient confidence. This means that the merged area is multiplied by 5, i.e., forming a set of 2 5= 32 pixels, all of whose histograms must be summed together so that peaks and background noise levels in the combined histogram can be found and distinguished from each other, thus allowing, for example, the SBR (for the entire group) to be calculated and whether it conforms to the configuration strategy. To obtain useful histogram content, 32 times the information must be assembled together, which means that for each individual pixel in the group, it may be necessary to continue collecting reflections 32 times longer than currently, in order to collect 32 times more photons as an initial estimate.

[0121] The continue / stop module can determine whether to continue collecting photon reflections at the SPAD of a pixel group based on the estimated exposure factor of the useful peak (as indicated by the hierarchical level of merging required at the grouping and compartmentation modules). For example, the continue / stop module can check whether the additional time required to collect photons for the pixel group might violate a configuration policy, such as consuming more energy than a specified quota. As another example, if a pixel group does not meet the conditions of the region of interest, i.e., is classified as belonging to an object that does not require high accuracy detection based on scene analysis, etc. (e.g., a distant tree in a car parking assist sensor use case), the continue / stop module can determine, based on the configuration policy, whether to continue activating the SPAD to achieve a more accurate measurement or deactivate the SPAD and resolve inaccurate results.

[0122] In some embodiments, the continue / stop module can acquire and use laser flash counts and / or stored flash counts per pixel to calculate a detection duration estimate (to estimate how many more flashes might be needed). For example, if 100 flashes have been performed so far while the pixel has been enabled, and the estimated exposure multiplier for the useful peak suggests 4x flashes are needed, the continue / stop module can determine that the pixel requires a total of 400 flashes, i.e., an additional 300 flashes. The continue / stop module can then decide whether to disable the pixel accordingly, based on a configuration policy and whether the added number of flashes conforms to that policy.

[0123] Now for reference Figure 9 , Figure 9 This is a schematic diagram of a combination of histogram data from multiple pixels of a sensor array, according to some embodiments.

[0124] like Figure 9 As shown, grouping adjacent pixels together allows previously unseen peaks to appear in the combined (cumulative) histogram. The size of the grouped regions can vary depending on the confidence level of the appearing peaks. This summation of the histograms of individual pixels can be used to identify objects and obstacles that were previously undetectable.

[0125] like Figure 9As shown, multiple noisy histograms (as shown on the left) can be summed to produce a single combined histogram (as shown on the right) to produce clearer peaks with improved confidence and accuracy.

[0126] Now for reference Figure 10 , Figure 10 This is a schematic diagram of pixel block grouping in a sensor array according to some embodiments.

[0127] exist Figure 10 In the diagram shown, the large shapes represent pixels that have been grouped together to increase confidence. The small shapes (i.e., unit rectangles) represent regions that have not yet been grouped together (the quality of these pixel histograms is high enough to measure available peaks without grouping pixels).

[0128] At each hierarchical level, a confidence metric can be evaluated for each pixel and / or group of pixels. If the confidence is low, pixel groups can be merged again to create a larger merged region until the desired confidence is achieved.

[0129] Now for reference Figure 11 , Figure 11 This is a block diagram of yet another exemplary module for dynamically gating a pixel array in a dToF LiDAR imaging system using analysis of previous frame data, according to some embodiments.

[0130] like Figure 11 As shown, in some embodiments, the same as described herein may be provided. Figure 7 The histogram monitoring module described and illustrated is similar to the histogram monitoring module, and the histogram monitoring module may also include a previous frame analysis module (not shown) and / or communicate with it to provide data and analysis of one or more previous frames to its histogram monitoring module and / or continue / stop module.

[0131] The previous frame analysis module can use 3D depth maps (and / or other image data) from previous frames to identify features and objects of interest, such as vehicles or obstacles in an exemplary use case of an automotive-mounted depth sensor for parking assistance, as referenced herein. Figure 5 and Figure 6 The discussed methods can improve the accuracy and correct peak confidence requirements of these objects, thereby providing greater certainty regarding the position and motion of objects relevant to the end use of depth sensors.

[0132] Data from one or more previous frames (i.e., depth maps and / or intensity images) captured by the sensor can be analyzed by the previous frame analysis module to further understand the scene. Scene information can also be derived from the current frame when acquiring the current frame, as discussed herein. This can be used to configure the sensor to require higher / lower accuracy in certain regions of interest (ROIs). For example, in a car scene, sky areas can be identified and discarded early (these areas have no measurable depth but consume high power). Objects detected as “objects of interest” (e.g., vehicles, pedestrians) may require higher 3D depth accuracy and therefore should allow for more laser flashes to achieve higher accuracy at the corresponding pixels. Scene regions with fast-moving objects may also have higher accuracy requirements. Scene regions corresponding to unknown or unidentifiable objects may also require higher accuracy to identify them.

[0133] Mobile vehicles typically use a range of techniques to determine their location, as well as the location of landmarks around them; one such method is simultaneous localization and mapping (SLAM). Landmarks identified in SLAM may require higher accuracy to improve self-localization performance and map-building accuracy. Therefore, these landmarks can be indicated as regions of interest for the sensors.

[0134] The continue / stop module can obtain the identification and / or classification of objects and / or ROIs in the current frame from the previous frame analysis module, as well as an indication of the accuracy requirements for each object, and accordingly and / or make a determination on the activation / deactivation of SPAD pixels based on a configuration strategy, which may vary by region based on the specified requirements.

[0135] It should be understood that the disclosed subject matter can determine per-pixel depth measurement performance in real time, as well as power consumption for reducing nearby objects or highly reflective objects.

[0136] It will also be understood that the disclosed topics provide hierarchical histogram grouping and compartmentalization, and predict when (and whether) a pixel will successfully generate a depth measurement, thereby enabling power consumption improvements for unmeasurable objects (e.g., the sky, the sun) that will never generate a peak in the histogram.

[0137] It will also be further understood that the disclosed subject provides a derivation of per-pixel energy consumption metrics for real-time decision-making, thereby enabling the identification of scene areas in the sensor that consume the most energy.

[0138] It will also be further understood that the disclosed subject matter can control SPAD activation (at the pixel / group level) in real time based on performance metrics and can shut down areas with excessive power consumption (e.g., blocking sunlight), where the identification and deactivation of affected pixels can even be completed in the early stages of measurement.

[0139] It will also be further understood that the disclosed subject provides different accuracy requirements for identifying objects within a scene through previous frame analysis, and their use in the power budget, which can therefore be allocated more efficiently to the region of interest of the application / scene.

[0140] It should be understood, specifically but not exclusively, that this article refers to... Figures 7 to 11 One or more of the described actions and / or functions can be executed by a computer program for a pixel array in a dynamically gated ranging sensor, the computer program including program instructions for performing the one or more of the described actions executable by at least one processor. Each of these programs can be provided on a non-transitory storage medium.

[0141] It should also be understood that one or more of the actions and / or functions described herein for dynamic gating of sensor arrays may be performed, implemented, and / or otherwise provided by processing circuitry, which may include, use, and / or otherwise facilitate one or more hardware modules (elements), such as electronic circuits, electronic components, integrated circuits (ICs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), central processing units (CPUs), graphics processing units (GPUs), artificial intelligence (AI) accelerators, etc.

[0142] In some embodiments, the processing circuitry may execute one or more software modules, such as processes, scripts, applications, agents, utilities, tools, operating systems (OS), etc. Each software module includes multiple program instructions stored in a non-transitory medium (program storage) coupled to, contained in, and / or otherwise communicated with the processing circuitry.

[0143] Additionally or alternatively, the processing circuitry may include, use, and / or otherwise facilitate memory and / or data storage devices, which may be provided with dedicated logic and / or in-memory processing to support simple metrics (e.g., total photon count, maximum value) for generating the histogram itself as it is collected and / or updated. For example, in some embodiments, the histogram's memory or data storage may count photon events entering the histogram to obtain a sum and maintain a record of the highest histogram bin value to date.

[0144] Similarly, one or more actions and / or functions for dynamic gating of sensor arrays as described herein may be executed, implemented, and / or otherwise provided by dedicated / hardwired control logic and / or other similar dedicated hardware, firmware, middleware, etc., which may optionally be deployed within the sensor and / or otherwise communicate with it to control its operation.

[0145] The descriptions of various embodiments are for illustrative purposes only, and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is intended to best explain the principles of the embodiments, their practical application, or technical improvements relative to existing technologies in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0146] It is anticipated that during the term of the patent formed by this application, many related ranging sensors, ranging imaging systems, and ranging tools and technologies will be developed, and the scope of the terms “ranging sensor,” “ranging imaging system,” “distance detection,” and / or “ranging” is intended to a priori include all these new technologies.

[0147] The term “approximately” as used in this article refers to 10%.

[0148] The terms “comprising,” “having,” and their variations mean “including but not limited to.” This term includes the terms “consisting of” and “substantially consisting of.”

[0149] The phrase “consistently of…” indicates that a composition or method may include other components and / or steps, provided that the other components and / or steps do not substantially alter the fundamental and novel characteristics of the claimed composition or method.

[0150] Unless the context clearly indicates otherwise, the singular forms “a” and “the” as used herein include the plural meaning. For example, the terms “a complex” or “at least one complex” can include multiple complexes, including mixtures thereof.

[0151] As used herein, the term "exemplary" means "as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as being more preferred or advantageous than other embodiments, and / or as excluding combinations of features of other embodiments.

[0152] As used herein, the term "optionally" means "provided in some embodiments and not provided in others." Any particular embodiment may include multiple "optional" features unless these features conflict with each other.

[0153] In this application, various embodiments are presented in a range format. It should be understood that the range format description is merely for convenience and brevity and should not be construed as a strict limitation on the scope of the embodiments. Therefore, the description of a range should be considered as having specifically disclosed all possible subranges and individual numerical values ​​within that range. For example, a description of a range, such as from 1 to 6, should be considered as having specifically disclosed subranges from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and individual numbers within that range, such as 1, 2, 3, 4, 5, and 6. This applies regardless of how broad the range may be.

[0154] All ranges of numbers specified herein refer to any referenced number (decimal or integer) within the specified range. The phrases “range between the first and second indicator numbers” and “range from the first to the second indicator number” are used interchangeably herein to refer to all fractions and integers including the first and second indicator numbers and those in between.

[0155] It should be understood that certain features of embodiments described in the context of a single embodiment for clarity may also be provided in combination in a single embodiment. Conversely, various features of embodiments described in the context of a single embodiment for brevity may also be provided individually or in any suitable sub-combination or appropriate in any other described embodiment. Certain features described in the context of various embodiments should not be considered as essential features of these embodiments unless the embodiment would be inoperable without these features.

[0156] While embodiments have been described in conjunction with their specific examples, it will be apparent to those skilled in the art that many alternatives, modifications, and variations will be readily apparent. Therefore, it is intended to cover all such alternatives, modifications, and variations that fall within the spirit and broad scope of the appended claims.

[0157] The applicant's purpose is that all publications, patents, and patent applications mentioned in this specification are incorporated herein by reference in their entirety, as if each individual publication, patent, or patent application were specifically and individually identified when referred to as being incorporated herein by reference. Furthermore, any reference or identification of any reference in this application shall not be construed as an admission that such reference is prior art to the invention. The use of section headings should not be construed as a necessary limitation. Additionally, the entire contents of any one or more priority documents of this application are incorporated herein by reference.

Claims

1. A processing circuit for dynamically strobing a pixel array in a ranging sensor using signal transmission and reflection, the processing circuit comprising: The processing circuit includes a memory and is used for: Analyze at least one histogram of the reflection arrival time counts for at least one pixel in the frame; Based on the analysis of the at least one histogram, at least one metric of the at least one pixel is determined; Based on the at least one metric and collection strategy, the reflection collection of the at least one pixel is selectively disabled.

2. The processing circuitry of claim 1, wherein, Analysis of the at least one histogram includes calculating at least one statistic based on the counts of the at least one histogram.

3. The processing circuit according to claim 2, characterized in that, Calculating the at least one statistic includes calculating at least one of the following: at least one peak of the counts of the at least one histogram; the cumulative sum of the values ​​of the counts of the at least one histogram.

4. The processing circuitry of claim 2, wherein, The at least one metric is calculated based on the at least one statistic, and the at least one metric includes at least one of energy consumption, confidence level, accuracy estimate, and detection duration estimate.

5. The processing circuitry of claim 2, wherein, It is also used to: count the number of transmissions corresponding to the activation timing of the pixel of the signal source of the ranging sensor for the pixel of the frame, and calculate the at least one metric based on the number of transmissions and the at least one statistic.

6. The processing circuitry of claim 1, wherein, The at least one histogram includes multiple histograms of adjacent pixels.

7. The processing circuitry of claim 1, wherein, The collection strategy varies depending on at least one region of interest.

8. The processing circuitry of claim 7, wherein, The at least one region of interest is determined by analyzing at least one of the depth map and intensity image of at least one other frame preceding the frame.

9. The processing circuitry of claim 7, wherein, The at least one region of interest is determined by analyzing scene information from an intensity image generated from the histogram of at least one subset of pixels.

10. The processing circuitry of claim 9, wherein, The analysis of the scene information of the intensity image is performed using a simultaneous localization and mapping (SLAM) process.

11. The processing circuitry of claim 9, wherein, It is also used for: counting the number of transmissions of the signal source of the ranging sensor corresponding to the activation timing of the pixel for the pixel of the frame; and using the number of transmissions to adjust the count value obtained from the count in the histogram of the pixel.

12. The processing circuitry of claim 11, wherein, The count value is adjusted based on the ratio between the number of transmissions counted for the pixel and the total number of transmissions performed by the signal source for the frame.

13. The processing circuitry of claim 1, wherein, The signal source of the ranging sensor is directional, and the processing circuit is further configured to selectively disable a corresponding beam portion of the signal source in a direction corresponding to the at least one pixel, based on the at least one metric and the collection strategy.

14. A method of dynamically strobing a pixel array in a ranging sensor using signal transmission and reflection, the method comprising: The method includes: Analyze at least one histogram of the reflection arrival time counts for at least one pixel in the frame; Based on the analysis of the at least one histogram, at least one metric of the at least one pixel is determined; Based on the at least one metric and collection strategy, the reflection collection of the at least one pixel is selectively disabled.

15. A computer program for dynamically gating a pixel array in a ranging sensor using signal transmission and reflection, characterized in that, The computer program includes program instructions that, when executed by at least one processor, cause the at least one processor to: Analyze at least one histogram of the reflection arrival time counts for at least one pixel in the frame; Based on the analysis of the at least one histogram, at least one metric of the at least one pixel is determined; Based on the at least one metric and a collection policy, selectively disabling reflection collection for the at least one pixel.

Images