Object grouping for time-of-flight ranging algorithms

Abstract

The invention relates to object grouping for time-of-flight ranging algorithms. According to an embodiment, a method for object abstraction is presented to distinguish different objects in a depth map generated from a time-of-flight sensor. The method includes performing a raster search or a spiral search on the depth map. The raster search includes systematically alternating between left and right scans and then up and down scans. The spiral search includes circumferential scanning. The method further includes detecting an edge of the object to guide continuation of the raster search or spiral search; calculating one or more advanced parameters related to the object; and repeating a raster search or a spiral search on cells in the depth map that have not been previously scanned to similarly identify other objects.

Description

Technical Field

[0001] The present disclosure relates generally to electronic imaging and, in particular embodiments, to time-of-flight ranging algorithms. Background Art

[0002] Time-of-flight (ToF) ranging devices are becoming increasingly common in applications such as autonomous vehicles, consumer electronics with gesture recognition capabilities, and 3D mapping technology. However, their accurate performance can be severely compromised by the generation of false targets. False targets are erroneous readings that can be mistaken for real objects within the device's field of view, resulting in inaccurate range measurements.

[0003] False targets can arise from a variety of sources, with lens blur and obscuring glare being two prominent examples. Blur occurs when the lens or foreign matter on the lens causes the light signal to diffuse, resulting in a diffuse reflection that lacks sharpness on the sensor. This diffusion can cause readings that falsely indicate the presence of an object or distort the perceived location of the actual object.

[0004] Obscuration glare is a non-ideal, uniform signal that spreads across the entire field of view of the sensor and is typically caused by reflections from lens components, the surrounding housing, or foreign matter such as dirt or dust. Obscuration glare significantly reduces contrast and can mask or mimic the presence of a real target, creating a false target.

[0005] Another source of false targets is the mismatch between the laser pulse shape and the response from single-photon avalanche diodes (SPADs). The long tail on vertical-cavity surface-emitting laser (VCSEL) pulses, in conjunction with the SPAD detector, can have after-pulse effects that exacerbate the problem by producing extended signals that last longer than the actual laser pulse duration. These long signals can be misinterpreted as additional, non-existent objects.

[0006] Flicker, which occurs at the detection limit, exacerbates this problem by introducing temporal fluctuations in the measurement signal. This flicker can be caused by various factors, such as changes in the reflectivity of the target surface or environmental conditions (such as changes in ambient lighting). The resulting intermittent readings lead to further uncertainty and false target generation.

[0007] ToF devices are designed to be highly sensitive, which, while beneficial for accurately detecting actual targets, paradoxically makes them more susceptible to these unwanted false targets. Sensitivity means that even small signal perturbations can be detected and potentially classified as objects by the system. Summary of the Invention

[0008] Technical advantages are generally achieved by embodiments of the present disclosure, which describe object grouping for a time-of-flight ranging algorithm.

[0009] A first aspect relates to a method for object abstraction to distinguish different objects in a depth map generated from a time-of-flight sensor. The method includes performing a raster search or spiral search on the depth map. The raster search systematically alternates between left-right scans and then up-down scans. The spiral search includes a circumferential scan. The method also includes detecting edges of objects to guide the continuation of the raster search or spiral search; calculating one or more high-level parameters related to the objects; and repeating the raster search or spiral search for previously unscanned cells in the depth map to similarly identify additional objects.

[0010] A second aspect relates to a method for pseudo-circumferential search to determine occlusion effects in a depth map generated from a time-of-flight sensor. The method comprises: determining a starting cell with a maximum signal value within the depth map; generating a first boundary region including cells adjacent to the starting cell; calculating a first ratio corresponding to the number of cells within the first boundary region having values ​​within a first threshold and the total number of cells within the first boundary region; in response to the first ratio being above a second threshold, generating a next boundary region including cells adjacent to the first boundary region; and in response to the first ratio being below the second threshold, stopping the pseudo-circumferential search, wherein the total number of cells within the second-to-last boundary region is used to determine the occlusion glare effect.

[0011] A third aspect relates to a method for flatness measurement to determine occlusion effects in a depth map generated from a time-of-flight sensor. The method comprises: determining a starting cell with a maximum signal value within the depth map; generating a first boundary region including cells adjacent to the starting cell; calculating a first flatness measurement for the first boundary region; generating a next boundary region including cells adjacent to the first boundary region; calculating a second flatness measurement for the next boundary region, wherein each flatness measurement is calculated based on an average range value and a normalized error value of cells within the corresponding boundary region; and calculating a total flatness measurement to determine occlusion glare effects.

[0012] Embodiments may be implemented in hardware, software, or any combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which:

[0014] Figure 1 is a block diagram of an embodiment system;

[0015] Figure 2 is an example of a depth map from a multi-zone ToF sensor;

[0016] Figure 3 is a flow chart of an embodiment method for grouping objects (i.e., object abstraction) using a depth map from a ToF sensor;

[0017] Figure 4 yes Figure 3 Implementation of the method on an exemplary depth map;

[0018] Figure 5 is an example from a bowl obscured from glare, represented as concentric range circles;

[0019] Figure 6 is a flow chart of an embodiment method for pseudo-circumferential search and flatness measurement for detecting occluding glare in a depth map from a ToF sensor;

[0020] Figure 7 is a graph showing the distribution of ranges of embodiments for detecting obscured glare;

[0021] Figure 8 is an example of a depth map from a multi-zone ToF sensor; and

[0022] Figure 9a-9c is based on Figure 6 Progress of methods for pseudo-circumferential search and flatness measurement. DETAILED DESCRIPTION

[0023] The present disclosure provides many applicable inventive concepts that can be embodied in a variety of specific contexts. Specific embodiments merely illustrate specific configurations and do not limit the scope of the claimed embodiments. Unless otherwise indicated, features from different embodiments may be combined to form additional embodiments. Various embodiments are shown in the accompanying drawings, where like components and elements are identified by like reference numerals, and repeated descriptions are omitted for brevity.

[0024] The changes or modifications described in one embodiment may also be applied to other embodiments. In addition, various changes, substitutions and alterations may be made herein without departing from the spirit and scope of this disclosure as defined by the appended claims.

[0025] While aspects of the present invention are primarily described in the context of time-of-flight sensors and sharpening algorithms, it should be understood that they can also be applied to other imaging devices and other types of algorithms. In particular, aspects of the present disclosure can be similarly applied to commercial or consumer electronic devices such as optical imaging devices, scanners, medical imaging devices, security cameras, and the like. Similarly, embodiments of the present disclosure can be applied to long-tail filtering, anti-flicker filtering, and glass detection algorithms, among others.

[0026] In an embodiment, an algorithm is presented that serves as a robust mechanism for grouping and abstracting objects based on data acquired from a time-of-flight sensor. The algorithm effectively helps eliminate false targets that may be present within data collected from a multi-zone time-of-flight ranging device. Implementation of this algorithm ensures that sensor data is accurately processed by distinguishing false readings and providing a clearer, more accurate representation of the environment detected by the time-of-flight sensor.

[0027] Traditional algorithms for filtering and processing data from time-of-flight (ToF) ranging devices face several significant shortcomings. The main problem is the loss of actual weak targets, which can be caused by the algorithm mistakenly classifying real signals as being caused by blur, glare, flicker, or the long tail effect of the laser pulse. The faintness of the target signal can make it very similar to the noise generated by these phenomena, leading to its accidental elimination during the cleaning process. Overall, these improvements enhance the reliability and accuracy of ToF devices, ensuring that they can provide the precise range and object detection required for critical applications.

[0028] Furthermore, some false targets caused by blur, obscuring glare, glint, or long tails can still evade detection by these algorithms and be mistakenly identified as legitimate targets. These lingering false targets impair system performance by introducing inaccuracies in range measurements. The core problem with these existing algorithms is their overly simplistic approach. They often lack the nuance required to effectively handle the complex nature of these interferences. As a result, they not only discard valid but weak signals but also fail to identify all inaccuracies.

[0029] In an example scenario, a ToF ranging device measures the distance to an object on a white table that includes a black keyboard. The keyboard and the tabletop may be at the same distance from the ToF ranging device, with the black color of the keyboard absorbing more light than the white tabletop. Consequently, the signal reflected to the sensor from the keyboard is weaker than that from the white tabletop. In contrast, the response from the white target is strong, resulting in occluding glare across the field of view. The weak reflection from the black keyboard and the resulting occluding glare from the white background may prevent conventional sharpening algorithms from distinguishing between the weak signal from the occluding glare of the lens and the weak signal from the true target (i.e., the black keyboard), thereby compromising object recognition and ranging accuracy.

[0030] The embodiment of the present disclosure classifies (i.e., groups) different objects (i.e., the black keyboard and white table here) into separate objects. By doing so, weak signals from different objects can be processed differently, and the effects of obscuring glare can be removed from data readout.

[0031] One mitigation technique for occlusion glare includes algorithms that identify and subtract occlusion glare contributions from the measured signal. For example, sharpening algorithms aim to clarify the actual edges of objects in the FoV by enhancing sharpness and contrast along these boundaries.

[0032] In an embodiment, enhancements to existing or new algorithms are proposed in sharpening, long-tail filtering, anti-flicker filtering, and glass detection algorithms. The sharpening algorithm counteracts the effects of blur by enhancing edges and improving object clarity in the sensor data. The long-tail filtering algorithm addresses issues such as extended signal decay times caused by VCSEL pulses and anomalous SPAD response, effectively truncating or reshaping the long tail to more accurately reflect the actual target distance. Anti-flicker filtering stabilizes the fluctuating signal caused by flicker, normalizing the readings over time to distinguish between false targets caused by near-limit detection issues and real object data. Furthermore, the glass detection algorithm identifies and compensates for signals distorted by transparent or reflective surfaces that might otherwise produce ghosting or other false targets within the sensor range.

[0033] Sharpening algorithms have been developed to enhance the quality of data from ToF sensors. Initially, basic sharpening techniques relied on simple thresholds to clarify the readings. These early methods were fundamental but limited in sophistication and ability to solve complex imaging problems that could arise from sensor data.

[0034] As technology advances, new sharpening methods have been developed that integrate range- and signal-based sharpening techniques. This marks an advancement in processing ToF sensor data by taking into account distance information and signal strength, thereby enabling more detailed enhancement of sensor readings. These algorithms focus not only on improving sharpness, but also on removing false targets that may be caused by lens blur (a common defect in imaging that can cause images to be out of focus) and other artifacts.

[0035] Subsequent iterations achieved further improvements by introducing simplified models to counteract occlusion glare, which is typically caused by stray light within the camera system and reduces contrast and removes image detail. Addressing this issue allows for a clearer distinction between actual targets and artifacts caused by lens imperfections, and improves the overall fidelity of ToF sensor readings.

[0036] Typically, lens sharpness plays a significant role in sharpening algorithms, as it directly impacts the sharpness of the raw sensor data; better initial image quality leads to more effective sharpening. Furthermore, module occlusion glare can severely degrade image quality; therefore, algorithms that effectively combat its effects are advantageous. Furthermore, the pulse segmenter's ability to distinguish targets within the region of interest enables it to clearly define objects in the field of view and meet specifications for spatial object discrimination.

[0037] Figure 1 1 shows a block diagram of an embodiment system 100. The system 100 includes a processor 102, a memory 104, a time-of-flight (ToF) sensor 106, a power supply unit (PSU) 108, and an interface 110, which may or may not be arranged as shown. Figure 1 1. One of each component (i.e., processor 102, memory 104, ToF sensor 106, power supply unit 108, and interface 110) is shown, but the number of components is not limiting, and a greater number is similarly contemplated in other embodiments. System 100 may include additional components not depicted, such as long-term memory (e.g., non-volatile memory, etc.), power management circuitry, security and encryption modules (e.g., Trusted Platform Module (TPM), etc.), etc. System 100 may be an electronic device (e.g., a smartphone, tablet, laptop, smartwatch, vehicle) or any system or subsystem capable of hosting ToF sensor 106.

[0038] In an embodiment, each component may communicate with any other component internal or external to the system 100. For example, each component may communicate using an I2C (Inter-Integrated Circuit) (also referred to as I2C or IIC) communication protocol, an I3C (Improved Inter-Integrated Circuit) communication protocol, or a Serial Peripheral Interface (SPI) specification, among others.

[0039] The processor 102 may be any component or collection of components suitable for performing computations or other processing-related tasks. In an embodiment, the processor 102 is an application processor, a baseband processor, or a microcontroller.

[0040] The memory 104 may be any component or collection of components suitable for storing programs or instructions for execution by the processor 102. In an embodiment, the memory 104 comprises a non-transitory computer-readable medium.

[0041] The ToF sensor 106 uses the speed of light to measure the distance to objects in its field of view. The ToF sensor 106 transmits a light signal, which travels to the target object, is reflected by it, and then captured by the ToF sensor 106. The time required for this round trip is measured, and because the speed of light is constant, the ToF sensor 106 can use this time measurement to accurately calculate the distance to the object.

[0042] The ToF sensor 106 includes a light source 112, which is typically an infrared (IR) LED, a laser diode, or a vertical cavity surface emitting laser (VCSEL). The light source 112 transmits a light signal toward the object to be measured. In one embodiment, the ToF sensor 106 uses a continuous light wave (i.e., indirect time-of-flight (iToF)). In another embodiment, the ToF sensor 106 uses a pulsed light signal (i.e., direct time-of-flight (dToF) application).

[0043] At the receiving end of the signal, there is a photodetector array 114 that is sensitive to a specific wavelength of the emitted light. The ToF sensor 106 may include a lens system 116 to focus the emitted light into a beam and ensure that the reflected light is directed onto the photodetector array 114. In an embodiment, the photodetector array 114 is a single photon avalanche diode (SPAD) detection array. The ToF sensor 106 may include additional components not shown, such as memory, a microcontroller, and a VCSEL driver.

[0044] The ToF sensor 106 may include a timing circuit 118 that is responsible for accurately measuring the interval between the time light is emitted and the time the light is detected after reflection. In an embodiment, the processor 102 receives data from the ToF sensor 106, interprets the timing data, and converts it into a distance measurement. The processor 102 may apply an algorithm to refine the data to compensate for factors such as ambient light noise or changes in object reflectivity to provide more reliable distance information. The ToF sensor 106 may be a multi-zone ToF sensor that can measure distances in several separate zones, such as 4×4, 8×8, or 16×16 zones.

[0045] The power supply unit 108 may be any component or collection of components that provides power to one or more components within the system 100. The power supply unit 108 may include various power management circuitry, charge storage components (ie, batteries), and the like.

[0046] The interface 110 may be any component or collection of components that allows the processor 102 to communicate with other devices / components or a user. For example, the interface 110 may be adapted to allow a user or the ToF sensor 106 to interact / communicate with the system 100 .

[0047] Figure 2 An example of a depth map 200 from a multi-region ToF sensor, such as the ToF sensor 106, is illustrated. The depth map 200 includes 64 cells 202 arranged in an 8×8 grid. i,j , where i=1 to 8 and j=1 to 8. Each cell (also referred to as a grid element, pixel, or region) can be identified by a subscript representing the X and Y coordinates in the depth map 200. For clarity, when discussing a cell in the depth map 200 generally, it is represented by cell 202. For example, cell 202 1,1 refers to the cell 202 located at the upper left corner of the depth map 200, and the cell 202 8,8 Refers to the cell 202 located at the lower right corner of the depth map 200 .

[0048] It should be understood that in embodiments, the depth map 200 may include a fewer or greater number of cells (e.g., 16 cells in a 4×4 grid). Thus, the number of cells 202 in the depth map 200 is non-limiting, and other arrangements are similarly contemplated in other embodiments.

[0049] Range profiling using the ToF sensor 106 involves measuring the distance to objects in different regions of the sensor's field of view by detecting the time required for light to be emitted from the ToF sensor 106, reflected from the object, and returned to a photodetector in the photodetector array 114. The principle behind ToF range profiling is the relationship between the time of flight of light and the distance it travels, where distance is calculated by multiplying the required time by the speed of light.

[0050] A multi-zone ToF sensor is designed to collect depth information from multiple specific areas within its field of view (typically simultaneously). In an embodiment, light source 112 is an array of emitter elements that projects light onto the scene, and a corresponding array of photodetectors 114 captures the reflected light. In an embodiment, each photodetector corresponds to a different zone or region in the scene. By controlling the illumination and readout of each zone, and by employing timing electronics, ToF sensor 106 can construct a detailed depth map representing the distance to each zone.

[0051] Each cell 202 within the depth map 200 corresponds to a specific photodetector within the photodetector array 114. In essence, the depth map 200 is a pixelated representation of the scene, where each cell 202 represents a volumetric portion of the sensor's field of view. Each cell 202 encodes the distance from the ToF sensor 106 to the object surface in that particular direction.

[0052] When light source 112 illuminates a scene, the light is reflected by objects and returns to a corresponding photodetector in photodetector array 114. Each photodetector records the time it takes for this light to return. These time measurements are then converted into distance values ​​using the known speed of light. Thus, in depth map 200, each cell 202 contains a value reflecting the measured distance for its particular region in space, which translates to how far away objects are relative to the sensor in that region.

[0053] The spatial resolution of depth map 200 (i.e., how finely it can represent surface details) is determined by the number and density of cells 202 in the map, corresponding to the number of photodetectors in photodetector array 114. A smaller grid with a higher density will produce a finer resolution depth map that captures more detailed information about the scene geometry. Conversely, a lower density with larger grid elements will produce a coarser representation that is sufficient for detecting larger objects or applications that require less spatial accuracy. The sensor's ability to quickly or simultaneously capture multiple regions allows for real-time range analysis, which is critical in dynamic environments or applications such as robotic navigation, gesture recognition, and 3D modeling.

[0054] The visual representation of the depth map can be similar to a grayscale image, where the intensity of each cell 202 corresponds to a distance value - darker cells 202 represent closer distances, and lighter cells 202 represent farther distances, and vice versa, depending on the chosen convention. Some depth maps can also be color-coded to enhance visual contrast and understanding of depth information.

[0055] In the numerical representation of the depth map, each cell 202 in the depth map 200 i,j The depth map includes values ​​representing the distance from the ToF sensor 106 to an object or surface at a specific point in the field of view. These numbers effectively replace the grayscale or color gradients typically used in visual depth maps.

[0056] 202 per unit i,j A number may be included that indicates how far the object is from the ToF sensor 106, e.g., a smaller number indicates closer space and a larger number indicates a greater distance. This numerical approach allows for precise distance quantification for each point on the unit 202, making it particularly suitable for applications where numerical data is superior to visual representations, such as in autonomous navigation systems or robotic control algorithms, where distances need to be processed algorithmically rather than visually.

[0057] The outline of an object 204 within the scene is presented on the depth map 200 as a collection of cells 202 that are distinguished from the background by representing a closer distance. Therefore, the number corresponding to the object 204 will be smaller than the number of the background, thereby revealing the location and shape of the object. For example, a cell with a value of '312' indicates that the object 204 is 312 units (e.g., millimeters) away from the sensor at that point, while a cell with a value of '326' is 326 units away from the sensor. As shown in the figure, for example, the cell 202 with a value of '312' 3,3 It is the cell 202 closest to the object 204 .

[0058] The edges or outlines of object 204 on the grid are identified by observing transitions between numerical values, where there is an abrupt change from a smaller (closer) to a larger (farther) number. This transition distinguishes the boundaries or outlines of object 204 from other objects or areas in depth map 200. By scanning across the grid, these outlines can be delineated and the shape of object 204 and its spatial orientation relative to other elements in the scene can be understood.

[0059] In some embodiments, each cell 202 may have multiple range values, for example, including a foreground range and a background range. This configuration may be relevant when the occluding glare from foreground object 204 produces a uniform range value across the entire FoV. Therefore, the background range outside the area covered by object 204 may be perceived only as a secondary range outside the initial range caused by the occluding glare.

[0060] Each cell 202 not only helps to outline the object 204, but also provides data that is crucial to understanding its three-dimensional structure. This information can be used to construct a 3D model of the object 204, or to determine its position and size relative to the ToF sensor 106 and other objects.

[0061] This obscuring glare artifact makes it difficult for a Time of Flight system to distinguish between direct signals from actual targets and scattered light from obscuring glare, which affects both covered and uncovered areas of the FoV. Therefore, without proper mitigation, a Time of Flight system may behave like a "single-zone" sensor, even though it is a multi-zone sensor capable of detecting different ranges, because it can read the range anywhere affected by obscuring glare.

[0062] Cells 202 adjacent to the edge or outline of object 204 may present a deceptive appearance of the object due to, for example, lens flare or VCSEL tail effects. Conventionally, a specified threshold is applied to the readings of cells 202, and the effects of obscuring glare are detected and removed.

[0063] In scenes with complex scenes, different target objects with varying distances and reflectivity can challenge the system 100's ability to distinguish and accurately measure range. ToF sensors are sensitive to even small signals, so obscuration glare becomes a significant issue. Obscuration glare is stray light that scatters within the device's optics and creates unwanted light contributions across the photodetector array 114.

[0064] In the example scenario, the knife-edge target covers half of the field of view (FoV) of the ToF sensor 106, and the background target is further away. The knife-edge target is directly between the ToF sensor 106 and the background target, obscuring part of the background. Because this stray light may come from brighter parts of the scene (in this case, potentially from the knife-edge or the background target), it may generate false range readings across all regions of the photodetector array 114 (e.g., blurring at the edges of the target and obscuring glare at the background).

[0065] Embodiments of the present disclosure provide deconvolution techniques and algorithms designed to mathematically reverse the effects of image blur. Grouping algorithms are proposed to more accurately segment pulse data into regions of interest. These algorithms further refine the clarification process and enhance the detection of spatial objects. These advances allow for the removal of false targets in ToF sensor data that may be caused by problems such as lens blur and obscuring glare, which can reduce the reliability of sensor output and complicate subsequent tasks such as object recognition and scene analysis.

[0066] Once the grouping algorithm has classified the different objects into different groups, a unique thresholding process is applied to each group. For example, a foreground target characterized by a robust signal undergoes a thresholding process, where any elements within its range group that exhibit a weak signal or are identified as occluding glare are eliminated. This ensures that only the most realistic representation of the foreground target remains. The same thresholding principle can be applied to background targets. Although the background signal is weaker than the foreground, background targets also produce occluding glare, albeit even weaker than foreground targets. By utilizing thresholding techniques, occluding glare can be removed from background targets.

[0067] Traditional algorithms for grouping objects take a relatively simple approach—sorting targets in ascending order. They first target the closest range within the first region, then continue with the next closest range, and so on. A threshold is applied to the gap between each sorted range within the region. If a range closely matches the previous range in order, connectivity is inferred and they are classified as the same object. Conversely, a significant gap between two sorted ranges indicates the presence of different objects, potentially indicating background objects.

[0068] However, this traditional approach has limitations due to its lack of spatial awareness. For example, two isolated areas on opposite ends of the array (e.g., both at 100 mm) may be incorrectly grouped as part of the same object due to the same range, despite a large spatial gap between them. In this case, if two objects have different reflective properties (such as a black keyboard on a white surface), if these items are incorrectly grouped or treated as a single object, a thresholding system designed to remove obscuring glare may mistakenly treat the less reflective item (the darker object) as part of the obscuring glare. This is because the system cannot recognize it as a separate object, resulting in an inaccurate representation of the imaged object. Therefore, while the simple algorithm is viable in initial, less complex applications, its shortcomings are clear in scenarios that require more precise spatial differentiation.

[0069] In an embodiment, the algorithm enhances spatial perception capabilities (i.e., object abstraction features). Initially, the goal is to accurately group actual objects in both temporal and spatial dimensions. Temporal grouping involves considering the range of an object, or the time it takes for a light pulse to travel to the object and back. Spatial grouping links corresponding ranges across each zone in the array, arranging them side by side or vertically. This allows the algorithm to distinguish if an object is physically discrete, then it appears to be a single object because the range readings resemble separate objects. By doing so, the system avoids inappropriately grouping multiple objects into a single entity.

[0070] Figure 3 Illustrated is a flow diagram of an embodiment method 300 for grouping objects (ie, object abstraction) using the depth map 200 from the ToF sensor 106 . Figure 4 The method 300 is illustrated as being implemented on the depth map 200. The method 300 may be executed, for example, by the processor 102 using instructions stored in the memory 104. The method 300 is implemented by Figure 4 Note that all steps outlined in the flowchart of method 300 are not necessarily required and may be optional. Additionally, changes to the arrangement of steps, removal of one or more steps and path connections, and addition of steps and path links may similarly be considered.

[0071] Method 300 allows for simultaneous consideration of temporal and spatial factors to distinguish and cluster objects indicated on a depth map from a ToF sensor. This dual-axis analysis significantly enhances the algorithm's ability to distinguish objects compared to conventional techniques, particularly when encountering objects within the same range from the ToF sensor 106. Advantageously, method 300 enables the identification and separation of two objects that exhibit a discernible gap in range measurements when within close-range proximity, effectively achieving a clear distinction. Subsequent algorithms can then process these individually clustered objects individually, improving the accuracy and effectiveness of system 100.

[0072] In an embodiment, method 300 employs object abstraction techniques to distinguish between different objects by performing a raster search or spiral search pattern, for example, by systematically alternating between left-right scanning and then up-down scanning. During this scanning process, the edges of the object are detected, thereby guiding the continuation of the search. After locating the object, the method allows for the determination of various high-level parameters associated with the object. Determining these high-level parameters includes, for example, calculating its center of mass, determining its size in the X and Y dimensions, establishing minimum, maximum, and median ranges, and evaluating its reflectivity by analyzing the return signal based on its range. This comprehensive set of parameters provides a detailed understanding of the object's characteristics.

[0073] In step 302, the grouping process begins by pinpointing the nearest target. This initiation requires selecting the cell with the smallest depth value recorded in the depth map 200, with the additional condition that it has not been merged into any group before. This ensures systematic and non-redundant coverage of all targets in the map. Figure 4 Unit 202 3,3 is the cell 202 closest to the object 204. Therefore, the grouping process starts from the starting cell 202. 3,3 start.

[0074] In step 304, the depth map 200 is scanned horizontally and vertically to search for the depth map starting from the starting cell (e.g., 202 3,3 ) continues. Specifically, scanning proceeds cell by cell to check for spatial connectivity with the previously inspected cell 202. This bidirectional scanning focuses on identifying and linking adjacent blocks that indicate continuous portions of an object. When it is determined that the range of a cell differs from the previously inspected cell by a certain threshold, scanning proceeds vertically by one cell based on a predetermined search algorithm. It then moves horizontally relative to the previous row. The search sequence continues until it is determined that all scanned edge cells exceed the specified threshold.

[0075] In an embodiment, after the initial starting point has been identified in step 302, a spiral scanning process begins across the depth map 200. This involves proceeding from one cell 202 to the next in a spiral pattern to ensure that each cell 202 has spatial connectivity with previously examined cells.

[0076] In an embodiment, the spiral scanning process involves a circular motion intended to detect and connect adjacent segments representing continuous portions of an object. If the extent of a cell 202 is within a predefined threshold compared to the extent values ​​of its immediate inner neighboring cells, it is considered to be part of a spiral pattern. When this condition is met, the scan continues to cover another cell in the spiral, expanding outward or cyclically according to the established search algorithm. This expansion simulates the formation of concentric "rings", resulting in a similar pattern to the Figures 9a to 9c The spiral search sequence is maintained until all scanned peripheral cells are observed to exceed a preset range threshold.

[0077] For example, in Figure 4 , horizontal scanning is performed in the left direction from the starting unit 202 3,3 Start to Unit 202 3,1 , and in the right direction from the starting unit 202 3,3 Start to Unit 202 3,7 In Unit 202 3,1 The search continues downward to cell 202. 4,1 , then right until unit 202 2,7 Since Unit 202 2,7 If the range value is lower than the specified threshold, the search will resume at unit 202 2,6 . It then moves down to unit 202 5,6 , then move left until unit 202 5,1 ,At this point, the first scanning direction ends because the range values ​​of all adjacent cells that have not been scanned to the previous cell exceed the specified threshold.

[0078] In the right direction, due to unit 202 3,7 If the range value is below the specified threshold, the search will return to the previous cell 202 3,6 , then move up to unit 202 2,6 , then move left until unit 202 2,1 The search continues up to unit 202 1,1 , then turn left until you reach unit 202 1,6 ,At this point, the second scanning direction ends because the range values ​​of all adjacent cells that have not been scanned to the previous cell exceed the specified threshold.

[0079] During the initial search phase, the system 100 is tasked with locating spatially connected targets. These targets are identified by their interconnectedness, based on the depth information they present and their coordinates in the plane of the depth map 200. To ensure comprehensive target identification, the search encompasses multiple attributes, including range, sensor signal, and reflectivity values.

[0080] In an embodiment, object parameters are determined, adding detailed characteristics for each identified object. Parameters that may be determined include centroid coordinates (x,y) (which may be determined via the centroid method), and size measurements along the x- and y-axes. In addition, the system calculates statistics for both range and rate: minimum, maximum, and median values, as well as standard deviation (sigma). Depth information is derived by analyzing the variance within the grouped range data, and reflectivity detail captures the extent to which light is reflected from the surface of the object, categorized between minimum and maximum values, or measured as a median reflectivity. Additionally, rate and its variance are tabulated in a similar manner. In an embodiment, the analysis extends to categorizing surface characteristics on a sliding scale from Lambertian reflectance to specular reflectance; this may require a separate search starting with the areas exhibiting the highest rates.

[0081] Step 306 establishes spatial connections between adjacent regions by applying threshold-based criteria. Based on depth values ​​and other relevant parameters, these criteria determine whether consecutive cells share enough commonality to be considered part of the same object. The threshold identifies regions within the same group to distinguish between different objects.

[0082] In an embodiment, the thresholding method is not only intended to achieve the current solution, but also involves additional refinements to enhance the distinction between connected objects and separate objects. These refinements can lead to more accurate grouping based on an improved understanding of the continuity of objects in the three-dimensional space depicted by the depth map.

[0083] Therefore, after the objects are identified and characterized, they are ranked. This ranking system or the assignment of importance factors can be based on several properties (such as size and speed). Such ranking can prioritize or further analyze objects based on their perceived importance or relevance to the application.

[0084] At step 308, the previous steps are repeated for additional objects within the depth map. Each loop targets the next closest object that is still ungrouped after the previous iteration. This continues until the entire map has been searched, with each object evaluated and, where applicable, grouped together with neighboring objects that meet a predetermined threshold. This can include objects located in the background. Method 300 is repeated to construct a comprehensive visual representation from depth map 200, encompassing all detectable objects within the field of view.

[0085] Thus, method 300 starts with the nearest detected target, which, while typically located at the center of the field of view, is not necessarily located at the center of the object. The search is then performed simultaneously in two lateral directions—to the right and to the left from this starting point. As the search expands outward, if it encounters an area where the range measurement is significantly different from the measurements of adjacent ranges, it can conclude that this new range may not belong to the same object. At this point, the algorithm stops moving laterally in that direction and redirects the focus to a vertical scan—either up or down. This process continues until the entire dimension of the object has been mapped. After the search algorithm reaches areas that no longer correlate with the established ranges (indicating that they are outside the defined object), it terminates the search in those directions.

[0086] In an embodiment, when a search encounters a region or cell 202 that is determined to no longer belong to the same object, based on, for example, the value of the cell 202 falling outside a specified threshold, it does not immediately end the search for that object. Instead, the search can be extended to adjacent rows or columns—whether above, below, to the right, or to the left—continuing the search to detect any portion of the body that can still be connected to the object. This approach can help identify objects with prominent features in a single direction, such as objects with multiple angular extensions. Thus, the proposed search ensures comprehensive coverage and considers the full range of the object's unique characteristics.

[0087] For example, method 300 may identify object 204 as a cup that is 5 cm wide and 10 cm tall. The process may also locate the center of the cup to provide a high-level descriptive analysis of the object. After this initial identification, method 300 analyzes data from other regions within depth map 200 to confirm the accuracy of the acquired information and determine whether these additional regions represent valid data or false positives.

[0088] When analyzing depth map 200, if a measurement of 320 units is determined in the lower right corner in addition to object 204, a conventional sorting algorithm may incorrectly classify object 204 and the object in the lower right corner together due to their similar range values. This indicates a flaw in the system because it cannot distinguish between separate objects with similar range values. This can be particularly problematic for two objects with different reflectivities and different signals at the same range.

[0089] Figure 5 An example of a bowl 500 of obscured glare is shown, which is represented as concentric range circles. As depicted, a typical response pattern obtained from obscured glare generally resembles a bowl shape. This obscured glare can come from targets occupying several central areas within the field of view, where legitimate targets produce responses within range 502, represented as hashed areas within bowl 500 (i.e., the legitimate response range). Figure 5As shown, the range of obscured glare tends to expand as the distance from the target increases, resulting in a bowl-shaped effect.

[0090] In one embodiment, the algorithm detects occulting glare based on its characteristic shape within the range data. The algorithm can be programmed to search for this specific contour, enabling it to identify areas potentially affected by occulting glare. Because few naturally occurring objects mimic this bowl-like shape, its detection serves as an effective metric for distinguishing occulting glare from valid targets in the viewing environment. This tool ensures that this glare is properly identified and accounted for during imaging.

[0091] Several indicators can identify an object as bowl-shaped, such as search rings and concentric circles. For example, when outlining the surface of a bowl, these contours, or "rings," serve as the outlines for analysis. To determine whether something has a bowl-like shape, two key properties can be evaluated: the flatness and curvature of each contour. Examining flatness allows us to discern the location of each concentric ring in space, while assessing curvature helps us understand the curvature of each ring, thereby aiding in the overall determination of a bowl-shaped pattern.

[0092] Given this bowl-shaped effect, a spiral search strategy or pseudo-peripheral search strategy can be implemented to detect the presence of obscuring glare. In one embodiment, the pseudo-peripheral search begins at the area closest to the observer and proceeds outward in a spiral trajectory. Alternatively, a method can be employed to gradually scan through successive concentric circles, examining each circular band in turn to determine if the bowl-shaped effect is present. In one embodiment, the search begins at the cell exhibiting the greatest signal or intensity, such as in the case of speckle due to reflection from glass or a shiny surface.

[0093] Figure 6 A flow chart of an embodiment method 600 for pseudo-circumferential search and flatness measurement to detect occlusion glare in the depth map 200 from the ToF sensor 106 is illustrated. Figure 7 Graph 700 of an embodiment range distribution for detecting obscured glare is illustrated. Method 600 may be executed, for example, by processor 102 using instructions stored in memory 104. Method 600 is performed by Figure 2 and Figure 7 It should be noted that not all steps outlined in the flowchart of method 600 are required, and steps may be optional. Furthermore, changing the arrangement of steps, removing one or more steps and path connections, and adding steps and path links are similarly contemplated.

[0094] In embodiments where the photodetector array 114 has a low resolution, rectangles may be used as suitable substitutes for circles for the depth map to accurately represent the circular shape.

[0095] At step 602, the pseudo circumferential search and flatness measurement begins by determining the starting cell corresponding to the true target center. In an embodiment, the true target center is the cell with the maximum signal (i.e., the maximum signal area) in the depth map 200. In an embodiment, the center of the true target is the cell closest to the object. In an embodiment, the center of the true target is the cell that generates the maximum signal from the object due to, for example, the surface reflectivity of the object, and is not necessarily the cell closest to the object. For example, in the depth map 200, the starting cell 202 3,3 In an embodiment, the center of the real target is determined by object search, for example, by object search and center of gravity method using method 300 .

[0096] At step 604, the first rectangular ring to be evaluated is determined by moving one cell outward in all directions from the starting cell (i.e., counting from the cell with the largest signal area). Subsequent rings to be evaluated are determined by expanding the size of the ring by moving one cell outward from the previous ring.

[0097] At step 606, the lower boundary 706 of the ring being evaluated is calculated. The lower boundary 706 is set to the average inner limit of the tracking unit 202 under the condition that it remains non-decreasing. The plotting begins at an initial point at zero radius (0).

[0098] In an embodiment, lower boundary 706 is set based on the range of the true target center because other cells in the object should not have signal values ​​below the maximum signal area (i.e., a cell with a signal below the maximum signal area value of the true target would indicate that the other cell is the center, which is impossible). Therefore, by setting lower boundary 706 to track the range of the true target center, any cell distribution that falls below lower boundary 706 will be outside the bowl shape (e.g., area 704d).

[0099] In an embodiment, the lower boundary 706 is set to the maximum of two evaluation criteria. The first criterion is the previous inner lower boundary (or the point range when the radius is equal to '1'). The second criterion is the average range of boundaries of a certain radius, where a margin (e.g., a margin of '0.8') is applied to account for noise.

[0100] At step 608, an upper boundary 702 of the evaluated ring is calculated. Upper boundary 702 is set to the average of the outermost limits of tracking unit 202, but is designed not to exceed a predefined merging distance (i.e., where two pulses overlap). Upper boundary 702 is set based on the predefined merging distance because it is guaranteed that there will be different targets outside of this range of values ​​(e.g., region 704h). In an embodiment, the predefined merging distance is set by the physical properties of the pulse width.

[0101] In an embodiment, the upper boundary 702 is set to the minimum of two evaluation criteria. The first criterion is the merging distance (i.e., the distance below which merging can occur). The second criterion is the distance from the true target center (e.g., the starting cell 202). 3,3 ) is the average range of the nth percentile of cells within a boundary of a certain radius (e.g., the 25th range percentile), where a margin to allow for noise is applied (e.g., a radius of '3' and a margin of '1.5').

[0102] In an embodiment, the calculation of the lower boundary 706 and the upper boundary 708 in steps 606 and 608 are automatically generated to describe the shape of the bowl.

[0103] At step 610, assuming that the data corresponding to the obscured glare is flat, the flatness of the ring being evaluated is analyzed. As defined herein, the "flatness" of a ring is more of a measure of the ring's roughness (or smoothness) and any tilt. In one embodiment, it is defined as the square root of the sum of the squares of the "errors" of the cells within the ring. The "error" is the normalized difference between the mean values ​​of the rings.

[0104] For each ring being evaluated, the analysis involves calculating a measure that quantifies how closely the range of the evaluated ring compares to the average range of the same ring in the depth map. This step is crucial for evaluating the consistency of the data. It is important to note that this process does not require a curve fitting procedure, as the expected gradient value is zero under the assumed influence of occlusion glare. Therefore, attempting to simulate this effect by applying a fit is unnecessary.

[0105] To further assess the accuracy of this assumption, we evaluated how closely the actual data ranges "fit" to the theoretical veiling glare. This was done by calculating the R-squared value of the "fit," which, while generally indicative of the fitting process, serves as an indicator of how well the data ranges meet the expected zero-gradient condition implied by the veiling glare effect. R-squared values ​​close to zero indicate that these ranges are more consistent with a veiling glare scenario, with few gradients and smooth surfaces, thus validating the initial assumption.

[0106] For each evaluated ring, calculate the average range value of the units within the evaluated ring and normalized error value (error z ) to provide a flatness map of the boundary area.

[0107] For example, the average range value can be calculated using the following equation where b is the boundary region (i.e., ring) being evaluated, B is the set of boundary cells, z is the cell, and m is the number of cells in the boundary region.

[0108] The normalized error value (error z ): where z is a cell that is a member of the boundary cell set (B).

[0109] In step 612, using uncertainty propagation To determine the overall flatness of the object (f), where n is the total number of valid cells in the depth map 200. For all valid cells, calculate the square root value (rsq z The square root value (rsq) can be calculated using, for example, the following equation: z ): z∈V, where V is the set of all valid cells within the merge distance. A "valid" cell is typically any valid range within the field of view (some ranges have already been deemed invalid by other algorithms). However, a "valid" cell can also be defined as a cell that has been identified as an object by the above method.

[0110] Therefore, the R-squared values ​​derived from each boundary region are aggregated. If each cell within those regions returns a high R-squared measure, the surface within the analyzed boundary region is not horizontal (i.e., not flat). Therefore, the cumulative flatness measure will be high. However, if the R-squared value is low, indicating minimal deviation from the ideal zero gradient or perfect flatness, the sum of these values ​​across all cells will also be low. Therefore, the algorithm uses this cumulative score to indicate flatness: a lower flatness measure indicates a smooth and uniform surface. Conversely, a higher value indicates an uneven or uneven surface.

[0111] Step 614 analyzes the evaluated ring based on the pseudo-circumferential search. The number of cells 202 within the boundary region that indicate an object or obscured glare is determined. The ratio of cells within the boundary region that indicate an object or obscured glare to the total number of cells within the boundary is calculated. If this ratio is above a set threshold, the process is repeated for the next ring at step 616. Conversely, if this ratio is below the set threshold, the method proceeds to step 618.

[0112] Figure 7 The distribution of cells within each range is shown in a diagram. 3,3 Each cell 202 includes a value (range and radial distance from the center) 704b-i. The radial distance of each cell to the true target center is plotted on Figure 7 The range of values ​​is plotted on the Y axis.

[0113] like Figure 7 As shown, each unit 202 is located from the center of the real target (eg, the starting unit 202 3,3 ) is based on the cell or zone width. For each radius, starting from the starting cell 2023,3 Starting from the range where the signal is at a maximum value (maximum signal region), an upper boundary 702 and a lower boundary 706 are determined.

[0114] During each ring evaluation, proceeding outward from the center of the true target, the position of each cell 202 is plotted based on its radius from the center of the true target, as Figure 7 . This allows for an assessment of whether a cell conforms to a bowl-like distribution. Establishing these upper and lower boundaries creates a filtering effect, outlining the expected contours of the circle. Stepwise plotting across the radius helps visualize and define the expected bowl-shaped distribution based on the data from each cell 202.

[0115] For example, cell 202 on depth map 200 1,5 202 from the starting unit 3,3 The horizontal and vertical distances of are two and two respectively. In order to calculate the unit 202 using the Pythagorean theorem 1,5 The radial distance (d) needs to be the square root of the sum of the squares of the horizontal and vertical distances. In this example, unit 202 1,5 202 from the starting unit 3,3 The radial distance is As another example, cell 202 on depth map 200 3,4 202 from the starting unit 3,3 The radial distance of is one ('1') because the vertical value is zero and the horizontal value is 1. Repeat and plot the radial distance of each cell in the object identified from method 300, similar to Figure 7 shown.

[0116] Therefore, to effectively detect and exploit bowl-shaped obscuration glare features, method 600 employs a radial search mechanism. This is similar to techniques used in glass detection, which involves searching for specific geometric patterns (such as concentric rings). Furthermore, method 600 can evaluate the flatness and curvature of the detected rings, as well as the average range across each curvature. This process accurately determines the presence and extent of obscuration glare within the sensor's field of view. Through this analysis, method 600 significantly enhances object detection and grouping capabilities in time-of-flight sensor applications.

[0117] Step 618 calculates the total number of cells in the rings whose ratio is above the threshold in the previous step. The last ring with a ratio below the threshold is not included in the sum.

[0118] At step 620, the results from step 618 (i.e., the number of cells within the ring with a ratio above a threshold) and from step 612 (i.e., the overall flatness measurement) are analyzed to derive a confidence measure to determine whether occlusion glare is present. In an embodiment, the confidence measure is derived by weighting, biasing, and clipping each sub-measurement (number of cells within the ring, flatness).

[0119] At step 622, based on the confidence measure, the method can determine whether occlusion glare is present on the depth map 200. If the confidence measure indicates that there is no occlusion glare, then all cells defined using, for example, method 300 are part of the object and not occlusion glare. Conversely, if the confidence measure indicates occlusion glare, then all cells defined using, for example, method 300 are occlusion glare, which can be removed (i.e., filtered out) from the depth map. After identification, occlusion glare can be filtered out on a cell-by-cell basis using more traditional methods (e.g., thresholding based on signal level—occlusion glare exhibits a low signal).

[0120] Figure 8 An example of a depth map 800 from a multi-region ToF sensor, such as the ToF sensor 106, is illustrated. The depth map 800 includes 64 cells 802 arranged in an 8×8 grid. i,j , where i = 1 to 8 and j = 1 to 8. Each cell can be identified by a subscript representing the X and Y coordinates in the depth map 800. For clarity, when discussing a cell in the depth map 800, it is generally represented by cell 802. It should be understood that in embodiments, the depth map 800 may include a fewer or greater number of cells (e.g., 16 cells in a 4×4 grid). Therefore, the number of cells 802 in the depth map 800 is non-limiting, and other arrangements are similarly contemplated in other embodiments.

[0121] In an embodiment, the depth map 800 corresponds to occluding glare at 400 mm, with 88% being background at 1400 mm. Shaded cells 802 in the depth map 800 indicate range values ​​below a threshold value, which indicates, for example, an object or occluding glare. In contrast, the unshaded cells 802 in the depth map 800 correspond to the background. In the depth map 800, the center of the true target is the starting cell 802. 5,5 , which corresponds to the maximum signal (ie, the maximum signal area), not necessarily the cell with the range value closest to the object.

[0122] Given a depth map 800, Figure 9a-9c The progress of pseudo-circumferential search and flatness measurement based on method 600 is shown. The depth maps 900, 920, 940 in these figures correspond to Figure 8 ; however, they are represented differently based on the progression of the loop being evaluated.

[0123] exist Figure 9a-9c In each figure, an inner rectangle and an outer rectangle are drawn, creating a corresponding ring or boundary region in each figure. For example, in Figure 9a In the example, the number of cells 802 in the boundary region is eight ('8'). Figure 9bIn the example, the number of cells 804 in the boundary region is sixteen ('16'). Figure 9c , the number of cells 802 within the boundary region is twenty-four ('24').

[0124] For each Figure 9a-9c , the ratio of cells in the boundary region indicating an object or obscuring glare to the total number of cells in the boundary region is calculated. In response to this ratio being greater than a threshold (e.g., 0.25), the search is expanded by moving another cell outward in all directions from the starting cell (i.e., counting from the cell with the largest signal area).

[0125] For each evaluated ring, calculate the average range value of the cells within the evaluated ring and normalized error value (error z ).

[0126] exist Figure 9a The search begins by moving in all directions with the starting cell 802 5,5 An adjacent unit. Therefore, Figure 9a The boundary region includes the region surrounding the starting cell 802 5,5 Seven ('7') units of Figure 9a , the number of cells 802 within the boundary region indicating an object or blocking glare is also seven ('7') (cells 802 are highlighted). Figure 9a In the case of the boundary region, the ratio of cells indicating objects or obstructing glare to the total number of cells in the boundary is one ('1'). Since this ratio is above the threshold of .25 (non-limiting), the search continues to Figure 9b .

[0127] Therefore, in Figure 9b The search continues from the previous Figure 9a Move one additional unit outward in all directions. Therefore, Figure 9b The boundary region in consists of 16 ('16') cells. Figure 9b In FIG, the number of cells 802 indicating an object or blocking glare within the boundary region is six ('6'). Figure 9b The ratio of cells in the boundary region indicating objects or obstructing glare to the total number of cells in the boundary is 6 / 16 or '0.375'. Since this ratio is still above the threshold of .25 (non-restrictive), the search continues to Figure 9c .

[0128] Therefore, in Figure 9c The search continues from the previous Figure 9b Move one additional unit outward in all directions. Therefore, Figure 9c The boundary region in includes twenty-four ('24') cells. Figure 9cIn FIG, the number of cells 802 indicating an object or blocking glare within the boundary region is three ('3'). Figure 9c In the example, the ratio of cells in the boundary region indicating objects or obscuring glare to the total number of cells in the boundary is 3 / 24 or '0.125'. The search ends when this ratio falls below a threshold of 0.25 (not limiting).

[0129] Based on the search results, any cells 802 that fall within the final boundary region and have a ratio below a threshold (e.g., 0.25) are not connected to the main object and are discarded. Similarly, any cells 802 that fall outside the final boundary region in the depth map 800 are also discarded. Finally, the overall flatness of the object (f) is determined using uncertainty propagation.

[0130] A first aspect relates to a method for object abstraction to distinguish different objects in a depth map generated from a time-of-flight sensor. The method includes performing a raster search or spiral search on the depth map. The raster search includes systematically alternating between left-right scans and then up-down scans. The spiral search includes circumferential scanning. The method also includes detecting edges of objects to guide the continuation of the raster search or spiral search; calculating one or more high-level parameters related to the objects; and repeating the raster search or spiral search for previously unscanned cells in the depth map to similarly identify additional objects.

[0131] In a first implementation form of the method according to the first aspect, the one or more high-level parameters of the object include center, size, minimum range, median range, maximum range, reflectivity or a combination thereof.

[0132] In a second implementation form of the method according to the first aspect or any previous implementation form of the first aspect, for each raster search or spiral search, the method further comprises determining a starting cell within the depth map that has not been previously scanned and has a lowest range value.

[0133] In a third implementation form of the method according to the first aspect or any previous implementation form of the first aspect, the raster search comprises scanning horizontally and vertically across the depth map starting from the starting cell. The spiral search comprises scanning circumferentially starting from the starting cell.

[0134] In a fourth implementation of the method according to the first aspect or any previous implementation of the first aspect, the raster search includes switching from horizontal scanning to vertical scanning in response to determining that the range value of the cell is above a threshold, and reverting to a previous cell before switching to vertical scanning. The spiral search includes moving in a circle or outward in a spiral from one cell to the next in response to determining that the range value of the cell is within the threshold compared to surrounding cells.

[0135] In a fifth implementation of the method according to the first aspect or any previous implementation of the first aspect, the raster search includes switching from vertical scanning to horizontal scanning in response to determining that the range value of the cell is above a threshold, and reverting to the previous cell before switching to horizontal scanning. The spiral search includes moving in a circular motion or outward in a spiral shape from one cell to the next until all scanned edge cells are determined to exceed the threshold.

[0136] In a sixth implementation form of the method according to the first aspect or any previous implementation form of the first aspect, the method further comprises: applying a different threshold value to eliminate the occlusion glare effect of each identified object.

[0137] A second aspect relates to a method for pseudo-circumferential search to determine occlusion effects in a depth map generated from a time-of-flight sensor. The method comprises: determining a starting cell with a maximum signal value within the depth map; generating a first boundary region including cells adjacent to the starting cell; calculating a first ratio corresponding to the number of cells within the first boundary region having values ​​within a first threshold and the total number of cells within the first boundary region; in response to the first ratio being above a second threshold, generating a next boundary region including cells adjacent to the first boundary region; and in response to the first ratio being below the second threshold, stopping the pseudo-circumferential search, wherein the total number of cells within the second-to-last boundary region is used to determine the occlusion glare effect.

[0138] In a first implementation form of the method according to the second aspect, the method also includes: calculating a second ratio, the second ratio corresponding to the number of cells in the next boundary area having range values ​​within the first threshold and the total number of cells in the next boundary area; in response to the second ratio being higher than the second threshold, generating a third boundary area including cells adjacent to the next boundary area; and in response to the second ratio being lower than the second threshold, stopping the pseudo-circumferential search.

[0139] In a second implementation form of the method according to the second aspect or any previous implementation form of the second aspect, the method further comprises: determining a flatness measure for each boundary region, wherein the masking glare effect is determined based on the flatness measure and the total number of cells within the penultimate boundary region.

[0140] In a third implementation form of the method according to the second aspect or any previous implementation form of the second aspect, the method further comprises: calculating a confidence measure based on the flatness measure and the total number of cells within the penultimate boundary region to determine the masking glare effect.

[0141] In a fourth implementation form of the method according to the second aspect or any previous implementation form of the second aspect, the flatness measure is determined by calculating an R-squared value for cells within each boundary region.

[0142] In a fifth implementation form of the method according to the second aspect or any previous implementation form of the second aspect, the method further comprises: calculating an upper boundary and a lower boundary of each boundary region, wherein the first threshold is determined based on the upper boundary and the lower boundary.

[0143] In a sixth implementation form of the method according to the second aspect or any previous implementation form of the second aspect, the lower boundary is set to the maximum of the following: the previous lower boundary, or the point range when the radial distance from the starting cell is equal to 1, or the average range of the boundary area. The upper boundary is set to the minimum of the following: the predefined merging distance, or the average range of the cells in the nth percentile of the boundary area, where n is greater than one.

[0144] A third aspect relates to a method for flatness measurement to determine occlusion effects in a depth map generated from a time-of-flight sensor. The method comprises: determining a starting cell with a maximum signal value within the depth map; generating a first boundary region including cells adjacent to the starting cell; calculating a first flatness measurement for the first boundary region; generating a next boundary region including cells adjacent to the first boundary region; calculating a second flatness measurement for the next boundary region, wherein each flatness measurement is calculated based on an average range value and a normalized error value of cells within the corresponding boundary region; and calculating a total flatness measurement to determine occlusion glare effects.

[0145] In a first implementation form of the method according to the third aspect, the method further comprises performing a pseudo-circumferential search. The pseudo-circumferential search comprises: calculating a first ratio corresponding to the number of cells within a first boundary region having a range value within a first threshold and the total number of cells within the first boundary region; in response to the first ratio being above a second threshold, generating a next boundary region including cells adjacent to the first boundary region; and in response to the first ratio being below the second threshold, stopping the pseudo-circumferential search, wherein the total number of cells within the penultimate boundary region and the total flatness measure are used to determine the occlusion glare effect.

[0146] In a second implementation form of the method according to the third aspect or any previous implementation form of the third aspect, the pseudo-circumferential search also includes: calculating a second ratio, the second ratio corresponding to the number of cells in the next boundary area having range values ​​within the first threshold and the total number of cells in the next boundary area; in response to the second ratio being higher than the second threshold, generating a third boundary area including cells adjacent to the next boundary area; and in response to the second ratio being lower than the second threshold, stopping the pseudo-circumferential search.

[0147] In a third implementation form of the method according to the third aspect or any previous implementation form of the third aspect, the method further comprises: calculating a confidence measure based on the total flatness measure and the total number of cells within the penultimate boundary region to determine the masking glare effect.

[0148] In a fourth implementation form of the method according to the third aspect or any previous implementation form of the third aspect, the pseudo circumferential search further comprises: calculating an upper boundary and a lower boundary of each boundary region. The first threshold is determined based on the upper boundary and the lower boundary.

[0149] In a fifth implementation form of the method according to the third aspect or any previous implementation form of the third aspect, the lower boundary is set to the maximum of the following: the previous lower boundary, or the point range when the radial range from the starting cell is equal to 1, or the average range of the boundary area. The upper boundary is set to the minimum of the following: the predefined merged range, or the average range of the cells at the nth percentile of the boundary area, where n is greater than one.

[0150] Although the description has been described in detail, it will be understood that various changes, substitutions and modifications may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims. In the various figures, the same elements are represented by the same reference numerals. In addition, the scope of the present disclosure is not intended to be limited to the specific embodiments described herein, as those of ordinary skill in the art will readily appreciate from this disclosure that currently existing or later developed processes, machines, manufactures, material compositions, means, methods or steps may perform substantially the same functions or achieve substantially the same results as the corresponding embodiments described herein. Therefore, the appended claims are intended to include processes, machines, manufactures, material compositions, means, methods or steps within their scope.

[0151] Therefore, the specification and drawings should be regarded as a brief description of the disclosure as defined by the appended claims, and all modifications, variations, combinations or equivalents falling within the scope of the disclosure should be covered.

Claims

1. A method for object abstraction to distinguish different objects in a depth map generated from a time-of-flight sensor, the method comprising: performing a raster search or a spiral search on the depth map, the raster search comprising systematically alternating between left-right scanning and then up-and-down scanning, the spiral search comprising circumferential scanning; detecting an edge of an object to guide continuation of the raster search or the spiral search; calculating one or more high-level parameters associated with the object; and The raster search or the spiral search is repeated for cells in the depth map that have not been previously scanned to similarly identify other objects. 2 . The method of claim 1 , wherein the one or more high-level parameters of the object include center, size, minimum range, median range, maximum range, reflectivity, or a combination thereof.

3. The method according to claim 1, further comprising: For each raster search or spiral search, a starting cell within the depth map that has not been previously scanned and has the lowest range value is determined.

4. The method of claim 3, wherein the raster search comprises: Scanning horizontally and vertically across the depth map starting from the starting cell, and wherein the spiral search comprises a circumferential scan starting from the starting cell.

5. The method of claim 4, wherein the raster search comprises: Switching from the horizontal scan to the vertical scan in response to determining that the range value of a cell is above a threshold and reverting to a previous cell before switching to the vertical scan, and wherein the spiral search includes: moving in a circle or outward in a spiral from one cell to the next in response to determining that the range value of a cell is within the threshold compared to surrounding cells.

6. The method of claim 4, wherein the raster search comprises: Switching from the vertical scan to the horizontal scan in response to determining that the range value of a cell is above a threshold and reverting to a previous cell before switching to the horizontal scan, and wherein the spiral search includes moving around in a circle or outward in a spiral from one cell to the next until all scanned edge cells are determined to exceed the threshold.

7. The method according to claim 1, further comprising: A different threshold is applied to remove the occlusion glare effect for each object identified.

8. A method for pseudo-circumferential search to determine occlusion effects in a depth map generated from a time-of-flight sensor, the method comprising: Determining a starting cell having a maximum signal value in the depth map; generating a first boundary region including cells adjacent to the starting cell; calculating a first ratio corresponding to a number of cells within the first boundary region having range values ​​within a first threshold and a total number of cells within the first boundary region; In response to the first ratio being higher than a second threshold, generating a next boundary region including cells adjacent to the first boundary region; as well as In response to the first ratio being below a second threshold, the pseudo-circumferential search is stopped, wherein the total number of cells within the penultimate boundary region is used to determine the obscured glare effect.

9. The method according to claim 8, further comprising: calculating a second ratio corresponding to the number of cells in the next boundary region having range values ​​within the first threshold and the total number of cells in the next boundary region; In response to the second ratio being higher than the second threshold, generating a third boundary region including cells adjacent to the next boundary region; and In response to the second ratio being lower than the second threshold, the pseudo-circumferential search is stopped.

10. The method according to claim 8, further comprising: A flatness measure is determined for each boundary region, wherein the obscured glare effect is determined based on the flatness measure and a total number of cells within the penultimate boundary region.

11. The method according to claim 10, further comprising: A confidence measure is calculated based on the flatness measure and the total number of cells within the penultimate boundary region to determine the obscured glare effect.

12. The method of claim 10, wherein the flatness measure is determined by calculating an R-squared value for the cells within each boundary region.

13. The method according to claim 8, further comprising: An upper boundary and a lower boundary of each boundary region are calculated, wherein the first threshold is determined based on the upper boundary and the lower boundary.

14. The method according to claim 13, The lower bound is set to the maximum of: The previous lower boundary, or the range of points where the radial distance from the starting cell is equal to 1, or the average extent of the boundary region, and The upper bound is set to the minimum of: a predefined merge distance, or The average range of the nth percentile cells of the boundary region, where n is greater than one.

15. A method for flatness measurement to determine occlusion effects in a depth map generated from a time-of-flight sensor, the method comprising: Determining a starting cell having a maximum signal value in the depth map; generating a first boundary region including cells adjacent to the starting cell; calculating a first flatness measure for the first boundary region; generating a next boundary region including cells adjacent to the first boundary region; calculating a second flatness measure for the next boundary region, wherein each flatness measure is calculated based on an average range value and a normalized error value of cells within a corresponding boundary region; as well as An overall flatness measurement is calculated to determine the effect of masking glare.

16. The method of claim 15, further comprising performing a pseudo-circumferential search, the pseudo-circumferential search comprising: calculating a first ratio corresponding to a number of cells within the first boundary region having range values ​​within a first threshold and a total number of cells within the first boundary region; In response to the first ratio being higher than a second threshold, generating a next boundary region including cells adjacent to the first boundary region; as well as In response to the first ratio being below a second threshold, the pseudo-circumferential search is stopped, wherein the total number of cells within the penultimate boundary region and the total flatness measure are used to determine the obscured glare effect.

17. The method according to claim 16, wherein the pseudo-circumferential search further comprises: calculating a second ratio corresponding to the number of cells in the next boundary region having range values ​​within the first threshold and the total number of cells in the next boundary region; In response to the second ratio being higher than the second threshold, generating a third boundary region including cells adjacent to the next boundary region; and In response to the second ratio being lower than the second threshold, the pseudo-circumferential search is stopped.

18. The method according to claim 16, further comprising: A confidence measure is calculated based on the overall flatness measure and the total number of cells within the penultimate boundary region to determine the obscured glare effect.

19. The method of claim 16, wherein the pseudo-circumferential search further comprises: An upper boundary and a lower boundary of each boundary region are calculated, wherein the first threshold is determined based on the upper boundary and the lower boundary.

20. The method according to claim 19, The lower bound is set to the maximum of: The previous lower boundary, or the point range when the radial range from the starting cell is equal to 1, or the average extent of the boundary region, and The upper bound is set to the minimum of: predefined scopes, or The average range of the nth percentile cells of the boundary region, where n is greater than one.

Images