SPOTTED LIGHTING FOR DEEP SENSORS

Abstract

A method includes projecting a first lighting pattern onto a scene. The first lighting pattern illuminates a first subset of the field of view. The method includes taking a first measurement corresponding to the light reflected from the scene in response to the first lighting pattern. The method includes projecting a second lighting pattern onto the scene. The second lighting pattern illuminates the second subset of the field of view. The method includes taking a second measurement corresponding to the light reflected from the scene in response to the second lighting pattern. The method includes comparing the first measurement with the second measurement and determining a depth value for an object in the scene based on this comparison. The method includes controlling the movement of a vehicle based on the depth value for the object.

Description

CROSS-REFERENCE TO RELATED REGISTRATIONS

[0001] This application claims priority under Section 35 § 119(e) of the preliminary US application with serial number 63 / 687,122, filed on August 26, 2024. The disclosure of that prior application is considered part of the disclosure of this application and is incorporated herein in full by reference. INTRODUCTION

[0002] The information provided in this section serves the purpose of providing a general overview of the context of the disclosure. Neither the work of the inventors currently named, to the extent described in this section, nor those aspects of the description that could not otherwise qualify as prior art at the time of filing, are expressly or implicitly recognized as prior art against the present disclosure.

[0003] This disclosure relates generally to depth sensing systems, and in particular to light detection and distance measurement systems (LiDAR systems) and other time-of-flight (ToF) systems used in vehicles. ToF systems determine distances to objects in a scene by illuminating the scene with a light source and detecting the reflected light with a sensor. The time it takes for the light to travel from the source, be reflected by an object, and return to the sensor corresponds to the object's distance. This process can be used to generate a point cloud or depth map of the scene. Some systems, known as direct ToF (dToF), emit short pulses of light and directly measure their round-trip time.Other systems, known as indirect Time-of-Flight (iToF) systems, emit a continuous, time-modulated light oscillation and determine the distance by measuring the phase shift between the emitted and received light. The illumination can be projected simultaneously across the entire field of view, or the scene can be scanned column by column, for example. SUMMARY

[0004] One aspect of the revelation provides a computer-implemented procedure, executed in data processing hardware, that causes the data processing hardware to perform operations. These operations include projecting a first illumination pattern onto a scene from a light-based detection and distance measurement sensor (LiDAR sensor). The first illumination pattern illuminates a first subset of the field of view and does not illuminate a second subset. The operations also include capturing, by several sensor pixels of the LiDAR sensor, a first measurement corresponding to light reflected from the scene in response to the first illumination pattern. Finally, the operations include projecting a second illumination pattern onto the scene from the LiDAR sensor. This second illumination pattern illuminates the second subset of the field of view and does not illuminate the first subset.The operations include capturing a second measurement, corresponding to light reflected from the scene in response to the second illumination pattern, using the multiple sensor pixels of the LiDAR sensor. The operations include comparing the first measurement with the second measurement. The operations include determining a depth value for an object in the scene based on this comparison. The operations include controlling the movement of a vehicle based on the depth value for the object.

[0005] Implementations of the disclosure may include one or more of the following optional features. In some implementations, comparing the first measurement with the second measurement involves determining a difference between the first and second measurements on a pixel-wise basis to isolate a direct light component from a disturbing light component. In these implementations, determining the difference between the first and second measurements may involve subtracting the second measurement from the first. Here, the first illumination pattern may contain a first checkerboard pattern of illumination, and the second illumination pattern may contain a second checkerboard pattern of illumination that is different from the first checkerboard pattern.In some examples, the first checkerboard pattern illuminates a first set of pixels, the second checkerboard pattern illuminates a second set of pixels, the second set of pixels is not illuminated by the first checkerboard pattern, and the first set of pixels is not illuminated by the second checkerboard pattern.

[0006] The operations may further include calibrating the LiDAR sensor to account for imperfect spatial modulation of the first or second illumination pattern. In some implementations, the first and second illumination patterns each contain rectangular spatial modulations. The first and second illumination patterns may each contain sinusoidal spatial modulations. In some examples, the LiDAR sensor includes a direct time-of-flight sensor. The LiDAR sensor may also include an indirect time-of-flight sensor.

[0007] Another aspect of the revelation provides a vehicle containing a light-based detection and distance measurement sensor (LiDAR sensor), data processing hardware, and storage hardware communicating with the data processing hardware. The storage hardware holds instructions which, when executed in the data processing hardware, cause the data processing hardware to perform operations. These operations include projecting an initial illumination pattern from the LiDAR sensor onto a scene. The initial illumination pattern illuminates a first subset of the field of view and omits any second subset. The operations also include capturing, by multiple sensor pixels of the LiDAR sensor, an initial measurement corresponding to light reflected from the scene in response to the initial illumination pattern.The operations involve projecting a second illumination pattern from the LiDAR sensor onto the scene. This second illumination pattern illuminates the second subset of the field of view while omitting the first subset. The operations also involve the LiDAR sensor's multiple sensor pixels capturing a second measurement corresponding to the light reflected from the scene in response to the second illumination pattern. Finally, the operations include comparing the first and second measurements. Based on this comparison, the operations include determining a depth value for an object within the scene. Finally, the operations include controlling the vehicle's movement based on this depth value.

[0008] Implementations of the disclosure may include one or more of the following optional features. In some implementations, comparing the first measurement with the second measurement involves determining a difference between the first and second measurements on a pixel-wise basis to isolate a direct light component from a disturbing light component. In these implementations, determining the difference between the first and second measurements may involve subtracting the second measurement from the first. Here, the first illumination pattern may contain a first checkerboard pattern of illumination, and the second illumination pattern may contain a second checkerboard pattern of illumination that is different from the first checkerboard pattern.In some examples, the first checkerboard pattern illuminates a first set of pixels, the second checkerboard pattern illuminates a second set of pixels, the second set of pixels is not illuminated by the first checkerboard pattern, and the first set of pixels is not illuminated by the second checkerboard pattern.

[0009] The operations may further include calibrating the LiDAR sensor to account for imperfect spatial modulation of the first or second illumination pattern. In some implementations, the first and second illumination patterns each contain rectangular spatial modulations. The first and second illumination patterns may each contain sinusoidal spatial modulations. In some examples, the LiDAR sensor includes a direct time-of-flight sensor.

[0010] Another aspect of the disclosure provides a computer program product encoded in a non-transient, computer-readable storage medium and containing instructions which, when executed by a data processing device, cause the data processing device to perform operations. The operations include projecting a first illumination pattern onto a scene from a light-based detection and distance measurement sensor (LiDAR sensor). The first illumination pattern illuminates a first subset of the field of view and omits a second subset. The operations include acquiring, by several sensor pixels of the LiDAR sensor, a first measurement corresponding to light reflected from the scene in response to the first illumination pattern. The operations include projecting a second illumination pattern onto the scene from the LiDAR sensor.The second illumination pattern illuminates the second subset of the field of view and does not illuminate the first subset. The operations include capturing a second measurement, corresponding to light reflected from the scene in response to the second illumination pattern, using the multiple sensor pixels of the LiDAR sensor. The operations include comparing the first measurement with the second measurement. The operations include determining a depth value for an object in the scene based on this comparison. The operations include controlling the movement of a vehicle based on the depth value for the object.

[0011] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Further aspects, features, and advantages will become clear from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described here serve only to illustrate selected configurations and are not intended to limit the scope of this disclosure; they show: Fig. 1 a schematic view of a vehicle performing a depth model to implement speckled lighting; Fig. 2 a schematic view of the depth model projecting an initial lighting pattern; Fig. 3 a schematic view of the depth model projecting a second lighting pattern; and Fig. 4. A flowchart of an exemplary design of operations for a computer-implemented procedure for performing speckled lighting.

[0013] Throughout the drawings, corresponding reference symbols denote corresponding parts. DETAILED DESCRIPTION

[0014] Exemplary configurations are now described in more detail with reference to the accompanying drawings. Exemplary configurations are provided to ensure that this disclosure is thorough and fully conveys its scope to those skilled in the art. Specific details, such as examples of specific components, devices, and processes, are presented to provide a precise understanding of the configurations of this disclosure. It is evident to those skilled in the art that specific details need not be used, that exemplary configurations can be embodied in many different forms, and that the specific details and exemplary configurations are not intended to limit the scope of the disclosure.

[0015] The terminology used here serves only to describe certain exemplary configurations and is not intended to be restrictive. As used here, the singular articles "a," "an," and "the" may be intended to include the plural forms unless the context clearly indicates otherwise. The terms "includes," "comprise," "contain," and "exhibit" are inclusive and therefore establish the presence of features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more further features, steps, operations, elements, components, and / or groups thereof.The procedural steps, processes, and operations described herein are not intended to necessarily require their execution in the specific order discussed or illustrated, unless explicitly identified as such. Additional or alternative steps may be employed.

[0016] When an element or layer is described as "attached to," "intervening with," "connected to," "attached to," or "coupled to" another element or layer, it may be directly attached to, intervening with, connected to, attached to, or coupled to that other element or layer, or there may be intermediate elements or layers. Conversely, when an element is described as "directly attached to," "directly intervening with," "directly connected to," "directly attached to," or "directly coupled to" another element or layer, there need not be any intermediate elements or layers. Other words used to describe the relationship between elements should be interpreted similarly (e.g., "between" or "directly between," "adjacent" or "directly adjacent," etc.).As used herein, the expression “and / or” includes all combinations of one or more of the associated listed elements.

[0017] The terms "first," "second," "third," etc., can be used here to describe different elements, components, areas, layers, and / or sections. These elements, components, areas, layers, and / or sections are not intended to be limited by these terms. These terms can only be used to distinguish one element, component, area, layer, or section from another. Terms such as "first," "second," and other numerical terms do not imply any sequence or order unless clearly indicated by the context.Thus, a first element, a first component, a first area, a first layer or a first section discussed below can be referred to as a second element, a second component, a second area, a second layer or a second section without deviating from the instructions of the exemplary configurations.

[0018] In this application, including the definitions below, the term "module" may be replaced by the term "circuit". The term "module" may refer to an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor (shared, dedicated, or a group) that executes code; a memory (shared, dedicated, or a group) that stores code executed by a processor; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip, being part of, or containing them.

[0019] The term "code," as used above, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared processor" includes a single processor that executes some or all of the code from multiple modules. The term "group processor" includes a processor that, in combination with additional processors, executes some or all of the code from multiple modules. The term "shared memory" includes a single memory that stores some or all of the code from multiple modules. The term "group memory" includes memory that, in combination with additional memory, stores some or all of the code from one or more modules. The term "memory" can be a subset of the term "computer-readable medium."The term "computer-readable medium" encompasses non-transient electrical and electromagnetic signals that propagate through a medium and can therefore be considered a physical, non-transient storage medium. Non-restrictive examples of non-transient storage include physical computer-readable media such as non-volatile memory, magnetic storage, and optical storage.

[0020] The devices and methods described in this application can be partially or completely implemented by one or more computer programs executed by one or more processors. The computer programs contain processor-executable instructions stored on at least one non-transient, computer-readable physical medium. The computer programs may also contain and / or access stored data.

[0021] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform a task. In some examples, a software application may be called an "application," an "app," or a "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0022] Non-transient memory can be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-transient memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (which is typically used, for example, for firmware such as boot programs).Examples of volatile memory include, but are not limited to, write / read memory (RAM), dynamic write / read memory (DRAM), static write / read memory (SRAM), phase change memory (PCM), and disks or tapes.

[0023] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in assembly / machine language. As used here, the terms "machine-readable medium" and "computer-readable medium" refer to a computer program product, a non-transient computer-readable medium, a device, and / or a setup (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, which includes a machine-readable medium that receives machine instructions as a machine-readable signal.The term "machine-readable signal" refers to a signal that is used to provide machine instructions and / or data to a programmable processor.

[0024] Various implementations of the systems and techniques described herein can be realized in digital electronics and / or an optical circuit arrangement, an integrated circuit arrangement, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include an implementation in one or more computer programs that are executable and / or interpretable in a programmable system comprising at least one programmable processor, which may be specialized or general-purpose, coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0025] The processes and logic operations described in this application can be performed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by working on input data and generating outputs. The processes and logic operations can also be performed by a special-purpose logic circuit arrangement, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing a computer program include, by way of example, general-purpose microprocessors, special-purpose microprocessors, and one or more processors of any type of digital computer. Generally, a processor receives instructions and data from read-only memory and / or read / write memory.The essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Generally, a computer also includes, or is functionally coupled to, a means of receiving data from and / or sending data to one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical media. However, a computer does not necessarily have to possess such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory); magnetic media (e.g., internal hard disks or removable media); magneto-optical media; and CD-ROM and DVD-ROM media.The processor and memory can be supplemented or incorporated by a logic circuit arrangement for a specific purpose.

[0026] To provide interaction with a user, one or more aspects of the disclosure may be implemented in a computer that has a display device, such as a CRT (cathode ray tube), an LCD (liquid crystal display), or a touchscreen for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, with which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as...Visual, auditory, or haptic feedback; and input can be received from the user in any form, including auditory, verbal, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to an internet browser in a user's client device in response to requests received from the internet browser.

[0027] Time-of-flight depth sensors (ToF depth sensors), which incorporate sensors for detecting and measuring distance using light (LiDAR sensors), are used to generate three-dimensional representations of an environment, often for use in navigation for autonomous vehicles. The basic operating principle of these systems relies on a set of assumptions about how light propagates from a source, is reflected by the scene, and returns to a sensor. In an ideal scenario, each pixel in the sensor array receives light exclusively from a single corresponding point in the scene that it directly observes. Based on this assumption, the time delay of the received light pulse is used to calculate the distance to that single scene point.

[0028] In practice, however, this one-to-one correspondence between a sensor pixel and a scene point is frequently violated. Pixels often receive light from multiple paths, leading to inaccurate depth measurements. A significant source of error is internal multipath interference, often referred to as blooming. This can occur when the scene contains highly reflective objects, such as retroreflective overhead traffic signs. Light returning from these objects is extremely bright. As the bright light enters the sensor, it can bounce between the sensor's optics and the sensor array itself before being absorbed. As a result, some of the light reaches pixels that are viewing other, more distant parts of the scene.

[0029] A pixel affected by blooming can therefore record multiple light reflections. Specifically, a valid reflection from a corresponding scene point and a disruptive reflection from the blooming noise. The disruptive reflection can be stronger and arrive earlier than the valid reflection, confusing the depth estimation algorithm. This can cause the system to perceive "phantom" or "ghost" objects that are not actually present, such as a vertical curtain of points appearing in the middle of a roadway. For a vehicle relying on this data, such a noise signal can be misinterpreted as an obstacle, potentially leading to unnecessarily hard braking.

[0030] Similar depth errors can arise from other sources of indirect light. External multipath propagation occurs when light from the source bounces between multiple objects in the scene before reaching the sensor, resulting in an overestimation of the object's true depth. Furthermore, adverse weather conditions such as fog or smoke can cause significant light scattering. This scattering creates an "infinite continuum of light paths," with some scattered light returning to the sensor sooner than direct reflection from an object, leading to an underestimation of depth and a low signal-to-noise ratio. These various sources of interfering light reduce the accuracy and reliability of the data generated by depth sensors.

[0031] Accordingly, implementations here focus on a depth model that projects a first illumination pattern from a LiDAR sensor onto a scene, and a second illumination pattern from the LiDAR sensor onto the scene. The depth model can capture a first measurement, corresponding to light reflected from the scene in response to the first illumination pattern, and a second measurement, corresponding to light reflected from the scene in response to the second illumination pattern, using multiple sensor pixels of the LiDAR sensor. Furthermore, the depth model determines a depth value for an object in the scene based on comparing the first and second measurements. In this context, the depth model can control the movement of a vehicle based on the depth value for the object.

[0032] Referring to the drawings and especially to Fig. 1- Fig. Figure 3 depicts a vehicle 10 as comprising various integrated components suitable for autonomous or assisted driving operations. The vehicle 10 includes data processing hardware 12 and storage hardware 14. The storage hardware 14 communicates with the data processing hardware 12. The storage hardware 14 stores executable instructions which, when executed in the data processing hardware 12, cause the data processing hardware 12 to perform operations related to depth sensing and comprehensive scene understanding of the vehicle 10's environment.

[0033] The vehicle 10 additionally contains one or more sensors for detection and distance measurement using light (LiDAR sensors) 20. Each LiDAR sensor 20 can include a light source 22 configured to emit light and an array of multiple sensor pixels 24. The LiDAR sensor 20 can be a direct time-of-flight (dToF) sensor or an indirect time-of-flight (iToF) sensor. A direct time-of-flight sensor typically operates by emitting discrete short pulses of light and then directly measuring the elapsed time required for the light to travel from the light source 22 to an object 30 in a scene and return to the multiple sensor pixels 24. In contrast, an indirect time-of-flight sensor can emit a continuous, time-modulated light oscillation and subsequently determine the distance by measuring the phase shift between the emitted light waveform and the received light waveform.Each LiDAR sensor 20 can have a field of view 26.

[0034] LiDAR sensors 20 are sensitive to the presence of interfering light components that lead to inaccurate or erroneous depth measurements. For example, internal multipath interference, known as blooming, can occur when highly reflective objects, such as retroreflective overhead traffic signs in a scene, cause extremely bright light to enter the LiDAR sensor 20. This bright light can then bounce between the optical components of the LiDAR sensor 20 and the sensor array before being absorbed. As a result, some of the light reaches the sensor pixels 24 that view other, more distant parts of the scene. A sensor pixel 24 affected by blooming can therefore record multiple light returns: one valid return from a corresponding scene point and one interfering return from the blooming interference signal.The interfering feedback can be stronger and arrive earlier than the valid feedback, potentially confusing a depth estimation algorithm. Therefore, the system perceives phantom or ghost objects that are not actually present, such as a vertical curtain of dots appearing in the middle of a roadway. For a vehicle 10 relying on this data, such an interfering signal can be misinterpreted as an obstacle, potentially leading to unnecessarily hard braking.

[0035] To mitigate the effects of such interfering light components, the vehicle 10 may contain a depth model 105. The depth model 105 operates under the command and control of the data processing hardware 12. The depth model 105 may contain a lighting device 110. The lighting device 110 is configured to instruct the LiDAR sensor 20, by means of commands 112, to project an initial lighting pattern 120 onto a scene. The lighting device 110 may receive an input signal 16 from one or more other sensors of the vehicle 10. The input signal 16 may contain various data types that indicate the operating environment and conditions of the vehicle 10 or the status of other systems of the vehicle 10. For example, the input signal 16 could use data from an inertial measurement unit (IMU) as a basis, providing information about the vehicle's motion, such as acceleration or angular velocity.In another example, the input signal 16 could use a signal from a global positioning system (GPS) receiver as a basis, indicating the location and speed of the vehicle 10. In yet another example, the input signal 16 could use data from an outside temperature sensor, a rain sensor, a light sensor, or a camera system as a basis, providing information about environmental conditions such as ambient light, precipitation, detected objects, or visibility. The lighting device 110 can generate the first lighting pattern 120 based on the input signal 16.

[0036] As in Fig. As shown in Figure 2, the first lighting pattern 120 is configured to illuminate a first subset of a field of view 122 (e.g., boxes shaded black). Temporally overlapping, the first lighting pattern 120 is configured to refrain from illuminating a second subset of a field of view 124 (e.g., unshaded boxes). An example of such a lighting pattern might be a checkerboard pattern, in which certain spatially defined areas (e.g., the first subset of the field of view 122) are actively illuminated, while adjacent or complementary spatial areas (e.g., the second subset of the field of view 124) are intentionally left unlit or with significantly reduced illumination.Depending on the specific optical design of the LiDAR sensor 20 and the desired scanning methodology, the first illumination pattern 120 can be applied uniformly across the entire field of view 26 of the LiDAR sensor 20, or it can be applied on a more granular substrate, such as a column-wise or row-wise substrate. As shown in . Fig. As shown in Figure 2, the first lighting pattern 120 is applied to a single column in the field of view 26.

[0037] Upon projection of the first illumination pattern 120 onto the scene, the multiple sensor pixels 24 of the LiDAR sensor 20 are configured to acquire an initial measurement 126. This initial measurement 126 corresponds to the light reflected from the scene in response to the projection of the first illumination pattern 120. During the acquisition of light reflected in response to the first illumination pattern 120, each sensor pixel 24 records the received light. The received light may include not only direct reflections originating from the illuminated sections of the scene but also contributions from interfering light components. Such interfering light components may be caused by phenomena such as internal multipath interference (e.g., blooming), external multipath interference, or scattering due to atmospheric conditions (e.g., fog, smoke). The interfering components may originate from highly reflective objects (e.g.,Retroreflectors 32) in the scene or from scattering media, even if these objects or sections of the scene are not directly in the illuminated section corresponding to a particular sensor pixel 24. The first measurement 126 therefore represents a superposition of the direct light signal, which carries the true depth information, and any interfering light present during the first illumination phase.

[0038] The lighting device 110 then instructs the LiDAR sensor 20, via commands 112, to project a second lighting pattern 130 onto the scene based on the input signal 16. As in Fig. As shown in Figure 3, the second lighting pattern 130 is configured to illuminate the second subset of the field of view 124. The second subset of the field of view 124 was intentionally not illuminated by the first lighting pattern 120. Simultaneously, the second lighting pattern 130 is configured to refrain from illuminating the first subset of the field of view 122, thus creating a complementary lighting state with respect to the first lighting pattern 120. For example, if the first lighting pattern 120 includes illumination of elements or areas with an even index in a spatial grid across the field of view, the second lighting pattern 130 may include illumination of elements or areas with an odd index in the same spatial grid.The complementary projection of lighting patterns is a design feature that facilitates the subsequent separation and isolation of direct light signals from interfering light components.

[0039] Following the projection of the second illumination pattern 130, the multiple sensor pixels 24 of the LiDAR sensor 20 acquire a second measurement 128. The second measurement 128 corresponds to the light reflected from the scene in response to the projection of the second illumination pattern 130. During this second acquisition, each sensor pixel 24 receives light containing contributions from various interfering light components. These interfering light components, such as those caused by internal multipath interference or blooming, can exist regardless of the specific illuminated subset and affect the sensor pixels 24. However, for the sensor pixels 24 that observe areas in the first subset of the field of view 122, the direct light component originating from these specific areas is largely absent, completely absent, or substantially reduced in the second measurement 128.This is because these areas are not directly illuminated by the second illumination pattern 130. Conversely, sensor pixels 24, which view areas in the second subset of the field of view 124, now receive a direct light component in addition to any interfering light. The recorded second measurement 128 therefore provides a further superposition of signals, but with a different distribution of direct and interfering light compared to the first measurement 126.

[0040] In some implementations, the first illumination pattern 120 and the second illumination pattern 130 each contain rectangular oscillation-shaped spatial modulations. Such rectangular oscillation-shaped modulations can include abrupt and varied transitions between illuminated areas (e.g., "on" states with high light intensity) and unilluminated areas (e.g., "off" states with minimal or zero light intensity). Thus, the rectangular oscillation-shaped modulations produce a distinct and varied on / off pattern across the entire field of view or within a specific sample column or row. For example, a rectangular oscillation-shaped spatial modulation might appear as a checkerboard pattern, where a defined pixel or group of pixels is either fully illuminated or completely dark.

[0041] In further implementations, the first illumination pattern 120 and the second illumination pattern 130 each contain sinusoidal spatial modulations. Unlike rectangular oscillations, sinusoidal spatial modulations involve a gradual and continuous fluctuation of light intensity across the pattern, resembling a continuous oscillation (e.g., a sine or cosine wave). The intensity varies smoothly from a minimum to a maximum and back, rather than exhibiting sharp, instantaneous transitions. Sinusoidal spatial modulation can offer several advantages in certain optical systems. For example, such modulations may be more robust against bandwidth limitations inherent in optical components or against optical softening phenomena that occur in real-world implementations due to diffraction, lens aberrations, or atmospheric effects.In some examples, the smooth transitions characteristic of sinusoidal patterns help maintain consistent signal quality and reduce noise, especially in systems where strong spatial contrasts are difficult to achieve or maintain with high fidelity. Both rectangular and sinusoidal patterns are suitable for generating the complementary illumination states desired for the effective operation of the depth model 105.

[0042] In some implementations, the depth model 105 includes a distance model 150. The distance model 150 is configured to determine a depth value 152 for one or more objects 30 in the scene. The distance model 150 determines the depth value 152 based on comparing the first measurement 126 with the second measurement 128. In some implementations, the distance model 150 compares the first measurement 126 with the second measurement 128 by determining a difference 154 between the first measurement 126 and the second measurement 128. In some examples, the depth model 105 performs the difference calculation on a pixel-wise basis. The purpose of this difference operation is to isolate a direct light component from a disturbing light component. For example, the disturbing light component may consist of internal multipath interference (e.g., blooming), external multipath interference (e.g.,Light bouncing between multiple objects in the scene) or scattered light due to adverse atmospheric conditions such as fog, smoke, or heavy precipitation. When the first illumination pattern 120 and the second illumination pattern 130 are used, each recorded first measurement 126 and second measurement 128 for a given sensor pixel 24 may contain a direct light component (e.g., when the corresponding scene point of a pixel was directly illuminated by this pattern) and approximately half of the total interfering light component. This approximation holds because the interfering light often originates from a wider area, and the magnitude of the interfering light is generally proportional to the total amount of light emitted into the scene.

[0043] By using the difference between the first measurement 126 and the second measurement 128 on a pixel-by-pixel basis, the common interfering light component, which is approximately consistent across both images for that sensor pixel, can be substantially reduced or effectively eliminated. In a specific implementation, the distance model 150 can determine the difference 154 by subtracting the second measurement 128 from the first measurement 126, or vice versa, to derive a resulting signal that essentially represents the direct light reflection from the scene. This process allows for a more accurate and reliable determination of the depth value 152, as the influence of unwanted interfering reflections or scattered light is significantly attenuated.

[0044] The depth model 105 can include an advanced driver assistance system (ADAS) 160. The ADAS 160 is configured to control the movement of the vehicle 10 based on the specified depth value 152 for the object 30. Specifically, the ADAS 160 can generate one or more control signals 162 that control the movement of the vehicle based on the depth value 152. The accurate depth values ​​152, which are essentially free from distortion caused by interfering light components, provide a more reliable and robust understanding of the vehicle 10's surroundings. This improved environmental perception can lead to enhanced decision-making by the ADAS 160. For example, the system can enable more precise and timely braking maneuvers, more accurate acceleration control, or more refined steering adjustments.Reducing the influence of interfering data also decreases the probability of a false positive result from obstacle detection, which could otherwise lead to unnecessary or erroneous vehicle responses. The reliable depth information obtained through this method contributes significantly to increased safety and overall performance of the vehicle's autonomous or assisted driving functions.

[0045] In some implementations, the depth model 105 can perform a calibration of the LiDAR sensor 20. The calibration is specifically designed to account for any imperfect spatial modulation of the first illumination pattern 120 and / or the second illumination pattern 130. As discussed, practical optical systems do not need to perfectly reproduce ideal rectangular or sinusoidal patterns. In this regard, the calibration can include a one-time routine to measure the actual spatial intensity profile of the projected patterns at each pixel and then derive correction factors (e.g., alpha and beta values). The depth model 105 can apply the correction factors during the comparison and subtraction process to ensure that the interfering light component is removed as completely and accurately as possible, even with real-world imperfections in the illumination patterns.

[0046] Fig.Figure 4 illustrates a computer-implemented method 400 for performing speckled illumination using depth sensors (e.g., LiDAR sensors 20). The data processing hardware 12 can perform the operations of method 400. In operation 402, method 400 includes projecting a first illumination pattern 120 from the LiDAR sensor 20 onto a scene. The first illumination pattern 120 illuminates a first subset of a field of view 122 and omits illumination of a second subset of the field of view 124. In operation 404, method 400 includes acquiring, by several sensor pixels 24 of the LiDAR sensor, a first measurement 126 corresponding to light reflected from the scene in response to the first illumination pattern 120. In Operation 406, Procedure 400 includes projecting a second illumination pattern 130 from the LiDAR sensor 20 onto the scene.The second illumination pattern 130 illuminates the second subset of the field of view 124 and does not illuminate the first subset of the field of view 122. In Operation 408, the procedure 400 comprises recording by the multiple sensor pixels 24 of the LiDAR sensor 20 a second measurement 128, corresponding to light reflected from the scene in response to the second illumination pattern 130. In Operation 410, the procedure 400 comprises comparing the first measurement 126 with the second measurement 128. In Operation 412, the procedure 400 comprises determining a depth value 152 for an object 30 in the scene based on comparing the first measurement with the second measurement. In Operation 414, the procedure 400 comprises controlling a movement of the vehicle 10 based on the depth value 152 for the object 30.

[0047] Thus, the Depth Model 105 provides robust depth sensing, particularly in LiDAR and other runtime systems, by strategically addressing the ubiquitous problem of interfering light components such as blooming, internal multipath propagation, external multipath propagation, and environmental scattering (e.g., fog or smoke). Unlike conventional systems that attempt to computationally correct these interference signals after data acquisition, this solution proactively attenuates them at the hardware level during the data acquisition process. By employing a complementary speckled illumination scheme and performing direct pixel-wise differential measurements, the Depth Model 105 effectively isolates the true direct light signal from external noise.

[0048] In particular, the depth model 105 employs the sequential projection of two different illumination patterns. The first illumination pattern 120 illuminates the first subset of the field of view 122, while leaving the second subset of the field of view 124 unilluminated, and a corresponding first measurement 126 is recorded. A second illumination pattern 130 (which is, for example, complementary to the first illumination pattern 120) is then projected, illuminating the previously unilluminated subset, and vice versa, and a second measurement 128 is recorded. The depth model 105 compares these two measurements, for example, by subtracting them on a pixel-wise basis. This operation effectively utilizes the property that interfering light components, which are largely a function of the total amount of emitted light and the system optics, occur consistently across both complementary illumination phases (and are approximately halved).Consequently, subtraction suppresses these unwanted components, resulting in a clean signal that represents only the direct light reflection from the scene. This unadulterated direct signal is then used to accurately determine depth values ​​for objects, significantly improving the reliability of vehicle control systems.

[0049] Furthermore, the Depth Model 105 is adaptable to various illumination profiles, including checkerboard patterns and sinusoidal spatial modulations, the latter offering increased robustness against optical imperfections. The Depth Model 105 also accounts for real-world imperfections through a calibration routine that measures imperfect spatial modulation and applies correction factors during the differential measurement process. This hardware-centric differential measurement strategy provides a superior solution to the challenges posed by interfering light, delivering more accurate and reliable depth information for critical applications such as autonomous vehicle navigation, where false positives or underestimated depths can have serious consequences.The ability to mitigate these complex light propagation problems directly at the detection stage represents a significant step forward in achieving high-fidelity environmental perception.

[0050] Several implementations have been described. However, it should be understood that various modifications can be made without deviating from the concept and scope of the disclosure. Accordingly, further implementations fall within the scope of the following claims.

[0051] The preceding description is provided for illustrative and descriptive purposes only. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular configuration are generally not restricted to that particular configuration but are, where applicable, interchangeable and may be used in any chosen configuration, even if not specifically shown or described. They may also be varied in many ways. Such variations are not to be considered a deviation from the disclosure, and it is intended that all such modifications are included within the scope of the disclosure. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 687,122

[0001]

Claims

[1] Computer-implemented procedure that is executed in data processing hardware and causes the data processing hardware to perform operations that include: Projecting a first illumination pattern onto a scene from a light-based detection and distance measurement sensor (LiDAR sensor), wherein the first illumination pattern illuminates a first subset of a field of view and omits illuminating a second subset of the field of view; Recording by multiple sensor pixels of the LiDAR sensor a first measurement corresponding to light reflected from the scene in response to the first illumination pattern; Projecting a second illumination pattern from the LiDAR sensor onto the scene, where the second illumination pattern illuminates the second subset of the field of view and omits illumination of the first subset of the field of view; The multiple sensor pixels of the LiDAR sensor capture a second measurement, corresponding to light reflected from the scene in response to the second illumination pattern; Comparing the first measurement with the second measurement; Based on comparing the first measurement with the second measurement, a depth value for an object in the scene is determined and Controlling the movement of a vehicle based on the depth value for the object. [2] Method according to claim 1, wherein comparing the first measurement with the second measurement comprises determining a difference between the first measurement and the second measurement on a pixel-wise basis in order to isolate a direct light component from a disturbing light component. [3] Method according to claim 2, wherein determining the difference between the first measurement and the second measurement comprises subtracting the second measurement from the first measurement. [4] Method according to claim 3, wherein the first lighting pattern includes a first checkerboard pattern of lighting and The second lighting pattern comprises a second checkerboard lighting pattern that is different from the first checkerboard pattern. [5] Method according to claim 4, wherein the first chessboard pattern illuminates a first set of pixels; the second checkerboard pattern illuminates a second set of pixels; the second set of pixels is not illuminated by the first checkerboard pattern and the first set of pixels is not illuminated by the second checkerboard pattern. [6] Method according to claim 1, wherein the operations further comprise performing a calibration of the LiDAR sensor to account for imperfect spatial modulation of the first illumination pattern or the second illumination pattern. [7] Method according to claim 1, wherein the first illumination pattern and the second illumination pattern each comprise rectangular oscillation-shaped spatial modulations. [8] Method according to claim 1, wherein the first lighting pattern and the second lighting pattern each comprise sinusoidal spatial modulations. [9] Method according to claim 1, wherein the LiDAR sensor comprises a direct time-of-flight sensor. [10] Method according to claim 1, wherein the LiDAR sensor comprises an indirect time-of-flight sensor.

Images