Method for depth estimation for autonomous driving

Abstract

The invention relates to a method / computer program / computer-readable storage / assistance system for depth estimation for an autonomous vehicle of a surrounding environment within a trajectory T for the

Description

FIELD OF THE INVENTION

[0001] The present invention relates to the field of automobiles. More specifically, the present invention relates to a method for depth estimation for autonomous driving according to claim 1. Furthermore, the present invention relates to a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as to a corresponding assistance system. BACKGROUND INFORMATION

[0002] Automated parking often requires very precise positioning to remain fully within the designated parking spot. This is particularly true for personal garages with limited space. To enable this precision, automated vehicles must also have precise depth measurements of their surrounding environment. This is critical for them to fit into tight parking spots while also avoiding collisions.

[0003] Unfortunately, most of an automated vehicle’s sensors are not optimized for these scenarios. An automated vehicle has many depth perception sensors, e.g. stereo cameras, LIDARs, RADARs. Yet most of them are optimized for medium to long range depth perception to support collision avoidance while driving. Thus, they are often inaccurate or simply unable to measure depths at these close ranges.

[0004] Ultrasonic sensors are better suited for low-speed parking scenarios. However, they are notoriously coarse and imprecise, limiting their usefulness for precise automated parking maneuvers. Therefore, these systems are often complemented by monocular cameras. Many automated vehicles have monocular cameras facing different areas of the vehicle’s surroundings. Since monocular camera images do not directly measure depth, depth must be inferred using semantic classifications and planar world assumptions or specialized machine learning algorithms. Unfortunately, these methods can fail when the camera image lacks sufficient visual cues or structure. For example, if an automated vehicle is trying to creep closer to a homogeneous concrete wall, a grille-mounted monocular camera may only see a homogeneous concrete surface that does not appear to change with distance. This may hinder depth estimation accuracy at close distances. SUMMARY OF THE INVENTION

[0005] It is an object of the invention to further develop a method in such a way that it enables a particularly safe drive during at least partially autonomous driving at minimal costs.

[0006] One aspect of the invention relates to a method for depth estimation of a surrounding environment of an autonomous vehicle within a trajectory for the autonomous vehicle. In this regard, it is intended that an activation of a capture mode upon request based on specific scenarios and / or parameters is performed. Subsequently, a projecting of a known structured light pattern using a light source into a surrounding environment of the vehicle is performed. Following that, a capturing of a deformed structured light pattern using at least one optical capturing device is performed. Subsequently, an analyzing of the deformed structured light pattern using an electronic computing device is performed. Finally, a deriving of depth estimations for the illuminated areas in the three-dimensional environment based on the captured deformed light patterns using an electronic computing device is performed.

[0007] The method for depth estimation for autonomous driving using structured light encompasses a series of components and steps that collaborate to generate or provide precise depth information, particularly for the vehicle's surroundings.

[0008] Key components include the light source, optical capturing device, and electronic computing device. The light source is typically provided by high-resolution vehicle headlights or other high-resolution light projectors capable of generating precise structured light patterns. The optical capturing device consists of one or more optical sensors, often in the form of vehicle cameras, positioned to capture the deformed structured light patterns reflected by surrounding objects. The electronic computing device, located inside the vehicle, processes the captured data, particularly using powerful hardware and specialized software algorithms.

[0009] The procedure commences with the activation of the capture mode, triggered upon request. This typically occurs in specific scenarios and / or based on predefined parameters. These scenarios and / or parameters may include approaching obstacles within a predefined distance range along the trajectory, malfunctions of conventional depth estimation sensors, or the detection of imprecise and / or uncertain depth estimations. The requirements of an environmental perception module can also initiate the activation process.

[0010] Following activation, a known structured light pattern is projected onto the vehicle's surroundings. This pattern may vary depending on the requirements of the current driving situation. The optical capturing device, consisting of vehicle cameras, then captures the deformed structured light pattern reflected by surrounding objects. The captured data is transmitted to the electronic computing device.

[0011] The electronic computing device also undertakes the analysis of the deformed light pattern. This is achieved by comparing the captured pattern to the known projected pattern and calculating the deformations caused by the three-dimensional environment. Based on this analysis, depth estimations for the illuminated areas in the three-dimensional environment are derived. These depth estimations are necessary to determine the precise position of the vehicle relative to the surrounding roadway.

[0012] The advantages of this method are manifold. It enables extremely precise depth estimations, particularly in situations where conventional depth estimation sensors are inaccurate or fail. The flexibility to select different structured light patterns and modes allows the system to adapt to various driving scenarios and environments. Additionally, it provides a redundant source of depth estimation, enhancing the functional safety of the autonomous vehicle.

[0013] Examples of applications for this method include automatic parking, obstacle avoidance, precise vehicle navigation, and accurate vehicle positioning within a three-dimensional environment, all of which are critical for autonomous driving.

[0014] In summary, the invention describes a structured light depth estimation system and method using high-resolution vehicle headlights and / or light projectors. Therefore, the inventive method solves the task of further developing a method that enables a particularly safe drive during partially autonomous driving with minimal costs.

[0015] In other words, this invention proposes to apply another depth estimation technique for automated vehicles that is better-suited for low-speed parking maneuvers. Specifically, it proposes using “structured light” depth estimation using the vehicle’s high-resolution headlights and / or light projectors. Structured light depth estimation is a well-known and well-validated technique. It depends on projecting a known pattern from a light projector onto a 3D structure and then capturing this structure from a camera at different location. It takes advantage of the way the light structures deform when they reflect from a 3D surface. By analyzing this deformation, these methods can estimate the depth of nearly every point illuminated by the pattern.

[0016] More specifically, the known pattern (i.e. “structured light”) is detected from the camera image. Then, correspondences are established between the observed pattern features and the projected pattern features. Given that the relative pose between the projector and camera are known, triangulation methods can be used to determine the depth of each illuminated feature from the camera and projector. This enables depth estimation for the illuminated 3D structure.

[0017] Modern automobiles can support this technique with minimal to no hardware modifications. This is because many automobiles have high-resolution LED headlights that can produce light projections with a resolution of millions of pixels. This system is primarily used for adaptive beam headlights that can block out certain regions so that they do not blind oncoming vehicles. However, they are also used for high-resolution projections, e.g. projecting a logo as part of a start-up animation sequence.

[0018] This invention proposes to take advantage of these high-resolution vehicle headlights to instead project the structured light pattern for depth estimation. This pattern can then be observed by a multitude of vehicle cameras.

[0019] Light projectors other than the headlights may also be used. For example, certain automobiles have a side-mirror mounted light projector that projects logos. These light projectors and their nearby cameras can also be used for structured light depth estimation.

[0020] The specific structured light pattern is implementation-specific and not a core tenet of this patent. For example, they may contain (but are not limited to) the following pattern types: points, lines, grids, speckle, checkerboard, binary-coded, gray-coded, color-coded, etc.

[0021] This system will only project structured light patterns during specific scenarios. In most scenarios, the lights are in their nominal operating mode. However, the system will switch to a “structured light depth estimation” mode when vehicle’s current depth estimation process is deemed insufficient for the current environment. In some implementations, this mode may be requested to the light control module by an automated driving or environmental perception control module.

[0022] The ’’structured light depth estimation” mode may be requested in the following scenarios:

[0023] The vehicle has previously detected an obstacle by other depth estimation sensors. However, the vehicle has moved towards the obstacle and it is now closer than the other sensor’s minimum depths. The structured light’s depth estimates are required to keeping tracking the obstacle and avoid a collision. This scenario may be common for automated parking systems that are pulling into tight parking spaces or garages. It may be the most practical implementation of this invention.

[0024] A nominal depth estimation sensor (LIDAR, RADAR, ultrasonics, etc.) entered a fault state, are unresponsive, or are providing data that fails health monitor checks. The system is missing necessary depth estimates or needs redundant estimates to meet functional safety requirements.

[0025] The environmental perception module / algorithms have determined that the current depth estimates are too imprecise and / or uncertain for reliable operation. To resolve this, the structured light depth estimates are requested as an additional input for the sensor fusion algorithm.

[0026] The vehicle just turned on so has not recently perceived or tracked its environment. Its automated driving system needs to ensure that there are no obstacles under or close to the vehicle before it can start moving. This region is within the minimum range of its typical depth estimation sensors. The structured light depth estimation mode may have several sub-modes that correspond to different structured lighting patterns. For example, some patterns are better suited to perceive objects with vertical edges while other patterns are better suited for horizontal edges. Some patterns are better suited for coarse estimates while others are better suited for finer refinements. Thus, the system may cycle through sub-modes or decide on sub-modes depending on the scenario.

[0027] For example:

[0028] The environmental perception module already has some depth estimates of its surrounding environment. However, it is requesting the structured light depth estimates as an additional input. The structured light depth estimation system may choose a structured light pattern that is best suited for the three-dimensional structure it expects. For example, if the environment is mostly vertical pillars, it may select a light pattern best suited for vertical edges. Or, if the environmental perception module already has a rough estimate of an obstacle’s depth, it may select a pattern that is best suited for that depth range.

[0029] When nothing is known about the surrounding environment, the structured light depth estimation system may cycle through several light patterns to ensure that it perceives a wide variety of possible structures. It may also employ a “coarse-to-fine” estimation scheme by starting with light patterns that provide coarse estimates and then refine those estimates with patterns that can estimate finer textures. In some implementations, the vehicle must be stopped or moving below a certain speed threshold while it is cycling through patterns. In fact, the structured light depth estimation module may even send a “vehicle stop” request. This may start a cycle of the vehicle stopping, the structured light depth estimation module cycling through different patterns to provide depth estimates, the vehicle moving through its perceived environment, and then the vehicle stopping to perceive again.

[0030] It is implementation-specific how the functionality is split among different control modules. In some implementations, the lighting control modules is simply given a request to enter a “structured light depth estimation” mode. The perception and analysis of the projected light is handled by another environmental perception module within the automated driving system. The lighting control module may also be given requests on which lighting patterns to project. Alternatively, the lighting control module can be given some additional information by other modules (e.g. information about its expected environment) and the lighting control module can decide which lighting pattern is best for that scenario.

[0031] This system may provide support for adding additional light projectors or cameras to vehicles. For example, many vehicles have small light projectors and fisheye cameras mounted underneath their side mirrors. However, vehicles only have fisheye cameras at the rear of the vehicle. If a small light projector integrated near the rear view camera, this system could take advantage of this pair to provide depth estimates of the area at the vehicle’s rear. Furthermore, the invention relates to a motor vehicle comprising at least the assistance system according to the preceding aspect. In particular, the motor vehicle is at least in part automated.

[0032] Advantageous embodiments of the method are to be regarded as advantageous embodiments of the computer program product, the non-transitory computer-readable storage medium, the assistance system, as well as the motor vehicle. The assistance system as well as the motor vehicle therefore comprises means for performing the method.

[0033] In the present disclosure, a computing unit may for example be understood as a data processing device with processing circuitry. A computing unit can therefore perform computing operations in order to process data. The computing operations may also include indexed accesses to a data structure, for example a look-up table, LUT.

[0034] In particular, a computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units.

[0035] A computing unit may also comprise one or more hardware and / or software interfaces and / or one or more memory units. Therein, a memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a non-volatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0036]

[0037] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figures and / or shown in the figure alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWING

[0038] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0039] The drawings shows in:

[0040] Fig. 1 an autonomous vehicle to represent a method for depth estimation of a roadway within a trajectory for the autonomous vehicle.

[0041] Fig. 2 a light pattern deformed based on a three-dimensional structure of the vehicle’s environment.

[0042] In the figures the same elements or elements having the same function are indicated by the same reference signs. DETAILED DESCRIPTION

[0043] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0044] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0045] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0046] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0047] In Fig. 1 an exemplary autonomous vehicle 1 is shown following its predefined trajectory T. The method involves several steps for estimating the depth of the roadway within the vehicle's path.

[0048] The method starts with the activation of a capture mode, triggered by specific scenarios and / or parameters. This activation is crucial to initiate the depth estimation process. A light source 2, which may consist of high-resolution vehicle headlights or advanced light projectors, projects a known structured light pattern into the vehicle's surrounding environment. This structured light pattern is depicted as a grid-like pattern surrounding the vehicle.

[0049] At least one optical capturing device 3, configured as a vehicle camera, captures the deformed structured light pattern, recording how it interacts with objects and surfaces in the environment.

[0050] An electronic computing device 4 processes and analyzes the captured deformed structured light pattern. This analysis involves comparing the deformed pattern with the known pattern and calculating deformations caused by the three-dimensional environment. Based on the analysis of the deformed light pattern, depth estimations are derived for the illuminated areas within the three-dimensional environment. These depth estimations are essential for the vehicle to understand the topography of the road and surrounding objects.

[0051] The method adapts parameters based on factors such as obstacle proximity, sensor performance, and the requirements of the environmental perception module 5 connected to the process. Furthermore, the method allows for flexibility by enabling the selection of different structured light patterns and modes to suit various driving scenarios.

[0052] In summary, Figure 1 illustrates the steps involved in depth estimation for autonomous vehicle navigation, utilizing structured light patterns projected into the vehicle's environment and captured by optical capturing devices for subsequent analysis and depth calculation.

[0053] Fig. 2 illustrates a structured light pattern 6. The parameters can be determined based on the approach to obstacles within a predefined distance range along the trajectory T and / or the occurrence of malfunctions and / or the detection of insufficient performance of conventional depth estimation sensors and / or the detection of imprecise and / or uncertain depth estimations and / or the requirements of an environmental perception module 5. The environmental perception module 5 can be the electronic computing device 4 or be integrated in the electronic computing device 4. The environmental perception module 5 has determined that an additional depth measurement is necessary. Therefore, it requests a “structured light depth estimation” mode from the light control module. The light control module then switches the headlights from nominal operation to projecting a structured light pattern 6, seen here as vertical bars in a binary pattern. This structured light is deformed based on the three-dimensional structure of the vehicle’s environment. Finally, the two headlights’ structured light projections are captured from a front-facing camera. The environmental perception module 5 detects this pattern and transforms the deformations into depth estimates. Reference Signs 1 autonomous vehicle 2 light source 3 computing device 4 optical capturing device 5 environmental perception module 6 light pattern T trajectory

Claims

1. Method for depth estimation of a surrounding environment of an autonomous vehicle within a trajectory (T) for the autonomous vehicle (1), comprising the following steps:- Activating a capture mode upon request based on specific scenarios and / or parameters;- Projecting a known structured light pattern using a light source (2) onto a surrounding environment of the vehicle;- Capturing a deformed structured light pattern using at least one optical capturing device (3);- Analyzing the deformed structured light pattern using an electronic computing device (4); and- Deriving depth estimations for the illuminated areas in the three-dimensional environment based on the captured deformed light patterns using an electronic computing device (4).

2. The method according to claim 1, characterized in thatthe parameters are determined based on the approach to obstacles within a predefined distance range along the trajectory (T) and / or the occurrence of malfunctions and / or the detection of insufficient performance of conventional depth estimation sensors and / or the detection of imprecise and / or uncertain depth estimations and / or the requirements of an environmental perception module (5).

3. The method according to claim 1 or 2, characterized in thata selection of different structured light patterns and / or modes is performed.

4. The method according to any one of the preceding claims, characterized in thatthe light source (2) is provided by high-resolution vehicle headlights and / or other light projectors.

5. The method according to any one of the preceding claims, characterized in thatthe capturing of the deformed structured light pattern is performed using at least one optical capturing device (3) configured as a vehicle camera.

6. The method according to any one of the preceding claims, characterized in thata relative position between the light source (2) and the optical capturing device (3) is used to perform triangulation methods for depth estimation.

7. A computer program product comprising program code means for performing a method according to any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium comprising at least the computer program product according to claim 7.

9. An assistance system for an autonomous vehicle (1), comprising at least one module, one capturing device, and one electronic computing device, wherein the assistance system is configured for performing a method according to any one of claims 1 to 6.15

Images