Computer-implemented method for generating a three-dimensional representation of the surroundings for a vehicle, training method, system, vehicle and computer program product

Abstract

The invention relates to a computer-aided method for creating a 3D representation of the surroundings for a vehicle. Two fisheye cameras capture image sequences. Neural networks project the features extracted from the images (first network for camera (1), copy of the first network for camera (2)) into a 3D space using calibration data. A second neural network, preferably comprising a cross-attention mechanism, fuses the projected features of the two cameras by finding correlations between them and taking into account spatial relationships between 2D and 3D data. Finally, the 3D surroundings are represented in voxels, the size of which increases as the distance from the vehicle increases, resulting in an efficient, multi-stage resolution. The voxels contain information regarding occupancy and object class.

Description

[0001] R.416409

[0002] - 1 -

[0003] Description

[0004] Computer-implemented method for generating a three-dimensional environment representation for a vehicle, training method, system, vehicle and computer program product

[0005] The present invention relates to a computer-implemented method for generating a three-dimensional representation of the environment for a vehicle, a system for generating a three-dimensional representation of the environment for a vehicle, a vehicle comprising such a system, and a computer program product stored on a computer-readable data carrier and comprising program code which, when executed on a computing unit, causes the unit to perform the steps of the method. The invention further relates to a training method for the method according to the invention. The method according to the invention enables precise and efficient three-dimensional environment sensing, which can be used for various driver assistance systems, in particular parking assistance systems.

[0006] State of the art

[0007] Document US12051001 B2 discloses a multi-task and / or multi-sensor fusion for three-dimensional object recognition, for example to promote the perception and control of autonomous vehicles.

[0008] Document US12051206B2 discloses a deep neural network (DNN) that performs panoptic segmentation by conducting pixel-level class segmentation and instance segmentation of a scene in a single pass of the DNN. In general, one or more images and / or other sensor data can be concatenated, stacked, and / or combined and fed into a DNN containing a common stem and multiple heads that predict different outputs. The DNN may include a class confidence head that predicts a confidence map representing pixels belonging to specific classes, an instance regression head that predicts object instance data for detected objects, an instance cluster head that predicts a confidence map of pixels belonging to specific instances, and / or a depth head that predicts range values. These outputs can be decoded. R.416409

[0009] - 2 - are used to identify boundary shapes, class labels, instance labels and / or distance values ​​for detected objects, and can be used to enable safe path planning and control of an autonomous vehicle.

[0010] The object of the present invention is to improve a method for generating a three-dimensional representation of the environment.

[0011] Disclosure of the invention

[0012] The above problem is solved according to the invention in accordance with independent claims 1 and 10 to 13.

[0013] The method according to the invention comprises capturing a sequence of camera images of a vehicle's surroundings using at least two vehicle cameras, wherein the capture areas of the two vehicle cameras overlap. The use of at least two vehicle cameras, which in particular each comprise a wide-angle lens or are designed as fisheye cameras, enables the capture of a large part of the vehicle's surroundings by each camera. Preferably, the vehicle's surroundings are captured almost completely by at least four vehicle cameras, wherein each of the vehicle cameras has a wide-angle lens and one of the vehicle cameras is arranged on each side of the vehicle, so that the capture areas of the vehicle cameras preferably differ. The wide-angle lenses cover a large field of view, which reduces the number of cameras required and thus saves costs.In the next step, or rather the first projection, initial image features or feature vectors are determined based on the camera images from the first vehicle camera by the backbone of a first neural network. Subsequently, after the backbone of the first neural network, these initial image features or feature vectors, determined based on the camera images from the first vehicle camera and using calibration data from the first vehicle camera, are projected into a three-dimensional space by an interpretation head of the first neural network. Using the first neural network for feature recognition and projection enables robust and efficient processing of the camera images, even under challenging conditions such as low light or rapidly changing scenes. Similarly, in an R.416409...

[0014] - 3 - In the next step, or rather in a second projection, second image features or second feature vectors are determined based on the camera images of the second vehicle camera by the backbone of the first neural network or preferably by a copy of the backbone of the first neural network. Subsequently, in this step, the second image features or second feature vectors determined based on the camera images of the second vehicle camera are projected into the (same) three-dimensional space by the interpretation head of the first neural network or preferably by a copy of the interpretation head of the first neural network, based on calibration data of the second vehicle camera. Preferably, therefore, the second projection is performed essentially simultaneously or in parallel with the first projection.The first neural network, or its copy, advantageously includes a backbone for determining the first and / or second image features or feature vectors, and a head or interpretation head for projecting these features into three-dimensional space. The first neural network also advantageously includes a transformer model. Subsequently, the projected first and second image features are fused, and a correlation between them is determined using a second neural network, which in particular features a cross-attention mechanism. This cross-attention mechanism specifically considers the spatial relationships between 2D image features and their corresponding 3D coordinates.Furthermore, the attention weights advantageously represent the correspondence between the features and the 3D coordinates. The cross-attention mechanism enables efficient and robust correlation of the image features by considering the spatial relationships between the features and their 3D coordinates. Finally, a three-dimensional environment representation is generated, which is subdivided into voxels. The voxels contain information about the occupancy and object class in this environment. The size of each voxel, or the resolution of the three-dimensional environment representation, varies incrementally depending on the distance to the vehicle, with the voxel size generally being larger for areas at a greater distance from the vehicle than in the vehicle's immediate vicinity. In other words, the invention relates to a computer-aided method for creating a 3D environment representation for a vehicle.At least two vehicle cameras R.416409.

[0015] - 4 - Capture image sequences. Two first neural networks project the features extracted from the images (first network for camera 1, copy of the first network for camera 2) into a 3D space using calibration data. A second neural network, preferably incorporating a cross-attention mechanism, fuses the projected features from both cameras by finding correlations between them and considering spatial relationships between 2D and 3D data. The 3D environment is then represented in voxels, the size of which increases with distance from the vehicle, resulting in efficient, multi-level resolution. The voxels contain information about occupancy and object class. The voxel-based representation allows for more efficient storage and processing of the 3D information.The stepwise changing or variable voxel size optimizes memory requirements and processing power by adjusting the resolution to the relevance of the information. In the near field, where high precision is important for tasks such as parking maneuvers, a higher resolution is advantageous, while in the far field, a lower resolution is sufficient for applications such as adaptive cruise control.

[0016] Advantageously, the voxel size for at least certain sub-areas of the environment can be adjusted depending on the current vehicle speed. This allows for dynamic adaptation of the resolution to the driving situation. At higher speeds, a lower resolution is advantageously sufficient in the immediate vicinity, as, for example, no parking maneuvers are performed. At lower speeds, a lower resolution is advantageously sufficient in the far vicinity, as, for example, no overtaking and / or following distance maneuvers are performed. This saves processing power in each case and enables faster updates of the environment rendering.

[0017] In one embodiment, the three-dimensional environment representation comprises at least three different voxel sizes, each representing a different distance range. In other words, the three-dimensional environment representation has at least three distinct voxel sizes that increase or are stepped depending on the distance. Each step of the voxel size advantageously represents a specific distance range. This enables efficient and scalable adjustment of the resolution to the distance. R.416409

[0018] - 5 -

[0019] Preferably, the object class assigned to a voxel includes at least one of the following classes: vehicle, pedestrian, road, and / or obstacle. Classifying the objects allows for differentiated handling of various road users and environmental features. This improves safety and enables specific functions such as pedestrian detection or obstacle avoidance.

[0020] Preferably, the three-dimensional environment representation is combined with a three-dimensional vehicle model by positioning the vehicle model as a synthetic model within the three-dimensional environment representation. This enables collision detection and precise maneuver planning, particularly for parking assistance systems.

[0021] In a further embodiment, the method includes the display of a virtual surround view based on the generated three-dimensional representation of the environment. For this purpose, the virtual surround view has a projection surface. Advantageously, the projection surface includes at least one lateral projection surface that extends at least partially around the vehicle. This lateral projection surface is deformed based on a predefined distance and as an envelope around the voxels labeled with occupancy information. The projection surface can additionally include a flat projection surface representing the ground or the surface beneath it. At least the currently captured camera images from the vehicle cameras are transformed and projected onto the projection surface.Additionally, it can be provided that stored camera images are projected onto the projection surface and that the projection surface is displayed to the user from the viewpoint or perspective of a virtual camera, particularly via a display or display device in the vehicle. The displayed surround view provides the driver with an overview of the vehicle's surroundings and facilitates parking and maneuvering in tight spaces. The deformation of the projection surface based on the voxels enables a realistic representation of the environment, reduces artifacts, and improves the perception of obstacles.

[0022] In an advantageous embodiment, the brightness of the pixels of the camera images projected onto the projection surface is determined depending on the information about the occupancy and / or the object class of voxels present at the location of the projection surface and / or in extension of the viewing direction of the perspective of R.416409

[0023] - 6 - The virtual camera is adjusted to the voxel lying in the projection plane. This improves the visualization of the environment and highlights important information.

[0024] Another aspect is that the voxel size is smaller within the specified distance compared to the voxel size in front of and behind the specified distance. This optimizes the resolution of the surround view in the relevant area and improves the rendering of obstacles.

[0025] In an additional embodiment, the method includes the planning and / or control of a vehicle's parking and / or maneuvering function based on the generated three-dimensional representation of the environment. The parking and / or maneuvering functions are specifically designed to prevent collisions between the vehicle and objects in its surroundings and / or to plan and / or at least partially automate vehicle maneuvers in confined spaces. This enables automated parking functions and assists the driver in maneuvering in challenging situations.

[0026] The training method according to the invention is designed to train the computer-implemented method for generating a three-dimensional representation of the environment for a vehicle. The training data comprises sequences of camera images from at least two vehicle cameras and distance data between the vehicle and objects in the vehicle's environment. The distance data is acquired using at least one distance sensor, in particular using another vehicle camera, a radar sensor, a LiDAR sensor, and / or an ultrasonic sensor, and / or determined by a stereovision and / or a structure-from-motion method. The training method advantageously enables the optimization of the three-dimensional representation of the environment by adjusting the parameters of the neural networks and the adaptation of the method to different vehicle types and environmental conditions.

[0027] The system for generating a three-dimensional representation of a vehicle's environment comprises at least two vehicle cameras, each in particular comprising wide-angle lenses, wherein each vehicle camera is configured to capture a sequence of camera images of the vehicle's surroundings. The system R.416409

[0028] - 7 - also includes a computing unit configured to execute the steps of the method. The system enables the practical implementation of the method according to the invention in vehicles.

[0029] The invention also relates to a vehicle which incorporates a system according to the invention. The integration of the system into a vehicle enables the use of the three-dimensional environment representation for various driver assistance systems, in particular parking assistance systems.

[0030] Finally, the invention relates to a computer program product stored on a computer-readable data carrier and comprising program code which, when executed on a computing unit, causes the unit to perform the steps of the method. The computer program product enables the software-based implementation of the method according to the invention.

[0031] Further advantages will become apparent from the following description of the exemplary embodiment with reference to Figure 1.

[0032] Figure 1: Flowchart of the procedure for generating a three-dimensional representation of the environment

[0033] Example of implementation

[0034] The following embodiment explains the inventive method for creating a three-dimensional representation of the environment. Figure 1 schematically shows a flowchart of the method for generating a three-dimensional representation of the environment as a block diagram. The method begins with the acquisition 10 of a sequence of images by at least two cameras (preferably having wide-angle lenses), which are each attached or arranged at different locations or positions on the vehicle. The acquired sequences of images are forwarded in parallel to two preferably identical first neural networks. In steps 20 and 30, the first neural network determines features or feature vectors based on the first and second image sequences, respectively, and projects these determined feature vectors into a three-dimensional space.Each branch of the first neural network includes a backbone (e.g., a Convolutional Neural Network) for extracting image features R.416409.

[0035] - 8 - or feature vectors and an interpretation head that projects these features into the common three-dimensional space, taking the camera calibration data into account. The image features obtained from the two camera images and projected are then combined and fused in a fusion stage in step 40. In step 40, a second neural network (e.g., a transformer model with a cross-attention mechanism) is used to correlate the identified and projected features from both perspectives and fuse correlated features. Subsequently, in step 50, a three-dimensional environment representation is generated, which is divided into voxels of different sizes. The voxel size of the three-dimensional environment representation varies depending on the distance to the vehicle—the further away, the larger the voxel. Each voxel contains information about its occupancy (whether an object is located there) and the object class (e.g., vehicle, car, etc.).B. vehicle, pedestrian, road). The three-dimensional environment representation can then be used for various applications, e.g. for surround view generation 60 or the execution 70 of a parking assistance procedure.

Claims

R.416409 - 9 - Claims 1. Computer-implemented method for generating a three-dimensional representation of the environment for a vehicle, the method comprising the following steps: a. Acquisition (10) of a sequence of camera images of a vehicle environment using at least two vehicle cameras, wherein the vehicle cameras in particular have wide-angle lenses (fisheye cameras); b. Initial determination (20) of first image features depending on the camera images of the first vehicle camera and projection of the respective determined first image features into a three-dimensional space using calibration data of the first vehicle camera by a first neural network; c.Second determination (30) of second image features depending on the camera images of the second vehicle camera and projection of the respective determined second image features into three-dimensional space using calibration data of the second vehicle camera by a copy of the first neural network; d. Fusion (40) of the projected first and second image features by determining a correlation between the projected first and second image features by means of a second neural network, which in particular has a cross-attention mechanism, wherein the cross-attention mechanism in particular takes into account the spatial relationships between 2D image features and their corresponding 3D coordinates, wherein the attention weights in particular represent the correspondence between the features and the 3D coordinates; and e.Generating (50) a three-dimensional environment representation that is subdivided into voxels, wherein the voxels contain information about the occupancy and the object class, and wherein the size of the respective voxel or the resolution of the three-dimensional environment representation varies stepwise depending on the distance to the vehicle, wherein the voxel size is larger for areas at a greater distance from the vehicle than in the near area of ​​the vehicle.

2. Method according to claim 1, wherein the voxel size is additionally adjusted depending on the current vehicle speed. R.416409 - 10 - 3. Method according to one of the preceding claims, wherein the generated three-dimensional environment representation comprises at least three different voxel sizes, each voxel size representing a different distance range.

4. Method according to any of the preceding claims, wherein the object class assigned to a voxel comprises at least one of the following classes: vehicle, pedestrian, road and / or obstacle.

5. Method according to one of the preceding claims, wherein the three-dimensional environment representation is combined with a three-dimensional vehicle model by positioning the vehicle model as a synthetic model in the three-dimensional environment representation.

6. A method according to any of the preceding claims, comprising the following step: f. Display (60) a virtual surround view depending on the generated three-dimensional environment representation, wherein the virtual surround view has a projection surface which includes at least one lateral projection surface, wherein the lateral projection surface is at least partially spanned around the vehicle, wherein the lateral projection surface is deformed based on a predetermined distance and as an envelope around the voxels marked with occupancy information, wherein at least the currently captured camera images of the vehicle cameras are transformed and projected onto the projection surface and the projection surface is displayed to a user from the perspective of a virtual camera.

7. Method according to claim 6, wherein the brightness of the pixels of the camera images projected onto the projection surface is adjusted depending on the information about the occupancy and / or the object class of voxels present at the location of the projection surface and / or in extension of the viewing direction of the perspective of the virtual camera in the projection plane.

8. A method according to claim 6 or 7, wherein the voxel size in the region of the predetermined distance is greater than the voxel size in the region in front of the R.416409 - 11 - specified distance and the area behind the specified distance is smaller.

9. A method according to any of the preceding claims, comprising the following step: g. Planning and / or controlling (70) a parking and / or maneuvering function of a vehicle depending on the generated three-dimensional representation of the environment, wherein the parking and / or maneuvering functions are in particular configured to avoid collisions between the vehicle and objects in the vicinity of the vehicle and / or to plan and / or at least partially automatically control a maneuver of the vehicle in confined spaces of the environment.

10. Training method, which is configured to train the computer-implemented method for generating a three-dimensional environment representation for a vehicle according to any one of claims 1 to 9, wherein the training data for the training method comprise sequences of camera images from at least two vehicle cameras and distance data between the vehicle and objects in the environment of the vehicle, wherein the distance data are acquired by means of at least one distance sensor, in particular by means of another vehicle camera, a radar sensor, a LiDAR sensor and / or an ultrasonic sensor, and / or are determined by a stereovision and / or a structure-from-motion method.

11. System for generating a three-dimensional representation of the environment for a vehicle, comprising: a. at least two vehicle cameras, in particular comprising wide-angle lenses, each configured to capture a sequence of camera images of a vehicle environment; b. a computing unit configured to perform the steps of the method according to any one of claims 1 to 9.

12. Vehicle comprising a system according to claim 11.

13. Computer program product stored on a computer-readable data carrier and comprising program code which, when placed on a computing unit R.416409 - 12 - is executed, the computing unit is caused to execute the steps of the method according to one of claims 1 to 9.

Images