Method for generating universal 3d representation of environment of vehicle

Abstract

The present disclosure includes a method for generating a generic 3D representation of an environment of a vehicle, with the steps of: generating first image data representing the environment based on a data source; extracting at least one image feature from the first image data by means of an ML model; generating a voxel-based 3D representation for the environment by means of an ML model by transforming the image features into corresponding voxel features, for the 3D representation of the voxel features storing information about the probability that the 3D range of voxels is occupied; training a machine learning model for extracting image features from the camera and determining occupancy of all 3D voxels, having the following steps: projecting information on 3D positions of voxel features in the 3D representation into the camera; determining a deviation between the cameras, the deviation indicating whether the two cameras see the same corresponding image point for the projected voxel feature (luminosity error); and adjusting at least one parameter of the ML model so as to minimize the deviation and improve the 3D representation.

Description

[0001] Technology Neighborhood

[0002] The present invention relates to a method for generating a generic 3D representation of a vehicle's environment. Background Art

[0003] Current methods for 2D or 3D object recognition known in the prior art typically use so-called machine learning models (ML models), which are trained on known and previously labeled objects such as vehicles, pedestrians, or traffic signs. These methods require a large amount of existing, annotated image data in order to train the neural network to achieve the desired quality.

[0004] However, this approach is not sufficient for the application field of autonomous driving. The reason is that an autonomous vehicle must be able to identify every object within a 360-degree radius of the vehicle's environment with which it could potentially collide. This task is solved with the help of so-called general object detection. The solutions for general object detection known in the prior art have so far generally followed a scheme of obtaining information about the depth and degree of motion of objects in the vehicle's environment by fusing stereo and temporal information. However, this approach can only be applied to single images or image pairs in the 2D range and usually means applying classic computer vision-based techniques, which have only very low performance compared to techniques based on machine learning models. Summary of the Invention

[0005] Therefore, the object of the present invention is to provide a solution by means of which a universal 3D object detection for the environment of a vehicle can be realized in an efficient and reliable manner.

[0006] This object is achieved by a method for generating a universal 3D representation of a vehicle's surroundings having the features of the independent claim.

[0007] According to a first aspect, the present disclosure relates to a method for generating a generic 3D representation of a vehicle's environment, the method comprising the following steps:

[0008] In a first step, first image data are generated based on at least one data source, which first image data represent the surroundings of the vehicle.

[0009] In a second step, at least one image feature is extracted from the first image data with the aid of a trainable M-model.

[0010] In a third step, a generic representation is generated for the vehicle's environment with the aid of the trainable ML model, which generic representation can be designed as a voxel-based 3D representation by transforming the at least one image feature in the 2D range into a corresponding voxel feature in the 3D range, wherein the generated 3D representation for the at least one voxel feature stores information about the probability that the 3D range of the voxel is occupied.

[0011] In a fourth step, a machine learning (ML) model is trained to extract at least one image feature from at least one camera (alternatively: for each pixel for each camera) and to determine the occupancy of all 3D voxels, which has the following steps:

[0012] In a fifth step, information about the 3D position of the at least one voxel feature in the generated 3D representation is projected into the first camera and at least one second camera of the vehicle.

[0013] In a sixth step, a deviation between the first camera and the second camera is determined, wherein the deviation indicates whether the two cameras see the same corresponding image point from the first image data for the at least one projected voxel feature.

[0014] In a seventh step, at least one parameter of the ML model is adjusted in order to minimize the determined deviation and thereby improve the generated 3D representation.

[0015] The basic idea of ​​the present invention is to generate a 3D representation of a scene or a vehicle's environment by using a machine learning (ML) model, which enables general object recognition based on 360 degrees, which can be used for various applications.

[0016] An important aspect of the present invention is that the proposed deep learning model predicts or determines the so-called occupancy of voxels in a 3D grid by utilizing the spatial and / or temporal effects of the image recording device to be recorded (such as a camera).

[0017] In this way, the ML model can independently perform generic object detection, while at the same time providing a rich 3D representation of the environment for multiple cameras, compared to a 2D representation of the environment.

[0018] Another aspect of the present invention is the knowledge that when projecting a voxel in three-dimensional space onto at least two adjacent cameras, a measurement error, such as a photometric error, can be calculated. If the relevant voxel is occupied, the photometric error is minimal because the two cameras have similar image content at the location in the image where the 3D voxel is projected.

[0019] Unoccupied voxels will have a higher photometric error because they are not at the correct depth for the corresponding pixel in the projected image. This photometric error can be aggregated or combined across all used camera pairs that are spatially or temporally adjacent to each other. This error can then be minimized by an optimization method in order to train the ML model so that better 3D representations can be generated.

[0020] Based on this knowledge, a generic 3D object representation of the vehicle's environment can be generated by generating a 3D voxel grid in the vehicle's environment. Each voxel in this 3D voxel grid is assigned an occupancy status that provides information about whether the voxel is occupied by an object. This allows for the generation of representations of arbitrary 3D shapes or 3D geometries of objects in the vehicle's environment that can be identified solely by the occupancy parameters of the corresponding voxels, regardless of their respective class affiliations.

[0021] By using an occupancy grid, the present invention advantageously combines the data efficiency resulting from so-called self-supervision with a 3D object representation in the vehicle environment.

[0022] Here, by utilizing the present invention, the following advantages are achieved:

[0023] - Generic object detection on labeled training data, rather than class-specific object detection on labeled training data.

[0024] - Efficient and independent training of the ML model.

[0025] -The ML model can be fed with new data at any time without the need to explicitly label the collected data in advance.

[0026] The ML model can be variably extended to different applications and areas of use (eg for autonomous driving of technical devices or vehicles, environmental observation, etc.), for example by varying the number of cameras used.

[0027] - Better prediction results of the ML model.

[0028] - Using sensors in a diverse and application-dependent manner to generate data for generating a 3D object representation, such as radar sensors, lidar sensors, ultrasound sensors, motion sensors and / or thermal sensors.

[0029] The ML model is better explainable because the generated 3D representation of the environment can be understood and evaluated by humans.

[0030] By means of the present invention and the generated 3D representation of the environment, any object with arbitrary geometry can be detected in the environment of the vehicle in order in this way to generate a corresponding trajectory for the vehicle.

[0031] One possible embodiment of the method provides for training the ML model using additional training data from a lidar data source, or for generating additional training data from the lidar data source. The lidar data can be used to create a rough estimate of the actual occupancy of voxels in order to facilitate the training process of the ML model. The data from the lidar data source is converted into a voxel-based representation, such as a voxel grid, which is then used as training data. This achieves the advantage that the ML model can be trained more simply and effectively, so that an accurate 3D object representation of the vehicle's environment can be generated efficiently.

[0032] One possible embodiment of the method provides for generating the photometric error by having a first camera and at least a second camera in the vehicle each record an image at the same time t1. This achieves the advantage that an accurate 3D object representation of the vehicle's surroundings can be generated efficiently.

[0033] One possible embodiment of the method provides for generating the photometric error by configuring the first camera and at least the second camera in the vehicle as a common camera, wherein the common camera is configured to record images at two different times t1 and t2. This achieves the advantage that an improved 3D object representation of the vehicle's surroundings can be generated in an efficient manner.

[0034] Therefore, photometric errors can be generated not only by spatially neighboring cameras, but also by temporally neighboring cameras. This significantly increases the overlap of neighboring camera ranges overall, which enables more precise determination of photometric errors. The overlap of the ranges seen by spatially neighboring cameras is typically very small - adding temporal neighbors greatly increases this overlap (see Figure 2 ).

[0035] One possible embodiment of the method provides for the at least one voxel feature to be extended by aggregating at least one further voxel feature from at least one previous point in time. This allows the ML model to obtain additional temporal information, thereby achieving the advantage that an improved 3D object representation of the vehicle's surroundings can be generated.

[0036] According to a second aspect, the present disclosure relates to a computer program comprising machine-readable instructions which, when executed on one or more computers and / or computer instances, cause the one or more computers and / or computer instances to perform a method according to the present invention.

[0037] According to a third aspect, the present disclosure relates to a machine-readable data carrier and / or a download product, the machine-readable data carrier and / or the download product having the computer program.

[0038] According to a fourth aspect, the present disclosure relates to one or more computers and / or computer instances having the computer program and / or having the machine-readable data carrier and / or the download product. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Further measures for improving the invention are presented in more detail below together with the description of preferred exemplary embodiments of the invention based on the drawings.

[0040] in:

[0041] Figure 1 shows a schematic flow chart of a method 100 for generating a generic 3D representation of an environment 50 of a vehicle 60;

[0042] Figure 2 A schematic diagram showing a detection system for performing method 100 according to an embodiment of the present invention; and

[0043] Figure 3 A schematic diagram of a signal flow graph for executing method 100 according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0044] Figure 1 A schematic flow chart of a method 100 for generating a generic 3D representation of an environment 50 of a vehicle 60 is shown.

[0045] In step 102 , first image data 10 are generated based on at least one data source 65 , which first image data represent the environment 50 of the vehicle 60 .

[0046] Alternatively, the first image data 10 may be generated from any data source, which may be based on the application of different sensor types.

[0047] In step 104 , at least one image feature 12 is extracted 104 from the first image data 10 .

[0048] In step 106 , a voxel-based 3D representation 70 is generated for the environment 50 of the vehicle 60 by transforming at least one image feature 12 within the 2D range 22 into a corresponding voxel feature 14 within the 3D range 24 , wherein the generated 3D representation 70 for at least one voxel feature 14 stores information 26 about the probability that the 3D range 24 of the voxel feature 14 is occupied.

[0049] In step 108 , a machine learning (ML) model 30 is trained for transforming at least one 2D image point 25 into at least one voxel feature 15 , which has the following steps:

[0050] In step 108 - 1 , the information 26 about the 3D position 27 of the at least one voxel feature 14 in the generated 3D representation 70 is projected into the first camera 72 and the at least one second camera 74 of the vehicle 60 .

[0051] By virtue of the fact that the first camera 72 and at least the second camera 74 in the vehicle 60 each record an image at the same time t1 (=spatial neighbors), a photometric error can be calculated.

[0052] Alternatively or additionally, the photometric error can be calculated by designing the first camera 72 and at least the second camera 74 in the vehicle 60 as a common camera, wherein the common camera is designed to record images at two different times t1 and t2 (=temporal neighbors).

[0053] In step 108 - 2 , a deviation 76 between the first camera 72 and the second camera 74 is determined, wherein the deviation 76 indicates whether the two cameras 72 , 74 see the same corresponding image point 25 from the first image data 10 for the projected at least one voxel feature 14 .

[0054] In step 108 - 3 , at least one parameter 32 of the ML model 30 is adjusted in order to minimize the determined deviation 76 and thereby improve the generated 3D representation 70 .

[0055] Optionally, the at least one voxel feature 14 , 15 is extended by aggregating, for example adding or averaging, at least one further voxel feature 16 from at least one previous point in time.

[0056] Optionally, additional training data may be generated from the lidar data source in order to train the ML model 30 .

[0057] Figure 2 A schematic diagram of step 108 - 1 is shown, by which the ML model 30 is trained to transform the first image data 10 into the 3D representation 70 .

[0058] In the left figure “Spatial Neighbors”, the two cameras 72 , 74 are each arranged spatially spaced apart from one another and each record an image of a 3D point Q(x, y, z) at the same point in time.

[0059] In the right figure "Temporal Neighbors", the two cameras 72, 74 are not spatially spaced apart from each other and can be designed as a single or shared camera. The shared camera then records an image at a first time t and a subsequent time t+1.

[0060] Then, for each voxel in the 3D grid recorded by the two cameras 72 , 74 , the photometric error formed in the case of each camera pair of the left and right images is calculated.

[0061] Figure 3 A schematic diagram of a signal flow graph for executing method 100 according to an embodiment of the present invention is shown.

[0062] Figure 3 Also shown is an occupancy network 200 for generating a 3D occupancy output 250 .

[0063] Previously known occupancy networks were initially used for the 3D representation of individual objects. In the prior art, it is also known to model the environment of an entire scene or vehicle by using an occupancy network. However, to date, large amounts of three-dimensional training data had to be generated and used to train such networks. With the present invention, these large amounts of training data for training the occupancy network are no longer necessary, as only image data are sufficient to train the model. This is achieved by determining the photometric error, which is obtained by projecting voxels into the different cameras on the vehicle 60 (see Figure 2 ) to form.

[0064] The proposed model is provided with input image data from preferably multiple cameras, which are or have been recorded at a defined point in time, as input 201. The occupancy network first extracts corresponding image features 210 therefrom.

[0065] The 3D positions of all 3D queries 220 are then geometrically projected onto the cameras 72, 74 (see Figure 2 ), at these cameras, these learned 3D queries 220 interact with image features 210 by means of occupancy transformers 230. The weighted values ​​derived and generated therefrom are then stored in a 3D voxel grid and are a representation of the occupancy probability 250 of each voxel. The model provides information about the probability that the relevant voxel is occupied in the defined 3D grid, as Figure 3 The output is 250.

[0066] Thus, the environment 50 around the vehicle 60 is modeled by means of a 3D voxel grid based on the input data 201. Each voxel in the 3D voxel grid contains information whether the voxel is occupied by a real object or not.

[0067] At the same time, a photometric error 240 is calculated for each voxel seen by cameras 72, 74 by comparing the original image content of the corresponding image pixel at the voxel's 3D voxel position. The model parameters of the ML network are trained using a loss function, such as an L2 loss, between the occupancy prediction and the photometric error using a suitable optimization method, such as gradient descent.

[0068] Based on input data 201 from a data source 65 sensing the vehicle's environment 50, feature extraction 210 is first performed, the results of which are fed to an occupancy transformer 230. The output of this occupancy transformer is fed to a photometric error module 240 to calculate a photometric error for each generated voxel, as described in detail above. Because these errors are in the same 3D grid as the learned output of the model, the predicted occupancy or predicted occupancy state can be directly compared to the corresponding photometric error. Here, a low photometric error corresponds to a high probability of occupancy for the voxel.

[0069] Using ML models trained in this way, the generated predictions for the occupancy state of individual voxels can be used for a variety of different tasks (see Figure 3 Module 270), in particular for general 3D object detection.

[0070] To improve the accuracy of the trained ML model, you can optionally perform the following steps (see Figure 3 ):

[0071] By introducing a temporal aggregation 221 of the predicted occupancy states (“occupancy predictions”), earlier points in time can be taken into account and moving objects can be identified and detected over a defined time course.

[0072] - Other sensor data, such as from LiDAR point clouds, can be easily integrated to give greater stability to the supervision of the ML model (see Figure 3 Module 260 in).

Claims

1. A method (100) for generating a generic 3D representation of an environment (50) of a vehicle (60), the method comprising the following steps: - generating (102) first image data (10) based on at least one data source (65), the first image data representing an environment (50) of the vehicle (60); - extracting (104) at least one image feature (12) from the first image data (10) by means of a trainable ML model (30); - generating (106) a voxel-based 3D representation (70) of the environment (50) of the vehicle (60) by means of the trainable ML model (30) by transforming the at least one image feature (12) in the 2D range (22) into a corresponding voxel feature (14) in the 3D range (24), wherein: storing, for the generated 3D representation (70) of the at least one voxel feature (14), information (26) about the probability that the 3D extent (24) of the voxel (14) is occupied; - training (108) the machine learning (ML) model (30) to extract at least one image feature (104) from at least one camera (72, 74) and to determine the occupancy of all 3D voxels, comprising the following steps: - projecting (108-1) information (26) about the 3D position (27) of the at least one voxel feature (14) in the generated 3D representation (70) into a first camera (72) and at least a second camera (74) of the vehicle (60); - determining (108-2) a deviation (76) between the first camera (72) and the second camera (74), wherein the deviation (76) indicates whether the two cameras (72, 74) see the same corresponding image point (25) from the first image data (10) for at least one projected voxel feature (14) (photometric error); and - adjusting (108-3) at least one parameter (32) of the ML model (30) in order to minimize the determined deviation (76) and thereby improve the generated 3D representation (70).

2. The method (100) according to claim 1, wherein: The ML model (30) is trained using additional training data from a LiDAR data source.

3. The method (100) according to any one of the preceding claims 1 or 2, wherein: The photometric error (76) is calculated by recording images by the first camera (72) and the at least second camera (74) in the vehicle (60) at the same time point t1.

4. The method (100) according to any one of the preceding claims 1 or 2, wherein: The photometric error (76) is calculated in such a way that the first camera (72) and the at least second camera (74) in the vehicle (60) are designed as a common camera, wherein the common camera is designed to record images at two different time points t1 and t2.

5. The method (100) according to any one of the preceding claims, wherein: The at least one voxel feature (14, 15) is extended by aggregating at least one further voxel feature (16) from at least one previous point in time.

6. A computer program comprising machine-readable instructions, which, when executed on one or more computers and / or computer instances, cause the one or more computers and / or computer instances to perform the method according to any one of claims 1 to 5.

7. A machine-readable data carrier and / or a download product having a computer program according to claim 6.

8. One or more computers and / or computer instances having a computer program according to claim 6 and / or having a machine-readable data carrier and / or a download product according to claim 7.

Images