Method for training machine learning model to generate voxel-based 3D representation of environment of vehicle

Abstract

To provide a method for training an ML model to generate a voxel-based 3D representation of an environment of a vehicle.SOLUTION: A method 100 includes: Step 102 of generating first image data, which represents an environment of a vehicle, based on at least one data source; Step 104 of extracting at least one image feature from the first image data using a trainable ML model; Step 106 of generating a 3D representation for the environment of the vehicle by transforming the at least one image feature in a 2D region into a voxel feature in a 3D region; Step 108 of rendering the at least one generated 3D representation based on color information and occupancy information included in the voxel feature to generate second image data; Step 110 of comparing the first image data with the second image data; and Step 112 of adjusting at least one parameter of the ML model in order to minimize ascertained deviation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for training a machine learning model for generating a voxel-based 3D representation of a vehicle's surroundings. [Background technology]

[0002] Prior Art Today's methods known in the prior art for 2D or 3D object recognition essentially use so-called machine learning models (ML models) that are trained on the basis of known objects characterized in existing data, such as vehicles, pedestrians or traffic signs. Such methods presuppose in this case a large amount of existing annotated image data in order to train a neural network so as to obtain a corresponding quality.

[0003] However, this approach is insufficient for the application domain of autonomous driving, since an autonomous vehicle must be able to recognize each object in a 360° radius around the vehicle with which it may collide. Such a problem is solved using so-called generic object detection. Until now, solutions known in the prior art for generic object detection have often pursued an approach that obtains information about the depth and mobility of objects around the vehicle by fusing stereo and temporal information. However, this approach is only applicable for single images or image pairs in the 2D domain, and often implies the use of techniques based on classical computer vision, which have a very low performance compared to techniques based on machine learning models.

[0004] The developed approach for generic object recognition is based on using so-called 3D voxel networks (=machine learning models), which may also be called 3D occupancy networks or 3D occupancy networks that predict occupancy. These 3D voxel networks are based on assigning an occupancy to each voxel in a 3D grid (=representation where the network's predictions are stored), i.e. whether the voxel is occupied by a real-world object or not. US2023222748 describes by way of example that a 3D grid based on voxels is used to recognize objects surrounding a vehicle.

[0005] However, until now, using 3D voxel networks has always required large amounts of data to train the neural network to obtain reasonably reliable object recognition, which is both expensive and laborious. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] US Patent Publication No. 2023222748 Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, the objective of the present invention is to provide a solution that allows training a machine learning model, such as a 3D occupancy network, in an efficient and low-cost manner to generate a voxel-based 3D representation of the vehicle's surroundings. [Means for solving the problem]

[0008] Disclosure of the Invention The above problem is solved by a method for training a machine learning (ML) model for generating a voxel-based 3D representation of the surroundings of a vehicle having the features of the independent claims.

[0009] According to a first aspect, the present disclosure relates to a method for training a machine learning (ML) model for generating a voxel-based 3D representation of a vehicle's surroundings.

[0010] In a first step, generating first image data representative of the vehicle's surroundings based on at least one data source is performed.

[0011] In a second step, extracting at least one image feature from the first image data using the trainable ML model is performed.

[0012] In a third step, a voxel-based 3D representation of the vehicle's surroundings is generated using a trainable ML model by converting at least one image feature in the 2D domain into a corresponding voxel feature in the 3D domain, each voxel feature including information regarding the occupancy of the 3D position of the voxel feature and color information.

[0013] In a fourth step, rendering of the generated 3D representation of the at least one voxel feature based on the color information and the information on occupancy is performed to generate second image data, whereby both the predicted color information and the predicted occupancy probability are used.

[0014] In a fifth step, a comparison is performed between the first input image data and the generated second output image data, and if a deviation is identified, in a sixth step, at least one parameter of the ML model is adjusted in order to minimize the identified deviation and thus to train the ML model, thereby improving the generated 3D representation of the ML model, where the at least one parameter is a weight of the ML model.

[0015] The idea behind the present invention is that a voxel-based 3D representation of the vehicle's surroundings is generated using a trainable ML model, where each generated voxel feature in the 3D domain, which corresponds to an image feature in the 2D domain, contains information about the occupancy of the 3D position of the voxel feature and color information.

[0016] By comparing the first input image data or first input image data with the generated second output image data or second output image data based on the generated 3D representation of the voxel features, deviations between the input image data and the output image data are detected by a loss function, which contributes to a continuous improvement of the ML model used.

[0017] Furthermore, a further important aspect of the present invention is that it is suitable for combining in an efficient manner efficient data processing by so-called self-supervised learning of so-called neural radiance fields (=NeRF) or volume rendering with 3D object representation by occupancy networks ("Occupancy Networks").

[0018] Thus, the present invention makes it possible to achieve the following advantages in an efficient and low-cost manner, without requiring large amounts of training data to train a neural network: Instead of class-specific object recognition based on labeled or annotated training data, generic object recognition is performed. The so-called self-supervised learning is implemented, i.e. independent and efficient training of neural networks taking into account data efficiency. New input data can be fed at any time to the neural network model to be trained (long-term learning) without the need to use common annotation and labeling techniques or so-called ground truth calculations on the collected data. Richer, more stereoscopic 3D views produce improved and more accurate forecasts. · Fine-grained, high-resolution occupancy information is improved. ML models can be variably scaled to different applications and fields of use (e.g. for automated driving of technical devices or vehicles, surrounding observation, etc.). -ML models produce better prediction results.

[0019] According to one possible embodiment of the method, the ML model is trained with additional training data from a lidar data source. In this case, the lidar data can be utilized to create a rough estimate of the actual occupancy of the voxels to facilitate the training process of the ML model. The data from the lidar data source is converted into a voxel-based representation, for example a voxel grid, and these representations are then used as training data. This achieves the advantage that the ML model can be trained more easily and better, and thus an accurate 3D object representation of the vehicle's surroundings can be efficiently generated.

[0020] According to one possible embodiment of the method, the rendering step is implemented as differentiable volume rendering, which achieves the advantage that deviations between the first input image data and the second output image data can be exploited to improve the ML model and thus the generated 3D representation.

[0021] According to one possible embodiment of the method, in the step of generating a voxel-based 3D representation, temporal information is used by extending at least one voxel feature by aggregating at least one further voxel feature from at least one preceding time point, thereby achieving the advantage that it is possible to generate an improved 3D object representation of the vehicle's surroundings on the basis of the thus improved ML model, since the ML model has additional information regarding time.

[0022] According to a second aspect, the present disclosure relates to a computer program comprising machine-readable instructions for causing one or more computers or computer instances to carry out a method according to the invention when the computer program is executed on one or more computers and / or computer instances.

[0023] According to a third aspect, the present disclosure relates to a machine-readable data carrier and / or a download product comprising a computer program.

[0024] According to a fourth aspect, the present disclosure relates to one or more computers and / or computer instances comprising a computer program and / or comprising a machine-readable data carrier and / or a download product.

[0025] Further means for improving the invention are shown in more detail below on the basis of the drawings in conjunction with a description of preferred embodiments of the invention. [Brief description of the drawings]

[0026] [Figure 1] 1 is a schematic flowchart of a method 100 for training a machine learning (ML) model 30 for generating a voxel-based 3D representation of the surroundings 50 of a vehicle 60. [Diagram 2] 1 is a schematic diagram of a system architecture for implementing a method 100 according to one embodiment of the present invention. [Diagram 3] FIG. 1 is a schematic diagram of a signal flow chart for implementing a method 100 according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Working Example FIG. 1 shows a schematic flow chart of a method 100 for training a machine learning (ML) model 30 for generating a voxel-based 3D representation of a surrounding 50 of a vehicle 60.

[0028] In step 102 , generation of first image data 10 representative of the surroundings 50 of the vehicle 60 based on at least one data source 65 is performed.

[0029] Optionally, the ML model 30 can be trained with additional training data from a lidar data source 9 (see FIG. 2).

[0030] In step 104 , extracting 104 at least one image feature 12 from the first image data 10 using the trainable ML model 30 is performed.

[0031] In step 106, an occupancy transformer 80 (see FIG. 2 ) uses the trainable ML model 30 to transform at least one image feature 12 in the 2D domain 22 into a corresponding voxel feature 14 in the 3D domain 24, thereby generating a voxel-based 3D representation 70 of the surroundings 50 of the vehicle 60, where each voxel feature 14 includes information regarding the occupancy 15 of the 3D position 18 of the voxel feature 14 and color information 16.

[0032] Optionally, in step 106 of generating a voxel-based 3D representation 70, temporal information 81 (see also FIG. 2) is used by extending at least one voxel feature 14 by aggregating at least one further voxel feature 14-2 from at least one preceding time point.

[0033] In step 108, a generated 3D representation 70 of at least one voxel feature 14 is rendered based on the predicted and determined color information 16 and the predicted and determined information regarding occupancy 15 to generate second image data 11.

[0034] Optionally, the rendering step 108 includes differentiable volume rendering. In this case, it is important to note that both the predicted occupancy probabilities 15 and the predicted color information 16 are used to generate the output image data 11. That is, the color information 16 is used only during rendering, i.e., during training. The color information 16 is no longer needed during generic object detection 130 (see FIG. 3 ), because generic object detection 130 uses only the occupancy probabilities 15.

[0035] In step 110, a comparison is performed between the first input image data 10 and the generated second output image data 11, and if a deviation 76 is identified, in step 112, an adjustment is performed of at least one parameter 32 of the ML model 30 in order to minimize the identified deviation 76 and thus to train the ML model, thereby improving the generated 3D representation 70 of the ML model 30. In this connection, it is noted that the comparison between the first input image data 10 and the generated second output image data 11 can preferably be performed when using differentiable volume rendering.

[0036] FIG. 2 (in combination with FIG. 1) illustrates a schematic diagram of a system architecture for implementing method 100 according to one embodiment of the present invention.

[0037] Based on first image data 10 representing the surroundings 50 of the vehicle 60 from at least one data source 65, extraction of at least one image feature 12 from the first image data 10 is performed using a trainable ML model 30 that includes an occupancy transformer 80.

[0038] An occupancy transformer 80 then uses the trainable ML model 30 to generate a voxel-based 3D representation 70 of the surroundings 50 (see FIG. 1 ) of the vehicle 60. In this case, at least one image feature 12 in the 2D domain 22 is transformed into a corresponding voxel feature 14 in the 3D domain 24. Each voxel feature 14 in this case includes information about the occupancy 15 of the 3D position 18 of the voxel feature 14, which may be additionally enriched by data from the lidar data source 9, and color information 16.

[0039] In this case, the occupancy transformer 80 may be provided with aggregated temporal information 81 for the voxel feature 14 or for a voxel feature 14-2 that was generated at an earlier time.

[0040] FIG. 2 further illustrates two paths: a first signal path 82 that is executed only during training of the model 30, and a second signal path 84 that is executed for interference and training of the model 30.

[0041] A rendering machine 90 renders the generated 3D representation 70 of the occupancy transformer 80 for the at least one voxel feature 14 to generate the second image data 11 .

[0042] A comparison is then performed between the first input image data 10 and the generated second image data 11. If a deviation 76 between the input image data 10 and the output image data 11 generated by the renderer 90 is identified, then an adjustment is performed to at least one parameter 32 of the ML model 30 in order to minimize the identified deviation 76. In this way, the ML model 30 is trained and the generated 3D representation 70 of the ML model 30 is improved.

[0043] FIG. 3 shows a schematic diagram of a signal flow chart for implementing the method 100 using an occupancy network 200 according to one embodiment of the present invention.

[0044] Previously known occupancy networks have been used for the time being for the 3D representation of individual objects. It is also known in the prior art to use occupancy networks to model the entire scene or surroundings of a vehicle. However, this has previously required the generation and use of large amounts of 3D training data to train such networks.

[0045] In accordance with the present invention, these large amounts of training data are not necessary to train the occupancy network because image data 10 from the surroundings 50 of the vehicle 60, from at least one data source 65, is sufficient to train and improve the ML model 30, as specifically and exemplarily shown in FIG.

[0046] The proposed model is provided with input image data 10 (see FIG. 2), preferably from multiple cameras, recorded or captured at a given time point, as input 201. The occupancy network 200 first extracts corresponding image features 210 from this input image data 10. For each image, multiple features can be extracted individually.

[0047] In block 230, the occupancy transformer 80 (see also FIG. 2) uses the trainable ML model 30 to generate a voxel-based 3D representation 70 of the surroundings 50 of the vehicle 60, with each voxel feature 14 including information regarding occupancy 15 or occupancy probability and color information 16 of the 3D position 18 of the voxel feature 14.

[0048] The determined occupancy probabilities, which are the final outcome of the proposed ML model 30, are provided for use in corresponding downstream tasks 130, 85, e.g. for 3D object recognition (see also FIG. 2).

[0049] During training of the ML model 30, the 3D occupancy probabilities 241 are fed to the rendering component 120 along with 3D color information 240 for each voxel, from which the input image data 10, 201 is reconstructed.

[0050] The 3D positions of all learned 3D queries 220 are then fed to an occupancy transformer 80 (see FIG. 2) where these learned 3D queries 220 interact with the image features 210 using an occupancy transformer 230. The resulting generated values ​​from this are then stored in a 3D voxel grid network, representing a representation of the occupancy probability 241 for each voxel. The model 30 provides as an output having color information 240 and occupancy information 241 for each voxel, information regarding the probability that the corresponding voxel in a given 3D grid network is occupied.

[0051] The first input image data 10 is then compared with the generated second output image data 11 and, if deviations 76 are identified, the output image data 11 is fed back to the input of the network by using an appropriate loss function. By adjusting at least one parameter 32 of the ML model 30, the identified deviations 76 are minimized and thus the ML model is improved or trained. In this way, the generated 3D representation 70 of the ML model 30 is successively improved.

[0052] In this case, optionally, temporal information 221 prior to generating the corresponding voxel-based 3D representation of the surroundings 50 of the vehicle 60 can be included in the prediction results of the ML model 30 by corresponding aggregation.

[0053] Additionally, the lidar sensor 140 can be used to explicitly monitor and train the occupancy predictions.

[0054] That is, based on the input data 201, the surroundings 50 around the vehicle 60 are modeled using a 3D voxel grid network, where each voxel in the 3D voxel grid network contains information about whether the voxel is occupied by a real-world object or not.

[0055] Based on input data 201 originating from a data source 65 of the vehicle 60 sensing its surroundings 50, a feature extraction 210 is first performed and the result of this feature extraction 210 is fed to an occupancy transformer 230. With the ML model thus trained, the generated predictions of the occupancy of individual voxels can be used for a wide variety of tasks (see module 130 in FIG. 3), in particular for generic 3D object detection.

[0056] To improve the accuracy of the trained ML model, the following steps can optionally be implemented (see FIG. 3). Introducing a temporal aggregation of “occupancy predictions” 221 makes it possible to take previous time points into account and recognize and detect moving objects over a given time course. To provide greater robustness to the supervised learning of the ML model, further sensor data, e.g. LIDAR point clouds, can be easily incorporated (see module 140 in Figure 3).

Claims

1. A method (100) for training an ML model (30) for generating a voxel-based 3D representation of a surrounding (50) of a vehicle (60), comprising: - generating (102) first image data (10) representative of said surroundings (50) of said vehicle (60) based on at least one data source (65); - extracting (104) at least one image feature (12) from said first image data (10) using a trainable ML model (30); - generating (106) a voxel-based 3D representation (70) of the surroundings (50) of the vehicle (60) by transforming (106) the at least one image feature (12) in the 2D domain (22) into a corresponding voxel feature (14) in the 3D domain (24) using the trainable ML model (30), each voxel feature (14) including information on the occupancy (15) of its 3D position (18) and color information (16); - rendering (108) the generated 3D representation (70) for at least one of the voxel features (14) based on the color information (16) and on information relating to the occupancy (15) to generate second image data (11); A step (110) of comparing the first input image data (10) with the generated second output image data (11); If a deviation (76) is identified, - adjusting (112) at least one parameter (32) of the ML model (30) in order to minimize the identified deviations (76) and thus to train the ML model, thereby improving the generated 3D representation (70) of the ML model (30); A method (100).

2. the ML model (30) is trained with additional training data from a lidar data source (9); 2. The method of claim 1 (100).

3. The rendering step (108) is implemented as differentiable volume rendering.

3. The method (100) of claim 1 or 2.

4. In a step (106) of generating the voxel-based 3D representation (70), temporal information (81) is used by extending the at least one voxel feature (14) by aggregating at least one further voxel feature (14-2) from at least one preceding time point. The method (100) of any one of claims 1 to 3.

5. A computer program comprising: The computer program comprises machine-readable instructions which, when executed on one or more computers and / or computer instances, cause the one or more computers or computer instances to carry out the method according to any one of claims 1 to 4. Computer program.

6. A machine-readable data carrier and / or download product comprising a computer program according to claim 5.

7. One or more computers and / or computer instances comprising a computer program according to claim 5 and / or comprising a machine-readable data carrier and / or a download product according to claim 6.

Images