Method of training ML model to generate voxel-based 3D representation of vehicle surroundings

Abstract

The invention relates to a method for training an ML model for generating a voxel-based 3D representation of the surroundings of a vehicle, comprising the following steps: generating first image data on the basis of at least one data source representing the surroundings of the vehicle; -extracting at least one image feature from the first image data by means of a trainable ML model; -generating a voxel-based 3D representation of the surroundings of the vehicle by means of a trainable ML model by transforming at least one image feature in the 2D domain into a corresponding voxel feature in the 3D domain, each voxel feature containing information about the occupancy and color information about the 3D position of the voxel feature; rendering the generated 3D representation of the at least one voxel feature based on the color information and the information about the occupancy in order to generate second image data; -comparing the first input image data with the generated second output image data, and in the event of a clear deviation-adjusting at least one parameter of the ML model in order to minimize the determined deviation and thus train the ML model and thereby improve the generated 3D representation of the ML model.

Description

Technical Field

[0001] The present invention relates to a method for training a machine learning model to generate a voxel-based 3D representation of a vehicle's surroundings. Background Art

[0002] In the 2D or 3D object recognition known in the prior art today, so-called machine learning models (ML models) are usually used, which are trained on the basis of known and labeled objects in existing data, such as vehicles, pedestrians or traffic signs. Such methods require a large amount of existing annotated image data in order to train a neural network for the corresponding quality.

[0003] However, such a solution is not sufficient for the application field of autonomous driving. Because an autonomously driven vehicle must be able to identify every object with which a collision could occur within a 360-degree radius of the vehicle's surroundings. Such tasks are solved with the help of so-called generic object detection. The solutions for generic object detection known in the prior art have so far generally adopted a solution that obtains information about the degree of motion and depth of objects around the vehicle by fusing stereo and temporal information. However, such a solution can only be applied to a single image or image pair in the 2D domain and usually means applying techniques based on classical computer vision, which have only very low performance compared to techniques based on machine learning models.

[0004] A more advanced approach for generic object recognition is based on the use of so-called 3D voxel networks (= machine learning models), which can also be called 3D-Occupancy networks or 3D occupancy networks, which predict occupancy. These 3D voxel networks are based on assigning an occupancy to each voxel in a 3D grid (= a representation in which the network predictions are stored), i.e. whether the voxel is occupied by an object in the real world. The use of a voxel-based 3D grid for identifying objects around a vehicle is described by way of example in Us2023222748A1.

[0005] However, the use of 3D voxel networks has always required a large amount of data to train the neural network and obtain correspondingly reliable object recognition. This effort is expensive and complex.

[0006] It is therefore the task of the present invention to provide a solution with the help of which a machine learning model such as a 3D occupancy network can be trained in an efficient and cost-effective manner in order to generate a voxel-based 3D representation of the vehicle surroundings. Summary of the invention

[0007] This object is achieved by a method for training a machine learning (ML) model for generating a voxel-based 3D representation of the surroundings of a vehicle, having the features of the independent claim.

[0008] According to a first aspect, the present disclosure relates to a method for training a machine learning (ML) model to generate a voxel-based 3D representation of a vehicle's surroundings.

[0009] In a first step, first image data are generated based on at least one data source representing the surroundings of the vehicle.

[0010] In a second step, at least one image feature is extracted from the first image data by means of a trainable ML model.

[0011] In a third step, a voxel-based 3D representation of the vehicle's surroundings is generated with the aid of a trainable ML model by transforming at least one image feature in the 2D domain into a corresponding voxel feature in the 3D domain, wherein each voxel feature contains information about occupancy and color information of the 3D position of the voxel feature.

[0012] In a fourth step, the generated 3D representation of the at least one voxel feature is rendered based on the color information and the information about occupancy in order to generate second image data. To this end, the predicted color information and the predicted occupancy probability are used.

[0013] In a fifth step, the first input image data are compared with the generated second output image data and, in the event of a clear deviation, at least one parameter of the ML model, here a weight of the ML model, is adjusted in a sixth step in order to minimize the determined deviation and thereby train the ML model and thus improve the generated 3D representation of the ML model.

[0014] The basic idea of ​​the invention is to generate a voxel-based 3D representation of the vehicle's surroundings with the help of a trainable ML model, wherein each generated voxel feature in the 3D domain corresponds to an image feature in the 2D domain and contains information about occupancy and color information of the 3D position of the voxel feature.

[0015] By comparing a first input (or Input) image data with a second output (or OUTPUT) image data generated on the basis of the generated 3D representation of the voxel features, the deviation between the input image data and the output image data is detected with the help of a loss function, which helps to continuously improve the ML model used.

[0016] Another important aspect of the invention is also that the invention is suitable for combining efficient data processing in an efficient way by so-called self-supervision of so-called Neural Radiance Fields (=NeRFs) or volume rendering of 3D object representations with Occupancy networks (“occupancy networks”).

[0017] Thus, with the present invention, the following advantages can be achieved in an efficient and cost-effective manner, without requiring a large amount of training data for training the neural network:

[0018] - Generic object recognition, rather than category-specific object recognition, based on labeled or annotated training data.

[0019] -Perform so-called self-supervision, i.e. independent and efficient training of neural networks taking data efficiency into account.

[0020] - The neural network model to be trained can be fed new input data at any time (long-term learning) without using common annotation and labeling techniques or so-called ground-truth calculation on the collected data.

[0021] - Improved and more accurate predictions generated through richer 3D volume representations.

[0022] - Improved high resolution and fine-grained occupancy information

[0023] - ML models can be scaled differently depending on the application and field of use (e.g. for autonomous driving of technical devices or vehicles, environmental observation, etc.).

[0024] -Better prediction results from ML models.

[0025] One possible embodiment of the method provides for the ML model to be trained with additional training data from a lidar data source. The lidar data can be used here to create a rough estimate of the actual occupancy of the voxels in order to facilitate the training process of the ML model. The data from the lidar data source are converted into a voxel-based representation, for example a voxel grid, which is then used as training data. The advantage achieved thereby is that the ML model can be trained more easily and better, so that an accurate 3D object representation of the vehicle's surroundings can be efficiently generated.

[0026] One possible embodiment of the method provides that the rendering step is implemented as a microscopic volume rendering. This achieves the advantage that the deviation between the first input image data and the second output image data can be used to improve the ML model and thus the generated 3D representation.

[0027] One possible embodiment of the method provides that in the step of generating a voxel-based 3D representation, time information is used by extending the at least one voxel feature by aggregating at least one further voxel feature from at least one previous point in time. This achieves the advantage that an improved 3D object representation of the vehicle surroundings can be generated based on the ML model improved in this way, since the ML model contains additional time information.

[0028] According to a second aspect, the present disclosure relates to a computer program comprising machine-readable instructions which, when executed on one or more computers and / or computer instances, cause the computers or computer instances to perform a method according to the invention.

[0029] According to a third aspect, the present disclosure relates to a machine-readable data carrier and / or a download product with a computer program.

[0030] According to a fourth aspect, the disclosure relates to one or more computers and / or computer instances having a computer program and / or having a machine-readable data carrier and / or a download product.

[0031] Further measures for improving the invention are shown in more detail below together with the description of preferred exemplary embodiments of the invention based on the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic flow chart of a method 100 for training a machine learning (ML) model 30 to generate a voxel-based 3D representation of a surrounding environment 50 of a vehicle 60 is shown;

[0033] Figure 2 A schematic diagram showing a system architecture for executing method 100 according to an embodiment of the present invention, and

[0034] Figure 3 A schematic diagram of signal flow planning for executing method 100 according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0035] Figure 1 A schematic flow chart of a method 100 for training a machine learning (ML) model 30 to generate a voxel-based 3D representation of the surroundings 50 of a vehicle 60 is shown.

[0036] In step 102 , first image data 10 are generated based on at least one data source 65 , which represents surroundings 50 of vehicle 60 .

[0037] Alternatively, it is possible to use data from a LiDAR data source 9 (see Figure 2) to train the ML model 30.

[0038] In step 104 , at least one image feature 12 is extracted 104 from the first image data 10 by means of the trainable ML model 30 .

[0039] In step 106, the trainable ML model 30 is used to transform the occupancy of the object using the occupancy transformer 80 (see Figure 2 ) generates a voxel-based 3D representation 70 of the surroundings 50 of a vehicle 60 by transforming at least one image feature 12 in the 2D domain into a corresponding voxel feature 14 in the 3D domain 24, wherein each voxel feature 14 contains information about occupancy 15 and color information 16 of the 3D position 18 of the voxel feature 14.

[0040] Optionally, in step 106 of generating the voxel-based 3D representation 70, the time information 81 (see also Figure 2 ) by extending the at least one voxel feature 14 by aggregating at least one further voxel feature 14-2 from at least one previous point in time.

[0041] In step 108 , the generated 3D representation 70 is rendered for the at least one voxel feature 14 based on the predicted and determined color information 16 and the predicted and determined information about the occupancy 15 in order to generate second image data 11 .

[0042] Optionally, the rendering step 108 comprises microscopic volume rendering. It is worth mentioning here that both the predicted occupancy probability 15 and the predicted color information 16 are used to generate the output image data 11. The color information 16 is therefore only used during rendering, i.e. during training. Figure 3 ) is no longer needed during the detection, since the generic object detection only uses the occupancy probability 15.

[0043] In step 110, the first input image data 10 are compared with the generated second output image data 11 and, in the case of a determined deviation 76, at least one parameter 32 of the ML model 30 is adjusted in step 112 in order to minimize the determined deviation 76 and thereby train the ML model and thus improve the generated 3D representation 70 of the ML model 30. It should be mentioned here in this context that the first input image data 10 can be compared with the generated second output image data 11 preferably using microvolume rendering.

[0044] Figure 2 (Combined Figure 1 ) shows a schematic representation of a system architecture for executing method 100 according to one embodiment of the present invention.

[0045] Based on first image data 10 from at least one data source 65 representing surroundings 50 of vehicle 60 , at least one image feature 12 is extracted from the first image data 10 by means of a trainable ML model 30 , which includes an occupancy transformer 80 .

[0046] The occupancy transformer 80 then generates a voxel-based 3D representation 70 of the surroundings 50 of the vehicle 60 with the aid of the trainable ML model 30 (see Figure 1 ). In this case, at least one image feature 12 in the 2D domain 22 is transformed into a corresponding voxel feature 14 in the 3D domain 24. Each voxel feature 14 contains information about occupancy 15, which can be additionally enriched with data from the lidar source 9 and color information 16 of the 3D position 18 of the voxel feature 14.

[0047] In this case, the time information 81 of the voxels 14 or the voxel features 14 - 2 generated at an earlier point in time can be fed in an aggregated manner into the occupancy transformer 80 .

[0048] exist Figure 2 Also illustrated are two paths—a first signal path 82 and a second signal path 84 , wherein the first signal path is traversed only when training the model 30 , and the second signal path is traversed for interference and training the model 30 .

[0049] The rendering machine 90 renders the generated 3D representation 70 of the occupancy transformer 80 for the at least one voxel feature 14 to generate the second image data 11 .

[0050] The first input image data 10 is then compared to the generated second image data 11. If a deviation 76 is identified between the input side image data 10 and the output image data 11 generated by the renderer 90, at least one parameter 32 of the ML model 30 is adjusted to minimize the determined deviation 76. In this way, the ML model 30 is trained and the generated 3D representation 70 of the ML model 30 is improved.

[0051] Figure 3 A schematic representation of a signal flow planning for performing the method 100 using an occupied network 200 according to one embodiment of the present invention is shown.

[0052] The occupancy networks known so far have primarily been used for the 3D representation of individual objects. Modeling the entire scene or surroundings of a vehicle by using an occupancy network is also known in the prior art. However, for this purpose, a large amount of three-dimensional training data had to be generated and used to train such networks.

[0053] With the present invention, these large amounts of training data for training the occupancy network become superfluous, since only image data 10 of the surroundings 50 of the vehicle 60 from at least one data source 65 are sufficient as described in Figure 2 The ML model 30 is trained and improved in a manner clearly and exemplarily shown in FIG.

[0054] The proposed model is provided as input 201 with input image data 10 from preferably a plurality of cameras (see Figure 2 ), these input image data are recorded or recorded at a defined time point. The occupancy network 200 first extracts corresponding image features 210 therefrom. Features can be extracted separately for each image.

[0055] In block 230, the switch 80 (see also Figure 3 ) A voxel-based 3D representation 70 of the surroundings 50 of the vehicle 60 is generated with the aid of a trainable ML model 30 , wherein each voxel feature 14 provides information about occupancy 15 or an occupancy probability and color information 16 about the 3D position of the voxel feature 14 .

[0056] The determined occupancy probabilities representing the final results of the proposed ML model 30 are fed for application to corresponding downstream tasks 130, 85, such as for 3D object recognition (see also Figure 2 ).

[0057] During training of the ML model 30 , the 3D occupancy probabilities 241 are fed into the rendering component 120 together with the 3D color information 240 for each voxel in order to reconstruct the input image data 10 , 201 therefrom.

[0058] The 3D positions of all learned 3D queries 220 are then fed into the occupancy transformer 80 (see Figure 3 ), in which these learned 3D queries 220 are interacted with image features 210 by means of an occupancy transformer 230. The resulting and generated values ​​therefrom are then stored in a 3D voxel grid network and depict a representation of the occupancy probability 241 of each voxel. The model 30 provides as output information about the probability of the associated voxel being occupied in the defined 3D grid network using the color information 240 and the occupancy information 241 of each voxel.

[0059] The first input image data 10 are then compared with the generated second output image data 11 and, in the case of a clear deviation 76, the image data 11 on the output side are returned to the input of the network by applying a suitable loss function. By adjusting at least one parameter 32 of the ML model 30, the determined deviation 76 is minimized and the ML model is thus improved or trained. In this way, the generated 3D representation 70 of the ML model 30 is continuously improved.

[0060] Optionally, in this case, time information 221 may flow into the prediction results of ML model 30 by corresponding aggregation before a corresponding voxel-based 3D representation of surroundings 50 of vehicle 50 is generated.

[0061] Additionally, the lidar sensor 140 may be used to explicitly monitor and train occupancy predictions.

[0062] The surroundings 50 around the vehicle 60 are thus modeled with the aid of a 3D voxel grid based on the input data 201. Each voxel in the 3D voxel grid network contains information whether the voxel is occupied by a real object.

[0063] Based on input data 201 from a data source 65 of the vehicle 60 sensing its surroundings 50, feature extraction 210 is first performed, the results of which are fed to an occupancy transformer 230. Using the ML model trained in this way, the predictions generated for the occupancy state of each voxel can be used for a variety of different tasks (see Figure 3 Module 130), in particular for general 3D object detection.

[0064] To improve the accuracy of the trained ML model, the following steps can optionally be performed (see Figure 3 ):

[0065] By introducing a temporal aggregation 221 of the predicted occupancy states (“occupancy predictions”), earlier points in time can be taken into account and moving objects can be identified and detected over a defined time course.

[0066] One can easily integrate additional sensor data (e.g. LiDAR point clouds) to provide greater stability for the supervision of ML models (see Figure 3 Module 140 in ).

Claims

1. A method (100) for training an ML model (30) to generate a voxel-based 3D representation of a surrounding environment (50) of a vehicle (60), the method comprising the following steps: - generating (102) first image data (10) based on at least one data source (65) representing the surroundings (50) of the vehicle (60); - extracting (104) at least one image feature (12) from the first image data (10) by means of a trainable ML model (30); - generating (106) a voxel-based 3D representation (70) of the surroundings (50) of the vehicle (60) by means of a trainable ML model (30) by transforming at least one image feature (12) in the 2D domain (22) into a corresponding voxel feature (14) in the 3D domain (24), wherein each voxel feature (14) contains information about occupancy (15) and color information (16) of the 3D position (18) of the voxel feature (14); - rendering (108) the generated 3D representation (70) of the at least one voxel feature (14) based on the color information (16) and the information about the occupancy (15) so as to generate second image data (11); - comparing (110) the first input image data (10) with the generated second output image data (11) and, in the event of a discrepancy (76), - adjusting (112) at least one parameter (32) of the ML model (30) in order to minimize the determined deviation (76) and thereby train the ML model and thereby improve the generated 3D representation (70) of the ML model (30).

2. The method (100) of claim 1, wherein the ML model (30) is trained using additional training data from a LiDAR data source (9).

3. The method (100) according to any of the preceding claims, wherein the step of rendering (108) is implemented as micro-volume rendering.

4. A method (100) according to any of the preceding claims, wherein time information (81) is used in the step of generating (106) a voxel-based 3D representation (70) by extending the at least one voxel feature (14) by aggregating at least one further voxel feature (14-2) from at least one previous time point.

5. A computer program comprising machine-readable instructions which, when executed on one or more computers and / or computer instances, cause the computers or computer instances to perform the method according to any one of claims 1 to 4.

6. A machine-readable data carrier and / or a download product having a computer program according to claim 5.

7. One or more computers and / or computer instances having a computer program according to claim 5 and / or having a machine-readable data carrier and / or a download product according to claim 6.

Images