Sensor virtualization

Abstract

The invention provides a method for training a neural network to predict an object around a vehicle, the method comprising:-obtaining physical sensor data from a physical sensor having one or more modalities; -performing a first mapping from the physical sensor data to the virtual sensor to obtain virtual sensor data; -performing a second mapping from the virtual sensor data to the 3D model space; and-training the neural network based on the virtual sensor data and the one or more annotations in the 3D model space.

Description

Technical Field

[0001] The invention relates to a method for training a neural network to predict objects in the surroundings of a vehicle and to a system for training a neural network to predict objects in the surroundings of a vehicle, wherein the system is configured to carry out the method of one of the preceding claims.

[0002] The invention also relates to a computer-readable storage medium storing a program code comprising instructions for executing such a method. Background Art

[0003] A key element of autonomous driving is the ability to build a 360-degree environment model. This environment model can be obtained by utilizing different sensor modalities. However, this involves training a different model for each sensor setup. Unfortunately, this solution is not scalable in terms of time and expense (e.g., training data collection, annotation, neural network modeling, and training). Therefore, a unified training approach is needed to overcome the above limitations. Summary of the invention

[0004] The object of the present invention is to provide a method for training a neural network to predict objects around a vehicle and a system for training a neural network to predict objects around a vehicle, wherein the system is configured to perform the method of one of the preceding claims, which overcomes one or more of the above-mentioned problems of the prior art.

[0005] A first aspect of the present invention provides a method for training a neural network to predict objects around a vehicle, the method comprising:

[0006] - obtaining physical sensor data from a physical sensor having one or more modalities;

[0007] - performing a first mapping from physical sensor data to one or more virtual sensors to obtain virtual sensor data;

[0008] - performing a second mapping from the virtual sensor data to the 3D model space; and

[0009] - Train a neural network based on virtual sensor data and one or more annotations in the 3D model space

[0010] The method of the first aspect has the advantage of using a sensor-agnostic perception setup. Thus, training data that has been acquired using different physical sensors can be combined.

[0011] Preferably, the physical sensor has at least two modalities.In addition, the physical sensor may include at least one sensor whose sensor data is not directly acquired in the 3D model space.

[0012] The training of the neural network can be performed, for example, based on ground truth training data including annotations. The annotations are not limited herein, i.e., they can be attributes of an object, objectness scores, and / or other predictions not directly related to one or more objects.

[0013] In a first implementation of the method according to the first aspect, the method further comprises obtaining training data using a fleet of vehicles, wherein the fleet of vehicles uses different physical sensors.

[0014] Since all physical sensor data from the fleet is mapped to virtual sensor data (which preferably reflects the same virtual sensor), uniform training can be performed even if the originally acquired sensor data is not uniform. For example, the physical sensors may differ in positioning on the vehicle, field of view, range, and / or many other acquisition parameters. These differences can be compensated by a first mapping, which can in effect modify the physical sensor data of multiple different physical sensors so that the corresponding virtual sensor data has characteristics as if acquired with one uniform virtual sensor (or, for example, a set of virtual sensors corresponding to different modalities).

[0015] Performing virtualization of several modalities (eg, having at least two virtual sensors corresponding to at least two virtual modalities) has proven to be particularly useful in experiments.

[0016] The fleet preferably comprises at least 10 vehicles, in particular at least 100 or 1000 vehicles, or more preferably at least 10000 vehicles.

[0017] In a further implementation of the method according to the first aspect, performing the first mapping comprises applying a transformation from the physical sensor data to obtain the virtual sensor data, wherein the transformation is based on a difference between actual physical properties of the physical sensor and virtual physical properties of the virtual sensor.

[0018] For example, a physical sensor may include one or more cameras having a given first focal length, while a virtual sensor may include one or more cameras having a second focal length different from the first focal length. Thus, a mathematical transformation may be used to transform the physical sensor data into virtual sensor data. In other words, the virtual sensor data then appear as if they were acquired with a camera having the second focal length. Thus, a fleet of vehicles may use cameras having different focal lengths, yet all acquired data will be available as virtual sensor data having a common second focal length.

[0019] In a further implementation of the method according to the first aspect, the 3D model space comprises a bird's eye view raster. This has the advantage that many annotations can be better estimated based on the 3D model space being a bird's eye view raster.

[0020] In another implementation of the method according to the first aspect, performing the second mapping includes: if a failure of a first sensor of the physical sensors is detected, the method includes populating with virtual sensor data obtained from a second sensor of the physical sensors, wherein the first sensor and the second sensor use different modalities.

[0021] This has the advantage that the method can compensate for failures of one or more physical sensors, thereby greatly improving the reliability of the method in practical applications. For example, in autonomous driving, it must be ensured that the vehicle can still navigate safely even if one of the cameras is obscured, for example by dust on the camera lens.

[0022] Filling of missing data can be performed as follows. Since different modalities are mapped into the model space, overlapping observations (feature maps) from different sensor modalities can be stored in the neural network. This provides sufficient redundancy in the event of sensor outages (i.e., one of the sensors fails to provide a signal for a specific timestamp). For example, at time t, the front camera fails to provide camera observations, but the long-range lidar operates normally. In this case, the feature map corresponding to the camera in front of the ego car in the model space will be empty, resulting in no detection without the additional sensor modality. However, the observations (feature maps) provided by the lidar are stored in the neural network, which allows the detection of objects even if the camera observations are disrupted.

[0023] In a further implementation of the method according to the first aspect, the virtual sensor comprises one virtual sensor for each virtual modality.

[0024] In other words, in this implementation, there is only one virtual sensor for each virtual modality. Experiments have shown that this mapping to one virtual sensor for each (virtual) modality simplifies the training data and may lead to better training results.

[0025] In a further implementation of the method according to the first aspect, the method further comprises the step of training parameters of an encoder and a decoder of the transformer model, wherein the encoder maps from the virtual sensor data to a latent space and the decoder maps from the latent space to a 3D model space.

[0026] This has the advantage that even for modalities where a well-defined mathematical transformation from virtual sensor data to 3D model space for that modality is not available, the mapping can be efficiently learned from the training data. Experiments show that the transformer model can produce good results for the mapping from virtual sensor data to 3D model space.

[0027] In a further implementation of the method according to the first aspect, the neural network comprises a feature mapping subnetwork that maps from the virtual sensor data to a feature map in the 3D model space, and a processing head that maps from the feature map to the annotation space.

[0028] In this implementation, the feature map in the 3D model space is an intermediate result, which is then used by the processing head to determine the annotations in the annotation space. Experiments show that this setup produces excellent results.

[0029] It should be understood that in this implementation, the annotation space may include annotations such as object labels and / or object (non-)existence, however, there is no limitation on the type of annotations used. In particular, they may also include scalar and / or vector values.

[0030] In a further implementation of the method according to the first aspect, the feature map comprises a feature sub-map in the 3D model space for each of the at least two modalities, wherein, preferably, each feature sub-map is fed into a processing head.

[0031] For example, there may be a first feature subgraph for camera data, a second feature subgraph for lidar data, and a third feature subgraph for radar data. Each of the feature subgraphs may include multiple values ​​for each position in the 3D model space. The entire feature graph may be considered to be stored in a large tensor.

[0032] In another implementation of the method according to the first aspect, one or more annotations include the presence of an object and / or a label of the object. The prediction may also include an objectness score, such as the probability that an object exists at a given location in the 3D model space. The label of the object may include, for example, a pedestrian label, a road label, a vehicle label, etc.

[0033] In another implementation of the method according to the first aspect, the method further comprises performing a fusion between at least two modalities in the 3D model space. A lightweight convolutional neural network may be used to perform the mapping.

[0034] In a further implementation of the method according to the first aspect, training the neural network comprises training the neural network multiple times, wherein, during at least one training session, data from one or more of the virtual sensors and / or the physical sensors is omitted.

[0035] This implementation has the advantage that the robustness of the trained neural network is increased.

[0036] In another implementation of the method of the first aspect, the first mapping includes one or more first parameters, the second mapping includes one or more second parameters, and the processing head for obtaining one or more annotations includes one or more third parameters, wherein training the neural network includes end-to-end training to obtain the first parameters, the second parameters, and the third parameters.

[0037] End-to-end training of all parameters has the advantage that only sufficient training data needs to be acquired and then all relevant parameters of the entire system can be automatically determined.

[0038] Another aspect of the present invention relates to a system for training a neural network to predict objects around a vehicle, wherein the system is configured to perform the method of the first aspect or one of the implementations of the first aspect. The system can be implemented, for example, on a server, in particular on multiple servers or in the cloud. The system can be configured to continuously receive new training data, for example from a fleet of vehicles that can be connected to the system.

[0039] Another aspect of the present invention relates to a computer-readable storage medium storing program code, the program code comprising instructions, which when executed by a processor, perform the method of the second aspect or one of the implementations of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical features of the embodiments of the present invention, the following briefly introduces the drawings provided for describing the embodiments. The drawings in the following description are only some embodiments of the present invention, and these embodiments can be modified without departing from the scope of the present invention as defined in the claims.

[0041] Figure 1 is a flow chart of a method according to an embodiment of the present disclosure,

[0042] Figure 2 is a block diagram illustrating a first mapping from physical sensor to virtual sensor data according to an embodiment of the present disclosure,

[0043] Figure 3 is a block diagram illustrating a second mapping from virtual sensor data to a feature map in a 3D model space according to an embodiment of the present disclosure, and

[0044] Figure 4 is a processing feature map according to an embodiment of the present disclosure (e.g., Figure 3 Flowchart of the feature diagram determined by the system). DETAILED DESCRIPTION

[0045] The foregoing description is only an implementation of the present invention, and the scope of the present invention is not limited thereto. Those skilled in the art can easily make any modification or replacement. Therefore, the protection scope of the present invention shall be subject to the protection scope of the attached claims.

[0046] Neural networks trained for 3D perception tasks are sensitive to the input sensors because they can implicitly learn the intrinsic properties of a particular sensor. For example, when a neural network is used to detect objects on a camera stream recorded with a different focal length than when the network was trained, the depth estimate often has significant errors due to the inherent mismatch in the cameras.

[0047] Embodiments of the present invention solve this problem by defining reference virtual sensors and mapping real sensors onto these virtual sensors.Preferably, the virtual sensors reflect a different physical property than at least one of the physical sensors.

[0048] Figure 1 is a flow chart of a method 100 according to the present invention. The method is used to train a neural network to predict objects around a vehicle. For example, the vehicle may be a (semi-autonomous or fully autonomous) vehicle.

[0049] The method comprises a first step 110 of obtaining physical sensor data from a physical sensor having at least two modalities. For example, the physical sensor may comprise a camera and a lidar or radar sensor. In other embodiments, there may be only one modality.

[0050] The method includes a second step 120 of performing a first mapping of physical sensor data to a virtual sensor to obtain virtual sensor data. For example, an image (video stream) from a camera may be mapped to obtain a virtual image (virtual video stream), and lidar and / or radar data may be mapped to obtain virtual lidar and / or virtual radar data. The mapping of the camera data may be based on the difference between the actual physical properties of the camera and the virtual physical properties of the virtual camera. Similarly, a mapping of data acquired by a lidar or radar may be mapped to a corresponding virtual lidar or radar. It should be understood that other such mappings may be performed, such as mapping from multiple sensors of a modality to one or more virtual sensors corresponding to the modality.

[0051] For example, a virtual camera may have predefined characteristics (e.g., focal length, principal point, position relative to the car coordinate system). The mapping from a physical camera to the virtual camera may be performed by obtaining a transformation from the physical camera to the virtual camera using extrinsic and intrinsic matrices. In the case of radar, the virtual sensor may have a predefined field of view and perception range. The physical radar signal is then preprocessed and projected onto a 3D model space raster grid, and the preprocessed signal is inserted into the bird's-eye view grid of the virtual radar (which may have a longer perception range and different angular resolution than the physical sensor). Physical-virtual lidar mapping works in a similar manner.

[0052] In some embodiments, such as when several different cameras and one type of lidar are used, there may be only one virtual sensor. The different cameras are mapped to the virtual camera sensor, but the lidar can be used without virtualization because the same type of physical lidar is installed to each recording vehicle.

[0053] The method further comprises a step 130 of performing a second mapping from the virtual sensor data to the 3D model space.

[0054] The method comprises a final step 140 of training a neural network based on the virtual sensor data and one or more annotations in the 3D model space.

[0055] In the case of a camera, the (first) mapping can be created using a camera matrix and projection geometry that ensures invariance to translation and rotation. On the other hand, 3D sensing devices typically transmit their signals in model space by design. Due to this property, sensor virtualization can be performed using an extrinsic matrix to provide translation and rotation invariance. An appropriate resolution match between the original 3D sensor and the virtual sensor can be ensured by scaling the input signal by a factor that can be obtained from the sensor configuration. In this way, sensor-agnostic perception systems can be developed. That is, the need to train a separate model when recordings from a new sensor setup become accessible can be avoided.

[0056] Sensor virtualization enables the training of 360-degree perception models with heterogeneous sensor setups. However, this sensor virtualization does not provide an efficient end-to-end solution to train the above models. The most common approach for implementing 360-degree perception systems is to train separate models for each sensor modality and then combine them to form an environment model. This solution is time-consuming and expensive because training convolutional neural networks requires a lot of power and computing power.

[0057] Since the ground truth around the ego car is well defined in model (world) space and the input sensors are mapped onto reference virtual sensors with known orientation and field of view, it is also beneficial to output the predictions in model space as a bird's eye view raster. During training, the neural network can be fed by the signals from all sensors and the predictions are defined in model space, which is also how the ground truth is represented. In this way, all inputs can be processed together in one step.

[0058] The proposed solution to improve efficiency is partly influenced by the use of a Siamese network with shared weights for both images. In the proposed implementation, a dedicated subnetwork is used for each sensor modality. For example, if the recording system has four pinhole cameras (front, back, left, right), only one backbone network is needed for feature extraction. If fisheye cameras are part of the sensor setup, their input can be used by the pinhole camera feature extractor network, or a new fisheye encoder network can be defined. The number of trainable network parameters can be reduced linearly by the number of sensors corresponding to the same sensor group.

[0059] Since the ground truth is preferably defined in a 3D model space and it is also desired to predict in 3D, the camera features are transformed from the 2D image space into the 3D world space. The dimensionality increase mapping can be performed in several ways. One option is to utilize a pseudo lidar solution. The preferred embodiment uses an encoder-decoder architecture. The encoder-decoder network comprises two sub-networks. The encoder is responsible for embedding the input into a latent space, which is typically a smaller dimensional space than the original input space. The decoder network is then fed with the encoded input and transforms it into the representation sought.

[0060] In a preferred embodiment, the encoder-decoder network is fed by camera images. The encoder is a convolutional neural network with asymmetric strides, while the decoder uses transposed convolutions with asymmetric strides. As a first step, the encoder extracts relevant features from the image in a trainable manner. The transformation from image space to model space can be further enhanced by a visual transformer using an attention mechanism, where the transformer module learns which image features are attended to which model space features. The output of the decoder is the final raster representation of the model space around the ego car, ensuring a bijective mapping between ground truth and predictions.

[0061] Different sensors have inherently different signal representations. For example, a camera image is represented as a dense grid with a color value for each pixel. The input to a lidar is completely different from a camera image and can be represented as a point cloud consisting of the X, Y, Z coordinates of the reflectivity and the intensity of the reflectivity. Radar may have a similar representation with an additional velocity component. Both lidar and radar have sparse representations compared to images. In addition, 3D sensing devices operate in model space by design, while camera images exist in 2D space. To avoid mixing of substantially different input representations, in a preferred embodiment, one subnetwork is used for each sensor modality. The subnetwork is fed by inputs from different sensors without introducing uncertainty due to heterogeneous sensor inputs. Uncertainty is handled with the help of sensor virtualization (i.e., mapping to virtual sensors).

[0062] Due to sensor virtualization and the known position and orientation of the reference virtual sensors, the environment model is end-to-end trainable. The training step operates on a set of sensor inputs and the network outputs predictions in the model space. Each sensor-specific subnetwork is fed by the corresponding sensor input and outputs its feature map in the model space relative to the reference sensor. A 360-degree environment model can be obtained by appropriately positioning and connecting the feature maps of the subnetworks to form a bird's-eye view raster. For example, the feature map of the rear camera is positioned at the raster in front of the ego car. Similarly, the feature map of the lidar corresponding to the area behind the car can be connected to the feature map of the rear camera.

[0063] Figure 2 is a block diagram illustrating a first mapping from physical sensor to virtual sensor data.

[0064] The system 200 includes the following physical sensors: a plurality of cameras 210a, 210b; a plurality of lidar sensors 220a, 220b; and a plurality of radar sensors 230a, 230b. In the illustrated embodiment, each of these physical sensors is processed in a similar manner to obtain virtual sensor data.

[0065] The images 212a, 212b obtained from the plurality of cameras 210a, 210b are processed by the virtual camera unit 214 to obtain virtual images 216a, 216b. The number of virtual images 216a, 216b obtained may correspond to the number of physical images 210a, 210b, but may be completely different in other embodiments. The mapping performed in the virtual camera unit 214 may involve applying a geometric transformation to the physical images 210a, 210b, such as adjusting the field of view or focal length and / or adjusting geometric distortion. In addition, the physical images 212a, 212b may have different resolutions and / or sizes compared to the virtual images 216a, 216b. The virtual images 216a, 216b are standardized and may be considered as normalized images.

[0066] The plurality of lidar sensors 220a, 220b generate physical lidar data 222a, 222b, which are processed by a virtual lidar module 224 to obtain virtual lidar data 226.

[0067] The plurality of radar sensors 230a, 230b generate physical radar data 232a, 232b. Preferably, the first lidar sensor 220a of the plurality of lidar sensors generates a first point cloud 222a, and the second lidar sensor of the plurality of lidar sensors generates a second point cloud 222b. These are processed by the virtual radar module 234 to obtain virtual radar data 236.

[0068] The LiDAR and RADAR use cases are similar to the camera example. Physical sensors with different parameters (e.g. sensing range, field of view) are mapped onto virtual sensors. Since LiDAR and RADAR operate in the model space, there is preferably only one combined virtual data. This can be seen in the case of RADAR, where the first radar sensor 230a generates 4 sensor signals in the first physical radar data 232a, the second physical radar sensor 230b generates 3 sensor signals in its second physical radar data 232b, and the virtual radar unit combines the two physical signals into one virtual data with 4 signals.

[0069] Preferably, no hand-crafted fusion is required since the mapping to the 3D model space has already been performed.

[0070] Figure 3 A second mapping from virtual data to a feature map in the 3D model space is shown.

[0071] Specifically, Figure 3 The system 300 shown receives as input (e.g., Figure 2The system 200 shown generates virtual camera data, virtual radar data and virtual lidar data, wherein the virtual camera data includes data from a virtual rear camera 310a and a virtual front camera 320b, the virtual radar data includes data from a virtual rear radar 320 and a virtual front radar 320b, and the virtual lidar data includes data from a first virtual lidar 330a and a second virtual lidar 330b.

[0072] The camera data 310a, 310b are processed by a pinhole encoder 312 to obtain camera embeddings 314. These embeddings 314 are then processed by a pinhole decoder 316 to obtain sensor virtualization parameters 318 associated with the camera data. From these, feature maps 340a, 340b are obtained for the rear and front cameras.

[0073] Data from the virtual rear radar 320a and the virtual front radar 320b are processed in the radar neural network 322 to obtain sensor virtualization parameters 234 related to the radar data. From these, feature maps 342a, 342b for the rear radar and the front radar are obtained.

[0074] Similarly, data from virtual lidars 330a, 330b are processed in a lidar neural network 332 to obtain sensor virtualization parameters 334 for the lidar data. From these, feature maps 344a, 344b for the lidar sensors are obtained.

[0075] The neural network 322, 332 can be trained using training data, where virtual sensor data and feature maps are available. Preferably, each feature map corresponds to a virtual sensor. Therefore, they can also be referred to as virtual sensor feature maps.

[0076] The sensor virtualization parameters may include parameters for the placement of the virtual sensors relative to the 3D model space. This is known during training time and may be considered as an input for constructing the bird's eye view feature map 410.

[0077] Since the first transformation is already from a physical sensor to a virtual sensor, information about the physical placement of the physical sensor is no longer needed at this stage. If the physical sensor includes a front camera, the virtual sensor will typically also include a front camera, however, the virtual camera may have a completely different field of view and even a different placement. Therefore, the details of the placement of the physical and virtual front cameras may be different, and it is the placement of the virtual camera that is important when generating the feature map.

[0078] For example, during virtualization of the front camera, it is known that the front camera is looking forward. Therefore, the corresponding virtual feature map can be inserted into the appropriate position in the final bird's eye view raster. If the virtual camera is the front camera, the feature map will start from the middle of the bird's eye view raster (i.e., from index (BEV raster width / 2)), because the ego car is located at the origin and the front camera perceives the environment in front of the ego car. If the virtual camera is the rear camera, the corresponding feature map will be inserted into the BEV raster starting from index 0 (which is behind the ego car).

[0079] The system 300 obtains feature maps, which can then be further processed to obtain estimated predictions.

[0080] Figure 4 A processing characteristic diagram 410 (eg, Figure 3 Flowchart of the feature diagram determined by the system).

[0081] Feature map 410 may be processed to obtain auxiliary camera prediction 412, auxiliary radar prediction 414, and / or auxiliary lidar prediction 416. Furthermore, feature map 410 is processed in fusion neural network 420. The fusion neural network generates final prediction 422 in 3D model space.

[0082] The 3D model space can be a bird's eye view raster, in particular a 306 degree bird's eye view raster. When a 360 degree bird's eye view raster storing a feature map in the model space is obtained, a prediction head can be added. For example, one head is responsible for learning whether an object exists in a raster pixel. At this point, the network can be trained. The head has been implemented as a convolutional layer, where the output channel size corresponds to the dimension of the predicted attribute. For example, the presence of an object within a raster pixel is a convolutional layer with one output channel, where the probability of objectness is predicted for each pixel.

[0083] However, it is beneficial to apply a sensor fusion subnetwork on top of the 3D model space raster feature map to train the network to appropriately combine features from different sensor modalities. In a preferred implementation, a lightweight CNN is used as a sensor fusion subnetwork. If a fusion subnetwork is used, an auxiliary head can be added to each virtual sensor model to facilitate training, and the intermediate predictions can be connected (similar to the method mentioned in the previous section) to the input of the sensor fusion subnetwork. The auxiliary losses calculated for the virtual sensor models are also added to the final loss calculated for the full perception range (i.e., on the bird's-eye view raster). The robustness of the perception system can be further improved by randomly turning off some sensors during training. This method is similar to dropout regularization, except that it masks sensor inputs instead of a group of neurons.

[0084] Preferably, Figure 3 The parameters of the neural networks 312, 322, 332 of the system 300 are trained together with the parameters of the fused neural network 420 that can act as a prediction head. In addition, the first mapping from physical sensor data to virtual sensor data can also include parameters (e.g., parameters of another one or more neural networks) that can also be obtained in the same training process.

[0085] Thus, end-to-end training can be performed from virtual sensor data (or starting from raw physical sensor data) to final predictions.

Claims

1. A method (100) for training a neural network (312, 322, 332, 420) to predict objects around a vehicle, the method (100) comprising: - obtaining (110) physical sensor data from a physical sensor having one or more modalities; - performing (120) a first mapping of the physical sensor data to one or more virtual sensors to obtain virtual sensor data; - performing (130) a second mapping from the virtual sensor data to the 3D model space; as well as - training (140) the neural network (312, 322, 332, 420) based on the virtual sensor data and one or more annotations in the 3D model space.

2. The method (100) of claim 1, further comprising obtaining training data using a fleet of vehicles, wherein: The fleet uses different physical sensors.

3. The method (100) according to any of the preceding claims, wherein: Performing (120) the first mapping includes applying a transformation from the physical sensor data to obtain the virtual sensor data, wherein the transformation is based on a difference between an actual physical characteristic of the physical sensor and a virtual physical characteristic of the virtual sensor.

4. The method (100) according to any of the preceding claims, wherein: The 3D model space includes a bird's eye view raster.

5. The method (100) according to any of the preceding claims, wherein: Performing (130) the second mapping includes: if a failure of a first sensor of the physical sensors is detected, the method (100) includes populating with virtual sensor data obtained from a second sensor of the physical sensors, wherein the first sensor and the second sensor use different modalities.

6. The method (100) according to any of the preceding claims, wherein: The virtual sensors include one virtual sensor for each virtual modality.

7. The method (100) according to any of the preceding claims, further comprising the step of training parameters of the encoder (312) and decoder (316) of the transformer model, wherein The encoder maps from the virtual sensor data to a latent space, and the decoder maps from the latent space to the 3D model space.

8. The method (100) according to any of the preceding claims, wherein: The neural network (312, 322, 332, 420) includes a feature mapping subnetwork that maps from the virtual sensor data to a feature map (340a, 340b, 342a, 342b, 344a, 344b, 410) in the 3D model space, and a processing head that maps from the feature map (340a, 340b, 342a, 342b, 344a, 344b, 410) to an annotation space.

9. The method (100) according to claim 8, wherein: The one or more modalities include at least two modalities, and the feature map (340a, 340b, 342a, 342b, 344a, 344b, 410) includes a feature submap in the 3D model space for each of the at least two modalities, wherein, preferably, each feature submap is fed into the processing head.

10. The method (100) according to any of the preceding claims, wherein: The one or more annotations include the existence of an object and / or a label of the object.

11. The method (100) according to one of the preceding claims, further comprising performing a fusion between the at least two modalities in the 3D model space.

12. The method (100) according to any of the preceding claims, wherein: Training the neural network (312, 322, 332, 420) includes training the neural network (312, 322, 332, 420) a plurality of times, wherein, during at least one training session, data from one or more of the virtual sensors and / or the physical sensors is omitted.

13. The method (100) according to any of the preceding claims, wherein: The first mapping includes one or more first parameters, the second mapping includes one or more second parameters, and the processing head for obtaining one or more annotations includes one or more third parameters, wherein training the neural network (312, 322, 332, 420) includes end-to-end training to obtain the first parameters, the second parameters, and the third parameters.

14. A system (200, 300) for training a neural network (312, 322, 332, 420) to predict objects around a vehicle, wherein: The system (200, 300) is configured to perform the method (100) of one of the preceding claims.

15. A computer-readable storage medium storing a program code, the program code comprising instructions which, when executed by a processor, perform the method (100) of any one of claims 1 to 13.

Images