Computer-implemented method and system for generating synthetic sensor data and training method

Abstract

The invention relates to a computer-implemented method and a system for generating synthetic sensor data (SSD) with superimposed range and intensity information (I1, I2) of a vehicle environment detection sensor, in particular a LiDAR sensor. The invention further relates to a computer-implemented method for providing a trained machine learning algorithm for generating synthetic sensor data (SSD) with superimposed range and intensity information (I1, I2) of a vehicle environment detection sensor, in particular a LiDAR sensor. Furthermore, the invention relates to a computer program and a computer-readable data carrier.

Description

Technical Field

[0001] This invention relates to a computer-implemented method for generating synthetic sensor data with superimposed distance and intensity information for vehicle environmental detection sensors, particularly LiDAR (Light Detection and Ranging) sensors.

[0002] The present invention also relates to a system for generating synthetic sensor data with superimposed distance and intensity information for vehicle environmental detection sensors, particularly LiDAR sensors.

[0003] The present invention also relates to a computer-implemented method for providing a trained machine learning algorithm for generating synthetic sensor data with superimposed distance and intensity information for vehicle environmental detection sensors, particularly LiDAR sensors.

[0004] Furthermore, the present invention also relates to a computer program and a computer-readable data carrier. Background Technology

[0005] Graphical user interfaces used to test highly automated driving functions of motor vehicles typically have multiple components that manage parameter sets, create virtual vehicle environments, and manage experiments.

[0006] The scene structure of the virtual vehicle environment, that is, the definition of the static and dynamic objects of the scene, is done by configuring and importing objects stored in the object library.

[0007] Typically, to generate LiDAR data, complex test drives in real-world environments are required to obtain the necessary data. Therefore, the goal is to synthesize and generate LiDAR sensor data. LiDAR point clouds primarily consist of two features: the intensity of the object and the distance of the object to the LiDAR sensor.

[0008] Distance can be relatively easily modeled geometrically, while intensity is based on the material's reflectivity, which in turn depends on the angle of incidence and the type of reflection.

[0009] Therefore, in order to model the intensity in a virtual environment, the material properties of the object to be modeled must be measured. Material measurement is expensive and can only be performed on a limited number of samples.

[0010] Meanwhile, modeling measurement noise and sensor noise characteristics in a model-based manner is very complex. The realism of synthetic data is limited by various factors, such as real surface structure, noise, multipath propagation, and lack of understanding of material properties.

[0011] Therefore, there is a need to improve existing methods and systems for generating synthetic sensor data from vehicle environment detection sensors, especially LiDAR sensors, so as to make it easier, more efficient and lower cost to generate virtual vehicle environments. Summary of the Invention

[0012] Therefore, the objective of this invention is to provide a computer-implemented method, system, training method, computer program, and computer-readable data carrier that can simplify, more efficiently, and at a lower cost generate synthetic sensor data from vehicle environmental detection sensors, particularly LiDAR sensors.

[0013] According to the invention, the task is solved by a computer-implemented method according to the invention for generating synthetic sensor data with superimposed distance and intensity information for vehicle environmental detection sensors, especially LiDAR sensors.

[0014] According to the invention, the task is solved by a system according to the invention for generating synthetic sensor data with superimposed distance and intensity information for vehicle environmental detection sensors, especially LiDAR sensors.

[0015] Furthermore, the task is solved by a computer-implemented method according to the invention for providing a trained machine learning algorithm for generating synthetic sensor data with superimposed distance and intensity information from vehicle environment detection sensors, particularly LiDAR sensors.

[0016] The task is also solved by a computer program according to the invention and a computer-readable data carrier according to the invention.

[0017] This invention relates to a computer-implemented method for generating synthetic sensor data with superimposed distance and intensity information for vehicle environmental detection sensors, particularly LiDAR sensors.

[0018] The method includes providing a hierarchical variational autoencoder having a first level, a second level, and a third level, or configured for communication of the third level with an external variational autoencoder.

[0019] The method further includes receiving sensor data from vehicle environment detection sensors, especially a first data record with distance information from synthesized and / or actually detected sensor data, via a first-level variational autoencoder. The first-level variational autoencoder assigns global features of the first data record of the sensor data to a first codebook vector.

[0020] The method further includes a first data record that receives sensor data from a vehicle environment detection sensor via a second-level variational autoencoder, wherein the second-level variational autoencoder assigns local features of the first data record of the sensor data to a second codebook vector.

[0021] The method further includes a second data recording adjustment (konditionieren) of sensor data from a vehicle environment detection sensor, which has distance and intensity information, consisting of a first feature vector encoded by a first-level variational autoencoder and a second feature vector encoded by a second-level variational autoencoder.

[0022] The method further includes merging the adjusted first feature vector and the adjusted second feature vector into a synthesized third feature vector, and decoding the synthesized third feature vector to generate a third data record of the synthesized sensor data of the vehicle environment detection sensor with superimposed distance and intensity information.

[0023] Synthetic sensor data from vehicle environment detection sensors is a computer-generated representation of the real vehicle environment detected by the sensors.

[0024] The present invention also relates to a system for generating synthetic sensor data with superimposed distance and intensity information for vehicle environmental detection sensors, particularly LiDAR sensors.

[0025] The system includes a hierarchical variational autoencoder having a first level, a second level, and a third level, or a third level configured to communicate with an external variational autoencoder.

[0026] The hierarchical variational autoencoder is constructed to receive sensor data from vehicle environment detection sensors, especially synthetically generated and / or actually detected sensor data, with distance information as a first data record through a first-level variational autoencoder. The first-level variational autoencoder assigns global features of the first data record of the sensor data to a first codebook vector.

[0027] The hierarchical variational autoencoder is configured to receive sensor data from a vehicle environment detection sensor via a second-level variational autoencoder. The second-level variational autoencoder assigns local features of the first data record of the sensor data to a second codebook vector.

[0028] The hierarchical variational autoencoder also constructs a second data record with distance and intensity information for sensor data from vehicle environment detection sensors, adjusting the first feature vector encoded by the first-level variational autoencoder and the second feature vector encoded by the second-level variational autoencoder.

[0029] The hierarchical variational autoencoder is also constructed to combine the regulated first feature vector and the regulated second feature vector into a synthesized third feature vector.

[0030] The hierarchical variational autoencoder also constructs a third data record for decoding the synthesized third feature vector to generate a superimposed distance and intensity information of the synthesized sensor data of the vehicle environment detection sensor.

[0031] Furthermore, the present invention relates to a computer-implemented method for providing a trained machine learning algorithm for generating synthetic sensor data of vehicle environment detection sensors, particularly LiDAR sensors, with superimposed distance and intensity information.

[0032] The method includes providing a hierarchical variational autoencoder having at least a first level and a second level.

[0033] The method further includes receiving input training data and output training data of a first autoregressive artificial neural network, particularly an artificial neural convolutional network, at the first level.

[0034] The method further includes training a first autoregressive artificial neural network, particularly an artificial neural convolutional network, at the first level in order to assign global features of the input training data to the first codebook vector.

[0035] The method further includes receiving input training data and output training data of a second-level autoregressive artificial neural network, particularly an artificial neural convolutional network.

[0036] Furthermore, the method includes training a second-level autoregressive artificial neural network, particularly an artificial neural convolutional network, to assign local features of the input training data to a second codebook vector, wherein the second-level autoregressive artificial neural network is modulated by the first-level autoregressive artificial neural network.

[0037] The idea of ​​this invention is to transform a predetermined data record of synthetic sensor data, especially LiDAR data containing distance information, using the above-described algorithm structure with a hierarchical variational autoencoder and an additional adjustment layer, so that a transformed or improved data record can be generated by adjusting the predetermined data record with another data record consisting of distance and intensity data from vehicle sensors. This data record has an approximation of synthetic sensor data with superimposed distance and intensity information from vehicle environment detection sensors.

[0038] Other embodiments of the present invention are described below with reference to the accompanying drawings.

[0039] According to a preferred extension of the present invention, the first data record of the sensor data is encoded by a first encoder of a hierarchical variational autoencoder, wherein the image resolution of the first data record is reduced by a predetermined factor, particularly 2. 4 Therefore, it is advantageous to generate a representation of the first data record with a dimension reduced by a predetermined factor.

[0040] According to another preferred extension scheme, the first data record of sensor data encoded by the first encoder is divided into a first level and a second level. The first data record of sensor data is encoded in the first level E1 by the second encoder of the hierarchical variational autoencoder. The image resolution of the first data record is reduced by a predetermined factor, specifically 2... 2 Therefore, the dimension of the first data record in the first level is further reduced by a predetermined multiple in a favorable manner.

[0041] According to another preferred extension scheme, the first data record of sensor data encoded into a first feature vector by the second encoder is assigned by the first autoregressive artificial neural network of the first level, especially the artificial neural convolutional network, to the first codebook vector that has the minimum distance from the first feature vector.

[0042] Therefore, the generated feature vector can be optimally assigned to the first codebook vector in a favorable manner, which integrates the digitally parameterizable properties of the first data record in a vector manner.

[0043] According to another preferred extension scheme, the first codebook vector is decoded by the first decoder of the hierarchical variational autoencoder, and the image resolution of the first codebook vector is increased by a predetermined multiple, especially 2. 2 This allows for an efficient, dimensionally increased representation of the first codebook vector in a favorable manner.

[0044] According to another preferred extension scheme, the first data record output by the first decoder of the first level and the first data record encoded by the first encoder of the hierarchical variational autoencoder are merged at the second level to form a synthesized third feature vector.

[0045] The first data record of the first level, which has the global characteristics of the original first data record, is therefore adjusted to the first data record of the second level.

[0046] According to another preferred extension scheme, the synthesized third feature vector is assigned to the second codebook vector that has the minimum distance from the synthesized third feature vector through the second autoregressive artificial neural network of the second level, especially the artificial neural convolutional network.

[0047] Therefore, the third feature vector can be assigned to the second codebook vector in a favorable manner as optimally as possible.

[0048] According to another preferred extension scheme, the second data record of the vehicle environment detection sensor is encoded by a third encoder at a third level, and the image resolution of the second data record is reduced by a predetermined factor, especially 2. 8 Therefore, it is advantageous to generate a representation of the second dataset with a dimensionality reduced by a predetermined factor.

[0049] According to another preferred extension scheme, the second data record of sensor data, which is encoded into a fourth feature vector by the third encoder, is assigned by the third autoregressive artificial neural network of the third level, especially the artificial convolutional neural network, to the third codebook vector that has the minimum distance to the fourth feature vector of the second data record.

[0050] Therefore, the fourth feature vector can be optimally assigned to the third codebook vector in a favorable manner.

[0051] According to another preferred extension scheme, the third codebook vector is decoded by the second decoder of the hierarchical variational autoencoder or an external variational autoencoder, and the second data record output by the second decoder adjusts the first feature vector encoded by the first-level variational autoencoder and the second feature vector encoded by the second-level variational autoencoder.

[0052] By adjusting the first and second levels using the output data from the third level, the first and second levels can be adjusted in a favorable manner.

[0053] Therefore, data records that can be transformed into first-level and second-level data records, or data records that can be transformed into a combination of first-level and second-level data records.

[0054] This has the advantage of being able to model or generate synthetic sensor data that incorporates distance information and transformed and thus improved intensity information.

[0055] The transformation of the first data record and the resulting improvements can be achieved by adjusting the sensor data actually detected by the second data record, especially the second data record.

[0056] According to another preferred extension scheme, a data recording identifier is used to adjust the first feature vector encoded by the first-level variational autoencoder and the second feature vector encoded by the second-level variational autoencoder. The data recording identifier indicates whether the sensor data is synthetically generated or actually detected sensor data.

[0057] By applying additional conditioning to the data at the first and second levels through data record identifiers, the synthetic sensor data to be output by the objective function or environmental detection sensor can be modeled in an advantageous manner through the autoregressive artificial neural network of the first and / or second levels.

[0058] The methods described in this paper are applicable to testing in a variety of virtual environments, such as autonomous vehicles, aircraft, and / or spacecraft. Attached Figure Description

[0059] To better understand the present invention and its advantages, reference is now made to the following description in conjunction with the accompanying drawings.

[0060] The invention will now be described in more detail with reference to exemplary embodiments schematically illustrated in the accompanying drawings. The drawings are as follows:

[0061] Figure 1 A flowchart illustrating a computer-implemented method for generating synthetic sensor data from a vehicle environmental detection sensor according to a preferred embodiment of the present invention is shown.

[0062] Figure 2 A detailed flow chart and system diagram of a method and system for generating synthetic sensor data from a vehicle environmental detection sensor according to a preferred embodiment of the present invention are shown.

[0063] Figure 3 An example matrix for modeling an objective function according to a preferred embodiment of the present invention is shown;

[0064] Figure 4 The representation of the sensory region in a conventional model with blind spots is not part of this invention.

[0065] Figure 5 A representation of the receptive region of a model or artificial neural network according to a preferred embodiment of the present invention is shown;

[0066] Figure 6 A flowchart illustrating the first layer of a hierarchical variational autoencoder according to a preferred embodiment of the present invention is shown.

[0067] Figure 7 A flowchart illustrating a method for training a second layer of a hierarchical variational autoencoder according to a preferred embodiment of the present invention is shown; and

[0068] Figure 8 A flowchart illustrating a method for providing a trained machine learning algorithm for generating synthetic sensor data for environmental detection sensors, according to a preferred embodiment of the present invention.

[0069] Unless otherwise stated, the same reference numerals in the accompanying drawings denote the same elements. Detailed Implementation

[0070] Figure 1 A flowchart illustrating a method for generating synthetic sensor data for a vehicle environmental detection sensor according to a preferred embodiment of the present invention is shown. Figure 2 A detailed flow chart and system diagram of a method and system for generating synthetic sensor data for a vehicle environmental detection sensor according to a preferred embodiment of the present invention are shown.

[0071] In a preferred embodiment of the present invention, the synthesized sensor data SSD of the environmental detection sensor is the sensor data of the vehicle's LiDAR sensor.

[0072] As an alternative, environmental detection sensors can be, for example, camera sensors or radar sensors.

[0073] When using a camera sensor, distance information contained in video image data can be calculated geometrically, for example. Intensity information can be encoded using grayscale values ​​or RGB color channels.

[0074] When using radar sensors, intensity information can be encoded, for example, through image brightness. Brighter pixel values ​​represent higher object reflectivity and thus higher image intensity, while darker pixel values ​​represent lower object reflectivity and thus lower image intensity.

[0075] The following explanation involves Figure 1 and Figure 2 The method and system include providing an S1 hierarchical variational autoencoder (HVAE) having a first level E1, a second level E2, and a third level E3.

[0076] As an alternative, the third level E3 may not be part of the hierarchical variational autoencoder HVAE, but rather form an outer level E3. In this case, the hierarchical variational autoencoder HAVE is configured to communicate with the outer third level E3 of the outer variational autoencoder.

[0077] Variational autoencoders employ artificial neural networks to learn effective data encodings in an unsupervised manner. The purpose of a variational autoencoder is to learn representations or encodings of data records, typically to reduce dimensionality.

[0078] Compared to traditional autoencoders, variational autoencoders are generative models. Their relationship with traditional autoencoders mainly stems from their architectural relationship, namely the encoder and decoder.

[0079] However, their mathematical formulations differ significantly. A variational autoencoder is a weighted probabilistic graphical model whose objective function is approximated by a neural network. The encoder generates feature vectors that synthesize the digitally parameterizable properties of a muster pattern in a vector manner.

[0080] Different features representing a pattern form different dimensions of the vector. The sum of all possible feature vectors is called the feature space. Feature vectors simplify automatic classification because they greatly reduce the number of attributes to be classified. Instead of the complete image, only a vector consisting of a given number of digits needs to be considered. The feature vectors generated by the encoder are then assigned to a pre-created codebook vector via an artificial neural network.

[0081] The method also includes receiving sensor data from the vehicle environment detection sensor S2, especially the synthesized and / or actually detected sensor data SSD and RSD, via the variational autoencoder VAE1 of the first level E1, and a first data record DS1 with distance information I1.

[0082] As an alternative, one could use only the synthesized or actual sensor data SSD or RSD.

[0083] The first-level variational autoencoder E1 (VAE1) assigns the global feature GM of the first data record DS1 of the sensor data to the first codebook vector CBV1.

[0084] The method also includes a first data record DS1 that receives sensor data from the vehicle environment detection sensor S3 via a variational autoencoder VAE2 of the second-level E2. The variational autoencoder VAE2 of the second-level E2 assigns the local features LM of the first data record DS1 of the sensor data to a second codebook vector CBV2.

[0085] The global feature GM of the first data record is understood as a coarse feature. Since this embodiment involves synthetic sensor data of a vehicle environment detection sensor, the global feature or coarse feature of the first data record DS1 is understood as an object identified in the graphic data or LiDAR point cloud, or an object identified as a global feature or coarse feature.

[0086] This may include, for example, buildings, stationary or moving vehicles, vegetation, traffic signs, people or similar objects.

[0087] Local features (LM) should be understood as the fine-grained features of objects contained in synthetic sensor data from vehicle environment detection sensors, especially LiDAR sensors. Fine-grained features, for example, can distinguish the type of an object. This means, for example, distinguishing pedestrian types such as age and / or gender, identifying vehicle types such as cars, two-wheelers, or commercial vehicles, or vegetation types.

[0088] The method also includes a second data record DS2 with distance and intensity information I1 and I2 from the sensor data of the vehicle environment detection sensor, and a second feature vector MV1 encoded by the variational autoencoder VAE1 of the first level E1 and the second feature vector MV2 encoded by the variational autoencoder VAE2 of the second level E2.

[0089] Subsequently, the adjusted first feature vector MV1 and the adjusted second feature vector MV2 are merged into a synthesized third feature vector MV3 by S5, and the synthesized third feature vector MV3 is decoded by S6 to generate a third data record DS3 of the synthesized sensor data SSD of the vehicle environment detection sensor with superimposed distance and intensity information I1 and I2.

[0090] refer to Figure 2 The timing of the method for generating synthetic sensor data SSD with superimposed distance and intensity information I1 and I2 from vehicle environment detection sensors is now described. The first data record DS1 of the sensor data is first encoded by the first encoder ENC1 of the hierarchical variational autoencoder HVAE. The image resolution of the first data record DS1 is reduced by a predetermined factor, specifically 2. 4 times.

[0091] The first data record DS1, encoded by the first encoder ENC1, is then divided into the first level E1 and the second level E2.

[0092] The first data record DS1 of the sensor data is encoded in the first level E1 by the second encoder ENC2 of the hierarchical variational autoencoder HVAE. The image resolution of the first data record DS1 is further reduced by a predetermined factor, especially 2. 2 times.

[0093] The sensor data is encoded by the second encoder ENC2 into a first data record DS1 of the first feature vector MV1, and then assigned by the first autoregressive artificial neural network KNN1 of the first layer E1, especially the artificial neural convolutional network, to the first codebook vector CBV1 that has the minimum distance to the first feature vector MV1.

[0094] The first codebook vector CBV1 is then decoded by the first decoder DEC1 of the hierarchical variational autoencoder HVAE. The image resolution of the first codebook vector CBV1 is increased by a predetermined factor, specifically 2. 2 times.

[0095] The first data record DS1 output by the first decoder DEC1 of the first level E1 is combined with the first data record DS1 encoded by the first encoder ENC1 of the hierarchical variational autoencoder HVAE on the second level E2 to form a synthesized third feature vector MV3.

[0096] The synthesized third feature vector MV3 is assigned by the second autoregressive artificial neural network KNN2 of the second layer E2, especially the artificial neural convolutional network, to the second codebook vector CBV2 that has the smallest distance to the synthesized third feature vector MV3.

[0097] Here, the second codebook vector CBV2, which has the smallest distance to the synthesized third feature vector MV3, has the highest similarity compared to other codebook vectors in the codebook or table. Commonly used distance metrics include Euclidean distance, weighted Euclidean distance, and / or Mahalanobis distance.

[0098] Furthermore, the second data record DS2, which contains sensor data from the vehicle environment detection sensor, is encoded by the third encoder ENC3 at the third level E3. Here, the image resolution of the second data record DS2 is reduced by a predetermined factor, specifically 2... 8 times.

[0099] The second data record DS2, which is encoded into a fourth feature vector MV4 by the third encoder ENC3, is assigned by the third autoregressive artificial neural network KNN3 of the third level E3, and in particular by an artificial convolutional neural network, to the third codebook vector CBV3 that has the minimum distance to the fourth feature vector MV4 of the second data record DS2.

[0100] The third codebook vector CBV3 is then decoded by the second decoder DEC2 of the hierarchical variational autoencoder HVAE or an external variational autoencoder. The second data record DS2 output by the second decoder DEC2 modulates the first feature vector MV1 encoded by the variational autoencoder VAE1 of the first level E1 and the second feature vector MV2 encoded by the variational autoencoder VAE2 of the second level E2.

[0101] Furthermore, the data recording identifier K is used to adjust the first feature vector MV1 encoded by the variational autoencoder VAE1 of the first level E1 and the second feature vector MV2 encoded by the variational autoencoder VAE2 of the second level E2. The data recording identifier K indicates whether the sensor data is synthetically generated or actually detected sensor data SSD and RSD.

[0102] The second feature vector MV2, encoded by the second variational autoencoder VAE2 of the second level E2, is then merged again with the third data record DS1 of the first level E1.

[0103] The resulting vector is decoded by the third decoder DEC3. This generates synthetic sensor data SSD, which combines distance and intensity information I1 and I2 from vehicle environmental detection sensors, particularly LiDAR sensors.

[0104] The generation of synthetic sensor data for the vehicle environment detection sensor therefore includes transforming the input data received by the hierarchical variational autoencoder, namely the first data record DS1, by adjusting the second data record DS2 to generate a third data record DS3. The third data record has improved intensity information I2 in addition to the distance information I1 contained in the first data record DS1.

[0105] Figure 3 Example matrices are shown for the autoregressive artificial neural networks KNN1, KNN2, and especially the artificial neural convolutional networks, representing the first level E1 and the second level E2. The artificial neural convolutional networks use... Figure 3 The masked convolution is shown. Here, an order is applied to each pixel Z, starting numerically from left to right, beginning at the top and ending at the bottom. The probability of the next pixel depends on one or more previously generated pixels. The model cannot read pixels below or to the right of the current pixel for prediction. Traditional artificial neural convolutional networks used for pixel generation often have blind spots (BF) in the receptive field that cannot be used for prediction, such as... Figure 4 As shown.

[0106] According to the present invention (see Figure 5 Using two convolution stacks (Faltungsstapel), namely a horizontal stack and a vertical stack, allows for the detection of the entire receptive region or receptive field.

[0107] Therefore, blind spots in the receptive region can be eliminated by combining two convolutional stacks. Horizontal stacking adjusts the current row up to the current pixel. Vertical stacking adjusts all rows above it. Vertical stacking without masking allows the receptive region to grow into a rectangle without blind spots, and the outputs of the two stacks are combined after each layer.

[0108] Each time a pixel is predicted, it is fed back into the convolutional neural network to predict the next pixel. This sequential nature is essential for generating high-quality images because it allows each pixel to be correlated with previous pixels in a highly non-linear and multimodal manner.

[0109] Each layer in a horizontal stack takes the output of the previous layer and the output of the previous stack as input.

[0110] Figure 6 A flowchart illustrating the first layer of a hierarchical variational autoencoder for training according to a preferred embodiment of the present invention is shown. Figure 7 A flowchart illustrating a method for training a second layer of a hierarchical variational autoencoder according to a preferred embodiment of the present invention is shown. Figure 8A flowchart illustrating a method for providing a trained machine learning algorithm for generating synthetic sensor data for environmental detection sensors, according to a preferred embodiment of the present invention.

[0111] The following describes the method for training the first layer E1 and the second layer E2 of the hierarchical variational autoencoder HVAE.

[0112] The method includes providing a hierarchical variational autoencoder (HVAE) S11, which has a first level E1 and a second level E2. In this embodiment, a third level E3 is also part of the hierarchical variational autoencoder (HVAE)3.

[0113] As an alternative, the third level E3 may not be part of the hierarchical variational autoencoder HVAE.

[0114] The method also includes receiving input training data TD1 and output training data TD2 from the first autoregressive artificial neural network KNN1, especially the artificial neural convolutional network, of the first layer E1 in S12 and S13.

[0115] The method also includes training a first autoregressive artificial neural network KNN1, particularly an artificial neural convolutional network, in the first layer E1 of S14, so as to assign the global features GM of the input training data TD1 to the first codebook vector CBV1.

[0116] Codebook vectors are generated using vector quantization. Vector quantization involves two steps. In the first step, training, a table or codebook is created using frequently occurring feature vectors. In the second step, for each of the other vectors, the codebook vector with the minimum distance is determined.

[0117] For data transmission, only the index of the codebook vector is needed; if the codebook is multidimensional, the index can also be a vector. The corresponding decoder must have the same codebook and can then generate an approximation of the original vector from the index.

[0118] Furthermore, the method includes receiving input training data TD3 and output training data TD4 of the second autoregressive artificial neural network KNN2, particularly the artificial neural convolutional network, of the second layer E2 in S15 and S16.

[0119] The method also includes training a second autoregressive artificial neural network KNN2, specifically an artificial neural convolutional network, in the second layer E2 of S17, to assign local features LM of the input training data TD3 to the second codebook vector CBV2. Here, the second autoregressive artificial neural network KNN2 of the second layer E2 is adjusted by the autoregressive artificial neural network KNN1 of the first layer E1.

[0120] While specific implementations have been described and illustrated herein, those skilled in the art will understand that various alternative and / or equivalent implementations exist. It should be noted that one or more exemplary implementations are merely examples and are not intended to limit the scope, applicability, or configuration in any way.

[0121] Conversely, the above overview and detailed description provide convenient guidance for those skilled in the art to implement at least one exemplary embodiment. It should be understood that various changes to the scope of function and arrangement of elements may be made without departing from the scope of the appended claims and their legal equivalents.

[0122] Generally speaking, this application is intended to cover changes, adjustments or variations of the embodiments shown herein.

Claims

1. A computer-implemented method for generating a synthetic sensor data (SSD) with superimposed distance and intensity information of an environment detection sensor of a vehicle, the method comprising the following steps: - providing (SI) a hierarchical variational autoencoder (HVAE) having a first level (El), a second level (E2) and a third level (E3), or being configured for communication with the third level of an external variational autoencoder; - receiving (S2) a first data record (DSl) with distance information (II) of sensor data of an environment detection sensor of a vehicle by a variational autoencoder (VAEl) of the first level (El), the variational autoencoder of the first level assigning global features (GM) of the first data record (DSl) of sensor data to a first codebook vector (CBVl); - receiving (S3) the first data record (DSl) of sensor data of an environment detection sensor of a vehicle by a variational autoencoder (VAE2) of the second level (E2), the variational autoencoder of the second level assigning local features (LM) of the first data record (DSl) of sensor data to a second codebook vector (CBV2); - conditioning (S4) a first feature vector (MVl) encoded by the variational autoencoder (VAEl) of the first level (El) and a second feature vector (MV2) encoded by the variational autoencoder (VAE2) of the second level (E2) with a second data record (DS2) of sensor data of an environment detection sensor of a vehicle with distance and intensity information; - merging (S5) the conditioned first feature vector (MVl) and the conditioned second feature vector (MV2) into a synthetic third feature vector (MV3); and - decoding (S6) the synthetic third feature vector (MV3) to generate a third data record (DS3) of synthetic sensor data (SSD) with superimposed distance and intensity information of an environment detection sensor of a vehicle.

2. The computer-implemented method according to claim 1, characterized in that the environment detection sensor is a LiDAR sensor; and / or the sensor data is synthetically generated and / or actually detected sensor data.

3. The computer-implemented method of claim 1, wherein, the first data record (DSl) of sensor data is encoded by a first encoder (ENCl) of the hierarchical variational autoencoder (HAVE), the image resolution of the first data record (DSl) being reduced by a predetermined factor.

4. The computer-implemented method of claim 3, wherein, The image resolution of the first data record (DS1) is reduced by a factor of 2 4 2.

5. The computer-implemented method of claim 3, wherein, the first data record (DSl) of sensor data encoded by the first encoder (ENCl) is divided into the first level (El) and the second level (E2), the first data record (DSl) of sensor data being encoded in the first level (El) by a second encoder (ENC2) of the hierarchical variational autoencoder (HAVE), the image resolution of the first data record (DSl) being reduced by a predetermined factor.

6. The computer-implemented method of claim 5, wherein, The image resolution of the first data record (DS1) is reduced by a factor of 2 2 .

7. The computer-implemented method of claim 3, wherein, A first data record (DS1) of sensor data encoded by a second encoder (ENC2) into a first feature vector (MV1) is assigned by a first auto-regressive artificial neural network of a first hierarchy (E1) to a first codebook vector (CBV1) having a smallest distance to the first feature vector (MV1).

8. The computer-implemented method of claim 7, wherein, The first auto-regressive artificial neural network is an artificial neural convolutional network.

9. The computer-implemented method of claim 7, wherein, The first codebook vector (CBV1) is decoded by a first decoder (DEC1) of the hierarchical variational autoencoder (HAVE), the image resolution of the first codebook vector (CBV1) being increased by a predetermined factor.

10. The computer-implemented method of claim 9, wherein, The image resolution of the first codebook vector (CBV1) is increased by a factor of 2 2 2.

11. The computer-implemented method of claim 9, wherein, The first data record (DS1) output by the first decoder (DEC1) of the first hierarchy (E1) is merged with the first data record (DS1) encoded by the first encoder (ENC1) of the hierarchical variational autoencoder (HAVE) on a second hierarchy (E2) into a synthetic third feature vector (MV3).

12. The computer-implemented method of claim 11, wherein, The synthetic third feature vector (MV3) is assigned by a second auto-regressive artificial neural network of the second hierarchy (E2) to a second codebook vector (CBV2) having a smallest distance to the synthetic third feature vector (MV3).

13. The computer-implemented method of claim 12, wherein, The second auto-regressive artificial neural network is an artificial neural convolutional network.

14. The computer-implemented method according to any one of claims 1 to 13, wherein, A second data record (DS2) of sensor data of an environment detection sensor of a vehicle is encoded on a third hierarchy (E3) by a third encoder (ENC3), the image resolution of the second data record (DS2) being reduced by a predetermined factor.

15. The computer-implemented method of claim 14, wherein, The image resolution of the second data record (DS2) is reduced by a factor of 2 8 2.

16. The computer-implemented method of claim 14, wherein, The second data record (DS2) encoded by the third encoder (ENC3) into a fourth feature vector (MV4) of sensor data is assigned by a third auto-regressive artificial neural network of the third hierarchy (E3) to a third codebook vector (CBV3) having a smallest distance to the fourth feature vector (MV4) of the second data record (DS2).

17. The computer-implemented method of claim 16, wherein, The third auto-regressive artificial neural network is an artificial convolutional neural network.

18. The computer-implemented method of claim 16, wherein, The third codebook vector (CBV3) is decoded by a second decoder (DEC2) of the hierarchical variational autoencoder (HAVE) or an external variational autoencoder, the second data record (DS2) output by the second decoder (DEC2) conditioning the first feature vector (MV1) encoded by the variational autoencoder (VAE1) of the first hierarchy (E1) and the second feature vector (MV2) encoded by the variational autoencoder (VAE2) of the second hierarchy (E2).

19. The computer-implemented method according to any one of claims 1 to 13, wherein, The first feature vector (MV1) encoded by the variational autoencoder (VAE1) of the first hierarchy (E1) and the second feature vector (MV2) encoded by the variational autoencoder (VAE2) of the second hierarchy (E2) are conditioned with a data record identifier (K) indicating whether the sensor data is synthetically generated sensor data or actually detected sensor data.

20. A computer-implemented method for providing a trained machine learning algorithm for generating synthetic sensor data (SSD) of an environment detection sensor of a vehicle having superimposed distance and intensity information, the method comprising the following steps: - providing (S11) a hierarchical variational autoencoder (HAVE) having at least a first level (E1) and a second level (E2); - receiving (S12, S13) input training data (TD1) and output training data (TD2) of a first auto-regressive artificial neural network of the first level (E1); - training (S14) the first auto-regressive artificial neural network of the first level (E1) in order to assign global features (GM) of the input training data (TD1) of the first auto-regressive artificial neural network to a first codebook vector (CBV1); - receiving (S15, S16) input training data (TD3) and output training data (TD4) of a second auto-regressive artificial neural network of the second level (E2); - training (S17) a second auto-regressive artificial neural network of the second level (E2) in order to assign local features (LM) of the input training data (TD3) of the second auto-regressive artificial neural network to the second codebook vector (CBV2), wherein, - conditioning the second auto-regressive artificial neural network of the second level (E2) by the first auto-regressive artificial neural network of the first level (E1).

21. The computer-implemented method according to claim 20, characterized in that the environmental detection sensor is a LiDAR sensor; and / or the first auto-regressive artificial neural network is an artificial neural convolutional network; and / or the second auto-regressive artificial neural network is an artificial neural convolutional network.

22. A system (1) for generating a synthetic sensor data (SSD) of an environmental detection sensor of a vehicle having superimposed distance and intensity information, having: - a hierarchical variational autoencoder (HAVE) having a first level (El), a second level (E2) and a third level (E3), or being configured for communication with a third level (E3) of an external variational autoencoder, the hierarchical variational autoencoder (HAVE) being configured for receiving, by a variational autoencoder (VAE1) of the first level (El), a first data record (DS1) of sensor data of an environment detection sensor of a vehicle having distance information (II), the variational autoencoder of the first level assigning global features (GM) of the first data record (DS1) of sensor data to a first codebook vector (CBV1), the hierarchical variational autoencoder (HVAE) being configured for receiving, by a variational autoencoder (VAE2) of the second level (E2), the first data record (DS1) of sensor data of the environment detection sensor of the vehicle, the variational autoencoder of the second level assigning local features (LM) of the first data record (DS1) of sensor data to a second codebook vector (CBV2), wherein the hierarchical variational autoencoder (HAVE) is configured to condition a first feature vector (MV1) encoded by a variational autoencoder (VAE1) of the first level (E1) and a second feature vector (MV2) encoded by a variational autoencoder (VAE2) of the second level (E2) with a second data record (DS2) of the sensor data of the environmental detection sensor of the vehicle having distance and intensity information, the hierarchical variational autoencoder (HAVE) is configured to merge the conditioned first feature vector (MV1) and the conditioned second feature vector (MV2) into a synthetic third feature vector (MV3), and the hierarchical variational autoencoder (HAVE) is configured to decode the synthetic third feature vector (MV3) to generate a third data record (DS3) of the synthetic sensor data (SSD) of the environmental detection sensor of the vehicle having superimposed distance and intensity information.

23. The system according to claim 22, characterized in that the environmental detection sensor is a LiDAR sensor; and / or the sensor data is synthetically generated and / or actually detected sensor data.

24. Computer program product with program code for performing a method according to any one of claims 1 to 21 when the computer program runs on a computer.

25. Computer-readable data carrier with program code of a computer program for performing a method according to any one of claims 1 to 21 when the computer program runs on a computer.

Images