Method for generating test data for simulating an assistance system of an at-partially assistant operating motor vehicle, computer program product, computer-readable

Abstract

The invention relates to a method for generating test data (30) for simulating an assistance system (12) of an at least partially assisted operation motor vehicle (10) by means of an electronic computing device (14), comprising the following steps: specifying actually acquired trajectory data (36) of at least one driving situation for the at least partially assisted driving operation of the motor vehicle (10); (S1) extracting a training trajectory from the trajectory data (36) by means of the electronic computing device (14); (S2) training a variable autoencoder (22) of the electronic computing device (14) using the extracted training trajectory; (S3, S4) generating potential test data (30) for the simulation by means of the trained variable autoencoder (22); (S5) comparing the actually acquired trajectory data (36) with the potential test data (30) by means of the electronic computing device (14); and identifying test data (30) for simulation on the basis of the comparison (S7). Furthermore, the invention relates to a computer program product, a computer readable storage medium and an electronic computing device (14).

Description

[0001] The invention relates to a method for generating test data for simulating an at least partially assisted operation of an assistance system of a motor vehicle using an electronic computing device. The invention also relates to a computer program product, a computer-readable storage medium, and an electronic computing device.

[0002] Automated or assisted driving of motor vehicles is known from the prior art. The motivation for the present invention is to increase safety, use resources more efficiently and improve comfort. The safety demonstration of advanced driver assistance systems (for highly automated driving) poses new challenges for verification and testing, because the previously common methods (which mainly focus on actual test drive data) require unreasonably long driving distances. For example, if the behavior of an emergency braking system is to be tested when another vehicle unexpectedly changes lanes on a highway, the developers of such a system must spend a lot of time in order to record enough such situations on the road. The recorded situations are then used to test the driver assistance system. In addition, new challenges arise, such as takeover scenarios between the system and the human driver.

[0003] In light of the above, simulation has proven to be a promising method for validating driver assistance systems. In other words, simulation makes it possible to generate a series of surrounding vehicle trajectories based on a limited number of real-world trajectories recorded during a test drive. Specifically, safety-relevant real-world driving situations are selected from all recorded test drive data and evaluated for their criticality. These situations are then simulated. Simulation is understood here as generating a set of trajectories for one or more neighboring vehicles that approximate the actual trajectories recorded during the test drive but represent a larger set of possible trajectories. The validity of the results depends crucially on the extent to which the simulated test cases faithfully reflect real-world traffic conditions.

[0004] Among them, US2022 / 100635 A1 describes a method for verifying autonomous control software for autonomous operation of a motor vehicle. For example, the autonomous control software is placed in a driving scenario for operation to observe the results of the autonomous control software. The verification model is placed in the driving scenario for operation multiple times to observe the results of the model each time. Whether the software understands the driving scenario is determined by the following fact, namely, whether the software results indicate that the virtual vehicle controlled by the software collides with other objects within a single time. Whether the verification model understands the driving scenario is determined based on the following, namely, whether the model results indicate that the virtual vehicle under the control of the model collides with other objects within one of multiple time points. The software is verified based on these judgments.

[0005] The object of the present invention is to create a method, a computer program product, a computer-readable storage medium and an electronic computing device, by means of which better generation of test data for an electronic computing device is possible.

[0006] This object is achieved by a method, a computer program product, a computer-readable storage medium and an electronic computing device according to the independent patent claims. Advantageous embodiments are specified in the dependent claims.

[0007] One aspect of the present invention relates to a method for generating test data for an assistance system of a motor vehicle for simulating at least partially assisted operation using an electronic computing device. Realistically acquired trajectory data of at least one driving scenario is predefined for the at least partially assisted driving operation of the motor vehicle. Training trajectories are extracted from the actually driven trajectory data using the electronic computing device. A variable autoencoder of the electronic computing device is trained using the extracted training trajectories. Potential test data for simulation is generated using the trained variable autoencoder. The actually acquired trajectory data is compared with the potential test data using the electronic computing device, and test data for simulation is identified based on the comparison.

[0008] In particular, a novel method based on artificial intelligence is proposed for generating realistic driving scenarios for the simulation verification of assistance systems. This improves the quality of the simulation verification by realistic test cases, while at the same time reducing the high complexity and workload required for the mathematical modeling of the driving scenarios. In particular, it is provided that, instead of mathematical models and physical parameters, an AI model (in particular a variable autoencoder) trained on real driving data is used to generate driving scenarios for the simulation verification. The present invention is not limited to the development of corresponding artificial intelligence models, but also covers the entire workflow from preprocessing the raw measurement data to integrating the newly generated data into the entire simulation environment with the help of the artificial intelligence model.

[0009] In particular, it can be provided that potential test data and / or test data are generated such that control of functional devices of the assistance system is possible using these potential test data and / or these test data. For example, an accelerator, a brake, a lateral acceleration, or a steering device can be considered as a functional device. Furthermore, it can be provided that control of the functional devices of the assistance system is performed based on the generated test data.

[0010] Therefore, if the generated data (test data) are specifically adapted to their intended technical use, they can be considered, in particular, as "functional data" for controlling technical devices (in particular, assistance systems). Thus, based on the actual generated test data (in particular, the corresponding parameters of the generated trajectories), it is possible to utilize the control of the assistance system, in particular, the driving of the motor vehicle, in a corresponding manner. For example, a driver assistance system can be designed to regulate speed, acceleration, a braking system, or a steering wheel.

[0011] It is further advantageous to extract test data with the aid of a classifier of an electronic computing device. In order to obtain sufficient data, in particular, for training the autoencoder, measurement sequences from test drives are used, from which continuous sequences are extracted, for example using a sliding window method, which can then be preprocessed. A so-called classifier is then used, in particular, to identify safety-relevant situations and extract these situations to form a training set. Labeled / tagged data is used for training the classifier, which represents safety-relevant situations. This training can be performed using actual historical data (i.e., data from real driving trajectories) or simulated data. Finally, a classifier is trained using the lateral position trajectories to identify safety-relevant situations and extract trajectory-relevant objects, in particular those that trigger safety-relevant scenarios, and their trajectories.

[0012] It is further advantageous to filter the trajectory data so that non-physical behaviors of at least one traffic participant are removed from the actually collected trajectory data. In this case, for example, objects that are present for a short time in the trajectory data can be identified as ghost objects and filtered out. In addition, it can be provided that the actually collected trajectory data are compared qualitatively and quantitatively with potential test data. In particular, the quality of the generated data depends crucially on the quality of the data used to train the autoencoder. Therefore, comprehensive preprocessing of the data is the basis for good performance when generating new data for simulation. Preprocessing refers to all transformations of the raw data. The starting point is, for example, the selection of suitable measurement signals to characterize the movement of the objects included in the measurement. An L-shape based on the analysis of the sensor data is used to identify the movement of traffic objects relative to the ego vehicle, in particular a motor vehicle equipped with corresponding sensors.

[0013] A cascade of different filters (e.g., min-max and Savitzky-Golay filters) is then used to smooth the measurement data and remove unphysical behavior (e.g., caused by signal jumps). Furthermore, measurement components with a low density of available measurements and therefore large uncertainties are ignored, and short-lived objects are identified as ghosts and filtered out.

[0014] Therefore, another aspect of the invention relates to a computer program product having program code means which, when processed by an electronic computing device, cause the electronic computing device to execute the method according to the above aspect.

[0015] Another aspect of the present invention relates to a computer-readable storage medium having a computer program product.

[0016] The present invention further relates to an electronic computing device for generating test data for simulating an assistance system of a motor vehicle in at least partially assisted operation, the electronic computing device comprising at least one variable autoencoder, wherein the electronic computing device is designed to carry out the method according to the aforementioned aspects. In particular, the method is carried out by means of the electronic computing device.

[0017] The electronic computing device has, for example, a processor, a circuit (in particular an integrated circuit) and other electronic components in order to be able to execute the corresponding method steps.

[0018] Advantageous embodiments of the method are to be regarded as advantageous embodiments of the electronic computing device, which for example has the subject features so that the corresponding method steps can be executed.

[0019] Further advantages, features and details of the invention are derived from the following description of preferred embodiments and from the accompanying drawings. The above-mentioned features and combinations of features mentioned in the description, as well as the features and combinations of features mentioned in the description of the drawings and / or shown alone in the drawings, can be used not only in the respective combination but also in other combinations or alone without departing from the scope of the invention.

[0020] in:

[0021] Figure 1 A schematic top view shows an embodiment of a motor vehicle having an embodiment of an assistance system and an embodiment of an electronic computing device;

[0022] Figure 2 shows a schematic flow chart according to an embodiment of the method;

[0023] Figure 3 shows a schematic block diagram according to an embodiment of a variable autoencoder;

[0024] Figure 4 shows a schematic top view of a scene with an embodiment of a motor vehicle; and

[0025] Figure 5 A schematic diagram of the data analysis is shown.

[0026] In the figures, identical or functionally identical elements are provided with the same reference signs.

[0027] Figure 1 A schematic top view of an embodiment of a motor vehicle 10 is shown. The motor vehicle 10 is operated at least partially with assistance. The motor vehicle 10 can also be operated fully with assistance. For this purpose, the motor vehicle 10 has, in particular, an assistance system 12 . The assistance system 12 has, in particular, an electronic computing device 14 .

[0028] In particular, an object is provided in the environment 16 of the motor vehicle 10, which is specifically shown below as another motor vehicle 18. The other motor vehicle 18 has a trajectory 20, which, for example, is traveling in the lane of the motor vehicle 10. This may be a safety-related situation, such as an emergency braking situation, in which the assistance system 12 must react accordingly if the motor vehicle 10 is in at least partially assisted driving mode.

[0029] Figure 2 In a first step S1, the actually acquired trajectory data 36 of the object are acquired by means of a measurement technique (see Figure 5 ), for example trajectory data 20. In a second step S2, the measurement sequence is extracted. In a third step S3, the measurement signal is preprocessed, in particular to identify non-physical behaviors and so-called ghost objects. In a fourth step S4, the objects are classified, in particular by means of a classifier trained with labeled simulated data. Then, in a fifth step S5, the variable autoencoder 22 ( Figure 3 ) generates data for a specific object class. To this end, provision can also be made for a qualitative and quantitative evaluation of these corresponding data in a sixth step S6, for example, using hyperparameter optimization. Then, in a seventh step S7, they are integrated into the existing simulation. Then, in an eighth step S8, the generated test data is simulated, and in particular, a safety-related evaluation thereof is performed.

[0030] Figure 3 A schematic block diagram of an embodiment of a variable autoencoder 22 is shown. The variable autoencoder 22 has at least one encoder 24 and a decoder 26, wherein both the encoder 24 and the decoder 26 can be designed as convolutional neural networks. Furthermore, a so-called latent space 28 is shown. Specifically, multivariate time series data is processed in the encoder 24 and reconstructed 34 in the decoder 26. Based on the reconstruction 34 and the latent space 28, test data 30 can be generated.

[0031] In particular, Figure 3A variable autoencoder 22 is shown as the basic architecture of an artificial neural network for generating test data 30. The autoencoder 22 consists of two basic parts, specifically an encoder 24 and a decoder 26. They are trained to reproduce the input data by transforming the input into a low-dimensional latent space 28 (specifically corresponding to the encoder part) and reconstructing the input from this latent space 28 (specifically corresponding to the decoder part). The encoder of the variable autoencoder 22 differs from a conventional autoencoder in that it can map the input data to a multivariate latent distribution. A sample is then drawn from this distribution and passed through the decoder 26, which creates a realistic reconstruction 34 of the input data. New data can be generated here by passing randomly drawn codes through the decoder 26.

[0032] Generative models are designed with multidimensional time series 32, which present particular challenges due to the temporal dependencies within the signal. Using convolutional neural networks, commonly used for image data, has proven promising for addressing this particular challenge. Alternatively, deep neural networks or recurrent neural networks can be used, for example. The network architecture and learning process are optimized using hyperparameter optimization in the form of a grid search.

[0033] To evaluate the performance of the autoencoder 22, and in particular the authenticity of the generated synthetic data, qualitative and quantitative validation techniques were used. To date, there is no standardized method for evaluating the performance of generative neural networks using time series signals as input data. To enable a differentiated and comprehensive evaluation, various data attributes were taken into account and assessed using selected metrics specifically designed for these technical applications.

[0034] Use the overall distribution of new data points for each feature (ignoring the temporal aspect of the data) and kernel density estimation to check whether the generated data covers the entire input data.

[0035] Another metric, specifically autocorrelation, considers the temporal aspect. It is used to check whether the generated data reflects the temporal dependencies of the test data 30. Specifically, autocorrelation is the correlation of a signal with a delayed version of itself. It measures the relationship between the current value of a signal and its original value.

[0036] In addition, a metric called MiVo (mean of the variance of the input output) can be used. This metric is based on a nearest neighbor distance metric for all training samples and all generated samples. MiVo makes it possible to assess not only the authenticity of the generated data, but also its diversity. The response of the assistance system 12 to a specific scenario is then tested using test data 30 generated for that scenario.

[0037] Figure 4 A schematic top view of a scene with an embodiment of a motor vehicle 10 is shown. In particular, an L-shape is shown based on the analysis of sensor data from one or more sensors 40. Due to its limited field of view, the sensor 40 perceives other motor vehicles 38 in the surrounding area as either behind or to the side (depending on which side the surrounding vehicle is facing the sensor during the maneuver). The perceived image resembles the letter L and is referred to as an L-shape 40. The L-shape 40, based on the analysis of the sensor data, is used to identify the movement of traffic objects relative to the host vehicle (in particular, the motor vehicle 10 equipped with the corresponding sensor 40).

[0038] Figure 5 A schematic diagram of the data analysis is shown. In particular, Figure 5 An example lane change maneuver is shown. The x-axis represents time t in seconds, and the y-axis represents lateral distance d in meters. Actual collected trajectory data 36 are represented by a dashed line, while test data 30 are represented by a solid line. The trajectories of adjacent motor vehicles are shown relative to motor vehicle 10.

[0039] In particular, the distribution of points in the training data set can be compared to the distribution of trajectory points in the generated set. In the figure, they are similar, but different. Edge cases are safety-related scenarios that correspond to real-world behavior of neighboring vehicles that was not captured during actual test drives.

Claims

1. A method for generating test data (30) for an assistance system (12) of a motor vehicle (10) for simulating at least partially assisted operation by means of an electronic computing device (14), the method comprising the following steps: - presupposing actually acquired trajectory data (36) of at least one driving situation for at least a partial assisted driving operation of the motor vehicle (10); (S1) - extracting a training trajectory from the trajectory data (36) by means of the electronic computing device (14); (S2) - training a variable autoencoder (22) of the electronic computing device (14) using the extracted training trajectory; (S3, S4) - generating potential test data (30) for the simulation using the trained variable autoencoder (22); (S5) - comparing the actually acquired trajectory data (36) with the potential test data (30) by means of the electronic computing device (14); (S6) and - Identifying test data for the simulation based on the comparison (30). (S7) 2. The method according to claim 1, It is characterized by: The potential test data (30) and / or the test data (30) are generated in such a way that it is possible to control functional devices of the auxiliary system (12) using the potential test data and / or the test data.

3. The method according to claim 1 or 2, It is characterized by: Based on the generated test data (30), control of functional devices of the auxiliary system (12) is performed.

4. The method according to any one of the preceding claims, It is characterized by: The training trajectory is extracted by means of a classifier of the electronic computing device (14).

5. The method according to any one of the preceding claims, It is characterized by: The trajectory data (36) is filtered so as to remove non-physical behaviors of at least one object (18) in the actually collected trajectory data (36).

6. The method according to claim 5, It is characterized by: Objects (18) that exist for a short time in the trajectory data (36) are identified as ghost objects and filtered out.

7. The method according to any one of the preceding claims, It is characterized by: The actual collected trajectory data (36) is compared qualitatively and quantitatively with the potential test data (30).

8. A computer program product having program code means which, when processed by an electronic computing device (14), causes the electronic computing device (14) to carry out the method according to any one of the preceding claims 1 to 7.

9. A computer-readable storage medium having the computer program product according to claim 8.

10. An electronic computing device (14) for generating test data (30) for an assistance system (12) of a motor vehicle (10) simulating at least partially assisted operation, the electronic computing device having at least one variable autoencoder (22), wherein the electronic computing device (14) is designed to carry out the method according to any one of claims 1 to 8.

Images