Methods for validating digitally manipulated (augmented) test scenarios

Abstract

The invention relates to a method 100 for validating digitally manipulated (augmented) test scenarios 120, comprising the steps of digitally recording a base scenario 110 with at least one real vehicle 200, 210 moving on a real road under defined real traffic conditions; digitally recording the base scenario 110 manipulated by selected traffic conditions 310 to create a reference scenario 130; manipulating the data of the digitally recorded base scenario 110 by selected traffic conditions 310 to create an augmented test scenario 120; and acquiring data of the traffic conditions in the augmented test scenario 120 and data in the reference scenario 130 with at least one sensor.The invention relates to an analysis 140 of the relationship between the test and reference data acquired by the at least one sensor, and an evaluation 150 of the validity of the data of the test scenario based on a result of the analysis 140 of the relationship. The invention further relates to a corresponding computer-implemented method and preferred uses of the method.

Description

[0001] The invention relates to the technical field of simulation technology and in particular to a method for validating digitally manipulated (augmented) test scenarios according to claim 1, a corresponding computer-implemented method according to claim 10, and preferred uses of the method according to claim 15.

[0002] The validation of environmental data sensing functions for automated driving systems (ADS) requires a significant amount of testing effort, including the use of simulated or recorded test data (test stimuli) from various scenarios. Sensor algorithms based on the 'sense-plan-act' approach for automated driving stacks are considered in all aspects of environmental information acquisition and processing. Since simulations are currently not sufficiently representative, real recordings based on open-world driving and / or test tracks are usually used. Collecting real data is complex and highly dependent on uncontrollable parameters such as weather or lighting conditions.

[0003] The German patent application DE 10 2021 128 704 A1 describes a computer-based method for creating a simulation scenario for land vehicles. It includes receiving and merging LIDAR point clouds, locating and classifying static and dynamic objects, and generating road information. Based on this, a simulation scenario is created that can be used to test autonomous driving functions.

[0004] An object of the present invention is therefore to circumvent these disadvantages and to provide realistic and easily producible test data for simulating autonomous vehicles.

[0005] This object is achieved by a method for validating digitally manipulated (augmented) test scenarios according to claim 1.The method according to the invention comprises the steps of digitally recording a base scenario with at least one real vehicle moving on a real roadway under defined real traffic conditions; digitally recording the base scenario manipulated by selected traffic conditions to create a reference scenario; manipulating the data of the digitally recorded base scenario by selected traffic conditions to create an extended test scenario; acquiring data of the traffic conditions in the extended test scenario and data in the reference scenario with at least one sensor; analyzing the interrelationship between the test and reference data acquired with the at least one sensor, and evaluating a validity of the data of the test scenario based on a result of the analysis of the interrelationship.

[0006] Analyzing the interrelationship between the test and reference data acquired with at least one sensor may also include comparing system behavior after applying the test stimuli (real / digital) to an ADS. In this case, it is examined to what extent the detection functions for acquiring environmental data lead to the same system behavior.

[0007] A key aspect of the method according to the invention is to maximize the use of recorded real data by digitally manipulating or augmenting the available data with respect to certain parameters. In this 'data augmentation,' the real recording is manipulated by changing individual parameters to generate additional instances of the recorded data. This method significantly increases the available test data at a reasonable cost.

[0008] Advantageous further developments of the method according to the invention are specified in the subclaims.

[0009] In a first advantageous embodiment of the method according to the invention, the traffic conditions include weather conditions, such as precipitation, lighting conditions, and / or roadway conditions, such as the type, course, and condition of a roadway, objects such as vehicles or people, and regulations such as traffic signs or traffic lights. This allows a wide variety of test scenarios to be developed and analyzed against reference scenarios.

[0010] In a second preferred embodiment of the method according to the invention, the selected traffic conditions comprise adding the effect of precipitation, in particular fog, drizzle, rain showers, sleet and / or snow, thus providing a range of precipitation which usually poses a particular challenge to the detection functions of a vehicle to avoid critical driving situations.

[0011] In a further preferred embodiment of the method according to the invention, digital noise and brightness changes are added to generate the effect of selected traffic conditions, which in particular allows a particularly simple simulation of precipitation.

[0012] While this can, in principle, simulate all weather conditions, it is also advantageous if the selected traffic conditions include the addition of the effect of additional objects, such as additional vehicles and / or people, to allow for simulations of specific traffic conditions.

[0013] In a further preferred embodiment of the method according to the invention, the digitized recording of the baseline and / or reference scenario includes imaging and / or distance-providing data, which are generated in particular by camera images and / or LIDAR (Light Detection and Ranging) point clouds and / or radar (Radio Detection and Ranging). This provides particularly precise reference data that improves the analysis of the test scenario.

[0014] In a further preferred embodiment of the method according to the invention, an evaluation (validation) of the validity of the test scenario data is carried out based on the analysis of the interrelationship between the data of the test scenario and that of the reference scenario. This makes it possible to determine the extent to which the test scenario data can actually be used to simulate real traffic conditions.

[0015] In a further preferred embodiment of the method according to the invention, in order to evaluate the validity of the data of the test scenario, compliance with the ISO 21448-2022 standard (scenario and system analysis) is determined, which allows an evaluation of the test data according to a defined standard that also allows a subsequent certification of the simulation and / or the recognition functions.

[0016] To validate the test scenario data, an evaluation of its validity is considered, as described above. In order to ultimately bring the test scenario closer to the reference scenario, in a further preferred embodiment of the method according to the invention, the data of the selected traffic conditions are then manipulated (optimized) accordingly based on this evaluation of the test scenario data.

[0017] On the one hand, this can be used to optimize the test scenario as described above. However, if the test scenario and the reference scenario sufficiently match, for example, because the test data can be assessed validly according to the ISO 21448-2022 standard, it is preferable to also manipulate detection-related algorithms of the at least one sensor for autonomous vehicles in order to improve the detection performance of the at least one sensor.

[0018] The above object is also achieved by a computer-implemented method for validating digitally manipulated (augmented) test scenarios according to claim 10, in which a computer communicates with at least one image and / or distance-providing sensor to record a base and reference scenario, and manipulates the base scenario by selected traffic conditions to create an extended test scenario, and communicates with the at least one sensor to record data of the traffic conditions in the reference scenario and the extended test scenario, and analyzes and evaluates an interrelationship between the data recorded by the at least one sensor in the test scenario and the reference data of the traffic conditions recorded by the at least one sensor in the reference scenario.

[0019] An essential point of the computer-implemented method according to the invention is that it can be largely automated and thus has a considerable time advantage compared to conventional open-world journeys and / or test sites.

[0020] Advantageous further developments of the computer-implemented method according to the invention are specified in the subclaims.

[0021] In a first preferred embodiment of the computer-implemented method, for evaluating (150) the validity of the data of the test scenario, compliance with the standard ISO 21448-2022 (scenario and system analysis) is determined, which allows compliance with specified standards and consequently a corresponding certification of an autonomous vehicle.

[0022] In a second preferred embodiment of the computer-implemented method, based on the evaluation of a validity of the data of the test scenario, the data of the selected traffic conditions are manipulated, which enables an automation of this step with a further time advantage and a defined validation of these data.

[0023] In a further preferred embodiment of the computer-implemented method, based on the analysis, detection-related algorithms of the at least one sensor for autonomous vehicles are manipulated. This occurs in particular if the data from the test scenario are evaluated as valid and the detection performance of the at least one sensor is to be further optimized.

[0024] In a further preferred embodiment of the computer-implemented method, optimization of the test scenario and / or detection-related algorithms of the at least one sensor is supported by a self-learning algorithm, in particular based on machine learning and / or neural networks. This allows a time advantage to be achieved in this optimization step as well, which is attributable to self-learning systems.

[0025] In principle, the method according to the invention can be used for simulations of a wide variety of different traffic conditions. Therefore, it should preferably be used for optimizing extended test scenarios, sensor optimization, and / or extended test scenarios for autonomous driving systems.

[0026] Further advantages, objects, and features of the present invention will be explained in the following description of the accompanying figures. Similar components may have the same reference numerals in the various embodiments. In the figures: Fig. 1 a functional diagram of a method sequence according to the invention for validating digitally manipulated (augmented) test scenarios; Fig. 2 a more detailed description of the subjects of all steps of the procedure according to Fig. 1, and Fig. 3 a comparison of the items of the baseline, test and reference scenario.

[0027] The Fig. 1 shows a functional diagram of a method sequence 100 according to the invention for validating a digitally manipulated (augmented) test scenario 120. In step 110, a real base scenario of traffic conditions is digitally recorded and, in a subsequent step 120, digitally expanded to form a test scenario. A parameter x of the data of the digital test scenario is changed to a specific value by expansion. The real base scenario 110 is digitally recorded as the reference scenario 130, but modified with the parameter x to reflect the same specific traffic conditions as for the expanded test scenario 120. In a further step 140, the interrelationship between the test stimuli of the test scenario 120 and the data of the reference scenario 130 is analyzed, and the validity of the expanded test data is subsequently assessed.This also involves comparing the system behavior after applying the test stimuli (real / digital) to an ADS, i.e., checking the extent to which the detection functions for collecting environmental data lead to the same system behavior. If these are found to be valid, for example, with regard to the SOTIF (Safety of the Intended Functionality) validation of the ISO 21448-2022 standard, the test scenario can be used to simulate a sensor, particularly an imaging and / or distance-measuring sensor. Therefore, this validation approach requires a specific detection system.

[0028] The validation approach is thus based on the use of a real-world test environment with high reproducibility. For validation, the interrelationships between nearly identical scenarios are analyzed both in the real world and with extended data. The validation argument and the evidence for data augmentation regarding specific parameters in representative scenarios can be used as evidence in a generalized validation argument for extended data.

[0029] The data augmentation mechanism is a key mechanism for multiplying existing real-world recorded data for the validation of sensor algorithms. To use this mechanism for SOTIF validation according to ISO 21448-2022, a case for the validity and appropriateness of the mechanism, including proof of its validity, must be presented. The validation approach described above strongly supports the SOTIF argument and is a valuable product in the context of autonomous driving in the European market, as well as a safety argument in the context of autonomous driving in the international market.

[0030] The Fig. 2 shows a more detailed representation of the items of all steps of the procedure according to Fig. 1. In step 110, a digital base scenario is created by creating a route 300 with an EGO (Autonomous) vehicle 200 in one lane and a PRU (Protected Road User), here a vehicle 210 in a parallel lane, moving toward each other. In the further step 120, a test scenario is created by digitally expanding selected traffic conditions 310', here precipitation, by adding digital noise and brightness reduction. Finally, in step 130, a reference scenario with digitally recorded selected traffic conditions 310, here real precipitation, is recorded for comparison with the test scenario.

[0031] In the next step 140, the correlation between the test stimuli of the test scenario and the data of the reference scenario is analyzed, and the validity of the data is assessed in the subsequent step 150. If the data is invalid, for example, according to the ISO 21448-2022 standard, the data does not realistically reflect the selected real-world traffic conditions. The data would therefore need to be optimized, for example, by appropriately manipulating the digital noise and brightness. This process can also be supported by self-learning systems such as neural networks, whose training data includes the extended data of the test scenario and the data of the reference scenario.

[0032] In principle, real-world recordings can serve as the baseline scenario, such as camera images and / or LIDAR point clouds of a scenario under slightly cloudy weather conditions. Digital noise and brightness changes are added to these recorded data to create the effect of rain or fog in the test scenario. This results in a case of the baseline scenario with altered weather conditions.

[0033] A corresponding procedure involves digitally recording a baseline scenario without precipitation in open-world driving and / or test areas. A reference scenario, exactly the same as the baseline scenario with a specific amount of precipitation, is then digitally recorded in open-world driving and / or test areas. Finally, the recorded baseline scenario (without precipitation) is augmented by adding digital noise to simulate precipitation. Finally, the correlation between the reference scenario and the augmented baseline scenario is analyzed to evaluate the validity of the augmentation.

[0034] Real recordings can also serve as a basis, such as camera images and / or LIDAR point clouds of an intersection-related scenario with a PRU, such as a vehicle. The information representing the PRU—the pixels for the camera and the point cloud partition for the LIDAR—can be extracted and reinserted into the scenario at a different position to simulate a different PRU within the scenario. This creates another instance of the base scenario with an additional PRU. A corresponding procedure here provides for a baseline scenario with a vehicle n meters in front of the EGO vehicle, centered on the oncoming lane, to be digitally recorded in open-world driving situations and / or test areas. Next, a reference scenario is digitally recorded. This is exactly the same scenario as the baseline scenario, but with the vehicle centered on the center lane marking in open-world driving situations and / or test areas. Finally, the recorded baseline scenario is extended by virtually / digitally moving the vehicle centered on the center lane marking to simulate object movement. Finally, the interaction between the reference scenario and the extended baseline scenario is analyzed to evaluate the validity of the extension.

[0035] The Fig.Figure 3 shows a comparison of the objects of the baseline, test, and reference scenarios from steps 110, 120, and 130. The EGO vehicle 200 and the oncoming vehicle 210 traveling in a parallel lane of the roadway 300 are moving toward each other at a speed of 20 km / h. The real baseline scenario is digitally recorded without selected traffic conditions 310, in this case, real rain. In the reference scenario of step 130, the baseline scenario with real rain was digitally recorded, which is then compared with the test scenario of step 120. In the test scenario, digital elements were added to the rule, for example, through digital noise and brightness changes. Finally, an interrelationship between the data of the test scenario and the data of the reference scenario is analyzed, i.e., it is examined whether the data is suitable for providing the at least one sensor used with an identical picture of the situation.If this is the case, the test data can be considered valid and used for further sensor tests, for example to optimize sensor performance.

[0036] The data augmentation procedure described above can be used to generate test stimuli. However, to generate valid test results that reflect the real-world behavior of the sensor algorithms, a case for the validity of the augmented data must be provided. This case can be based on representative sample validations that compare the interrelationship of the augmented data with real-world data. For this, exactly the same scenarios (augmented and real-world) must be available. Due to the high reproducibility capabilities, established test sites can be used to create accurate replicas of augmented scenarios, providing reliable data for correlation analysis. List of reference symbols 100 procedures for validating digitally augmented test scenarios 110 Recording a baseline scenario 120 Recording an extended test scenario 130 Recording a reference scenario 140 Analyzing the Interrelationship Between Test and Reference Data 150 Evaluation of the validity of sensor data from test scenario 200 EGO (Autonomous) Vehicle 210 additional vehicles 300 track 310 Selected traffic conditions in the reference scenario 310' Selected traffic conditions in the test scenario

Claims

[1] Method (100) for validating digitally manipulated, in particular augmented, test scenarios (120), comprising the steps: Digitally recording a base scenario (110) with at least one real vehicle (200, 210) moving on a real route under defined real traffic conditions; Digitally recording the base scenario (110) manipulated by selected traffic conditions (310) to create a reference scenario (130); Manipulating the data of the digitally recorded base scenario (110) around selected traffic conditions (310') to create an extended test scenario (120); Collecting data of the traffic conditions in the extended test scenario (120) and the reference scenario (130) with at least one sensor; analyzing (140) the interrelationship between the test and reference data acquired with the at least one sensor, and Evaluating (150) a validity of the test scenario data based on a result of analyzing (140) the interrelationship. [2] Method (100) according to claim 1, characterized by that the traffic conditions include weather conditions, such as precipitation (310, 310'), lighting conditions, and / or roadway conditions, such as the type, course and condition of a roadway (300), objects such as vehicles (200, 210) or persons, regulations such as traffic signs or traffic lights. [3] Method (100) according to claim 1 or 2, characterized by that the selected traffic conditions (310') include adding the effect of precipitation, in particular fog, drizzle, rain showers, sleet and / or snow. [4] Method (100) according to one of the preceding claims, characterized by that digital noise and brightness changes are added to create the effect of selected traffic conditions (310'). [5] Method (100) according to one of the preceding claims, characterized by that the selected traffic conditions (310, 310') include adding the effect of further objects, such as further vehicles (210) and / or persons. [6] Method (100) according to one of the preceding claims, characterized by that the digitalized recording of the base (110) and / or reference scenario (130) comprises imaging and / or distance-providing data. [7] Method (100) according to one of the preceding claims, characterized by that compliance with ISO 21448 is established to assess the validity (150) of the test scenario data. [8] Method (100) according to one of the preceding claims, characterized by that based on the evaluation (150) of a validity of the data of the test scenario, the data of the selected traffic conditions (310') are manipulated. [9] Method (100) according to one of the preceding claims, characterized by that, based on the analysis (140) of the interrelationship between the data of the test scenario (120) and those of the reference scenario (130), detection-related algorithms of the at least one sensor for autonomous vehicles are manipulated. [10] Computer-implemented method (100) according to one of the preceding claims for validating digitally manipulated, in particular augmented, test scenarios (120), in which a computer communicates with at least one image and / or distance-providing sensor to record a base scenario (110) and a reference scenario (130), and manipulates the base scenario (110) by selected traffic conditions (310') to create an extended test scenario (120), communicates with the at least one sensor to record data of the traffic conditions in the reference scenario (130) and the extended test scenario (120), and analyzes (140) and evaluates (150) an interrelationship between the data recorded by the at least one sensor in the test scenario (120) and the data of the traffic conditions recorded by the at least one sensor in the reference scenario (130). [11] Computer-implemented method (100) according to claim 10, characterized bythat, for the purpose of assessing (150) the validity of the data of the test scenario (120), compliance with the ISO 21448-2022 standard is established. [12] Computer-implemented method according to claim 10 or 11, characterized by that based on the evaluation (150) of a validity of the data of the test scenario, the data of the selected traffic conditions (310') are manipulated. [13] Computer-implemented method (100) according to one of claims 10 to 12, characterized by that, based on the analysis (140), detection-related algorithms of the at least one sensor for autonomous vehicles are manipulated. [14] Computer-implemented method (100) according to one of claims 10 to 13, characterized by that an optimization of the test scenario and / or detection-related algorithms of at least one sensor is supported by a self-learning algorithm, in particular based on machine learning and / or neural networks. [15] Use of the method according to one of claims 1 to 9 for optimizing extended test scenarios and / or sensors for autonomous driving systems.

Images