Method for verifying digitally-operated (augmented) test scenario

Abstract

To provide a method for providing test data which is realistic and easily generated in order to simulate an autonomous vehicle.SOLUTION: A method for verifying a digitally operated test scenario includes a step of digitally recording a basic scenario by at least one real vehicle on an actual road under a defined actual traffic condition, a step of generating a reference scenario by digitally recording the basic scenario operated by the selected traffic condition, a step of generating an augmented test scenario by operating data on the digitally recorded basic scenario by the selected traffic condition, a step of acquiring data on the traffic condition in the augmented test scenario and the reference scenario with the use of at least one sensor, and a step of evaluating propriety of data on the test scenario on the basis of a result obtained by analyzing correlation between test data and reference data acquired by at least the one sensor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the technical field of simulation engineering, in particular to a method for verifying digitally manipulated (extended) test scenarios according to claim 1, a computer-implemented method according to claim 10 and preferred applications of the method according to claim 15. [Background technology]

[0002] Validating an Automated Driving System's (ADS) ability to sense environmental data requires numerous tests, including the use of simulated or recorded test data (test stimuli) from various scenarios. Sensor algorithms, in the sense of the "sense-plan-act" part of the ADS stack, are considered in all aspects of acquiring and processing environmental information. Currently, simulations are not sufficiently representative, so it is common to use open-world driving or real-world recoding based on test sites. Collecting real data requires significant effort and is highly dependent on parameters that cannot be influenced, such as weather and lighting conditions. Summary of the Invention

[0003] Therefore, one object of the present invention is to avoid these drawbacks and provide realistic, easily generated test data for simulating autonomous vehicles.

[0004] This problem is solved by a method for verifying a digitally manipulated (extended) test scenario as set forth in claim 1. The method according to the present invention includes the steps of digitally recording a base scenario with at least one real vehicle driving on a real road under defined real traffic conditions; digitally recording the base scenario manipulated by selected traffic conditions to create a reference scenario; manipulating data of the digitally manipulated base scenario by the selected traffic conditions to create an extended test scenario; acquiring data of the traffic conditions of the extended test scenario and data of the reference scenario by at least one sensor; analyzing the correlation between the test data acquired by the at least one sensor and the reference data; and evaluating the validity of the data of the test scenario based on the results of the correlation analysis.

[0005] Analyzing the correlation between the test data recorded by the at least one sensor and the reference data can also include comparing the system behavior after applying a test stimulus (real / digital) to the ADS, in which case it is checked to what extent the detection function for recording the environmental data leads to the same system behavior.

[0006] A key aspect of the method of the present invention is to maximize the use of recorded real data by digitally manipulating or augmenting the available data with respect to specific parameters. In this "data augmentation," real recordings are manipulated so that individual parameters are changed to generate additional instances of the recorded data. This process significantly increases the available test data at a reasonable cost.

[0007] The subclaim set describes further advantageous embodiments of the method according to the invention.

[0008] In a first advantageous embodiment of the method according to the invention, it is provided that the traffic conditions comprise weather conditions such as precipitation, lighting conditions and / or road conditions such as road type, course, nature, objects such as vehicles and people, regulations such as traffic signs and signals, etc. This allows a wide variety of test scenarios to be created and analysed against a reference scenario.

[0009] In a first preferred embodiment of the method according to the invention, the selected traffic conditions include adding the effects of precipitation, in particular fog, drizzle, showers, sleet, snow, etc., thereby providing a range of precipitation that usually poses particular challenges to the vehicle's detection capabilities in order to avoid critical driving situations.

[0010] In a further preferred embodiment of the method according to the invention, digital noise and brightness variations are added to generate the effects of selected traffic conditions, which makes it possible in particular to easily simulate precipitation.

[0011] In principle, all weather conditions can be simulated, but it is advantageous if the selected traffic conditions are combined with the influence of other objects, such as other vehicles or people, to allow the simulation of specific traffic conditions.

[0012] In a further preferred embodiment of the method according to the invention, the digitized record of the base scenario and / or reference scenario consists of image data and / or distance data, in particular generated by camera images and / or LIDAR (Light Detection And Ranging) point clouds and / or radar (Radio Detection And Ranging), which provides particularly accurate reference data and improves the analysis of the test scenario.

[0013] In a further preferred embodiment of the method according to the invention, a validation of the data of the test scenario is carried out based on an analysis of the correlation between the data of the test scenario and the data of the reference scenario, which makes it possible to determine to what extent the data of the test scenario can be used to simulate real traffic conditions.

[0014] In a further preferred embodiment of the method according to the invention, the validity of the test scenario data is evaluated according to the ISO 21448-2022 standard (Scenario and System Analysis), whereby the test data is evaluated according to a defined standard which also allows for the subsequent certification of the simulation and / or detection functions.

[0015] To verify the data of the test scenario, the above-mentioned plausibility assessment is taken into account, and in a further preferred embodiment of the method according to the invention, the data of the selected traffic conditions are manipulated (optimized) accordingly based on this assessment of the data of the test scenario, in order to ultimately bring the test scenario closer to the reference scenario.

[0016] On the one hand, the test scenario can be optimized as described above, but if the test scenario and the reference scenario match sufficiently, e.g., if the test data can be validly evaluated according to standard ISO 21448-2022, it may be preferable to also manipulate the non-detection related algorithms of at least one sensor for the autonomous vehicle in order to improve the detection performance of the at least one sensor.

[0017] The aforementioned problem is also solved by a computer-implemented method for validating digitally manipulated (extended) test scenarios as set forth in claim 10. The computer communicates with at least one image- and / or distance-providing sensor to record a base scenario and a reference scenario, manipulates the base scenario by selected traffic conditions to create an extended test scenario, communicates with at least one sensor to record data of traffic conditions in the reference scenario and the extended test scenario, and analyzes and evaluates correlations between data recorded by the at least one sensor in the test scenario and reference data of traffic conditions recorded by the at least one sensor in the reference scenario.

[0018] An important aspect of the computer-implemented method according to the present invention is that it can be automated on a large scale and offers considerable time advantages over traditional open-world drives and / or test sites.

[0019] Advantageous further embodiments of the computer-implemented method according to the invention are set forth in the dependent claims.

[0020] In a first preferred embodiment of the computer-implemented method, compliance with the ISO 21448-2022 standard (scenario and system analysis) is determined to assess (150) the validity of the test scenario data, which allows compliance with the defined standard and subsequently the corresponding certification of the autonomous vehicle.

[0021] In a second preferred embodiment of the computer-implemented method, the data of the selected traffic conditions are manipulated based on an assessment of the validity of the data of the test scenario, which allows this step to be automated, with further time advantages and a defined validation of this data.

[0022] In a further preferred embodiment of the computer-implemented method, a detection-related algorithm of at least one sensor for the autonomous vehicle is manipulated based on the analysis, particularly when data from the test scenario is evaluated as valid to further optimize the detection performance of the at least one sensor.

[0023] In a further preferred embodiment of the computer-implemented method, the optimization of the test scenarios and / or detection-related algorithms of the at least one sensor is supported by self-learning algorithms, in particular based on machine learning and / or neural networks, which means that time advantages can also be achieved in this optimization step due to the self-learning system.

[0024] In principle, the method according to the invention can be used to simulate a wide range of different traffic conditions, which makes it desirable to use it for extended test scenarios for the optimization of sensors and / or for the optimization of extended test scenarios for autonomous driving systems.

[0025] Further advantages, objects and features of the present invention will be explained with reference to the following description taken in conjunction with the accompanying drawings, in which: Similar components may have the same reference numerals in the various embodiments. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 2 is a functional diagram of a method sequence according to the invention for verifying a digitally manipulated (extended) test scenario. [Figure 2] 2 shows a more detailed representation of the subject matter of all steps of the method sequence according to FIG. 1. [Figure 3] A comparison of objects in the base scenario, test scenario, and reference scenario. DETAILED DESCRIPTION OF THE INVENTION

[0027] FIG. 1 shows a functional diagram of a method sequence 100 according to the present invention for validating a digitally manipulated (augmented) test scenario 120. In step 110, a base scenario with real traffic conditions is digitally recorded, followed by digital augmentation to form a test scenario in step 120. In this process, the parameter x of the data of the digital test scenario is modified by augmentation to a specific value. The real base scenario 110 is digitally recorded as a reference scenario 130, which has been modified by the parameter x to the same specific traffic conditions as the augmented test scenario 120. In a further step 140, the correlation between the test stimuli of the test scenario 120 and the data of the reference scenario 130 is analyzed to evaluate the validity of the augmented test data. The system behavior after application of the test stimuli (real / digital) to the ADS is also compared here. That is, it is checked to what extent the detection function for recording environmental data results in the same system behavior. If these are considered valid, for example for SOTIF (Safety of the Intended Functionality) validation of the ISO 21448-2022 standard, test scenarios can be used to simulate sensors, especially image and / or distance measurement sensors. This validation method therefore requires a specific detection system.

[0028] Therefore, the validation approach is based on the use of a highly reproducible, real-world test environment. For validation, correlations between nearly identical scenarios are analyzed in both the real environment and the augmented data. Evidence from validation arguments and data augmentation for specific parameters in representative scenarios can be used as evidence for generalized validation arguments on the augmented data.

[0029] The data extension mechanism is an important mechanism for extending existing real-world data for the validation of sensor algorithms. To use this mechanism for SOTIF validation according to the ISO 21448-2022 standard, a validity and validity argument, including proof of mechanism, must be presented. The above validation approach strongly supports the claims of SOTIF as a valuable product in the context of autonomous driving in the European market, as well as a safety claim in the context of autonomous driving in the international market.

[0030] Figure 2 shows in more detail the subject matter of all steps of the method sequence according to Figure 1. In step 110, a path 300 is created in which the vehicles 210 move towards each other, with the EGO (autonomous vehicle) 200 in one lane and the PRU (protected road user) in a parallel lane. In a further step 120, a test scenario is created by digitally enhancing the selected traffic conditions 310'. Finally, in step 130, a reference scenario with digitally recorded selected traffic conditions 310 (here, actual precipitation) is recorded for comparison with the test scenario.

[0031] In the next step 140, the correlation between the test stimuli of the test scenario and the data of the reference scenario is analyzed, followed by an assessment of the validity of the data in the subsequent step 150. For example, if the data is invalid compared to the standard ISO 21448-2022, the data does not realistically reflect the selected real traffic conditions. Therefore, the data must be optimized, for example by manipulating digital noise or brightness accordingly. This process can also be supported by a self-learning system such as a neural network whose training data includes augmented data of the test scenario and data of the reference scenario.

[0032] In principle, real-world recorded data, such as camera images or LIDAR point clouds from a slightly cloudy weather scenario, can be used as a base scenario. Digital noise and brightness variations can be added to this recorded data to create the effects of rain or fog in the test scenario. The result is a modified version of the base scenario with altered weather conditions.

[0033] The corresponding method involves digitally recording a base scenario without precipitation on an open-world drive and / or test site. Next, a reference scenario, which is an identical scenario to the base scenario but with a certain amount of precipitation added, is digitally recorded on an open-world drive and / or test site. Finally, the recorded base scenario (without precipitation) is augmented with digital noise to simulate precipitation. Finally, the correlation between the reference scenario and the augmented baseline scenario is analyzed to evaluate the validity of the augmentation.

[0034] However, it can also be based on real records, such as camera images and / or LIDAR point clouds of an intersection-related scenario where a PRU, such as a real vehicle, is present. Information representing the PRU (pixels in the case of a camera, or point cloud partitions in the case of LIDAR) can be extracted and reinserted into the scenario at a different position to simulate another PRU in the scenario. This creates another instance of the base scenario with the added PRU.

[0035] The approach to this case is to digitally record a base scenario in an open-world driving and / or proving ground, with the vehicle centered in the oncoming lane, n meters ahead of the EGO vehicle. Next, a reference scenario identical to the base scenario is digitally recorded, but with the vehicle centered on the center lane marker. Finally, the recorded base scenario is augmented by virtually / digitally moving the vehicle around the center of the center lane marker to simulate object movement. Finally, the correlation between the reference scenario and the augmented base scenario is analyzed to assess the validity of the augmentation.

[0036] Figure 3 shows a comparison of the base scenario, test scenario, and reference scenario from steps 110, 120, and 130. An EGO vehicle 200 and an oncoming vehicle 210 are traveling toward each other in parallel opposing lanes of a roadway 300 at a speed of 20 km / h. The actual base scenario is digitally recorded without the selected traffic conditions 310 (in this case, actual rain). For the reference scenario in step 130, the base scenario including actual rain is digitally recorded and compared with the test scenario in step 120. In the test scenario, digital rules are added, for example, through digital noise or brightness changes. The correlation between the data from the test scenario and the data from the reference scenario is finally analyzed, i.e., whether the data are suitable to provide the same situation image to at least one sensor used. In this case, the test data is evaluated as valid and can be used for further sensor testing, for example, to optimize sensor performance.

[0037] The data augmentation process described above can be used to generate test stimuli. However, to generate valid test results that reflect the behavior of sensor algorithms in the real world, an argument must be provided demonstrating the validity of the augmented data. This argument can be based on the validation of a representative sample comparing the correlation between the augmented data and real-world data. To do this, the exact same scenario (augmented and real-world data) must be available. With high repeatability capabilities, an exact replica of the augmented scenario can be created at an established test site, providing reliable data for correlation analysis. [Explanation of symbols]

[0038] 100 Digitally Augmented Test Scenarios Validation Methods 110 Recording the Basic Scenario 120 Recording Extended Test Scenarios 130 Recording Reference Scenario 140 Analyze the correlation between test data and reference data 150 Validation of Sensor Data Using Test Scenarios 200 EGO (autonomous vehicle) 210 Another vehicle 300 roads 310 Traffic conditions selected in the reference scenario 310' Traffic conditions selected in the test scenario

Claims

1. A method (100) for validating a digitally manipulated (augmented) test scenario (120), comprising: a. Digitally recording a baseline scenario (110) with at least one real vehicle (200, 210) on a real road under defined real traffic conditions; b. Digitally recording the base scenario (110) driven by the selected traffic conditions (310) to create a reference scenario (130); c. Manipulating the digitally recorded data of the base scenario (110) with selected traffic conditions (310') to create an extended test scenario (120); d. recording data of traffic conditions in the extended test scenario (120) and the reference scenario (130) using at least one sensor; e. analyzing (140) correlations between test data collected by at least one sensor and reference data; f. Evaluating (150) the validity of the test scenario data based on the results of analyzing (140) the correlation; A method (100) comprising:

2. The method (100) according to claim 1, characterized in that the traffic conditions include weather conditions such as precipitation (310, 310'), lighting conditions, and / or road conditions such as the type, course and nature of the road (300), objects such as vehicles (200, 210) and people, and regulations such as traffic signs and signals.

3. 3. The method (100) according to claim 1 or 2, characterized in that the selected traffic conditions (310') include adding the effects of precipitation, in particular fog, drizzle, showers, sleet, snow, etc.

4. A method (100) according to any of claims 1 to 3, characterized in that digital noise and brightness variations are added to generate the effect of the selected traffic conditions (310').

5. The method (100) according to any of the preceding claims, characterized in that the selected traffic conditions (310, 310') comprise adding the influence of further vehicles (210) and / or further objects such as people.

6. The method (100) according to any of the preceding claims, characterized in that the digital record of the base scenario (110) and / or the reference scenario (130) comprises image and / or distance data.

7. 7. The method (100) according to any of claims 1 to 6, characterized in that compliance with the ISO 21448 standard (Scenario and System Analysis) is established for assessing the validity (150) of the test scenario data.

8. A method (100) according to any of the preceding claims, characterized in that, starting from an evaluation (150) of the validity of the data of the test scenario, the data of the selected traffic conditions (310') are manipulated.

9. 9. The method (100) according to any one of claims 1 to 8, characterized in that the method (100) comprises operating a detection-related algorithm of at least one sensor for an autonomous vehicle, starting from an analysis (140) of correlations between data of the test scenario (120) and data of the reference scenario (130).

10. A computer-implemented method (100) for verifying a digitally manipulated (extended) test scenario (120) according to any of claims 1 to 9, comprising: The computer communicates with at least one image and / or distance providing sensor to record the base scenario (110) and the reference scenario (130); Manipulating the base scenario (110) according to selected traffic conditions (310') to generate an extended test scenario (120); communicating with at least one sensor to obtain data on traffic conditions in the reference scenario (130) and the extended test scenario (120); A computer-implemented method (100) for analyzing (140) and evaluating (150) correlations between data acquired by at least one sensor in the test scenario (120) and data of traffic conditions acquired by at least one sensor in a reference scenario (130).

11. 11. The computer-implemented method (100) of claim 10, wherein compliance with the ISO 21448 standard (Scenario and System Analysis) is determined to assess the validity (150) of the test scenario data.

12. 12. The computer-implemented method (100) according to claim 10 or 11, characterized in that the data of the selected traffic conditions (310') are manipulated based on an assessment (150) of the validity of the data of the test scenario.

13. A computer-implemented method (100) according to any of claims 10 to 12, characterized in that starting from said analysis (140), a detection-related algorithm of at least one sensor for an autonomous vehicle is operated.

14. 14. A computer-implemented method (100) according to any of claims 10 to 13, characterized in that the optimization of the test scenarios and / or detection-related algorithms of the at least one sensor is supported by a self-learning algorithm, in particular based on a machine learning neural network.

15. Use of the method according to any of claims 1 to 9 for optimizing extended test scenarios and / or sensors of an autonomous driving system.

Images