Method for modifying test data for autonomous driving

Abstract

The invention relates to a Method for modifying test data for autonomous driving wherein the test data (DTA) is vehicle sensor (SNR) data (DTA) for testing a driver assistance system (ADS), wherein the change of test data (DTA) is made from a first scenario (FSC) to a second modified scenario (SMS), wherein the test data (DTA) includes optical camera (OPC) data (DTA) and / or radar (RDR) data (DTA) and / or LIDAR (LDR) data (DTA). It is proposed, that the modification of test data (DTA) is made using AI technology (AIT), wherein an input (INT) into the modification process comprises the sensor (SNR) data (DTA) of the first scenario (FSC) and specifications (ODD) for modifying the sensor (SNR) data (DTA), wherein an output comprises vehicle sensor (SNR) data (DTA) according to the second modified scenario (SMS).

Description

[0001] Method for modifying test data for autonomous driving

[0002] The present invention relates to a method for modifying test data for autonomous driving. Furthermore, the invention relates to a device for modifying test data for autonomous driving.

[0003] The general field of the invention is testing autonomous driving systems - often referred to as advanced driver assistance systems [ADAS]. More specifically the invention aims to perform the testing not in real traffic but to simulate the traffic scenarios in a laboratory environment putting the ADAS-modules to be tested in a HIL, MIL, SIL, VIL-testing environment.

[0004] Herein, the abbreviations are known as

[0005] - HIL (Hardware-in-the-Loop) Testing: HIL testing involves testing a system or component by integrating real hardware with a simulated or emulated environment.

[0006] - MIL (Model-in-the-Loop) Testing: MIL testing involves testing a system or component using a simulation model of its behavior.

[0007] - SIL (Software-in-the-Loop) Testing: SIL testing involves testing the software components of a system in an isolated software environment.

[0008] - VIL (Vehicle-in-the-Loop) Testing: This type of testing involves integrating a physical vehicle with a simulated or emulated environment.

[0009] In the context of ADAS an Operational Design Domain [ODD] is a concept used in the field of autonomous vehicles. It refers to the specific conditions and circumstances in which a self-driving vehicle is designed to operate safely. The ODD encompasses factors such as the geographic area, weather conditions, road types, speed limits, and other environmental and operational constraints under which the autonomous vehicle is intended to function properly. Defining the ODD is crucial as it helps ensure that the autonomous vehicle operates within its capabilities and avoids situations where it may not be able to safely navigate or respond appropriately.

[0010] The association for standardization of automation and measuring systems [ASAM], a German standardization body published a new international standard concept for defining ODDs under: https: / / www.asam.net / standards / detail / openodd / . This ASAM OpenODD concept enables governments and the automotive industry to access ODD descriptions that are exchangeable, comparable, and processable. The ODD is often written by engineers dealing with autonomous vehicle safety or functional requirements. Current works in automotive industry manually interpret the ODD requirements to engineering requirements, then integrate them into scenario descriptions in simulation or real data logging. This method requires manual tuning to specific ODD details, specific ODD expansion, i.e. adapt a scenario from an Ell to Asian street, and the generated data is still less realistic than real data scenarios.

[0011] The importance of the ODD in the field of ADAS / AV development is significant since those scenarios and data are used for ADAS testing and validation and since the requirement is to generate scenarios that - in total - cover the ODD. Due to these requirements the ODDs are gradually expanded. For example, the test scenarios should not only comprise dry and sunny scenes but also rainy or snowy weather conditions. Other examples are different surrounding cars or different street / road styles (i.e., streets in Ell, USA, or Asia) or even different traffic situations like road constructions blocking the way.

[0012] The data can be generated via either simulation or a real data logging campaign. Both options have advantages and disadvantages. The real data campaigns are very expensive. Simulating data is significantly cheaper than collecting real data, but normally requires manual tuning to specific ODD details like adapt a scenario from an EU street to an Asian street. Typically, simulation data is less realistic and accurate than real-world data. Using only real data campaign data is insufficient for covering the full range necessary to fulfill the ODDs.

[0013] In the last years, experts have been working with multiple Mi L, Hi L, and ViL testing projects on several autonomous driving vehicle platforms (Ford Focus, Simrod, Ranger Rover Evoque). Experts also used simulation tools for scenario generation for example HEEDs (=> Hierarchical Evolutionary Engineering Design System, developed by Siemens Digital Industries Software, is a commercial software tool used for design exploration and optimization) and falsification techniques, for example: Surpassing ADAS safety and comfort expectations while reducing extensive testing. One example may be found under: https : / / www. youtu be . com / watch?v=XP 16qm1YBoo.

[0014] One object of the invention is bridging the gap between simulation and real data, i.e.

[0015] Sim2Real (=> Simulation to Reality, refers to a research and development paradigm in artificial intelligence and robotics) and Real2Sim (=> mutatis mutandis) topics, overcoming the limitations of simulation data and models such that they are beneficial for physical system development.

[0016] From Matt Cragun: "DRIVE Sim Novel Al Tools Advance AV Deployment / NVIDIA Blog", 23 March 2022, pages : / / blogs. nvidia.com / blog / drive-sim- omniverse-neural-ai-digital-twin / it is known to use a twostep approach. The traffic environment is either virtually (as a 3D-model) or neurally (a neural network outputs images from an input of sensor data) reconstructed.

[0017] The invention is based on the problem of improving the known method for modifying test data such that a further optimization is reached regarding the above explained challenges.

[0018] In order to overcome the limitations of current test data generation and adaption the invention proposes a method according to claim 1 and a device according to the independent product claim preferably prepared to carry out a method according to the invention.

[0019] Proposed is a method for modifying test data for ADAS and / or autonomous driving wherein the test data is vehicle sensor data for testing a driver assistance system, wherein the change of test data is made from a first scenario to a second modified scenario, wherein the test data includes optical camera data and / or radar data and / or LIDAR data.

[0020] Essential is that the modification of test data is made using Al technology, wherein an input into the modification process comprises the sensor data of the first scenario and specifications for modifying the sensor data, wherein an output comprises vehicle sensor data according to the second modified scenario.

[0021] According to one embodiment it is proposed, the first scenario represents a specific traffic situation, and the modification of the test data results in a second modified scenario representing a modified traffic situation.

[0022] According to one embodiment it is proposed that the modification of the test data includes optical camera data by fitting certain visual features in the vehicle sensor data.

[0023] According to one embodiment it is proposed that the modification of the test data includes radar data by modifying the distances and velocities of objects in the vehicle sensor data. According to one embodiment it is proposed that the modification of the test data includes LIDAR data by adjusting the spatial detection of objects in the vehicle sensor data.

[0024] According to one embodiment it is proposed that the Al technology includes machine learning and / or neural networks to perform the modification of the test data for autonomous driving.

[0025] According to one embodiment it is proposed that the specifications for modifying the test data of autonomous driving include parameters such as speed, position, vehicle behavior or road conditions.

[0026] According to one embodiment it is proposed that the modified test data for autonomous driving to test and evaluate the performance and reliability of an advanced driver assistance system in various scenarios.

[0027] According to one embodiment it is proposed that the Al technology includes image-to-image translation and the method comprises modifying the test data for autonomous driving by transforming optical camera data into modified scenarios.

[0028] According to one embodiment it is proposed that the Al technology includes style transfer or neural style transfer and the method comprises modifying the test data for autonomous driving by applying different styles and visual features to the optical camera data.

[0029] According to one embodiment it is proposed that the Al technology includes Object Segmentation and Replacement, and the method comprises modifying the test data for autonomous driving through the segmentation and replacement of certain objects in the optical camera data.

[0030] According to one embodiment it is proposed that the Al technology includes scene reconstruction and the method comprises modifying said test data for autonomous driving by the partial reconstruction of the data, wherein the modifications include at least one of the following features: generating new 3D models, changing textures, and changing lighting conditions. According to one embodiment it is proposed that the Al technology includes Al-assisted 3D modeling and rendering techniques and the method comprises modifying the test data for autonomous driving through the application of advanced Al algorithms to generate and display 3D models of the vehicle's environment.

[0031] According to one embodiment it is proposed that said specifications are textual specifications.

[0032] According to one embodiment it is proposed that during a first step sensor data are acquired during a real data logging campaign respectively during a test drive with a vehicle.

[0033] In a second step these test data may be used as said first scenario test data in a method according to the invention or one of its embodiments.

[0034] Another teaching of the invention relates to a device for modifying test data for autonomous driving, in particular for carrying out a method according to the invention as defined herein.

[0035] Essential feature of the device for modifying test data for autonomous driving is that the device comprises a data processing unit and a memory arrangement to store test data for autonomous driving, wherein the test data is vehicle sensor data for testing a driver assistance system, wherein the change of test data is made from a first scenario to a second modified scenario, wherein the test data includes optical camera data and / or radar data and / or LIDAR data.

[0036] According to one embodiment it is proposed, an Al-module comprising

[0037] - Al-technology wherein said Al-technology configured for modifying the test data from a first scenario to a second modified scenario;

[0038] - a communication channel to receive the test data;

[0039] - an Al algorithm based on Al technology, capable of analyzing, transforming, and modifying the test data;

[0040] - an output unit to provide the modified test data for use in autonomous driving testing.

[0041] According to a preferred embodiment, an Al technology module is proposed that is configured to modify the test data from a first scenario to a second modified scenario; which Al technology module has a communication channel capable of receiving the test data; which comprises an Al-algorithm based on Al-technology capable of analyzing, transforming, and modifying the test data; which has an output unit to provide the modified test data for use in autonomous driving tests.

[0042] According to one embodiment it is proposed, that said Al technology module is configured such that it includes at least one of image-to-image translation, style transfer, object segmentation and replacement, scene reconstruction, Al-powered 3D modeling and rendering techniques.

[0043] The combination of simulation and generative Al according to the invention may allow for greater flexibility and diversity in scenario generation compared to purely simulation-based or purely data-driven approaches. The Al technology used according to the invention may overcome limitations of synthetic data while maintaining the ability to generate diverse scenarios based on ODD requirements. This approach may bridge the gap between simulation and real-world data, potentially improving the testing and validation of autonomous driving systems.

[0044] All explanations given with regard to the method for modifying test data are fully applicable to the device respectively to the Al-module as well.

[0045] This involves training an Al model to learn the style or characteristics of winter scenes and then applying that style to the original summer scene.

[0046] There are several options to train the Al model to perform according to the above objectives. Below some possibilities are outlined:

[0047] - Supervised Learning: In this approach, a dataset, e.g., of summer and winter scenes, are paired together. Providing Al model with examples of summer scenes and their corresponding winter versions enables the Al model to perform the mapping between the two styles. This may be done for any kind of modification the Al is expected to perform. This approach requires labeled data, which can be time-consuming and costly to obtain if the labelling is done manually.

[0048] - Unsupervised Learning: without access to e.g., paired summer and winter scenes, unsupervised learning techniques may be used. One possibility is to use generative adversarial networks (GANs). GANs consist of two neural networks, a generator and a discriminator, which compete against each other to produce realistic images. By training a GAN on a large dataset of both, e.g., summer and winter scenes, new winter scenes based on the learned style can be generated. - Transfer Learning: Another option is to use a pre-trained model that has already been trained on a large dataset of images or videos. For example, a pre-trained model like CycleGAN may be used, which is specifically designed for image-to-image translation tasks. By fine-tuning the pre-trained model on the specific summer-to-winter task or any other modification task, good results may be achieved with less training time and data.

[0049] The method for modifying test data for autonomous driving may comprise as a previous or preparatory step a step of training an Al module to alter a given test data of a traffic situation, wherein the Al module modifies the traffic situation itself (road blocking, pedestrian entering the road, additional traffic lights, more other vehicles, etc.) or in terms of weather conditions, lighting conditions, or any type of style - such as country specific road style, e.g., US road, European road, Asian road - to produce a realistic and coherent representation of the altered movie scenario.

[0050] This additional step may comprise training the Al module by providing a variety of training data covering various recorded test data, including traffic situations on US roads, EU roads and Asian roads, as well as different weather conditions (preferably of the same location), lighting conditions (preferably of the same location) and styles.

[0051] This additional step may comprise generating the training data by acquiring and recording real test campaign data using sensors and cameras in different geographical regions or weather conditions.

[0052] This additional step may involve the Al module comprising a neural network architecture to learn and generate the modification of the movie scenario based on the input parameters obtained.

[0053] This additional step may comprise training the Al module using training data for modifying the film scenario such that the Al module is also capable of taking into account and adapting traffic flow patterns, traffic rules and behavior of road users in the altered film scenario. In general, the training data for the Al-module may comprise variations of the above parameters together with specifications of the respective scenario respectively explanatory text about the parameters in an ODD text style.

[0054] The properties, features and advantages of this invention described above, as well as the manner they are achieved, become clearer and more understandable in the light of the following description and embodiments, which will be described in more detail in the context of the figures. This following description does not limit the invention on the contained embodiments. Same components or parts can be labeled with the same reference signs in different figures. In general, the figures are not for scale. It shall be understood that a preferred embodiment of the present invention can also be any combination of the dependent claims or above embodiments with the respective independent claim.

[0055] Figure 1 shows a schematic depiction of the method and a device according to the invention as a flow diagram,

[0056] Figure 2 shows a schematic depiction of how an ADAS is tested using the modified scenario sensor data.

[0057] Figure 1 shows in a simplified, schematic flow diagram the method according to the invention for modifying test data for testing a driver assistance system ADS. An example of a device DVC for modifying test data DTA for testing an autonomous driving system is shown, too. The test data DTA is vehicle sensor SNR data DTA. An Ego-vehicle EVH is shown comprising sensors SNR illustrating that the test data DTA includes optical camera OPC data DTA and / or radar RDR data DTA and / or LIDAR LDR data DTA.

[0058] The method starts with providing a first scenario FSC. Said first scenario FSC is changed to a second modified scenario SMS, wherein the example shows four different second modified scenario SMS. The original scene is changed to a scene with snow conditions (upper left UPL), with a road construction ahead (upper right UPR), to a scene with Asian style environment (lower left LWL) and to a Belgian style environment (lower right LWR). Carbrands and models are changed to adjust to the different road situations as to be expected.

[0059] The modification of test data DTA is made using Al technology AIT, wherein an input INT into the modification process comprises the sensor SNR data DTA of the first scenario FSC and specifications ODD for modifying the sensor SNR data DTA.

[0060] The Al technology AIT applied includes machine learning MLR and / or neural networks NRN to perform the modification of the test data DTA for autonomous driving. The specifications ODD for modifying the test data DTA of autonomous driving include parameters such as speed, position, vehicle behavior or road conditions or any other textual description for scenario modification. The Al technology AIT includes:

[0061] - image-to-image translation I IT and

[0062] - style transfer SYT or neural style transfer SYT by applying different styles and visual features VSF to the optical camera OPC data DTA and

[0063] - Object Segmentation OSG and Replacement RPL modifying the test data DTA through the segmentation and replacement RPL of certain objects in the optical camera OPC data DTA and - scene reconstruction RCN modifying said test data DTA by the partial reconstruction RCN of the data DTA, wherein the modifications include at least one of the following features: generating new 3D models, changing textures, and changing lighting conditions and

[0064] - Al-assisted 3D modeling and rendering techniques modifying the test data DTA through the application of advanced Al algorithms to generate and display 3D models of the vehicle's environment.

[0065] The specifications ODD is textual input, like for the upper left UPL example of the modified scenario this may be: “the traffic / obstacle cars are Mercedes cars, German street style and typical German road, Snowing conditions”.

[0066] An output OPT comprises vehicle sensor SNR data DTA according to the respective second modified scenario SMS, here four different second scenarios are illustrated.

[0067] The upper right UPR modification may serve as an example, wherein the first scenario FSC represents a specific traffic situation STS (no specific road blocking), and the modification of the test data DTA resulting in said second modified scenario SMS representing a modified traffic situation MTS - here, a road construction ahead.

[0068] The Al-technology comprises fitting certain visual features VSF in the vehicle sensor SNR data DTA, e.g., into the optical camera OPC data DTA. This can be seen in the upper right UPR modification, wherein the road blocking elements of the construction are inserted.

[0069] In case the test data DTA includes radar RDR data DTA the modifying may be about e.g., changing the distances and velocities of objects in the vehicle sensor SNR data DTA. When dealing with LIDAR LDR data DTA modifying may be about adjusting the spatial detection of objects in the vehicle sensor SNR data DTA.

[0070] As shown in figure 2 the modified test data DTA is used for autonomous driving to test and evaluate the performance and reliability of an advanced driver assistance system ADS in various scenarios.

[0071] Said device for modifying test data DTA comprises a data DTA processing unit PRU and a memory arrangement MMA to store the test data DTA. A communication channel CMC is provided to receive the test data DTA before an Al algorithm based on Al technology AIT analyses, transforms, and modifies the test data DTA. An Al-module AIM is configured to modifying the test data DTA from a first scenario FSC to a second modified scenario SMS using Al technology AIT. The modified test data DTA is provided via an output unit OPU as output OPT for use in autonomous driving testing.

Claims

Claims1. Method for modifying test data for autonomous driving wherein the test data (DTA) is vehicle sensor (SNR) data (DTA) for testing a driver assistance system (ADS), wherein the change of test data (DTA) is made from a first scenario (FSC) to a second modified scenario (SMS), wherein the test data (DTA) includes optical camera (OPC) data (DTA) and / or radar (RDR) data (DTA) and / or LIDAR (LDR) data (DTA), characterized in that the modification of test data (DTA) is made using Al technology (AIT), wherein an input (I NT) into the modification process comprises the sensor (SNR) data (DTA) of the first scenario (FSC) and specifications (ODD) for modifying the sensor (SNR) data (DTA), wherein an output comprises vehicle sensor (SNR) data (DTA) according to the second modified scenario (SMS).

2. Method for modifying test data according to claim 1 , wherein the first scenario (FSC) represents a specific traffic situation (STS), and the modification of the test data (DTA) results in a second modified scenario (SMS) representing a modified traffic situation (MTS).

3. Method for modifying test data according to claim 1 or 2, wherein the modification of the test data (DTA) includes optical camera (OPC) data (DTA) by fitting certain visual features (VSF) in the vehicle sensor (SNR) data (DTA).

4. Method for modifying test data according to one of the preceding claims, wherein the modification of the test data (DTA) includes radar (RDR) data (DTA) by modifying the distances and velocities of objects in the vehicle sensor (SNR) data (DTA).

5. Method for modifying test data according to one of the preceding claims, wherein the modification of the test data (DTA) includes LIDAR (LDR) data (DTA) by adjusting the spatial detection of objects in the vehicle sensor (SNR) data (DTA).

6. Method for modifying test data according to one of the preceding claims, wherein the Al technology (AIT) includes machine learning (MLR) and / or neural networks (NRN) to perform the modification of the test data (DTA) for autonomous driving.

7. Method for modifying test data according to one of the preceding claims, wherein the specifications (ODD) for modifying the test data (DTA) of autonomous driving include parameters such as speed, position, vehicle behavior or road conditions.

8. Method for modifying test data according to one of the preceding claims, the method comprising using the modified test data (DTA) for autonomous driving to test and evaluate the performance and reliability of an advanced driver assistance system (ADS) in various scenarios.

9. Method for modifying test data according to one of the preceding claims, wherein the Al technology (AIT) includes image-to-image translation (I IT) and the method comprises modifying the test data (DTA) for autonomous driving by transforming optical camera (OPC) data (DTA) into modified scenarios.

10. Method for modifying test data according to one of the preceding claims, wherein the Al technology (AIT) includes style transfer (SYT) or neural style transfer (SYT) and the method comprises modifying the test data (DTA) for autonomous driving by applying different styles and visual features (VSF) to the optical camera (OPC) data (DTA).11 . Method for modifying test data according to one of the preceding claims, wherein the Al technology (AIT) includes Object Segmentation (OSG) and Replacement (RPL) and the method comprises modifying the test data (DTA) for autonomous driving through the segmentation and replacement (RPL) of certain objects in the optical camera (OPC) data (DTA).

12. Method for modifying test data according to one of the preceding claims, characterized in wherein the Al technology (AIT) includes scene reconstruction (RCN) and the method comprises modifying said test data (DTA) for autonomous driving by the partial reconstruction (RCN) of the data (DTA), wherein the modifications include at least one of the following features: generating new 3D models, changing textures, and changing lighting conditions.

13. Method for modifying test data according to one of the preceding claims, wherein the Al technology (AIT) includes Al-assisted 3D modeling and rendering techniques and the method comprises modifying the test data (DTA) for autonomous driving through theapplication of advanced Al algorithms to generate and display 3D models of the vehicle's environment.

14. Method for modifying test data according to one of the preceding claims, wherein said specifications (ODD) are textual specifications (ODD).

15. A device for modifying test data (DTA) for testing an autonomous driving system, in particular for carrying out a method according to at least one of the previous claims, comprising: a data (DTA) processing unit (PRU) and a memory arrangement (MMA) to store test data (DTA) for autonomous driving, wherein the test data (DTA) is vehicle sensor (SNR) data (DTA) for testing a driver assistance system (ADS), wherein the change of test data (DTA) is made from a first scenario (FSC) to a second modified scenario (SMS), wherein the test data (DTA) includes optical camera (OPC) data (DTA) and / or radar (RDR) data (DTA) and / or LIDAR (LDR) data (DTA), characterized in an Al-module (AIM) comprising Al-technology (AIT) is configured to modifying the test data (DTA) from a first scenario (FSC) to a second modified scenario (SMS); a communication channel (CMC) to receive the test data (DTA); an Al algorithm based on Al technology (AIT), capable of analyzing, transforming, and modifying the test data (DTA); an output unit (CPU) to provide the modified test data (DTA) as output (OPT) for use in autonomous driving testing.

16. A device according to claim 15, wherein said Al-module (AIM) comprising Al- technology (AIT) is configured such that it includes at least one of image-to-image translation (I IT), style transfer (SYT), object segmentation (OSG) and replacement (RPL), scene reconstruction (RCN), Al-powered 3D modeling and rendering techniques.

Images