Method and system for generating scenarios for testing and validating automated driving functions

Abstract

The invention relates to a method for generating scenarios for testing and validating automated driving functions and / or an advanced driver assistance system (ADAS), wherein an automated driving function (15) is part of a first agent (10); comprising the method steps: - Simulating the behavior of the automated driving function (15) in a simulation environment defined by at least one test case (T i ) is defined with a scenario; - Changing the test case (T i ) into a modified test case (T j ) and / or modifying the simulation environment by a second agent (20), wherein the second agent (20) is designed as an adversarial agent in relation to the first agent (10); - Simulating the behavior of the automated driving function (15) in the modified simulation environment and generating simulation results (370); - Evaluating the simulation results (370) and generating evaluation results (470) and / or an evaluation index (490); - Saving the modified test case (T j ) and / or the modified simulation environment in a test database (200) if the evaluation results (470) and / or the evaluation index (490) exceed or fall below a predefined criterion, such as a threshold.

Description

[0001] The invention relates to a method and a system for generating scenarios for testing and validating automated driving functions and / or a driver assistance system (ADAS).

[0002] Automated driving features, also known as autonomous driving, refer to the ability of a vehicle to drive and navigate without direct human intervention. These features are achieved through the use of advanced sensors, computer systems, and algorithms that allow the vehicle to perceive its surroundings, make decisions, and control its movements. Vehicles with automated driving features typically use a combination of sensors such as radar, LIDAR (Light Detection and Ranging), cameras, and ultrasonic sensors. These sensors enable a detailed, real-time understanding of the vehicle's surroundings, including the detection of objects, pedestrians, other vehicles, and road conditions. The sensors collect data and feed it into sophisticated perception algorithms that interpret and analyze the information to create a comprehensive picture of the surroundings.This includes detecting and classifying objects, predicting their behavior, and identifying potential hazards or obstacles. Based on its perception of the environment, the vehicle's on-board computing system processes the data and, using complex algorithms, makes decisions about the actions to be taken. This includes determining the appropriate speed, driving position, and lane changes to navigate the road safely and efficiently. Once the decisions are made, the vehicle's control systems take over the execution of the actions. These control systems regulate the vehicle's acceleration, braking, and steering, follow the specified route, or react to real-time conditions and traffic situations. Automated driving functions often rely on high-resolution maps and precise localization techniques to precisely position the vehicle on the road.The maps provide detailed information about lane markings, traffic signs, and other relevant features, while the localization systems use a combination of GPS, inertial measurement devices, and other sensors to determine the vehicle's exact position.

[0003] The ADAC has defined five levels of automation, ranging from Level 1 (assisted driving) to Level 5 (autonomous driving). These levels categorize the degree of human involvement and the capabilities of the automated driving system. Lower levels may require driver supervision and intervention, while higher levels enable fully automated operation.

[0004] The higher the level of automation, the greater the requirements for functional safety. To verify and validate automated driving functions, extensive tests are conducted in various traffic scenarios to ensure the safety, reliability, and efficiency of the system. However, due to the multitude of possible traffic scenarios, it is not possible to conduct real road tests under real traffic conditions for all conceivable traffic scenarios. Therefore, virtual simulation tests are increasingly being carried out using modern software tools. Simulations enable controlled testing in a virtual environment, allowing different scenarios and the system's response to be evaluated without causing real incidents. Simulations can be used to simulate various traffic situations, including collision scenarios and rare events that are difficult to reproduce in real tests.This requires the development of a wide range of test scenarios covering various traffic situations. These scenarios can include highway driving, urban environments with intersections, roundabouts, pedestrian crossings, construction sites, harsh weather conditions, and challenging road geometries. Each scenario should test specific functions and capabilities of the automated driving system.

[0005] During the simulation, extensive data is collected from various sensors and vehicle systems. This data includes sensor inputs, system outputs, vehicle dynamics, and environmental conditions. The collected data is crucial for analyzing system behavior, identifying potential problems, and improving algorithms and decision-making processes. Metrics and safety criteria are defined to evaluate system performance, such as detection accuracy, decision quality, response time, robustness to various scenarios, and the ability to handle failures or unexpected events. The system's performance is compared against defined benchmarks and requirements. Based on the test results, the system's algorithms, software, and hardware components are iteratively developed and refined.The insights gained from the tests are used to improve the performance of the system, fix identified vulnerabilities, and increase security and reliability.

[0006] Conducting extensive and diverse tests in diverse traffic scenarios is therefore crucial for verifying and validating the performance and safety of automated driving functions. These tests help identify limitations, refine algorithms, and build confidence in the capabilities of automated driving functions before they are deployed in a public transport vehicle.

[0007] However, it is important to develop a large pool of diverse test scenarios, as only a diverse set of test scenarios ensures comprehensive coverage of the various real-world conditions, situations, and edge cases. This helps identify potential problems, vulnerabilities, or limitations that may not be visible in a limited number of scenarios. A broad range of possible scenarios enables comprehensive testing to evaluate the reliability, robustness, performance, functionality, and responsiveness of an automated driving function. A large pool of test scenarios helps detect vulnerabilities, program errors, or unexpected behavior. By exposing an automated driving function to different scenarios, developers can identify potential problems that may arise under certain conditions.This results in more robust, reliable and adaptable automated driving functions that can effectively meet the needs of different users in a variety of scenarios.

[0008] Various methods are known for expanding the pool of available scenarios. One possibility, for example, is the iterative modification of parameters and parameter values ​​that describe a scenario. Another possibility is the use of two adversarial AI agents to generate new traffic scenarios.

[0009] US 11 443 130 B2 discloses test coverage for automated driving functions of a motor vehicle by two adversarial artificial intelligences in a simulation of a test case. A first agent A1 is equipped with the function to be tested and pursues a goal. The second adversarial agent A2 pursues the goal of disrupting agent A1's function. The adversarial agent A2 receives a reward if it causes agent A1 to fail. Test cases that cause agent A1 to fail are stored in a memory. New test cases are simulated based on their similarity to the first test case and saved as a new test case if a similarity threshold is exceeded.

[0010] DE 10 2018 128 290 A1 discloses a method for evaluating the driving performance of an automated driving function in a variety of driving situations and driving conditions, wherein the driving scenarios are changed by a parametric variation, such as weather, road users.

[0011] The invention is based on the object of creating possibilities for improving the generation of new, previously unknown traffic scenarios for testing and verifying automated driving functions in a simulation in order to save computing capacity for carrying out simulations and to carry out a more precise validation, verification or calibration of the automated driving function.

[0012] This object is achieved according to the invention with respect to a method by the features of patent claim 1, with respect to a system by the features of patent claim 11, and with respect to a computer program product by the features of patent claim 15. The further claims relate to preferred embodiments of the invention.

[0013] The present invention can provide a pool of meaningful scenarios for testing and validating an automated driving function. This ensures that automated driving functions can be tested in a broad spectrum of real-world driving situations, such as different road conditions, traffic scenarios, weather conditions, and potential hazards. This coverage increases the reliability of system performance in real-world environments. Furthermore, the use of the second agent can identify limitations or weaknesses of the automated driving function by detecting scenarios in which the first agent or automated driving function struggles or does not respond effectively. This identification helps to refine and improve the algorithms and functions of the automated driving function in order to increase its performance and eliminate potential safety issues.By generating new requirement scenarios, potential areas for improvement can be identified and adaptability to changing driving conditions and new technologies can be improved. This future-proofing ensures that automated driving functions remain functional in the face of changing environments and requirements.

[0014] The present invention can thus improve the reliability, safety and efficiency of automated driving functions, but also of driver assistance systems in different driving scenarios and contribute to their continuous improvement and adaptation.

[0015] The present invention provides a method for generating scenarios for testing and validating automated driving functions and / or a driver assistance system (ADAS). A scenario is a traffic event in a chronological sequence and is defined by a selection of parameters and associated parameter values. An automated driving function is part of a first agent. The method comprises the following steps: - Simulating the behavior of the automated driving function in a simulation environment defined by at least one test case with a scenario; - Changing the test case into a modified test case and / or changing the simulation environment by changing parameters and parameter values ​​by a second agent, wherein the second agent is designed as an adversarial agent to the first agent; - Simulating the behavior of the automated driving function in the modified simulation environment, which was modified directly or by the modified test case, and generating simulation results; - Evaluating the simulation results and generating evaluation results and / or an evaluation index; - Saving the modified test case and / or the modified parameters and / or parameter values ​​of the simulation environment in a test database if the evaluation results and / or the evaluation index exceed or fall below a specified criterion, such as a threshold value.

[0016] In a further development, it is provided that the second agent changes the parameters and parameter values ​​using sensor data that is stored in a parameter database and to which the second agent can access.

[0017] In an advantageous embodiment, the sensor data includes measurement signals from measurement systems such as cameras, LiDAR sensors, radar sensors, ultrasonic sensors, acceleration sensors, speed sensors, torque sensors, humidity sensors and / or temperature sensors, GPS data, weather data, traffic data, environmental data such as road geometry, arrangement of buildings and vegetation, as well as data on the behavior of road users such as pedestrians, cyclists and vehicles.

[0018] In a further embodiment, it is provided that the test cases modified by the second agent and the modified simulation environment are evaluated by a third agent with regard to compliance with system limits such as a maximum and minimum speed.

[0019] In particular, it is intended that the test cases modified by the second agent and the modified simulation environment are evaluated using a loss function with regard to compliance with system limits such as a maximum and minimum speed.

[0020] Advantageously, it is provided that the first agent, the second agent and the third agent are designed as models with artificial intelligence algorithms (AI models), wherein the algorithms comprise at least one neural network.

[0021] In an advantageous embodiment, the neural network is designed as a convolutional neural network (CNN) or as a deep neural network.

[0022] Advantageously, reinforcement learning algorithms are provided for training the behavior of the first agent and / or the second agent and / or the third agent.

[0023] In particular, the system boundaries are defined by system values ​​stored in the parameter database, whereby at least the third agent can access the system values.

[0024] In one further training course, a 3D open-world simulation is planned.

[0025] According to a second aspect, the invention provides a system for generating scenarios for testing and validating automated driving functions and / or a driver assistance system (ADAS). A scenario is a traffic event in a temporal sequence and is defined by a selection of parameters and associated parameter values. The system comprises a first agent, a second agent, a test database, a simulation module, an evaluation module, and a parameter database, wherein an automated driving function is part of the first agent, and wherein the second agent is configured as an adversarial agent to the first agent. The simulation module is configured to simulate the behavior of the automated driving function in a simulation environment defined by at least one test case with a scenario.The second agent is configured to transform the test case into a modified test case and / or the simulation environment by modifying parameters and parameter values. The simulation module is configured to simulate the behavior of the automated driving function in the modified simulation environment, which was modified directly or by the modified test case, and to generate simulation results. The evaluation module is configured to evaluate the simulation results and generate evaluation results and / or an evaluation index. The test database is configured to store the modified test case and / or the modified parameters and / or parameter values ​​of the simulation environment if the evaluation results and / or the evaluation index exceed or fall below a predefined criterion, such as a threshold value.

[0026] In a further development, it is provided that the second agent changes the parameters and parameter values ​​using sensor data that is stored in a parameter database and to which the second agent can access, wherein the sensor data includes measurement signals from measurement systems such as cameras, LiDAR sensors, radar sensors, ultrasonic sensors, acceleration sensors, speed sensors, torque sensors, humidity sensors and / or temperature sensors, GPS data, weather data, traffic data, environmental data such as road geometry, arrangement of buildings and vegetation as well as data on the behavior of road users such as pedestrians, cyclists and vehicles.

[0027] In an advantageous embodiment, it is provided that the test cases modified by the second agent and the modified simulation environment are evaluated by a third agent or a loss function with regard to compliance with system limits such as a maximum and minimum speed.

[0028] In a further embodiment, it is provided that the first agent, the second agent and the third agent are designed as models with artificial intelligence algorithms (AI models), wherein the algorithms comprise at least one neural network which is designed as a convolutional neural network (CNN) or as a deep neural network, and wherein reinforcement learning algorithms are provided for training the behavior of the first agent and / or the second agent and / or the third agent.

[0029] According to a third aspect, the invention provides a computer program product comprising executable program code configured to carry out the method according to the first aspect when executed.

[0030] The invention is explained in more detail below with reference to embodiments shown in the drawing.

[0031] It shows: Fig. 1 is a block diagram illustrating an embodiment of a system according to the invention; Fig. 2 a flow chart explaining the individual method steps of a method according to the invention; Fig. 3 is a block diagram of a computer program product according to an embodiment of the third aspect of the invention.

[0032] Additional features, aspects and advantages of the invention or embodiments thereof will become apparent from the detailed description taken in conjunction with the claims.

[0033] The Fig. The system 100 according to the invention shown in Figure 1 comprises a first agent 10, a second agent 20, a test bank module 200, a simulation module 300, an evaluation module 400 and a parameter database 500. In addition, the system 100 can comprise a third agent 3.

[0034] The first agent 10, the second agent 20, the third agent 30, the test bank module 200, the simulation module 300, the evaluation module, and the parameter database 500 can be implemented as standalone computing units or as a cloud-based solution. In particular, they can each be provided with a processor and / or a memory unit.

[0035] In the context of the invention, a “processor” can be understood to mean, for example, a machine or an electronic circuit. A processor can in particular be a main processor (Central Processing Unit, CPU), a microprocessor or a microcontroller, e.g. an application-specific integrated circuit or a digital signal processor, optionally in combination with a memory unit for storing program instructions, etc. A processor can also be a virtualized processor, a virtual machine or a soft CPU. It can also be, for example, a programmable processor that is equipped with configuration steps for carrying out the aforementioned method according to the invention or is configured with configuration steps such that the programmable processor implements the inventive features of the method, the modules or other aspects and / or sub-aspects of the invention.In particular, the processor can contain highly parallel computing units and powerful graphics modules.

[0036] In the context of the invention, a "storage unit" or "storage module" and the like can be understood to mean, for example, a volatile memory in the form of random access memory (RAM), a permanent memory such as a hard disk or a data storage device, or, for example, a removable storage module. The storage module can also be a cloud-based storage solution.

[0037] In the context of the invention, a "module" can be understood, for example, as a processor and / or a memory unit for storing program instructions. For example, the processor is specifically configured to execute the program instructions in such a way that the processor and / or the control unit performs functions to execute or implement the method according to the invention or a step of the method according to the invention.

[0038] The term "database" refers to both a storage algorithm and the hardware in the form of a storage unit. In particular, the test database 200 and the parameter database 500 can be implemented as a cloud computing infrastructure.

[0039] In particular, it is intended that the three agents 10, 20, 30, the simulation module 300 and the evaluation module 400 are integrated in a cloud computing infrastructure.

[0040] A cloud computing infrastructure offers the ability to expand or reduce resources as needed, allowing computing power, storage space, or network resources to be easily adapted to changing requirements. This scalability enables cost optimization and efficient resource allocation without large investments in hardware. Furthermore, users can access applications and data from anywhere with internet access. This accessibility enables collaboration and seamless integration across different devices and locations. Cloud computing also offers flexibility in software deployment, allowing companies to deploy and update applications quickly and without disruption.

[0041] In particular, cloud computing infrastructure is known for its high computing efficiency, as cloud providers maintain large data centers with a vast pool of computing resources. This resource pooling allows multiple users and applications to share and efficiently utilize the same underlying infrastructure. By consolidating resources and dynamically allocating them as needed, cloud providers can achieve higher resource utilization and avoid resource waste. Furthermore, cloud computing relies heavily on virtualization technologies such as virtual machines (VMs) or containers. Virtualization allows the partitioning of physical resources into virtual instances, allowing multiple workloads to run concurrently on the same hardware. This virtualized environment enables efficient resource allocation, scalability, and isolation, leading to greater computing efficiency.Furthermore, cloud infrastructure offers elasticity, allowing users to scale resources up or down as needed. This scalability ensures that computing resources can be quickly provisioned and removed to meet workload demands. By dynamically adjusting resources to current needs, cloud computing optimizes resource utilization, avoids congestion, and reduces unnecessary costs. Furthermore, cloud providers use load balancing mechanisms to distribute the workload evenly across multiple servers or instances. Load balancing ensures that computing resources are used efficiently and avoids resource bottlenecks or overloading of individual servers. Through intelligent routing and load balancing, cloud infrastructure optimizes resource utilization and ensures high computing efficiency.The hardware utilizes energy-efficient processors and high-density memory systems. These hardware optimizations reduce power consumption, increase performance, and improve overall computing efficiency.

[0042] Since simulations are highly computationally intensive, high IT efficiency with minimization of resource and energy consumption, as well as computing time, while simultaneously maximizing output is important. The present invention enables resource and energy consumption to be reduced through the use of adversarial agents 10, 20, and 30, as relevant scenarios can be found more quickly, thus enabling the testing and verification of automated driving functions to be carried out more quickly and efficiently. Reducing the number of simulations results in, in particular, energy savings, thereby reducing environmental impact and improving sustainability. Since simulations process large amounts of data, the development of resource-efficient applications is particularly important.

[0043] The present invention is based on a concept from the field of artificial intelligence (AI) in which an adversarial AI agent attacks another AI agent. Adversarial attacks are deliberate and malicious attempts to manipulate or deceive AI systems by exploiting vulnerabilities in their design or algorithms. In these attacks, carefully crafted adversarial inputs are introduced with the intent of misleading the AI ​​system's decision-making process. The attacks can be directed against various AI applications, such as image recognition, natural language processing, or autonomous vehicles, and aim to cause misclassifications, incorrect results, or security vulnerabilities. In adversarial machine learning, vulnerabilities of AI systems and their defenses against attacks are deliberately investigated.Adversaries are agents who aim to compromise the performance or integrity of AI models. These attackers can use techniques such as data poisoning, model evasion, or model extraction to undermine the effectiveness or reliability of AI systems. The goal of adversarial machine learning is to develop robust models and algorithms capable of resisting and mitigating adversarial attacks to protect the AI ​​system from such attacks.

[0044] The first agent 10, the second agent 20, and the third agent 30 are each designed as an AI model. The AI ​​model specifically includes neural networks, but can also include other types of machine learning. In an image processing application for object recognition, such as is important for an automated driving function, a combination of neural networks and machine learning techniques is used.

[0045] A neural network consists of neurons arranged in multiple layers and connected to each other in various ways. A neuron is able to receive information at its input from outside or from another neuron, evaluate this information in a specific way, and pass it on in a modified form at the neuron's output to another neuron or output it as the final result. Hidden neurons are located between the input neurons and the output neurons. Depending on the type of network, there can be several layers of hidden neurons. They ensure the forwarding and processing of information. Output neurons finally provide a result and pass it on to the outside world. The arrangement and interconnection of the neurons gives rise to different types of neural networks, such as feed-forward networks, recurrent networks, or convolutional neural networks.The networks can be trained through unsupervised or supervised learning.

[0046] Specifically, the Kl model is based on a convolutional neural network (CNN), which is well-suited for image analysis tasks. CNNs consist of multiple layers of interconnected neurons, including convolutional layers for feature extraction and pooling layers for spatial subsampling. Using these layers, the model learns hierarchical representations of image features. To train the model, a large dataset of annotated images is required. This dataset contains images of various objects with their corresponding class labels. The training data plays a crucial role in training the model to correctly recognize different object categories. The neural network model goes through a training phase in which it learns to recognize objects from the provided annotated images.The training data is fed into the network, the weights and biases of the neurons are adjusted through backpropagation, and the model parameters are optimized using algorithms such as stochastic gradient descent or its variants. Transfer learning can also be used to further improve model performance. Transfer learning uses pre-trained models trained on large datasets such as ImageNet. The knowledge and properties of the previously trained model can be used as a starting point for the current task, leading to faster convergence and improved accuracy. In addition to neural networks, other machine learning techniques can also be used for feature extraction.For example, techniques such as scale-invariant feature transformation can extract informative features from images, which can then be used as input to the neural network. These techniques improve the model's ability to capture fine-grained details and distinguish between different objects.

[0047] The AI ​​model may also incorporate reinforcement learning (RL) algorithms. Reinforcement learning can be used to train an agent to make appropriate decisions in an environment. The AI ​​model learns to perform actions that maximize a reward signal over a given period of time. By interacting with the environment, the agent tries out different actions and learns from the feedback it receives in the form of rewards or punishments. The goal of the model is to find an optimal strategy that maximizes the expected long-term gain. RL enables the AI ​​model to adapt and optimize its behavior in dynamic environments. By constantly interacting with the environment, the model can adapt its decision-making process based on the observed rewards and changes in the environment.This adaptability enables the model to handle situations in which the environment is subject to fluctuations, uncertainties, or temporal changes. A Markov decision process is preferably used as the reinforcement learning algorithm. However, a temporal difference learning (TD learning) algorithm can also be used. With the TD learning algorithm, actions are adjusted not only when the agent receives the reward, but after each action based on an estimate of the expected reward. Algorithms such as Q-learning and SARSA, actor-critic, or Monte Carlo simulations are also conceivable, allowing for dynamic programming and strategy adaptation through iterative procedures.

[0048] In a further development, the AI ​​model can also include deep reinforcement learning algorithms, which combine reinforcement learning with deep neural networks. Deep RL algorithms such as Deep Q-Networks (DQN) or Proximity Policy Optimization (PPO) use deep neural networks as function approximators to handle high-dimensional state and action spaces.

[0049] In addition to an AI model, agents 10, 20, and 30 each contain algorithms for accessing the test database 200 and / or the parameter database 500 to obtain the information and data required for a specific task.

[0050] The first agent 10 comprises at least one automated driving function 15, which is tested in the simulation module 300. Within the scope of the invention, the automated driving function 15 can also be embodied as an Advanced Driver Assistance System (ADAS).

[0051] An ADAS system refers to a set of technologies and features that assist the driver while driving a vehicle. These systems use sensors, algorithms, and computer-based components to enhance safety, comfort, and convenience. ADAS features include adaptive cruise control, lane-keeping assist, automatic emergency braking, blind-spot detection, and parking assistance, among others. ADAS features are primarily designed to assist the driver but also require their active participation and attention. An automated driving feature, often referred to as autonomous driving or self-driving, goes beyond ADAS assistance. It refers to the ability of a vehicle to drive and navigate without constant intervention from a human driver.

[0052] Automated driving functions use a combination of sensors, machine vision, artificial intelligence, and decision-making algorithms to perceive the environment, make driving decisions, and control the vehicle. The transition from an ADAS system to an automated driving function thus concerns the increasing degree of autonomy at which the vehicle can operate without constant driver intervention.

[0053] The behavior of the automated driving function 15 is simulated in the simulation module 300 in a simulation environment, wherein the simulation environment is defined by the scenario of a test case T i is determined.

[0054] A scenario is a traffic event in a temporal sequence, such as crossing a motorway bridge, turning into a turning lane, driving through a tunnel, entering a roundabout, or stopping at a pedestrian crossing. An overtaking maneuver, also known as a cut-in scenario, can be described as a scenario in which a first vehicle initially follows another vehicle, then changes lane and increases speed to overtake the other vehicle.

[0055] The scenario is typically defined by parameters and parameter values. An example of a scenario parameter is a physical quantity, a chemical quantity, a torque, a rotational speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person, or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographical feature such as a gradient, a time of day, a temperature, a precipitation value, and / or a season.

[0056] The test cases T i are stored in the test database 200. The data for a test case T itypically contain a test identification number (test ID), a user name, a creation time, a name of the underlying parameterized scenario, parameters, and parameter values. The test cases T i are typically derived from data from real and / or virtual traffic scenarios, which are processed with suitable extraction and classification algorithms.

[0057] The first agent 10 selects a test case T i from the test database 200 and passes it on to the simulation module 300. The selection of the test case T i can be based on a decision of the first agent 10 or the test case T to be selected i is selected by a user via a user interface to which agent 10 is connected.

[0058] The simulation module 300 comprises a software application 350 for carrying out a simulation of the behavior of the automated driving function 15 or another system to be tested, such as in particular a driver assistance system (ADAS), in a test case T i specified simulation environment. The simulation module 300 is particularly designed to perform a 3D open-world simulation.

[0059] A 3D open-world simulation is a computer-generated virtual environment that represents a three-dimensional world in which various objects, entities, and interactions can take place. It is characterized by its openness, which allows users or agents to freely explore and interact with the simulated world without predetermined paths or restrictions. The simulation represents a virtual world that is usually rendered three-dimensionally using computer graphics techniques. This allows for a realistic representation of objects, buildings, landscapes, and other elements within the simulation. The users or agents in the simulation have the freedom to move around the virtual world and explore it in a non-linear manner.They aren't restricted by predefined paths or fixed objectives and can move freely and interact with various elements and activities within the simulation. The goal of an open-world 3D simulation is often to convey a sense of realism and immersion in the virtual world. This includes realistic physics, graphics, sound effects, and interactions with the environment to make the virtual world believable and engaging.

[0060] The simulation also enables interactions between objects and entities in the virtual world. These can be physical interactions such as collisions, object manipulation, or environmental interactions such as weather effects or day-night cycles. These interactions can, in turn, be controlled by simulated physics and rules. In particular, a 3D Open Dynamics simulation can represent events and changes over time, where objects can have properties that can change, and the state of the world can change depending on various factors or user actions. The users or agents participating in the simulation have a certain degree of freedom of action, allowing them to make decisions, influence the environment, and affect outcomes.This can include completing tasks or missions, creating or building structures, interacting with other people or objects, and performing various activities within the simulated world. Open-world simulations provide a dynamic, interactive, and immersive environment for users or agents to explore, learn, and master complex systems and scenarios.

[0061] During the simulation, simulation results 370 are generated using various data. This includes synthetic sensor data representing the inputs received by the sensors of the automated driving function 15. This can be data from cameras, LiDAR (Light Detection and Ranging) sensors, radar sensors, or other relevant sensors. The simulated sensor data provides information about the environment, including the position, speed, and properties of objects such as vehicles, pedestrians, and obstacles. In addition, the simulation provides control data representing the commands or actions generated by the automated driving function 15 in response to the sensor inputs. These can be steering commands, acceleration or deceleration commands, braking signals, or other control signals that influence the vehicle's behavior in the simulated scenario.This can include information such as road layout, traffic patterns, weather conditions, lighting conditions, or specific events that occur during the simulation. Scenario data helps evaluate the performance of the automated driving function 15 in different scenarios and assess its robustness and effectiveness under different conditions. In particular, event data is generated that records specific events or interactions during the simulation. This can include events such as vehicle-to-vehicle communication, pedestrian crossings, lane changes, traffic signal changes, or other important events relevant to the simulated scenario. Event data provides insight into how the automated driving function 15 reacts to certain events and can assist in analyzing behavior and decision-making processes.In addition, visualization data can be generated to provide a visual representation of the simulated environment and the performance of the automated driving function. This can be rendered images or videos showing the simulated scene, the behavior of objects, and the actions performed by the automated driving function 15. The visualization data helps to visually understand and analyze the performance of the automated driving function 15.

[0062] The simulation results 370 generated in a simulation are transferred to the evaluation module 400. The evaluation module 400 comprises evaluation software 450 for comprehensive analysis and data testing of the simulation results 370 in order to generate evaluation results 470 therefrom. In this process, the performance of the automated driving function 15 is determined in the test case T idefined traffic scenario. Furthermore, potential areas for improvement can be identified to ensure the reliability and safety of the automated driving function 15 before practical use.

[0063] In particular, the simulation results 370 are evaluated using performance metrics, particularly in the form of key performance indicators. Performance metrics can be defined in relation to safety, efficiency, accuracy, or other relevant aspects. For example, key performance indicators for collision avoidance rate, reaction time, distance to obstacles, or trajectory accuracy can be determined in order to evaluate the performance of the automated driving function 15 in various scenarios. The evaluation results 470 thus generated serve in particular to validate and verify the performance of the automated driving function 15.

[0064] According to the invention, from the totality of the evaluation results 470 for a specific test case T i an evaluation index 490 is determined, which serves as a measure of the performance of the automated driving function 15 in the respective test case T i For example, the rating index 490 is a number on a scale of 0 to 100. However, a different scale may also be provided.

[0065] The evaluation results 470 and / or the evaluation index 490 are transferred to the test database 200 and compared with the corresponding test case T i linked.

[0066] In particular, it is intended that the AI ​​model of the first agent 10 be equipped with reinforcement learning algorithms, so that the first agent 10 receives a reward for a high evaluation index of 490. The first agent 10 can be trained to optimize its behavior using this reward.

[0067] However, the level of the rating index 490 depends on the difficulty of the underlying scenario. For example, it is not very difficult for an automated driving function 15 to master an easy scenario and receive a high rating index of 490. However, the situation is different for critical scenarios or so-called corner cases.

[0068] Critical scenarios or corner cases refer to situations that are rare, difficult, or extreme, but have a significant impact on the performance, safety, or effectiveness of an automated driving function. One example is adverse weather conditions. By simulating scenarios with heavy rain, fog, blizzards, or poor visibility, the ability of the automated driving function 15 to accurately perceive the environment and respond appropriately can be tested. These conditions can put the system's sensor performance, object recognition, and decision-making capabilities to the test. Low light conditions are another example of critical scenarios. Testing the performance of an automated driving function 15 in low-light scenarios, such as at night, at dusk, or at sunrise, can be crucial.These scenarios can affect the performance of cameras, LiDAR, and other sensors, making object detection difficult.

[0069] Added to this is the unpredictable behavior of pedestrians and cyclists. Simulating scenarios in which pedestrians or cyclists exhibit unpredictable behavior, such as suddenly crossing the roadway, crossing without appropriate signals, or unpredictable movements, helps evaluate the ability of the automated driving function 15 to detect and respond to vulnerable road users in challenging situations. Furthermore, the influence of road geometry, e.g., at complex intersections, plays a major role. Testing the behavior of the automated driving function 15 at busy and complex intersections with multiple lanes, traffic lights, pedestrian crossings, and crossing vehicles is crucial. These scenarios require accurate perception, efficient decision-making, and precise path planning to ensure safe and smooth navigation.

[0070] Then there are construction site areas that are difficult to manage. Road construction site simulations assess the ability of the automated driving function 15 to navigate through changing lane configurations, temporary signage, traffic cones, and construction vehicles. These scenarios require adaptability, object recognition, and the ability to understand and follow temporary traffic regulations.

[0071] Furthermore, sensor failures must not be neglected. Simulating scenarios in which one or more sensors fail or malfunction is important for testing the redundancy and fallback mechanisms of the automated driving function 15 or the driver assistance system. The automated driving function 15 should be able to detect and manage sensor failures to ensure the safety and functionality of the vehicle.

[0072] Another challenge is object detection. The creation of scenarios with difficult-to-detect objects or obstacles, such as low-contrast objects, partially obscured objects, or objects with unusual shapes, is also an important aspect for evaluating the robustness and accuracy of the automated driving function 15 in object detection and object tracking.

[0073] In addition, unusual traffic situations such as wrong-way drivers, sudden lane changes, or aggressive driving maneuvers frequently occur in road traffic. In these scenarios, the ability of the automated driving function 15 to handle unusual traffic behavior and make appropriate decisions is assessed.

[0074] Sensor malfunctions such as electromagnetic interference or interference from neighboring vehicles or infrastructure are less common, but must also be considered. In addition, wear and tear such as sensor degradation, contamination, or computational bottlenecks can occur.

[0075] There are therefore a whole series of critical scenarios or corner cases that an automated driving function must master, many of which occur with only a very low probability, so typical test cases do not reflect all possible critical scenarios. However, these critical scenarios are crucial for determining whether the robustness, safety, and reliability of an automated driving function are sufficient to function reliably in the multitude of difficult real-world situations.

[0076] According to the invention, the second agent 20 is designed as an adversary agent that attempts to find gaps and weak points in the behavior of the first agent 10 or the automated driving function 15. For example, the automated driving function 15 of the first agent 10 is designed as a parking function that is trained to approach a parking space in the shortest possible time. The second agent 20, on the other hand, aims to block this parking function of the first agent 10.

[0077] According to the invention, the adversary agent 20 now creates a scenario by deliberately influencing and modifying the vehicle's environment to test the robustness, safety, or vulnerabilities of this parking function. This also applies to other automated driving functions 15. The new scenarios created by the modification are intended to challenge the performance of the automated driving function 15 and reveal vulnerabilities.

[0078] Another example relates to the training data set of the first agent 10. It is possible that the automated driving function 15 of the first agent 10 was trained with a data set that focused on images of European roads. The second agent 20 now changes the test cases T i or directly in the simulation, the environment of the vehicle, so that the automated driving function 15 has difficulty making the right decisions. Accordingly, the simulation results 370 are evaluated by the evaluation module 400 with a lower evaluation index 490. The lower the evaluation index 490, the more successful the strategy of the second agent 20 for changing the selected test case T i or the simulation environment.

[0079] To change the scenarios, the second agent 20 also has access to the test database 200 with the test cases T iand the evaluation results 470 or the respective evaluation index 390. In addition, the second agent 20 has access to the parameter database 500, in which sensor data 520 and system values ​​570 are stored.

[0080] The sensor data 520 is data from various sensors and detection systems. The sensor data 520 includes, in particular, the camera data recorded by a vehicle camera. This data comprises images or video sequences depicting a traffic scene from the camera's perspective. It can contain information about objects, lanes, traffic signs, traffic lights, and road markings. Further sensor data 520 is LiDAR data. This is 3D point cloud data generated by emitting laser beams and measuring their reflections. The LiDAR data provides information about the distances, positions, and shapes of objects in the environment, allowing the automated driving function 15 to perceive the environment in three dimensions. Radar data can also be provided. Radar data contains information about the distance, speed, and sometimes also the angle of the detected objects.Radar data helps detect and track objects, estimate their speed, and assess their proximity to the vehicle. Sensor data 520 may also include ultrasonic data. This represents the distance between the vehicle and nearby objects or obstacles. Ultrasonic sensors are often used for parking assistance and distance detection. Furthermore, GPS data may be provided for determining the vehicle's position and navigation. Additionally, other data from acceleration sensors, speed sensors, torque sensors, humidity sensors, temperature sensors, etc. may be included in sensor data 520. Environmental data may also be provided, which includes information about lighting conditions, weather conditions (such as rain, fog, or snow), road surface conditions, or other factors such as vegetation or development that may influence the performance of automated driving function 15.

[0081] For this purpose, the second agent 20 changes the test cases T i significant by incorporating sensor data 520 into at least some of the test cases T i so that now modified test cases T j In addition, the second agent 20 can directly intervene in the simulation and change the simulated environment, e.g., the type of vegetation, the number of buildings, the behavior of pedestrians and other vehicles, etc. This creates new scenarios that must be mastered by the automated driving function 15. In addition, data poisoning can be used, in which the second agent 20 inserts malicious or misleading data into the test cases T i or the simulation environment to disrupt the behavior or performance of the first agent 10.

[0082] The test cases T modified by the second agent 20 j or the test cases T that can be derived from the change in the simulation environmentj , which received a low evaluation index 490 due to weak or mediocre evaluation results 470, are stored in the test database 200, as they reflect critical scenarios that are difficult for the automated driving function to handle. The test database 200 can further be equipped with modification software 250, which is designed to, based on the test cases T modified by the agent 20, j further test cases T k so that even without the action of the second agent 20, the pool of available scenarios, which particularly cover critical traffic situations, is increased. In particular, the modification software 250 determines the parameters and parameter values ​​of a new test case T j modified iteratively, so that in this way a wide range of further test cases T k can be generated.

[0083] In particular, the AI ​​model of the second agent 20 is equipped with reinforcement learning algorithms, so that the second agent 20 receives a reward for successfully intervening against the activities of the first agent 10. The second agent 20 is thus trained by the reward to optimize its behavior.

[0084] In a further development of the invention, a third agent 30 is provided, which acts as a kind of supervisor and monitors the behavior of the second agent 20. For this purpose, the third agent 30 has access to the system values ​​550 stored in the parameter database 500. The system values ​​550 can be stored in the form of numbers, images, graphics, time series, parameters, etc. For example, target and target values ​​that define a safety standard can be specified.

[0085] These system values ​​550 define the system limits for an automated driving function 15, which are crucial for safe and reliable operation. For example, setting a predefined maximum speed is a crucial limit to ensure that the automated driving function 15 operates within safe speed ranges. The same applies to a minimum speed limit, which helps ensure that the automated driving function 15 functions properly within a certain range. This limit prevents the automated driving function 15 from activating or using certain functionalities if the vehicle speed drops below the specified limit. In addition, environmental conditions can be specified, such as defining acceptable weather conditions, road types (e.g.Highways, city streets) or geographical boundaries to restrict the activation of the automated driving function 15 in unsuitable or unsupported environments. Another aspect concerns the definition of the coverage and range of the sensors of the automated driving function 15. This establishes the limits of the sensor functionality and ensures that the automated driving function 15 can accurately detect objects within the specified area. Outside the specified boundaries, the automated driving function 15 may need to resort to alternative methods or indicate limited functionality to the driver.In addition, scenarios can be defined in which certain functionalities of the automated driving function 15 can be deactivated or their effectiveness restricted, for example in adverse weather conditions, in certain traffic scenarios or in the presence of certain objects or road conditions.

[0086] In particular, setting limits for the design of the human-machine interface (HMI) is crucial to ensure safe and effective interaction between the driver and the automated driving function 15. This includes setting limits on the type and frequency of warnings, notifications, or requests for driver intervention to avoid driver overload or distraction.

[0087] When considering the system boundaries of an automated driving function, ethical aspects must also be considered. Safety and risk minimization should be paramount when defining the system boundaries. The automated driving function should be designed to avoid dangerous situations and minimize the likelihood of accidents or injuries to the driver, vehicle occupants, pedestrians, and other road users. The automated driving function should have reliable mechanisms to appropriately handle critical situations or incidents. Furthermore, the social and environmental impacts of the automated driving function must be considered. This includes considering potential impacts on traffic patterns, energy consumption, infrastructure requirements, and sustainability in general.System boundaries should aim to minimize negative impacts and contribute to a more sustainable transport ecosystem. Incorporating these ethical considerations into the definition of system boundaries can ensure that the technology is developed and deployed responsibly.

[0088] Establishing these system boundaries helps ensure that the automated driving function 15 operates within safe and reliable parameters. This allows the automated driving function 15 to effectively support the driver while preventing it from intervening in situations that exceed its capabilities or could compromise safety.

[0089] The third agent 30 checks whether the results obtained by the second agent 20 in the test cases T ior changes made directly in the simulation environment comply with the system boundaries defined in the system values ​​550. In particular, the AI ​​model of the third agent 30 is equipped with reinforcement learning algorithms, so that the third agent 30 receives a reward for successfully intervening against the activities of the second agent 20. The third agent 30 is thus trained by the reward to optimize its behavior. In a further development, it can be provided that the system values ​​550 are changed by the third agent 30 depending on the reward, so that in this way the system boundaries of the automated driving function are generated in a self-learning manner.

[0090] By monitoring the second agent 20 by the third agent 30, it can be ensured that the adversary scenarios are created in such a way that responsible and safe testing procedures are guaranteed, since unrealistic and / or unethical scenarios are excluded.

[0091] However, it is also possible to determine a loss function for the behavior of the second agent 20 instead of a third agent 30. A loss function, also called a cost function or objective function, is a mathematical function used in machine learning to quantify the difference between the predicted and actual values ​​of an AI model's output. The purpose of the loss function is to measure the performance of the second agent 20 and to guide the learning process by providing feedback on the performance of the second agent 20. The loss function compares the changes made by the second agent 20 with an actual target value, particularly with regard to the defined system boundaries. The loss function calculates a numerical value that represents the error or "loss" between a scenario created by the second agent 20 and a realistic scenario.In particular, a mean square error can be used as a loss function.

[0092] In Fig. Figure 2 shows the process steps for generating scenarios for testing and validating automated driving functions and / or a driver assistance system (ADAS).

[0093] In a step S10, the behavior of the automated driving function 15 is simulated in a simulation environment that is characterized by at least one test case T i defined with a scenario.

[0094] In a step S20, the test case T i into a modified test case T j and / or the simulation environment is changed by changing parameters and parameter values ​​by a second agent 20, wherein the second agent 20 is designed as an adversarial agent to the first agent 10.

[0095] In a step S30, the behavior of the automated driving function 15 is simulated in the modified simulation environment, which can be determined directly or by the modified test case T j has been changed and simulation results 370 are generated.

[0096] In a step S40, the simulation results 370 are evaluated and evaluation results 470 and / or an evaluation index 490 are generated,

[0097] In a step S50, the modified test case T j and / or the changed parameters and / or parameter values ​​of the simulation environment are stored in a test database if the evaluation results and / or the evaluation index exceed or fall below a specified criterion, such as a threshold value.

[0098] Fig.3 schematically illustrates a computer program product 900 comprising executable program code 950 configured to perform the method according to the first aspect of the present invention.

[0099] The present invention can provide a pool of meaningful scenarios for testing and validating an automated driving function. This ensures that automated driving functions can be tested in a broad spectrum of real-world driving situations, such as different road conditions, traffic scenarios, weather conditions, and potential hazards. This coverage increases the reliability of system performance in real-world environments. Furthermore, the use of the second agent can identify limitations or weaknesses of the automated driving function by detecting scenarios in which the first agent or automated driving function struggles or does not respond effectively. This identification helps to refine and improve the algorithms and functions of the automated driving function in order to increase its performance and eliminate potential safety issues.By generating new requirement scenarios, potential areas for improvement can be identified and adaptability to changing driving conditions and new technologies can be improved. This future-proofing ensures that automated driving functions remain functional in the face of changing environments and requirements.

[0100] The present invention can thus improve the reliability, safety and efficiency of automated driving functions, but also of driver assistance systems in different driving scenarios and contribute to their continuous improvement and adaptation. Reference symbol 10 first agent 15 automated driving functions 20 second agent 30 third agent 100 systems 200 test database 250 modification software 300 simulation module 350 software applications 370 simulation results 400 evaluation module 450 evaluation software 470 review results 490 rating index 500 parameter database 520 sensor data 520 system values 900 computer program product 950 program code QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 11 443 130 B2

[0009] DE 10 2018 128 290 A1

[0010]

Claims

[1] Method for generating scenarios for testing and validating automated driving functions and / or a driver assistance system (ADAS), wherein a scenario is a traffic event in a temporal sequence and is defined by a selection of parameters and associated parameter values, and wherein an automated driving function (15) is part of a first agent (10); comprising the method steps: - Simulating (S10) the behavior of the automated driving function (15) in a simulation environment which is characterized by at least one test case (T i ) is defined with a scenario; - Changing (S20) the test case (T i ) into a modified test case (T j ) and / or changing the simulation environment by changing parameters and parameter values ​​by a second agent (20), wherein the second agent (20) is designed as an adversarial agent to the first agent (10); - Simulating (S30) the behavior of the automated driving function (15) in the modified simulation environment, which is determined directly or by the modified test case (T j ) was changed, and generating simulation results (370); - evaluating (S40) the simulation results (370) and generating evaluation results (470) and / or an evaluation index (490); - Save (S50) the modified test case (T j ) and / or the changed parameters and / or parameter values ​​of the simulation environment in a test database (200) if the evaluation results (470) and / or the evaluation index (490) exceed or fall below a predetermined criterion, such as a threshold value. [2] The method of claim 1, wherein the second agent (20) changes the parameters and parameter values ​​by sensor data (520) stored in a parameter database (500) and accessible to the second agent (30). [3] The method according to claim 2, wherein the sensor data (520) comprises measurement signals from measurement systems such as cameras, LiDAR sensors, radar sensors, ultrasonic sensors, acceleration sensors, speed sensors, torque sensors, humidity sensors and / or temperature sensors, GPS data, weather data, traffic data, environmental data such as road geometry, arrangement of buildings and vegetation, and data on the behavior of road users such as pedestrians, cyclists and vehicles. [4] Method according to one of claims 1 to 3, wherein the test cases (T j ) and the changed simulation environment is evaluated by a third agent (30) with regard to compliance with system limits such as a maximum and minimum speed. [5] Method according to one of claims 1 to 3, wherein the test cases (T j) and the changed simulation environment is evaluated using a loss function with regard to compliance with system limits such as a maximum and minimum speed. [6] Method according to one of claims 1 to 5, wherein the first agent (10), the second agent (20) and the third agent (30) are designed as models with artificial intelligence algorithms (AI models), the algorithms comprising at least one neural network. [7] Method according to claim 6, wherein the neural network is designed as a convolutional neural network (CNN) or as a deep neural network. [8] Method according to one of claims 6 or 7, wherein reinforcement learning algorithms are provided for training the behavior of the first agent (10) and / or the second agent (20) and / or the third agent (30). [9] Method according to one of claims 4 to 8, wherein the system boundaries are defined by system values ​​(550) stored in the parameter database (500), wherein at least the third agent (30) can access the system values ​​(550). [10] Method according to one of claims 1 to 9, wherein a 3D open world simulation is carried out as the simulation. [11] System (100) for generating scenarios for testing and validating automated driving functions and / or a driver assistance system (ADAS), wherein a scenario is a traffic event in a temporal sequence and is defined by a selection of parameters and associated parameter values, with a first agent (10), a second agent (20), a test database (200), a simulation module (300), an evaluation module (400) and a parameter database (500); wherein an automated driving function (15) is part of the first agent (10); wherein the second agent (20) is designed as an adversarial agent to the first agent (10); wherein the simulation module is designed to simulate the behavior of the automated driving function (15) in a simulation environment that is defined by at least one test case (T i ) is defined with a scenario; wherein the second agent (20) is designed to carry out the test case (T i) into a modified test case (T j ) and / or to change the simulation environment by changing parameters and parameter values; wherein the simulation module (300) is designed to modify the behavior of the automated driving function (15) in the modified simulation environment, which can be determined directly or by the modified test case (T j ) has been changed, and to generate simulation results (370); wherein the evaluation module (400) is designed to evaluate the simulation results (370) and to generate evaluation results (470) and / or an evaluation index (490); and wherein the test database (200) is designed to store the changed test case (T j ) and / or to store the changed parameters and / or parameter values ​​of the simulation environment if the evaluation results (470) and / or the evaluation index (490) exceed or fall below a predetermined criterion, such as a threshold value. [12] The system (100) of claim 11, wherein the second agent (20) changes the parameters and parameter values ​​by sensor data (520) stored in a parameter database (500) and accessible to the second agent (30), wherein the sensor data (520) includes measurement signals from measurement systems such as cameras, LiDAR sensors, radar sensors, ultrasonic sensors, acceleration sensors, speed sensors, torque sensors, humidity sensors and / or temperature sensors, GPS data, weather data, traffic data, environmental data such as road geometry, arrangement of buildings and vegetation, and data on the behavior of road users such as pedestrians, cyclists and vehicles. [13] System (100) according to claim 11 or 12, wherein the test cases (T j) and the changed simulation environment is evaluated by a third agent (30) or a loss function with regard to compliance with system limits such as a maximum and minimum speed. [14] System (100) according to one of claims 11 to 13, wherein the first agent (10), the second agent (20) and the third agent (30) are designed as models with artificial intelligence algorithms (AI models), wherein the algorithms comprise at least one neural network which is designed as a convolutional neural network (CNN) or as a deep neural network, and wherein reinforcement learning algorithms are provided for training the behavior of the first agent (10) and / or the second agent (20) and / or the third agent (30). [15] A computer program product (900) comprising an executable program code (950) configured to carry out the method according to any one of claims 1 to 10 when executed.

Images