Methods for statistical virtual validation of driver assistance systems

The method for statistical virtual validation of driver assistance systems addresses the challenge of covering rare scenarios by using a simulation environment with real-world data to test and improve the performance of driver assistance systems, ensuring safety and effectiveness in real-world operations.

DE102023117943B3Active Publication Date: 2024-11-28DR ING H C F PORSCHE AG
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
DE102023117943
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-07-07
Publication Date
2024-11-28
Estimated Expiration
2043-07-07

AI Technical Summary

Technical Problem

Existing methods for validating driver assistance systems and automated driving systems struggle to effectively cover rare and stochastic 'corner case' scenarios, which are not accounted for in predefined databases, leading to potential risks in real-world operations.

Method used

A method for statistical virtual validation using a simulation environment with a virtual driving environment and ego vehicle, where measurement data from real-world driving is used to create a realistic simulation, allowing for the generation of various driving scenarios, including stochastic unknown scenarios, and evaluating the system-under-test's performance.

Benefits of technology

Enables cost-effective testing of previously unknown scenarios, ensuring the system's readiness for real-world conditions by recreating these scenarios in a controlled simulation, thereby enhancing system quality and safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

The present invention relates to a method for the statistical virtual validation of driver assistance systems, wherein a simulation environment with a virtual world featuring virtual road traffic and a virtual ego-vehicle (13) placed in this virtual world is formed, wherein in the simulation environment the virtual ego-vehicle is at least partially controlled by a system-under-test (11), wherein a driver assistance system (ADAS) or an automated driving system (ADS) is selected as the system-under-test, wherein a sequence of driving scenarios is generated through the interaction of the system-under-test with the virtual world within the simulation environment, wherein, under a simulation of everyday traffic events, actors acting on the basis of stochastic models are represented and arbitrary journeys of the ego-vehicle with a sufficiently long driving duration are carried out.where the sufficiently long driving duration is chosen such that at least one stochastically occurring unknown scenario is found among the multitude of driving scenarios, and where this at least one unknown scenario is identified and evaluated among the multitude of driving scenarios. Furthermore, a simulation system is used for the statistical virtual validation of driver assistance systems.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The present invention relates to a method for statistical virtual validation of driver assistance systems. Furthermore, a simulation system for statistical virtual validation of driver assistance systems is claimed.

[0002] Approaches to the virtual validation of an advanced driver assistance system (ADAS) and / or an automated driving system (ADS) can use a test case-based or scenario-based testing approach. A major disadvantage of this approach is that the scenarios to be tested must first be modeled. While there are many sources from which information about scenarios occurring in road traffic can be obtained, there is always the risk of not covering so-called corner-case scenarios, which occur rarely and thus lie outside a previously defined database of scenarios to be tested.

[0003] In statistical on-road testing, the ADAS / ADS to be tested are operated in customer-oriented operation, verifying the correct behavior of the systems without any clearly defined test cases. This can also result in random or stochastical corner-case scenarios. Such corner-case scenarios are also, in principle, possible in simulation environments where driver assistance systems are statistically tested.

[0004] A simulation environment typically consists of various interacting simulation models, e.g., a simulation model representing the driving environment, a simulation model representing driver behavior, and / or a simulation model representing the vehicle controlled by the driver or the "system under test" (SUT), also referred to as the ego vehicle. All of these models are available in varying degrees of realism.

[0005] The publication DE 20 2022 106 107 U1 discloses a system for testing Level 3 automated driving systems, in which multiple cameras capture images of vehicles and roads. Test cases are determined based on the camera images and an analysis of the camera images.

[0006] From the publication DE 10 2021 132 722 A1, a system for selecting predefined test scenarios is known in order to evaluate the performance of autonomous vehicles.

[0007] Document DE 10 2021 202 335 A1 describes a method for testing an at least partially autonomous robot or motor vehicle using a statistical approach. The testing is based on predefined test cases.

[0008] Document DE 10 2019 216 836 A1 describes a method for training at least one algorithm for a control unit of a motor vehicle, wherein the algorithm is trained by a self-learning neural network. In addition to providing a computer program product module for an automated or autonomous driving function, a simulation environment, and a mission for the motor vehicle, at least one behavioral parameter of the motor vehicle is modified such that it lies beyond permissible limits.

[0009] Document DE 10 2017 007 136 A1 describes a self-learning algorithm for an automated vehicle with a predefined automation module by generating learning situations. A traffic simulation and a vehicle dynamics simulation are performed, and a relevant learning situation is identified based on selection criteria determined on the basis of predefined metrics.

[0010] Against this background, it is an object of the present invention to provide a method for validating driver assistance systems, wherein the validation is carried out within a simulation environment. The simulation environment should include both a virtual driving environment and a virtual ego vehicle. Particular attention should be paid to stochastically occurring, unknown scenarios, which could pose a potential hazard to driving operation during validation in a real world. Furthermore, a control unit should be provided on which the method is executed.

[0011] To solve the aforementioned problem, a method for statistical virtual validation of driver assistance systems is proposed, wherein, during vehicle journeys in a real world with real road traffic, vehicle sensors measure measurement data and determine settings of vehicle actuators, and store the measurement data and settings. The stored measurement data and settings form a basis for a realistic simulation environment with a virtual world featuring virtual road traffic and a virtual ego vehicle placed in the virtual world. In the simulation environment, the virtual ego vehicle is at least partially controlled by a system under test, wherein a driver assistance system (ADAS) or an automated driving system (ADS) is selected as the system under test.Through the interaction of the system under test with the virtual world, a multitude of driving scenarios are created within the simulation environment. In a simulation of everyday traffic events, actors acting based on stochastic models are represented, and arbitrary journeys of the ego vehicle with a sufficiently long journey time are executed. The sufficiently long journey time is chosen such that at least one stochastically occurring unknown driving scenario occurs among the multitude of driving scenarios. This at least one unknown scenario is triggered by an evaluation event. The system under test is evaluated based on the at least one unknown driving scenario. Based on the evaluation, the system under test is applied and executed while controlling a real vehicle.

[0012] The vehicle sensors can include, for example, a lidar and / or other radar systems that monitor the real traffic surrounding the vehicle, as well as measurement sensors related to the vehicle itself, such as an acceleration sensor. The vehicle actuators can, for example, relate to a turn signal adjustment or an accelerator pedal adjustment.

[0013] According to ISO 21448, driving scenarios are literally divided into "known or unknown," whereby, purely theoretically, unknown driving scenarios result from the set of all possible driving scenarios minus all known driving scenarios. Furthermore, for the purposes of this disclosure, driving scenarios are also considered unknown that may occur frequently in real or simulated traffic situations, but have not (yet) been used as a training basis for the respective SUT or for which no application of the respective SUT has yet taken place. At least a subset of all unknown driving scenarios may therefore be known to a higher authority, but is unknown to the respective SUT to be validated.

[0014] The virtual ego vehicle moves within the simulated world of the simulation environment during random journeys without specific test cases being specified. Even the stochastically occurring unknown scenario does not have to be a predefined test case. Simple situations, such as a newly detected traffic sign or a newly detected target vehicle, can randomly result in a variety of driving scenarios in the simulation. During their evaluation, it is checked whether, for example, the newly detected traffic sign was also correctly detected. Simulation data is used as so-called ground truth. The method according to the invention thus advantageously enables the generation of a data basis for the unknown scenarios without endangering people or equipment.

[0015] The evaluation event or trigger event for a driving scenario to be evaluated is a trigger or condition that indicates that a driving scenario evaluable for the respective SUT will follow during the simulation. The respective evaluation event differs depending on the SUT. According to the invention, an adaptive cruise control system is selected as the SUT, in which an evaluation event is generated when another (simulated) vehicle enters the detection range of (simulated) vehicle sensors. A driving scenario following this evaluation event, e.g., how the other vehicle moves, is then part of the stochastic simulation. If, for example, the other vehicle has a vehicle outline previously unknown to the SUT, this is an unknown driving scenario.In order to be able to use the respective driving scenario for the evaluation, it is therefore important to identify in the simulation when an evaluable driving scenario begins.

[0016] According to the invention, a traffic sign recognition system is formed as the SUT. For each road in the simulation environment, a ground truth is stored in advance, e.g., with a respective speed limit. A change in the speed limit occurring during a simulated journey on a selected road is an evaluation event, since the change is indicated by a traffic sign, and thus the SUT, as a traffic sign recognition system, also generates an output that can be compared with the ground truth, with the result of the comparison being evaluable. In this example, an unknown driving scenario is formed by a traffic sign unknown to the traffic sign recognition system (recognized as a traffic sign by the standard outline of a traffic sign).

[0017] An application of the SUT based on the evaluation using the at least one unknown scenario is designed, for example, in such a way that the SUT shows a desired behavior when the at least one unknown scenario is run through again.

[0018] In one embodiment of the method according to the invention, the system under test is embedded in the simulation environment using an X-in-the-loop approach. A driver, the ego vehicle, and a driving environment are simulated.

[0019] According to the invention, actions of the virtual driver of the ego vehicle are represented by a driver model. In the case of an ADAS as a system under test, the driver partially or completely assumes vehicle control. Therefore, the driver model must be equipped with the capabilities to analyze traffic situations and act based on stochastic decisions. At least one operating action performed by the driver of the system under test is modeled. In this process, an activation behavior or takeover behavior must be represented. For example, it must be modeled whether the virtual driver would assume the driving task themselves, depending on a given driving scenario, and deactivate the driver assistance system in the process. This can be the case, for example, if the vehicle is tailgating too closely.

[0020] In the simulation environment considered in the method according to the invention, there are no changes with regard to a vehicle model compared to test case-based testing.

[0021] In a further embodiment of the method according to the invention, the driving environment is represented by a driving environment model. The driving environment model models at least one of the following scenarios: • Landscapes with road networks in which actors operate; • Road users are independent actors; • realistic road infrastructure with stochastically occurring malfunctions.

[0022] The road networks and landscapes that must necessarily be modeled in the method according to the invention to represent the realistic simulation environment are in contrast to test case-based testing, in which only a scenery relevant to the respective test case, such as road infrastructure and buildings, is modeled.

[0023] Road users such as vehicles, pedestrians, or cyclists are modeled as independent actors that analyze traffic situations following their own logic and make decisions and execute actions based on stochastic models. This is only possible in the realistic simulation environment created by the method according to the invention, so that free interaction between the independent actors can take place.

[0024] A realistic modeling of road infrastructure includes consideration of malfunctions, such as traffic lights or faulty maps. The occurrence of such errors is also modeled stochastically.

[0025] The simulation of the virtual world enables a simple implementation of different countries, so that the method according to the invention can advantageously also be used to consider a wide variety of driving scenarios in a country-specific manner, for example, taking country-specific traffic signs into account.

[0026] In a further embodiment of the method according to the invention, in the case of an ADAS as the system under test, part of a driving task is taken over by a driver model. In the case of an ADS as the system under test, the driving task is completely taken over by the ADS, and the driver model represents system activation and, optionally, takeover situations.

[0027] In yet another embodiment of the method according to the invention, the at least one unknown driving scenario is formed by at least one corner-case scenario with a low probability of occurrence. If necessary, a simulation duration is extended such that such a corner-case scenario has occurred or has been indicated by a corresponding evaluation event.

[0028] In a further embodiment of the method according to the invention, after identification of a driving scenario with a low probability of occurrence, this is inserted into the simulation environment for a further virtual drive.

[0029] In a further embodiment of the method according to the invention, the at least one unknown scenario is simulated in a real world using a real ego vehicle. The real ego vehicle is controlled by the system under test, with the system under test being adapted to handle the at least one unknown scenario. Bug fixes can also be defined and implemented in the system under test if the system under test does not function as desired. The simulation in the real world can take place, for example, on a test track.

[0030] In a further embodiment of the method according to the invention, the incorrect behavior of individual actuators compared to everyday actions or the occurrence of incorrect behavior of individual actuators compared to everyday probabilities of occurrence are forced in the simulation environment. This advantageously results in more unknown scenarios occurring in a shorter time. This is achieved, for example, by • Criticality metrics (e.g. “time to collision”) are integrated in order to analyze the respective driving scenarios that occur stochastically in the simulation based on their respective criticality; • Criticality serves as a “reward” for a decision model of the driver model and other road users; • Develop learning models that, using criticality as a reward, learn to guide the system under test in situations that evoke unknown scenarios. Such models can, for example, learn that other road users brake abruptly or increase speed before a traffic light changes from green to yellow. This also requires learning applicable traffic rules and laws.

[0031] Furthermore, a simulation system for a statistical virtual validation of driver assistance systems is claimed, wherein the simulation system comprises a computing unit and the computing unit is configured to execute an algorithm according to a method according to the invention.

[0032] The statistical virtual validation according to the invention allows previously unknown driving scenarios to be identified and tested cost-effectively in simulation. These driving scenarios can then be recreated, at least partially, in prototype vehicles on test tracks. If the respective SUT meets the specifications, another previously unknown driving scenario is tested and passed. If an error occurs, targeted bug fixes can be developed. This approach advantageously increases the system quality of the SUT, for example, until the SUT receives safety approval and can be used in real road traffic.

[0033] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

[0034] It is understood that the features mentioned above and those to be explained below can be used not only in the combination specified in each case, but also in other combinations or on their own, without departing from the scope of the present invention. Fig. 1 shows a relationship diagram for statistical validation in an embodiment of the system according to the invention.

[0035] In the Fig.1 shows a relationship diagram 10 for statistical validation in one embodiment of the system according to the invention. At the center is a system under test (SUT) 11 to be statistically validated, which at least partially controls a virtual ego vehicle 13, i.e., depending on whether the system under test 11 is an ADAS or an ADS. The system under test 11 is executed within a driving environment simulation 14, with a driver simulation 12 taking place in the virtual ego vehicle 13. This is necessary for a simulation that is as realistic as possible, since even a real driver in a real (ego) vehicle deactivates or "overrides" the ADAS or ADS in certain situations. To validate the SUT 11, not all driving scenarios running in the simulation are used; rather, a stochastical selection is made from among these. List of reference symbols 10 Relationship diagram statistical validation 11 System under Test (SUT) 12 Stochastic driver simulation 13 (First Person) Vehicle Simulation 14 Stochastic driving environment simulation

Claims

[1] A method for statistical virtual validation of driver assistance systems, wherein, when a vehicle is traveling in a real world with real road traffic, measurement data is measured by vehicle sensors and settings of vehicle actuators are determined and the measurement data and settings are stored, wherein the stored measurement data and settings form a basis for a realistic simulation environment with a virtual world having virtual road traffic and a virtual ego vehicle (13) placed in this virtual world, wherein in the simulation environment the virtual ego vehicle (13) is at least partially controlled by a system under test (11), wherein a driver assistance system (ADAS) or an automated driving system (ADS) is selected as the system under test (11),wherein the interaction of the system under test (11) with the virtual world within the simulation environment creates a plurality of driving scenarios, wherein actors acting on the basis of stochastic models are represented in a simulation of everyday traffic events, and arbitrary journeys of the ego vehicle (13) are carried out with a sufficiently long journey time, wherein the sufficiently long journey time is selected such that at least one stochastically occurring unknown driving scenario occurs among the plurality of driving scenarios, wherein the at least one unknown driving scenario is triggered by an evaluation event, wherein the evaluation event is a trigger indicating that a driving scenario evaluable for the respective system under test (11) follows in the temporal course of the simulation, and the respective evaluation event differs depending on the system under test,wherein the system-under-test (11) is evaluated based on the at least one unknown driving scenario, wherein the system-under-test is applied on the basis of the evaluation and executed when controlling a real vehicle., [2] Method according to claim 1, wherein an adaptive cruise control system is selected as the system under test (11) and the evaluation event is formed by the entry of another vehicle into a detection range of vehicle sensors, or wherein a traffic sign recognition system is selected as the system under test (11) and the evaluation event is formed by a change in a speed limit. [3] Method according to one of the preceding claims, wherein the system-under-test (11) is embedded in the simulation environment via an X-in-the-loop approach, wherein a driver (12), the ego vehicle (13) and a driving environment (14) are simulated, wherein actions of the driver (12) of the ego vehicle (13) are represented by a driver model, wherein at least one operating action of the system-under-test (11) performed by the driver (12) is modeled. [4] Method according to claim 3, wherein the driving environment (14) is represented by a driving environment model, wherein the driving environment model models at least one of the following scenarios: • Landscapes with road networks in which actors operate; • Road users are independent actors; • realistic road infrastructure with stochastically occurring malfunctions. [5] Method according to one of the preceding claims, wherein in the case of an ADAS as system-under-test (11) part of a driving task is taken over by a driver model, and wherein in the case of an ADS as system-under-test (11) the driving task is taken over entirely by the ADS and an activation of the system and optionally takeover situations are represented by the driver model. [6] Method according to one of the preceding claims, wherein the at least one unknown driving scenario is formed by at least one corner case scenario with a low probability of occurrence. [7] Method according to one of the preceding claims, wherein after identification of a driving scenario with a low probability of occurrence, this is inserted into the simulation environment for a further virtual drive. [8] Method according to one of the preceding claims, wherein the at least one unknown driving scenario is simulated in a real world with a real ego vehicle, wherein the real ego vehicle is controlled by the system under test (11), and wherein the system under test (11) is adapted to cope with the at least one unknown driving scenario. [9] Method according to one of the preceding claims, wherein in the simulation environment, a misbehavior of individual actuators is forced in comparison to everyday actions or an occurrence of the misbehavior of individual actuators is forced in comparison to everyday probabilities of occurrence. [10] Simulation system for a statistical virtual validation of driver assistance systems, wherein the simulation system comprises a computing unit and the computing unit is configured to execute an algorithm according to a method according to one of claims 1 to 9.

Citation Information

Patent Citations

  • Method and device for training self-learning algorithms for an automated driving vehicle

    DE102017007136A1

  • Method for training at least one algorithm for a motor vehicle control unit, computer program product and motor vehicle

    DE102019216836A1

  • SELECTING TEST SCENARIOS TO EVALUATE AUTONOMOUS VEHICLE PERFORMANCE

    DE102021132722A1

  • Method for diagnosing faults in an electrically powered vehicle with an electric drivetrain

    DE102021200393A1

  • Method and apparatus for testing a technical system

    DE102021202335A1