METHOD AND SYSTEM FOR TESTING A DRIVER ASSISTANCE SYSTEM FOR A VEHICLE

Description

[0001] The invention relates to a computer-implemented method and a system for testing a driver assistance system for a vehicle, wherein a scenario in which the vehicle is located is simulated and the driver assistance system is operated in an environment of the vehicle based on the simulated scenario, and wherein the driving behavior of the driver assistance system is observed in the environment of the vehicle.

[0002] The use of advanced driver assistance systems (ADAS), which in their further development enable autonomous driving (AD), is constantly increasing in both passenger cars and commercial vehicles. Driver assistance systems make a significant contribution to increasing active road safety and enhance driving comfort.

[0003] In addition to systems that primarily serve driving safety, such as ABS (Anti-lock Braking System) and ESP (Electronic Stability Program), a wide variety of driver assistance systems are offered in the area of ​​passenger cars and commercial vehicles.

[0004] Driver assistance systems already used to increase active road safety include parking assist, adaptive cruise control (ACC), which adjusts the vehicle's speed to a distance from the vehicle ahead based on the driver's selected speed. Other examples include ACC stop-and-go systems, which, in addition to ACC, automatically resume driving in traffic jams or when vehicles are stationary; lane keeping or lane assist systems, which automatically keep the vehicle in its lane; and pre-crash systems, which, in the event of a potential collision, prepare or initiate braking to dissipate kinetic energy from the vehicle and, if necessary, initiate further measures if a collision is unavoidable.

[0005] These driver assistance systems increase road safety by warning the driver in critical situations, up to and including initiating autonomous intervention to avoid or mitigate accidents, for example by activating an emergency braking function. Additionally, driving comfort is enhanced through features such as automatic parking, lane keeping assist, and adaptive cruise control.

[0006] The safety and comfort benefits of a driver assistance system are only perceived positively by vehicle occupants if the support provided by the driver assistance system is safe, reliable and, as far as possible, comfortable.

[0007] Furthermore, depending on its function, each driver assistance system must be able to handle traffic scenarios with maximum safety for its own vehicle and without endangering other vehicles or other road users.

[0008] The respective degree of automation of vehicles is divided into so-called automation levels 1 to 5 (see, for example, standard SAE J3016). The present invention relates in particular to vehicles with driver assistance systems of automation levels 3 to 5, which are generally considered autonomous driving.

[0009] The challenges of testing such systems are manifold. In particular, a balance must be struck between testing effort and test coverage. The main task when testing ADAS / AD functions is to demonstrate that the driver assistance system functions reliably in all conceivable situations, especially critical driving situations. Such critical driving situations pose a certain risk, as a lack of response or an incorrect response from the respective driver assistance system can lead to an accident.

[0010] Testing driver assistance systems therefore requires considering a large number of driving situations that can arise in various scenarios. The range of possible scenarios is generally spanned by many dimensions (e.g., different road characteristics, the behavior of other road users, weather conditions, etc.). From this virtually infinite and multidimensional parameter space, it is particularly relevant for testing driver assistance systems to extract those parameter constellations for critical scenarios that could lead to unusual or dangerous driving situations.

[0011] As in Fig. 1 As depicted, such critical scenarios have a far lower probability of occurrence than usual scenarios.

[0012] Scientific publications suggest that operating a vehicle in autonomous mode is statistically safer than operating a human-driven vehicle only if the corresponding driver assistance system has accumulated 275 million miles of accident-free driving time to validate it. This is practically impossible to achieve through real-world test drives, especially given the very tight development cycles and quality standards required in the automotive industry. Furthermore, it would be unlikely that a sufficient number of critical scenarios, or the driving situations arising from them, would be included in such tests for the aforementioned reason.

[0013] It is known from the prior art to use real test drive data from a real fleet of test vehicles to validate and verify driver assistance systems and to extract scenarios from the recorded data. Furthermore, it is known to use full factorial experimental designs for validation and verification.

[0014] FLORIAN HAUER ET AL: "Scenario Optimization for the Safety of Automated and Autonomous Driving Systems" reveals a methodology that employs a metaheuristic search to optimize scenarios. This requires the creation of a suitable search space and a suitable fitness function. Starting with abstract descriptions of the system functionality and use cases, parameterized scenarios are derived. The parameters define a search space within which suitable scenarios must be found. Using search-based techniques guided by a fitness function, those scenarios in which the system exhibits its worst performance are identified.

[0015] DUAN JIANLI ET AL: "Test Scenario Generation and Optimization Technology for Intelligent Driving Systems" reveals a new scenario generation algorithm called Combinatorial Testing Based on Complexity (CTBC), which is based on both the combinatorial testing (CT) method and the test matrix technique (TM) and is intended for intelligent vehicle systems.

[0016] It is an object of the invention to enable the testing of driver assistance systems, in particular driver assistance systems for autonomous driving, in critical scenarios. Specifically, it is an object of the invention to identify critical scenarios for driver assistance systems. This object is achieved by the teaching of the independent claims. Advantageous embodiments are found in the dependent claims.

[0017] A first aspect of the invention relates to a computer-implemented method for testing a driver assistance system for a vehicle, comprising the following steps: Simulating a scenario in which the vehicle is located; operating the driver assistance system in the vehicle's environment based on the simulated scenario; observing the driving behavior of the driver assistance system in the vehicle's environment; determining a driving situation that arises from the driving behavior of the driver assistance system in the vehicle's environment; determining the quality of the simulated scenario depending on the hazard level of the resulting driving situation; checking at least one termination condition; and modifying the simulated scenario based on the determined quality until at least one termination condition with regard to quality is met, whereby when modifying the simulated scenario, a new scenario is created in which parameters have been exchanged, parameters have been omitted, and / or new parameters have been added.

[0018] A second aspect of the invention relates to a system for testing a driver assistance system for a vehicle, comprising: Means for simulating a scenario in which the vehicle is located; means for operating the driver assistance system in the vehicle's environment based on the simulated scenario; means for observing the driving behavior of the driver assistance system in the vehicle's environment; means for determining a driving situation that arises from the driving behavior of the driver assistance system in the vehicle's environment; means for determining the quality of the simulated scenario depending on the hazard level of the resulting driving situation; means for checking at least one termination condition; and means for modifying the simulated scenario based on the determined quality until a termination condition with respect to the quality is met, wherein when the simulated scenario is modified, a new scenario is created in which parameters have been exchanged, parameters have been omitted, and / or new parameters have been added.

[0019] A third aspect of the invention relates to a system for testing a driver assistance system for a vehicle, comprising an agent, wherein the agent is configured to generate a scenario and to provoke an error of the driver assistance system by changing the scenario, and wherein a strategy for changing the scenario is continuously improved by means of a reinforcement learning methodology through interaction of the agent with the driver assistance system during operation until a termination condition is reached.

[0020] A vehicle environment within the meaning of the invention is preferably formed at least by the objects relevant for vehicle control by the driver assistance system. In particular, a vehicle environment comprises a scene and dynamic elements. The scene preferably includes all stationary elements. A scenario within the meaning of the invention is preferably formed from a temporal sequence of, in particular, static scenes. The scenes specify, for example, the spatial arrangement of at least one other object relative to the ego object, e.g., the constellation of road users. A scenario can, in particular, contain a driving situation in which a driver assistance system at least partially controls the vehicle equipped with the driver assistance system, referred to as the ego vehicle, e.g., autonomously executes at least one vehicle function of the ego vehicle.

[0021] A driving situation within the meaning of the invention preferably describes the circumstances that must be considered for selecting suitable behavior patterns of the driver assistance system at a specific time. A driving situation is therefore preferably subjective, representing the perspective of the ego-vehicle. It further preferably includes relevant conditions, possibilities, and influencing factors of actions. A driving situation is further preferably derived from the scene through an information selection process based on transients, e.g., mission-specific, as well as permanent goals and values.

[0022] Driving behavior within the meaning of the invention is preferably a behavior of the driver assistance system through action and reaction in an environment of the vehicle.

[0023] A quality level, as defined in the invention, preferably characterizes the simulated scenario. A quality level is preferably understood to be a quality or characteristic of the simulated scenario with regard to its suitability for testing the driver assistance system. A more critical scenario preferably has a higher quality level. Preferably, the dangerousness of a driving situation arising from the respective scenario for the tested driver assistance system is a measure of the scenario's quality level.

[0024] Reinforcement learning is a machine learning method in which an agent independently learns to perform tasks within an environment. This occurs when the agent tries different actions in the environment and receives either a reward or a punishment based on feedback from the environment. After a learning phase, the agent is able to perform a task within the environment that earns it the highest possible reward.

[0025] An agent within the meaning of the invention preferably refers to a computer program or a module of a data processing system that is capable of a certain independent and self-dynamic, in particular autonomous, behavior. This means that, depending on various states, in particular different statuses, a predetermined processing operation takes place without any further start signal being given from the outside or any external control intervention occurring during the process.

[0026] The invention is based on the idea of ​​iteratively improving simulated scenarios to make them as suitable as possible for testing a driver assistance system. In other words, the simulated scenarios are improved to be as suitable as possible for detecting or inducing a potential fault in the driver assistance system. This allows simulated scenarios to be specifically optimized for a particular driver assistance system or a function of a driver assistance system.

[0027] Preferably, a termination condition can be provided, upon reaching which the iterative process is terminated. If such a termination condition is not reached, a further termination condition can be, for example, a maximum test time or a maximum number of test kilometers covered by the vehicle in the simulated scenarios. The quality of the simulated scenario, i.e., the parameter that evaluates the simulated scenarios, preferably depends on a defined criterion relating to a driving situation that occurred in each iteration step.

[0028] The dangerousness of the driving situation can be used as a criterion for assessing the system's quality. Preferably, this measure of quality is a calculated time until a collision occurs, an accident probability, and / or inadequate driving behavior by the driver assistance system. Such inadequate driving behavior could include, for example, a violation of a traffic rule and / or a maneuver with an excessive risk of damage, particularly personal injury.

[0029] The invention follows a game-like approach with two "players," whereby the method or system for testing the driver assistance system attempts to iteratively generate scenarios of increasing complexity until a predefined criterion, in particular a (safety-)critical metric relating to the driver assistance system functionality, is violated. In this case, the testing method or system has "won." If such a violation, i.e., the fulfillment of a termination condition, is not triggered, then the tested driver assistance system "wins."

[0030] The invention significantly reduces the number of kilometers required to test a driver assistance system, as it intuitively identifies the scenarios that are particularly critical for that system. The majority of scenarios can normally be handled easily by the system. However, in such cases, weaknesses of the driver assistance system cannot be detected.

[0031] During the test procedure, a so-called agent, which may preferably be trained as a software module or sub-algorithm in the test procedure, learns from the behavior of the tested driver assistance system and continuously improves the quality of the simulated scenario in order to cause a malfunction of the investigated driver assistance system.

[0032] The test procedure is repeated iteratively until a change in the simulated scenario leads to a behavior of the driver assistance system that violates a predefined target value, which serves as a termination condition. Such a termination condition could be, for example, a duration until a collision of less than 0.25 seconds or a specific time budget, e.g., 600 hours of maximum simulation time.

[0033] Assuming a sufficiently long time budget, the invention can achieve a high probability that the ADAS or AD system will function correctly. The reliability of the tests performed according to the invention depends on the algorithm used to modify the simulated scenarios. Ideally, such an algorithm possesses a human-like intuition that allows the respective driver assistance system to be pushed to its limits.

[0034] In an advantageous embodiment of the procedure for testing a driver assistance system, a speed, in particular an initial speed, of the vehicle and / or a trajectory of the vehicle are specified when simulating the scenario. This allows empirical data to be taken into account during testing. In this way, an agent can be put on the right track from the outset in order to develop a critical scenario.

[0035] In a further advantageous embodiment of the method for testing a driver assistance system, the values ​​of parameters within a scenario are changed when the simulated scenario is modified. In this case, the existing scenario is iteratively adapted and thus improved for testing the respective driver assistance system. This approach is particularly advantageous when a specific scenario needs to be "optimized" for testing a particular driver assistance system.

[0036] In a further advantageous embodiment of the method for testing a driver assistance system, the parameters of the scenario are selected from the following group, depending on the type of driver assistance system to be tested: speed, in particular an initial speed, of the vehicle; trajectory of the vehicle; lighting conditions; weather; road surface condition; number and position of static and / or dynamic objects; speed and direction of movement of the dynamic objects; condition of signaling systems, in particular traffic signal systems; traffic signs; vertical elevation, width and / or drivability of lanes, lane layout, number of lanes.

[0037] In a further advantageous embodiment of the method for testing a driver assistance system, changing the simulated scenario generates a new scenario, preferably consisting of sequentially combined scenarios. A new scenario is preferably characterized by the fact that new driving tasks must be mastered. For example, the scenario of approaching an intersection is fundamentally different from the scenario of driving on a highway.

[0038] Replacing simulated scenarios with new ones offers the advantage that a single test drive can cover many different driving situations and allow for testing numerous functions of the driver assistance system in a wide variety of environments. This significantly increases the validity of the test procedure. When creating a new scenario, entirely new parameters can also be introduced to modify existing parameter values.

[0039] In a further advantageous embodiment of the method for testing a driver assistance system, a fictitious reward is credited when determining its performance, and the modification is based on a function designed to maximize the reward. Preferably, the algorithm used in the invention, in particular an agent that applies this algorithm, learns which changes to the existing simulated scenario or which changes to a new scenario are conducive to achieving the desired effect, i.e., which changes lead to critical scenarios and may provoke a malfunction of the driver assistance system or an accident.

[0040] In a further advantageous embodiment of the procedure for testing a driver assistance system, the quality is higher the more dangerous the driving situation that arises, in particular the shorter the calculated duration until a collision occurs.

[0041] In a further advantageous embodiment, the simulated scenario is modified using evolutionary algorithms. Evolutionary algorithms are also known as genetic algorithms. When modifying such algorithms, different algorithms are combined and mutated. From the resulting algorithms, candidates for the next iteration step are generated.

[0042] This selection can be made by choosing those candidates that are most likely to trigger critical scenarios. Genetic evolutionary algorithms offer a high degree of flexibility for optimizing existing scenarios with respect to predefined criteria.

[0043] In a further advantageous embodiment of the method for testing a driver assistance system, a utility function is approximated based on the determined performance, which describes the value of a specific scenario. The algorithm or agent views this value of the simulated scenario as a kind of reward and is preferably configured to maximize both the value and the reward.

[0044] In a further advantageous embodiment of the method for testing a driver assistance system, the driver assistance system is simulated. Simulating the driver assistance system is particularly advantageous because, in this case, no test bench is required for testing the actual components of a real driver assistance system. In particular, the method according to the invention can be executed faster than real time in this case. The speed of the simulation is limited only by the available computing power.

[0045] In a further advantageous embodiment of the inventive method for testing a driver assistance system, a strategy for changing the scenario is continuously improved using a reinforcement learning methodology based on the determined performance during the test operation until the termination condition is reached. When using reinforcement learning, an algorithm or agent independently learns a strategy to maximize a received reward. Both positive and negative rewards can be assigned for actions taken. The use of reinforcement learning allows for particularly effective optimization of the simulated scenarios.

[0046] In a further advantageous embodiment of the method for testing a driver assistance system, historical data from previous test runs of the driver assistance system, particularly the system itself, are considered during the initial scenario simulation. The use of historical data can be employed to pre-train the algorithm or agent. This can reduce the time required to identify critical scenarios. Furthermore, algorithms or agents trained on a different, particularly similar, ADAS or AD system can also be used. In particular, this allows for the execution of regression tests to ensure that changes to previously tested parts of the driver assistance system's software do not introduce new errors.

[0047] In a further advantageous embodiment of the method for testing a driver assistance system, data relating to the vehicle's environment are fed into the driver assistance system during its operation, and / or the driver assistance system, particularly its sensors, are stimulated based on the vehicle's environment. In this case, the method according to the invention can be used to test a physically existing driver assistance system. Preferably, the vehicle is simulated. However, it is also conceivable that the entire vehicle, including the driver assistance system, is tested on a test bench in this manner. This embodiment offers the advantage that the driver assistance system, with all its components, can be tested under conditions that are as realistic as possible.

[0048] In an advantageous embodiment of the system for testing a driver assistance system, the agent is configured to observe a driving situation that arises from the driving behavior of the driver assistance system in an environment of the vehicle based on the simulated scenario, and to determine the quality of the scenario depending on the danger of the resulting driving situation.

[0049] In a further advantageous embodiment of the system for testing a driver assistance system, the agent is pre-trained based on historical data. This data is taken into account by the agent during the initial simulation of the scenario.

[0050] The features and advantages described above relating to the first aspect of the invention also apply accordingly to the second and third aspects of the invention and vice versa.

[0051] Further features and advantages will become apparent from the following description in relation to the figures. They show, at least partially schematically: Figure 1 a diagram of the probability of occurrence of scenarios depending on their complexity; Figure 2a an example of a scenario; Figure 2b an example of a scenario with higher complexity than the one from Figure 2a ; Figure 3 an example implementation of a procedure for testing a driver assistance system; and Figure 4 An example of a system for testing a driver assistance system.

[0052] Figure 1 This shows the probability of occurrence of scenarios depending on their complexity. The probability of occurrence is the likelihood with which scenarios happen in real-world road traffic.

[0053] In Fig. 1It is noticeable that the majority of scenarios are of relatively low complexity, which also corresponds to the general life experience of a driver. The scope of these scenarios is in Fig. 1 Designated with "A". Scenarios of high complexity, whose scope lies in Fig. 1 Scenarios designated with "B" are relatively rare. However, those "B" scenarios with high complexity are particularly relevant for investigating the functionality of driver assistance systems.

[0054] In order to achieve a sufficient number and diversity of different scenarios with high complexity "B" during the testing of a driver assistance system, a very high number of scenarios must therefore be run through, based on the distribution curve shown.

[0055] Figure 2aFigure 3 shows a first scenario in which a pedestrian 6 crosses a zebra crossing and a vehicle 1 controlled by a driver assistance system 2, as well as another vehicle 5a in the opposite lane, approach the zebra crossing. The driver assistance system 2 controls both the longitudinal and lateral movement of vehicle 1.

[0056] At the in Fig. 2 In the first scenario 3 shown, the pedestrian 6, the road layout, the crosswalk, and the oncoming vehicle 5a are all clearly visible to the driver assistance system 2 via sensors. In the illustrated example, the driver assistance system 2 will recognize that it needs to reduce the vehicle speed to allow the pedestrian 6 to cross at the crosswalk. The movement of the other vehicle 5a is not expected to play a role in this. Figure 2a This therefore concerns scenario 3 with comparatively low complexity.

[0057] In the second scenario 3, which is in Fig. 2b As depicted, there is no zebra crossing. Furthermore, next to the lane of the vehicle controlled by driver assistance system 2, there are other vehicles 5b, 5c, 5d parked, which prevent pedestrian 6 from being detected by the sensors of driver assistance system 2, or prevent their detection.

[0058] In addition to pedestrian 6 and parked vehicles 5b, 5c, 5d, there is another vehicle 5a in the vicinity of vehicle 1 controlled by driver assistance system 2, which, as shown in Figure 2a , which is approaching vehicle 1 controlled by the driver assistance system 2.

[0059] Behind this further vehicle 5a is a motorcyclist 4. Whether this motorcyclist is perceptible in the vicinity of the vehicle 1 controlled by the driver assistance system 2 can be determined from Figure 2bdo not interpret. In the depicted scenario 3, the motorcyclist 4 will attempt to overtake the other vehicle 5a in the opposite lane. At the same time, the pedestrian 6 will attempt to cross the road in the depicted scenario. He is unaware of the vehicle 1 controlled by the driver assistance system 2.

[0060] Depending on how the driver assistance system 2 reacts or acts in scenario 3, i.e., what driving behavior the driver assistance system 2 exhibits in the vicinity of vehicle 1, a driving situation will arise that is more or less dangerous. For example, if vehicle 2, as in Figure 2bAs indicated by the arrows, if the vehicle 1, controlled by the driver assistance system 2, continues driving at undiminished speed in scenario 3, a collision between the vehicle 1 and the motorcycle 4 is likely to occur. Such a driving situation would be extremely dangerous.

[0061] Due to the large amount of information that the driver assistance system 2 of vehicle 1 receives in the Figure 2b The second scenario 3 demonstrates how the system must process the scenario 3 shown and the possible problems that can arise from the constellation of road users 5a, 5b, 5c, 5d, which are visible to the driver assistance system 2 in the surrounding area. Figure 2b a comparatively high level of complexity, especially compared to that in Figure 2a the first scenario shown.

[0062] Figure 3 Figure 1 shows an embodiment of a method 100 for testing a driver assistance system for a vehicle 1.

[0063] In a first step, 101, a scenario 3, in which vehicle 1 is located, is simulated. Preferably, the environment of vehicle 1 with all dynamic elements is simulated. Fig. 2b for example, pedestrian 6, the other vehicle 5a and the motorcycle 4, as well as stationary elements, in Fig. 2b the other vehicles 5b, 5c, 5d and the road, simulated.

[0064] Based on this simulation, a driver assistance system 2, optionally together with the vehicle 1 controlled by it, can be simulated, preferably in a test rig 12. In this case, the sensors of the driver assistance system 2 are preferably stimulated in such a way as to replicate the simulated scenario 3 or the environment of the vehicle 1 resulting from this simulated scenario 3. For this purpose, suitable stimulators, such as those known from the prior art, are used.

[0065] Preferably, the driver assistance system 2 or only the software of the driver assistance system 2 can be integrated into the simulation of scenario 3 in the manner of a hardware-in-the-loop test.

[0066] Finally, it is also possible to simulate the driver assistance system 2 or even just the software of the driver assistance system 2.

[0067] Preferably, when simulating the scenario, a speed, in particular an initial speed of vehicle 1 and / or a trajectory of vehicle 1, is specified. Furthermore, historical data from previous test runs of either the tested driver assistance system 2 or other driver assistance systems can preferably be taken into account when simulating the scenario. This historical data is particularly useful in determining an initial simulated test scenario. Furthermore, preferably, such historical data can be used to train a so-called agent that tests the driver assistance system, or an agent that has already been used to test another driver assistance system can be used.

[0068] In a second step 102, the driver assistance system 2 is operated in the vehicle's environment based on the simulated scenario 3. If the driver assistance system 2 is also only simulated, then its operation in the environment of the vehicle 1 controlled by the driver assistance system 2 is also only simulated.

[0069] In a third work step 103, the driving behavior of the driver assistance system 2 is observed in the environment of the vehicle 1 driven by the driver assistance system 2.

[0070] Based on the data obtained through observation, a driving situation can be determined at any given time which arose from the driving behavior of the driver assistance system 2 in the vicinity of the vehicle 1. This is preferably carried out in a fourth work step 100.

[0071] Due to the resulting driving situation, the driver assistance system 2 must make new decisions about how to behave and control the vehicle 1 it controls.

[0072] In simulated scenario 3, the resulting driving situation can be objectively examined with regard to its hazard potential. In particular, based on the information available through scenario 3, an accident probability and the time until a collision can be calculated for each time step of the simulation.

[0073] In scenario 3 of the Figure 2b For example, the time until the collision could be calculated if the vehicle 1, controlled by the driver assistance system 2, continues straight ahead at undiminished speed.

[0074] The probability of an accident can be influenced, for example, by an assessment of the adequacy of the driving behavior of the driver assistance system 2. An accident probability in scenario 3 of Fig. 2 a or Fig. 2b The risk would be increased, for example, if the vehicle 1 controlled by the driver assistance system 2 were traveling at a significantly excessive speed.

[0075] In a fifth step, the quality of the simulated scenario 3 is determined based on a predefined criterion in relation to the resulting driving situation. The quality is derived primarily from the complexity of the simulated scenario, with greater complexity indicating higher quality. Preferably, the quality indicates how the driving situation resulting from the behavior of the driver assistance system 2 is to be assessed with respect to a predefined criterion. Such a criterion could, for example, be the dangerousness of the resulting driving situation, which is characterized by the probability of an accident and / or the time until a potential collision. Furthermore, the dangerousness could also be characterized by the probability of inadequate driving behavior by the driver assistance system 2.

[0076] In a sixth step, it is checked whether a termination condition is met. Such a termination condition can be defined by a predefined criterion relating to the driving situation, for example, a limit value for the maximum time until a collision or a maximum probability of an accident. A maximum test time can also be specified as a further termination condition.

[0077] If the termination condition is reached, the simulated scenario or the parameter values ​​of the simulated scenario and / or the quality of the simulated scenario are output in an eighth step 108, in particular in the form of a test report.

[0078] If the termination condition is not met, a seventh step (107) involves modifying the simulated scenario 3 based on the determined performance. Here, the test procedure starts again from the first step (101). The steps are performed iteratively, preferably until at least one of the termination conditions is met.

[0079] There are basically two different approaches to changing the simulated scenario in the seventh step (107). Firstly, only the values ​​of parameters within the simulated scenario can be changed. In this case, the modified simulated scenario always builds upon the simulated scenario used in the previous iteration step. This approach is particularly useful when the simulated scenario 3 is modified using so-called evolutionary algorithms.

[0080] Alternatively, changing the simulated scenario creates a new scenario. In such a new scenario, parameters can be exchanged, omitted, and / or new parameters can be added.

[0081] This approach is particularly useful when the strategy for changing the scenario employs a reinforcement learning methodology based on the determined performance. In reinforcement learning, this strategy is continuously improved during the test run until the termination condition is reached. Preferably, in this case, a utility function is approximated based on the determined performance, which describes the performance value of a specific simulated scenario.

[0082] As mentioned previously, the algorithm that modifies the scenarios can preferably be trained as a so-called agent. In this case, the testing procedure resembles a two-player game, with the agent playing against Driver Assistance System 2 to provoke an error in Driver Assistance System 2.

[0083] Preferably, the performance is characterized by a fictitious reward, and changes are made based on a cost function or an optimization of the cost function. Preferably, the cost function is designed to maximize the fictitious reward. The performance is preferably higher the more dangerous the resulting driving situation is, in particular the shorter the calculated time until a collision.

[0084] A workflow that includes a procedure for testing a driver assistance system may further comprise the following steps: In a first step, the driver assistance system being tested is integrated, along with a suitable vehicle dynamics model, e.g., VSM®, into a suitable modeling and integration platform, e.g., MobiConnect®. A 3D simulation environment is also preferably provided on this integration platform.

[0085] In a second step, a scenario template is created that generically describes the road properties (e.g., using OpenDRIVE®), road users, and vehicle maneuvers (e.g., using OpenSCENARIO®). Optionally, virtual scenarios can be generated based on data from real-world driving. Suitable data sources include GPS data, sensor data, object lists, etc.

[0086] In a third step, the parameters that an algorithm can use to modify scenarios are identified and selected. Value ranges are then defined for these parameters, specifying the parameters within which they can fluctuate. For example, the algorithm could be configured to continuously assign values ​​between 5 and 35 m / s for the scenario parameter "vehicle speed of the preceding vehicle." Preferably, not only parameters relating to the environment of vehicle 1, controlled by the driver assistance system 2, can be selected, but also the trajectories of vehicle 1 can be modified by the algorithm. Trajectories for other road users can also be modified as parameters. Individual trajectories for each road user are defined using waypoints and time steps.The trajectories can then be changed by altering the position of the waypoints and the distance between the waypoints.

[0087] In a fourth step, specific criteria are predefined to control the iterative generation of scenarios. If the predefined criterion is the time until a collision occurs, the algorithm will attempt to minimize this time and search for parameter values ​​in the scenarios that would lead to an accident.

[0088] In a fifth step, suitable termination conditions are defined. A possible termination condition is, for example, that the time until a collision occurs is 0.25 seconds or that a maximum number of iterations has been reached.

[0089] In a sixth step, an initial set of parameter values ​​is generated. These are created randomly, selected manually, or based on real test drives. Together with the previously generated scenario template, concrete scenarios can be created in this way, which can then be executed in the 3D simulation.

[0090] The procedure for testing a driver assistance system, as described above, can then be carried out.

[0091] Figure 4 shows an exemplary embodiment of a system for testing a driver assistance system.

[0092] This system 10 preferably comprises means 11 for simulating a scenario in which the vehicle 1 is located, means 12 for operating the driver assistance system 2 in an environment of the vehicle 1 based on the simulated scenario, means 13 for observing a driving behavior of the driver assistance system 2 in the environment of the vehicle 1, means 14 for determining a driving situation which arises from the driving behavior of the driver assistance system 2 in the environment of the vehicle, means 15 for determining a quality of the simulated scenario 3 depending on a predefined criterion with regard to the driving situation, in particular a dangerousness of the driving situation that arose, means for checking a termination condition 16 and means 17 for changing the simulated scenario 3 based on the determined quality until a termination condition is reached.

[0093] Preferably, the aforementioned means are formed by a data processing system. However, the means 12 for operating the driver assistance system in the vicinity of the vehicle 1 can also be formed by a test bench, in particular a test bench for a driver assistance system or a vehicle. In this case, the means 13 for observing the driving behavior of the driver assistance system 2 can be formed, at least in part, by sensors.

[0094] The means 17 for changing the simulated scenario can preferably be trained as an agent.

[0095] Preferably the system has an interface 18, which may preferably be designed as a user interface or as a data interface.

[0096] It should be noted that the exemplary embodiments are merely examples and are not intended to restrict the scope of protection, application, or structure in any way. Rather, the preceding description provides the person skilled in the art with a guideline for implementing at least one exemplary embodiment, whereby various modifications, particularly with regard to the function and arrangement of the described components, can be made without departing from the scope of protection as defined by the claims and these equivalent combinations of features. Reference sign list

[0097] A, B Area of ​​scenarios 1 Vehicle 2 Driver assistance system 3 Scenario 4 Motorcycle 5a, 5b, 5c, 5d Other vehicles 6 Pedestrians 11 Means for simulating 12 Means for operating a driver assistance system 13 Means for observing driving behavior 14 Means for determining a driving situation 15 Means for determining quality 16 Means for changing a scenario 17 Means for checking a termination condition 18 Interface

Claims

1. Computer-implemented method (100) for testing a driver assistance system (2) for a vehicle (1), comprising the following steps: simulating (101) a scenario (3) in which the vehicle (1) is located; operating (102) the driver assistance system (2) in an environment of the vehicle (1) based on the simulated scenario (3); observing (103) a driving behavior of the driver assistance system (2) in the environment of the vehicle (1); determining (104) a driving situation that arises from the driving behavior of the driver assistance system (2) in the environment of the vehicle (1); determining (105) a quality of the simulated scenario (3) as a function of a predefined criterion with respect to the resulting driving situation, in particular a danger of the resulting driving situation; checking (106) at least one termination condition of the method (100); and changing (107) the simulated scenario (3) on the basis of the determined quality until the at least one termination condition is fulfilled, wherein, when changing the simulated scenario (3), a new scenario is created in which parameters have been exchanged, parameters have been omitted, and / or new parameters have been added.

2. Method (100) according to claim 1, wherein, when simulating the scenario, a speed, in particular an initial speed, of the vehicle (1) and / or a trajectory of the vehicle (1) is specified.

3. Method (100) according to claim 1 or 2, wherein when changing the simulated scenario, only values of parameters of the simulated scenario are changed.

4. Method (100) according to one of the preceding claims 1 to 3, wherein when the simulated scenario (3) is changed, a new scenario is created which consists of scenarios combined in succession.

5. Method (100) according to one of the preceding claims 1 to 4, wherein when determining the quality, the quality is characterized by a fictitious reward and the change is made on the basis of a cost function which is designed to maximize the fictitious reward.

6. Method (100) according to one of the preceding claims 1 to 5, wherein the modification of the simulated scenario (3) is performed using evolutionary algorithms.

7. Method (100) according to one of the preceding claims 1 to 6, wherein, on the basis of the determined quality, a utility function is approximated which describes the quality value of a specific simulated scenario (3).

8. Method (100) according to one of the preceding claims 1 to 7, wherein the driver assistance system (2) is simulated.

9. Method (100) according to one of the preceding claims 1 to 8, wherein a strategy for changing the scenario (3) is continuously improved during test operation by means of a reinforcement learning methodology based on the determined quality until the termination condition is reached.

10. Method (100) according to one of the preceding claims 1 to 9, wherein historical data from previous test operations of a driver assistance system, in particular the driver assistance system (2) to be tested, are taken into account during the initial simulation of the scenario.

11. Method (100) according to one of the preceding claims 1 to 10, wherein, when operating the driver assistance system (2), data relating to the environment (4) of the vehicle (1) is fed into the driver assistance system (2) and / or the driver assistance system (2), in particular its sensors, are stimulated on the basis of the environment (4) of the vehicle (1).

12. System (10) for testing a driver assistance system (2) for a vehicle (1), comprising: means (11) for simulating a scenario (3) in which the vehicle (1) is located; means (12) for operating the driver assistance system (2) in an environment of the vehicle (1) based on the simulated scenario (3); means (13) for observing a driving behavior of the driver assistance system (2) in the environment of the vehicle (1); means (14) for determining a driving situation resulting from the driving behavior of the driver assistance system (2) in the environment of the vehicle (1); means (15) for determining a quality of the simulated scenario (3) as a function of a predefined criterion relating to the driving situation, in particular a danger of the driving situation that has arisen; means (16) for checking at least one termination condition of the method (100); and means (17) for changing the simulated scenario (3) on the basis of the determined quality until the at least one termination condition is fulfilled; wherein, when the simulated scenario (3) is changed, a new scenario is created in which parameters have been exchanged, parameters have been omitted, and / or new parameters have been added.

13. System (10) for testing a driver assistance system (2) for a vehicle (1), in particular according to claim 12, comprising an agent (16), wherein the agent (16) is configured to generate a scenario (3) and to provoke an error in the driver assistance system by changing the scenario (3), and wherein a strategy for changing the scenario (3), in particular by means of a reinforcement learning methodology, is continuously improved by interaction of the agent (21) with the driver assistance system (2) during operation until a termination condition is reached.

14. System (10) according to claim 13, wherein the agent (16) is configured to observe a driving situation arising from the driving behavior of the driver assistance system (2) in an environment (4) of the vehicle (1) based on the simulated scenario (3) and to determine a quality of the scenario (3) as a function of the danger of the driving situation that has arisen.

15. System (10) according to claim 13 or 14, wherein the agent (16) is pre-trained based on historical data and this data is taken into account by the agent (16) during the initial simulation of the scenario (3).