Method for testing a driver assistance system of a vehicle - Patents.com

Abstract

The present invention relates to a system (10) for generating scenarios for testing a driver assistance system of a vehicle (1), and a corresponding method, the system comprising: - means (11) for simulating a traffic situation (3) comprising a vehicle and at least one further road user (1, 4, 5a, 5b, 5c, 5d, 6), the first road user (4) being capable of being controlled by the first user (2); a first user interface (12) for outputting a virtual environment based on a virtual traffic situation (3) to a first user (2) via the first user interface (12), in particular an at least visual one; a second user interface (13) for capturing inputs of the first user (2) for controlling at least one first road user (4) in the virtual environment; means (14) for operating a driver assistance system (2) in the virtual environment of the vehicle (1) based on a simulated traffic situation (3); A means for importing the scenario (15); and A means for determining the quality factor of the obtained scenario (16); Equipped with.

Description

[Technical field]

[0001] The present invention relates to a computer-implemented method for testing at least one driver assistance system for a vehicle.The present invention further relates to a corresponding system for testing at least one driver assistance system. [Background technology]

[0002] The penetration of driver assistance systems (Advanced Driver Assistance Systems - ADAS), which, if further advanced, would enable autonomous driving (Autonomous Driving - AD), continues to increase in both passenger and commercial vehicle sectors. Driver assistance systems make an important contribution to increasing actual road safety and serve to improve driving comfort.

[0003] In addition to systems which act in driving safety, such as ABS (Anti-lock Braking System) and ESP (Electronic Stability Program), a number of driver assistance systems are being promoted in the passenger car and commercial vehicle sector.

[0004] Driver assistance systems already in use to increase safety on real roads are parking assistance and adaptive automatic vehicle spacing control, also known as adaptive cruise control (ACC), which appropriately adjusts the desired speed selected by the driver to the distance from the vehicle driving in front. Further examples of such driver assistance systems are, in addition to ACC, ACC stop-and-go systems that automatically move the vehicle further in traffic jams or stationary traffic, lane departure warning or lane assistance systems that automatically keep the vehicle in its lane, and pre-crash systems that, for example, prepare or initiate braking in case of a possible collision in order to drain the vehicle's kinetic energy, as well as potentially initiating further measures if a collision is unavoidable.

[0005] These driver assistance systems increase safety on the roads, both by warning the driver of dangerous situations and by initiating automatic intervention to avoid or minimize accidents, for example by activating an emergency braking function. In addition, functions such as automatic parking, automatic lane keeping and automatic proximity control increase driving comfort.

[0006] The safety and comfort benefits of a driver assistance system are only positively perceived by vehicle occupants when the assistance provided by the driver assistance system is safe, reliable, and, where possible, convenient.

[0007] Furthermore, every driver assistance system, depending on its function, has to handle a given traffic scenario with maximum safety for the vehicle itself and without endangering other vehicles or other road users, respectively.

[0008] The degrees of automation of a vehicle are divided into so-called automation levels 1 to 5 (see, for example, the SAE J3016 standard). The present invention particularly relates to vehicles with driver assistance systems at automation levels 3 to 5, which are generally considered to be autonomous.

[0009] There are many and varied challenges in testing such systems. In particular, a balance needs to be found between the test effort and the test scope. The main challenge when testing ADAS / AD functions is to thereby show a guaranteed function of the driver assistance system in all possible situations, including particularly dangerous driving situations. Such dangerous driving situations contain a certain degree of danger, since a non-reaction or an incorrect reaction by the respective driver assistance system may lead to an accident.

[0010] Testing of driver assistance systems therefore needs to allow for a large number of driving situations that may occur in various scenarios. The range of possible scenarios therefore generally covers many aspects (e.g., different road characteristics, behavior of other road users, weather conditions, etc.). From this practically infinite and multi-dimensional parameter range, it is particularly important in testing driver assistance systems to derive these parameter configurations for dangerous scenarios that may lead to abnormal or dangerous driving situations.

[0011] As shown in FIG. 1, such dangerous scenarios have a much lower probability of occurrence relative to normal scenarios.

[0012] To validate the corresponding driver assistance systems, scientific publications consider that operating a vehicle with autonomous operation will only be statistically safer than a human-controlled vehicle when each driver assistance system has completed 275 million miles of accident-free driving. This is practically impossible to achieve in real test drives, especially considering the very tight development cycles and quality standards required by the automotive industry. For the reasons mentioned above, it is unlikely that a sufficient number of dangerous scenarios, or driving situations resulting from these scenarios, respectively, will be incorporated.

[0013] It is known from the prior art to use real test drive data from a real fleet of test vehicles to validate and verify driver assistance systems and to derive scenarios from recorded data. It is further known to use full factorial designs for validation and verification. [Prior art documents] [Non-patent literature]

[0014] [Non-Patent Document 1] “Metrics for assessing the criticality of traffic situations and scenarios” publication, P. Junietz et al., “11th Driver Assistance Systems and Automated Driving Workshop”, FAS 2017 Summary of the Invention [Problem to be solved by the invention]

[0015] One object of the present invention is to be able to test driver assistance systems, in particular automated driving functions, in dangerous scenarios. In particular, it is an object of the present invention to identify dangerous scenarios for driver assistance systems. [Means for solving the problem]

[0016] This problem is solved by the teaching of the independent claims. Advantageous embodiments can be found in the dependent claims.

[0017] A first aspect of the invention relates to a computer implemented method for testing a driver assistance system of a vehicle, comprising the following operational steps: - simulating a virtual traffic situation comprising a vehicle and at least one further road user, the first road user being controllable by the first user and the further road user not being controllable by the user being controlled automatically, in particular by artificial intelligence or by logic-based control; - outputting a virtual environment of at least one first road user to the first user via a first, in particular at least visual, user interface based on the virtual traffic situation; capturing inputs of a first user (2) for controlling at least one first road user in the virtual environment of the first road user via a second user interface, the captured inputs of the first user and the resulting interaction of the at least one first road user with the virtual environment being integrated into a simulation of a virtual traffic situation; operating a driver assistance system in the virtual environment of the vehicle based on the simulated traffic situation, the driver assistance system displaying a driving behavior; - capturing a scenario resulting from a driving behavior of a driver assistance system in a virtual environment of the vehicle; - determining a quality factor of the obtained scenario depending on predefined criteria, in particular the hazard of the obtained scenario; outputting the quality factor to a user via a first or second user interface, in particular a display; It is.

[0018] A second aspect of the present invention relates to a system for generating scenario data for testing a driver assistance system of a vehicle, the system comprising: - means for simulating a virtual traffic situation comprising a vehicle and at least one further road user, the first road user being controllable by the first user and the further road user not being controllable by the user being controlled automatically, in particular by artificial intelligence or by logic-based control; a first, in particular at least visual, user interface for outputting a virtual environment of at least one first road user to a first user via the first, in particular at least visual, user interface based on the virtual traffic situation; a second user interface for capturing inputs of the first road user for controlling at least one first road user in the virtual environment of the first road user via the second user interface, the captured inputs of the first user and the resulting interaction of the at least one first road user with the virtual environment being integrated into a simulation of a virtual traffic situation; means for operating a driver assistance system in the vehicle's virtual environment based on the simulated traffic situation, whereby the driver assistance system displays a driving behavior; and means for capturing scenarios resulting from driving behavior of a driver assistance system in the vehicle virtual environment; A means for determining the quality factor of the obtained scenarios according to predefined criteria, in particular the hazard of the obtained scenarios; Equipped with Here, the first or second user interface, particularly a display, is further configured to output the quality factor to a user.

[0019] Further aspects of the invention relate to computer programs and computer readable media.

[0020] A user within the meaning of the present invention is a natural person, ie a human being.

[0021] A driver assistance system within the meaning of the present invention is preferably a driver assistance system configured to support the driver during the driving process or to at least partially control the vehicle, in particular at automation levels 3 to 5, or more particularly an automated driving function.

[0022] A road user within the meaning of the invention is preferably any object which performs an active part of the traffic, in particular a road user is a person, an animal or a vehicle.

[0023] Extraction within the meaning of the present invention preferably denotes separation or isolation.

[0024] In particular, the scenarios are separated or respectively isolated from the scenario data, and data ranges are thereby preferably selected within the scenario data.

[0025] Scenario data within the meaning of the present invention is preferably characterized by the positions and movements of road users and by the positions of static objects with respect to the scenario.

[0026] A scenario within the meaning of the invention is preferably formed from a series of, in particular static, scenes in a time series. Thereby, the scene shows the spatial configuration of at least one other object relative to its own object, for example the arrangement of road users. The scenario preferably incorporates dynamic and static content. A model for the systematic description of scenarios is preferably used here, which is more preferably the model of the PEGASUS project (https: / / www.pegasusprojekt.de), with six independent layers: 1. Roads (geometry,...), 2. Road facilities and rules (traffic signals,...), 3. Temporary changes and events (road works,...), 4. Moving objects (vehicles moving relative to the vehicle under test, objects related to traffic, such as pedestrians), 5. Environmental conditions (lighting situation, road weather,...), 6. Digital information (V2X, digital data / maps,...). The scenarios include, inter alia, driving situations in which the driver assistance system at least partially controls a vehicle equipped with the driver assistance system, referred to as the ego vehicle, which automatically performs at least one vehicle function of the ego vehicle.

[0027] A traffic situation within the meaning of the present invention preferably represents the totality of all road user environments in the traffic included in a defined spatial region and / or a defined time period or time point. These road user environments are preferably incorporated into the selection of applicable behavior patterns at a particular time point. Preferably, the traffic situation includes all relevant states, possibilities and determinants of actions. Although not required, the traffic situation can also be represented in terms of road users or objects.

[0028] The simulated measurement variables within the meaning of the present invention are preferably selected from the following group: the speed of the road user, in particular the initial speed, the direction of movement, in particular the path of the road user, the lighting conditions, the weather, the road surface, the temperature, the number and position of static and / or dynamic objects, the speed and direction of movement of dynamic objects, in particular the path, the state of the signal system, in particular a traffic light system, traffic signs, the number of lanes, acceleration or braking deceleration of the road user or object.

[0029] A predefined arrangement of measured variables within the meaning of the present invention is preferably an arrangement of values ​​of one or more measured variables, in particular a chronological sequence.

[0030] Labelling within the meaning of the present invention preferably means providing a classification designation.

[0031] The risk of a scenario within the meaning of the present invention preferably indicates a spatial or temporal proximity to a traffic situation without obtaining any possible accident-free outcome (by one's own efforts and taking into account significant uncertainties). The risk is at its maximum when an accident can no longer be avoided. Preferably, the risk is also called criticality. When considering the driving behavior or driving ability of the driver assistance system, the risk can characterize the probability of an accident and / or the calculated duration until the time point of impact. The maximum risk is preferably present when the calculated duration is 0 seconds and / or when the probability of an accident is P=1. The increasing accident probability can be triggered in particular by driving maneuvers, such as, for example, steering, braking, evasive reactions during acceleration, or significant gradient changes (so, for example, the vehicle swerves due to a sharp maneuver). The increasing accident probability can also force a violation of the driving task or its actual route (through evasive maneuvers), in particular with respect to other road users (in logical or AI-based guidance) and in dangerous driving situations, respectively. An increased accident probability may also result from external factors affecting the first road user or other road users, especially if the driver is, for example, not visible.

[0032] A quality factor within the meaning of the present invention preferably characterizes a simulated scenario. The quality factor is preferably understood as the quality or nature and / or relevance of the simulated scenario with respect to the danger of the driving situation for a particular driver assistance system.

[0033] Relevance within the meaning of the present invention is preferably understood as the frequency with which a scenario occurs in road traffic, for example a backlight scenario is more relevant than a scenario where an aircraft lands on a road, relevance also preferably depends on the region to which road traffic is relevant, for example a scenario may be relevant in Germany but irrelevant in China.

[0034] The vehicle environment within the meaning of the present invention is preferably formed at least by road users and other objects related to the vehicle guidance provided by the driver assistance system. In particular, the vehicle environment includes the scenery and dynamic elements. The scenery preferably includes all static elements.

[0035] Abort conditions within the meaning of the present invention are preferably defined objectively or may be effected by user input.

[0036] Means within the meaning of this specification may be designed as hardware and / or software, in particular as a processing unit, in particular as a digital processing unit, in particular as a microprocessor unit (CPU), preferably in data or signal connection to a memory and / or a bus system and / or having one or more programs or program modules. The CPU may be configured to process commands implemented as programs stored in the memory system, to take input signals from the data bus and / or to send output signals to the data bus. The memory system may include one or more, in particular various, storage media, in particular optical, magnetic, solid-state and / or other non-volatile media. The programs may be designed to perform or be capable of performing the methods described herein, such that the CPU may execute the steps of such methods and then in particular generate the scenarios.

[0037] The present invention is based on an approach that generates scenarios by running real people, but without the need for any test drives in real traffic.

[0038] According to the invention, that is, a real driver drives a vehicle in a simulated traffic situation, in particular in a virtual environment or a virtual environment created by the simulated traffic situation, respectively. The invention allows a crowdsourcing approach to generate scenarios. One or more users can then navigate their selected road users through the virtual traffic situation in the simulator. When simulating a virtual traffic situation, as in real traffic, a virtually infinite number of different scenarios can arise due to the virtually infinite possibilities of navigating options for the road user(s) as well as other chance mechanisms. The invention uses predefined criteria to decide between the occurrence of a known scenario or the occurrence of a new scenario. To that end, the simulation process and in particular the simulation data generated therefrom are continuously analyzed / monitored.

[0039] The crowdsourcing approach can thereby take advantage of people's natural behavioral instincts. The method of the present invention, or even a suitable system, can be provided to users in this way. These users can then drive the simulated traffic "for fun". Alternatively, a task can be set for the users, for example to get from point A to point B as quickly as possible while observing traffic rules or with the need to collect several objects at various locations. Furthermore, the users may not be able to concentrate when navigating through the simulated traffic, for example, they may need to provide certain voice inputs or the like.

[0040] The simulation physics correspond to reality, thereby generating the most realistic scenario data possible. This applies in particular to the physics of the road users and the physics of the environment. It is not possible to drive through objects or the like. In particular, preferentially, multiple users navigate multiple road users in a simulated traffic.

[0041] The feedback of the quality factor of the scenario to the user conveys the quality or danger of the scenario resulting from his activity. The user can then try to increase the quality factor by adapting his behavior in controlling the road users he controls. Preferably, the user can request the simulation means simulating the scenario to repeat a previously run scenario. In this way, the user can optimize his control behavior as much as necessary until the highest possible quality factor, i.e. the highest danger, is reached for a particular scenario, in particular in compliance with traffic regulations.

[0042] Preferably, the resulting driving situation is recorded as scenario data so that it can be reproduced at a later time in a simulated scenario.

[0043] In one advantageous embodiment of the method, the scenario data generated thereby is already labeled, in particular the objects of the virtual traffic situation, since information about the properties of the objects is available in the simulation and thus information can be associated with the objects.

[0044] This is particularly advantageous for data from real test runs where all objects need to be labelled, which is generally very time-consuming as this can only be done by real people.

[0045] In one advantageous embodiment of the method, the simulated scenario of the identified quality factor is modified until an abort condition is reached. Preferably, the simulated scenario is outputted, additionally or alternatively, until its quality factor reaches an abort condition. More preferably, the simulated scenario is outputted only when the quality factor reaches an abort condition. In this case, the test procedure is resumed or respectively repeated until the user's driving behavior and / or the simulation of the scenario results in a behavior of the driver assistance system that violates a predefined target value that serves as an abort condition. Such an abort condition can be, for example, of duration less than 0.25 seconds until the time point of the crash, or even a specific time budget, for example, a maximum simulation time of 600 hours.

[0046] Assuming a sufficiently long and sufficient time budget, the present invention makes it possible to achieve a high probability of proper functioning of the ADAS or AD system.

[0047] In a further advantageous embodiment of the method, the modification is at least partly performed by the user via the first user interface or the second user interface using an editor, where the user's actions are preferably recorded and stored in a control table. The possibility of modifying the scenario not only allows the user to influence the obtained driving situation through his driving behavior, but also to directly influence the simulated driving scenario. Thereby, the user has an additional influence on the quality factor of the obtained driving situation. The user can therefore optimize the existing driving scenario in order to enable a driving situation with the highest possible quality factor in cooperation with the user's driving behavior.

[0048] In a further advantageous embodiment of the method, the vehicle speed, in particular the initial speed, and / or the vehicle path are predefined when simulating a scenario, these default values ​​being boundary conditions in the testing of the driver assistance system and preferably also modifiable by the user.

[0049] In a further advantageous embodiment of the method, when modifying a simulated scenario, only the parameter values ​​of the simulated scenario are changed. Thus, the nature of the respective simulated scenario remains. No new parameters are added, for example by adding additional vehicles. Thereby, the user remains in the respective simulated scenario under test and does not exist in a completely newly created scenario.

[0050] In a further advantageous embodiment of the method, the parameters of the scenario are selected from the following group depending on the type of driver assistance system being tested: the speed of the road user, in particular the initial speed, the direction of movement, in particular the path of the road user, the lighting conditions, the weather, the road surface, the temperature, the number and position of static and / or dynamic objects, the speed and direction of movement of dynamic objects, in particular the path, the state of the signal system, in particular a traffic light system, the traffic signs, the number of lanes, the acceleration or braking deceleration of the road user or object.

[0051] Changing these parameters can greatly affect the risk of a scenario.

[0052] In a further advantageous embodiment of the method, the quality factor is characterized by a reward, in particular a conceptual reward, for the user. In particular, the user receives a reward granted, for example in a virtual account, depending on the achieved quality factor. This can be achieved by a user who always tries to achieve driving situations with a high quality factor.

[0053] In a further advantageous embodiment of the method, the quality factor is higher the more dangerous the respectively obtained scenario is, in particular the shorter the calculated duration to the time point of collision is. This has the advantage that the most dangerous possible driving situations are identified, even if they occur only very rarely in normal traffic. The number of test kilometers to be completed can also be reduced accordingly.

[0054] In a further advantageous embodiment of the method, the simulated scenario is modified using an evolutionary algorithm, also called a genetic algorithm. Modification by such an algorithm involves different algorithms being crossed and altered. Candidates for the next iteration step, i.e. the next variant of the simulated scenario, are formed from the algorithms thereby obtained.

[0055] In a further advantageous embodiment of the method, a utility function is approximated based on the determined quality factor specifying the quality factor value exhibited by a particular simulated scenario, which allows the calculation of a reward for the user.

[0056] In a further advantageous embodiment of the method, the driver assistance system is simulated. This means that, according to the concept of "software in the loop", only the software or actual code of the driver assistance system, respectively, is taken into account or implemented when simulating a virtual traffic situation. This allows a purely simulation-based test of the driver assistance system. Simulation or the provision of signals to the actual driver assistance system is not required thereby.

[0057] In a further advantageous embodiment of the method, historical data from previous test operations of a driver assistance system, in particular a driver assistance system, is incorporated into the initial simulation of the scenario.

[0058] The use of historical data can be used to pre-train algorithms used to simulate traffic scenarios. Doing so can reduce the duration required to determine dangerous scenarios. Furthermore, it is also possible to use algorithms that have been trained in another, particularly similar, ADAS / AD system. Thus, in particular, so-called regression tests can be performed to ensure that changes to already tested parts of the driver assistance system software do not introduce new errors.

[0059] In a further advantageous embodiment of the method, data relating to the vehicle environment is provided to the driver assistance system during operation of the driver assistance system and / or the driver assistance system, in particular its sensors, are stimulated based on the vehicle environment. This embodiment allows testing of real driver assistance systems or even real vehicles with such real driver assistance systems.

[0060] Doing so allows testing of the driver assistance system, in particular its software or its entire hardware, on a test bench, in particular hardware-in-the-loop methods can be used for that purpose.

[0061] The features and advantages explained above in relation to the first aspect of the invention also apply to the other aspects of the invention, and vice versa, accordingly.

[0062] Further features and advantages are shown in the following description of exemplary embodiments with reference to the figures, which are at least partially illustrated diagrammatically. [Brief description of the drawings]

[0063] [Figure 1] A diagram of the probability of a scenario occurring as a function of its criticality and / or hazard. [Diagram 2] FIG. 2 is a block diagram of an exemplary embodiment of a method for generating a scenario. [Figure 3a]This is a first example of a simulated virtual traffic situation. [Figure 3b] 2 is a second example of a simulated hypothetical traffic situation. [Figure 4] 3 is a third example of a simulated hypothetical traffic situation. [Diagram 5] 1 is an exemplary embodiment of a system for generating scenario data for testing a driver assistance system of a vehicle. [Figure 6] 1 is an exemplary embodiment of a means for operating a driver assistance system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0064] Figure 1 shows the probability of occurrence of a scenario as a function of its criticality and / or danger. The probability of occurrence is the probability that the scenario occurs in real road traffic.

[0065] It is noted in FIG. 1 that the majority of scenarios are of relatively low complexity and / or risk, which also correspond to the normal life experience of a motor vehicle driver. The range of these scenarios is designated as "A" in FIG. 1. In contrast, high complexity scenarios, the range of which is designated as "B" in FIG. 1, occur with a relatively low frequency. However, it is precisely these "B" scenarios with high complexity that are of major relevance for analyzing the functionality of a driver assistance system.

[0066] Therefore, obtaining a sufficient number and diversity of different scenarios of high "B" complexity during testing of driver assistance systems usually requires running a very large number of scenarios based on the distribution curves shown. An alternative approach is described below, in which these high complexity scenarios are introduced specifically for testing driver assistance systems, thus reducing the number of scenarios to be run and therefore the number of kilometers to be covered.

[0067] A method for generating a large number of different scenarios with maximum possible complexity for testing a driver assistance system is described below with reference to Figures 2 to 3b.

[0068] A virtual traffic situation 3 in which a driver assistance system 7 is to be tested is simulated in a first work step 101. The virtual traffic situation 3 has a plurality of virtual road users 1, 4, 5a, 5b, 5c, 5d, 6, in which a vehicle 1 is controlled by a driver assistance system 7 under test. Preferably, at least one further first road user 4 of the plurality of road users can be controlled by a first user 2, and these road users 5a, 5b, 5c, 5d, 6 that cannot be controlled by a user are automatically controlled in this virtual traffic situation 3. Preferably, artificial intelligence or logic-based control is used here. Preferably, there can be a plurality of road users controlled by users, i.e. humans, in the simulated virtual traffic situation 3. A plurality of driver assistance systems 7 in a single vehicle or in a plurality of vehicles can also be tested simultaneously.

[0069] In particular, a traffic flow model such as PTV-Vissim® or Eclipse SUMO, in particular version 1.8.0, is preferably used to simulate the virtual traffic situation.

[0070] There are essentially two approaches to simulating a virtual traffic situation: the simulation is based on data obtained in a real test drive. In this case, the parameters of the individual objects, such as for example the speed of the road users, can be changed or used as captured during the real test drive. In an alternative embodiment of the method, the traffic situation 3 is generated based on a purely mathematical algorithm. Preferably, there can be a mixture of the two approaches.

[0071] An example of such a simulated traffic situation 3 is shown in Fig. 3a. In the traffic situation 3 shown in Fig. 3a, a pedestrian 6 is crossing the road. A motorcycle 4 controlled by a first user approaches the pedestrian 6 in the lane at his side. Other vehicles 5b, 5c, 5d are parked alongside in the lane, making the pedestrian 6 invisible or only slightly visible to the motorcycle 4 controlled by the first user 2. A further vehicle 5a at the same level as the pedestrian 6 is driving in the second lane as oncoming traffic. The vehicle 1 approaches the rear of the other vehicle 5a, whose longitudinal and lateral control is provided by the driver assistance system 7. It cannot be deduced from Fig. 3a whether this motorcycle 4 is visible to the vehicle 1 controlled by the first user 2.

[0072] The other vehicles 5a, 5b, 5c, 5d, the pedestrians 6 as well as the motorcycle 4 form the virtual environment of the vehicle 1 controlled by the driver assistance system in the traffic situation 3. Correspondingly, the further vehicles 5a, 5b, 5c, 5d and the pedestrians 6 form the virtual environment of the motorcycle 4 controlled by the user in the traffic situation 3.

[0073] Depending on how the first user 2 reacts or behaves in the first scenario resulting from the traffic situation 3, i.e. on which driving behavior the first user 2 exhibits in the virtual environment of the motorcycle 4 that he controls, a more or less dangerous scenario is obtained. Generally, the more complex the scenario, the more dangerous it is and therefore the more difficult it is for the driver assistance system 7 to process. For example, if the first user 2 brakes to stop the motorcycle 4, as shown in Fig. 3a by the bar in front of the movement arrow of the motorcycle 4, the vehicle 1 controlled by the driver assistance system can pass the oncoming vehicle 5a in the other lane without any trouble.

[0074] Figure 3b shows the same hypothetical traffic situation 3 as in Figure 3a, with a motorcycle controlled by a first user 2 in the same initial traffic situation as in Figure 3a. The first user 2 continues to drive the motorcycle at an unreduced speed, as indicated by the movement arrow away from the motorcycle 4 controlled by the first user 2.

[0075] In this case, there is a high probability of developing a scenario in which the motorcycle 4 collides with the vehicle 1 controlled by the first driver assistance system 7. This is also shown in Figure 3b. Such a driving situation / scenario corresponds to a very high risk.

[0076] The initial speed and / or the initial route of the vehicle 1 and / or motorcycle 4 are preferably pre-determined by simulating a traffic situation 3 .

[0077] In a second work step 102 the virtual traffic situation 3 is output to the first user 2 via the first user interface 12 .

[0078] Examples of possible user interfaces 12 are shown in FIG. 4 and preferably include a visual user interface, in particular a screen, an audio user interface, in particular a speaker, and / or a user interface for stimulating the sense of balance of the first user 2.

[0079] In a third work step 103 , inputs of the first user 2 for controlling the motorcycle 4 in the virtual environment of the first road user 1 are captured via the second user interface 13 .

[0080] A second user interface 13 is also shown in Figure 4. Preferably, these relate to the steering wheel, gear shift, hand brake, brake pedal, clutch, and / or accelerator, and other possible controls available to the driver in the vehicle.

[0081] However, depending on the type of road user 1, 4, 5a, 5b, 5c, 5d, 6 controlled by the user 2, other input means may also be provided as user interface 13, for example a type of joystick.

[0082] As explained above, the first road user 1 in Figures 3a and 3b is a motorcycle 4. The captured inputs of the first user 2 and the resulting interactions of the motorcycle 4, i.e. the first road user, with its virtual environment are incorporated into a simulation of the traffic situation 3 shown in Figures 3a and 3b.

[0083] The interactions in the traffic situation 3 shown in Figures 3a and 3b are, for example, how the first user 2 reacts to the initial scenario. Depending on the reaction of the first user 2 to this initial scenario, other road users, in particular the oncoming vehicle 1, as well as further oncoming vehicles 5a and pedestrians 6, also react. For example, when the driver assistance system 7 notices that the vehicle 1 controlled by the first user 2 does not reduce its speed, it is expected that the oncoming vehicle 1 will brake. These interactions then affect the evolution of the virtual traffic situation 3.

[0084] Generating the traffic situation 3 is therefore a continuous process, running continuously in a loop, as indicated by the arrows in FIG. 2, thereby generating simulation data.

[0085] During the simulation, objects that are part of the virtual traffic situation 3 are preferably labeled. This concerns both static as well as dynamic objects, so that subsequent data that can be obtained from the simulation data contains so-called ground truth information.

[0086] When the scenario data is used, for example, to test the driver assistance system 7, it is possible to understand which objects the driver assistance system 7 correctly detected and which ones it incorrectly detected. Examples of such labels are trees, pedestrians, cars, trucks, etc.

[0087] More preferably, an action is set in the driving situation 3 that encourages activity from the first user. For example, in the traffic situation 3 of Fig. 3a and Fig. 3b, this can be a further vehicle that follows the motorcycle 4 driven by the first user 2 and forces it to accelerate. An unexpected movement path of a pedestrian 6, for example, running away, can also be such an action.

[0088] In a fourth working step 104, the driver assistance system 2, and therewith the vehicle 1 which it controls, are operated in a vehicular virtual environment resulting from the position and orientation of the vehicle 1 in a simulated traffic 3 created by the simulation and control of a motorcycle 4 by a first user 2. As will be described below, only the hardware of the driver assistance system 7, with or without sensors, as well as the software of the driver assistance system 7 can thereby be tested in operation. Missing hardware components can also be simulated.

[0089] An example of such an operation in a virtual environment is shown in Fig. 4. A vehicle 1 with the functionality to be tested or a driver assistance system 2 to be tested is operated in a virtual road system including at least one virtual traffic situation 3. On its way from a starting point to a predefined target point, the vehicle 1 has to drive near a construction site created by the user 2, maneuver through simulated traffic, deal with traffic from a vehicle 4 with a real driver, etc., so as to affect the traffic situation. The dashed travel path is thereby obtained as an example. The road system with the prepared traffic situation 2 is preferably a base structure in which a developer, for example a vehicle manufacturer, can drive his vehicle, preferably anonymously. This means that the vehicle is known to be driving autonomously, and not the manufacturer and / or manufacturer. Preferably, not only the functionality but also the driving dynamics are thereby simulatedly displayed. The user manually controlling the road user can then let loose and try to confuse the driver assistance system 7 of the vehicle 1.

[0090] A scenario resulting from the driving behavior of the driver assistance system 7 for the vehicle 1 controlling in the traffic situation 3, or the environment of the vehicle 1, is imported in a fifth work step 105.

[0091] Both already known scenarios that have been performed before or that have been predefined as templates, as well as scenarios that have not yet been predefined, can thereby be verified.

[0092] Both types of scenarios are preferably defined by predefined configurations of simulated measurement variables that can be determined from the virtual traffic situation 3. These predefined configurations form templates for the scenarios or correspond to elementary maneuvers from which the occurrence of the scenario can be inferred. This can be for example a strong braking deceleration of the vehicle 1 in Figures 3a and 3b, which is used as a trigger condition for the occurrence of a scenario that is not yet predefined.

[0093] In a sixth working step 106, preferably a quality factor of the obtained scenario is determined depending on a predefined criterion, where the quality factor is preferably characterized by the criticality of one of the scenarios. Preferably, the more critical the obtained scenario is, the higher the quality factor. The criticality is preferably determined by the so-called "time to X" metric, such as that described, for example, in "Metrics for assessing the criticality of traffic situations and scenarios" publication, P. Junietz et al., "11th Driver Assistance Systems and Automated Driving Workshop", FAS 2017. In particular, what can thereby be used as criteria are the duration of time to impact point (time to impact), time to kickdown, time to maneuver, time to react, distance to nearest encounter, time to nearest encounter, worst time to impact, etc. Furthermore, preferably, the criticality is characterized by the probability of an accident.

[0094] In a seventh working step 107, the determined quality factor is output to the user 2, preferably via one of the aforementioned interfaces. This can be done after each new scenario is created or after running each newly created scenario in its entirety. However, it is preferred that this can only be done after running several scenarios in their entirety.

[0095] Further preferably, the first user 2 is granted a reward, in particular a conceptual reward, depending on the quality factor of the presented scenario. Further preferably, the conceptual reward is indicative of the quality factor.

[0096] In an eighth work step 108, first user inputs are captured, preferably via the first user interface 12, or the second user interface 13. These further inputs serve to modify the simulated traffic situation 3.

[0097] Preferably, the first user 2 can thereby change, replace or remove objects in the simulation of the traffic situation 3. The user can further add new objects. The first user 2 can thereby try to configure the traffic situation 3 to be as complex as possible. Doing so makes the user induce errors in the driver assistance system 7 when the vehicle 1 is driven. Thus, the user 2 can for example also specify, as a further road user, that the trailer of a truck is of the same colour as the sky, making it difficult for the light sensor of the driver assistance system to detect. The user can thereby preferably make use of multiple possibilities for modifying the traffic situation. Below are listed some examples where they are used individually or together: changing the surface pattern of an object (e.g. road signs printed on a truck), setting up a building site, e.g. with warning traffic lights in the middle of the road, running dinosaurs across the street (e.g. in connection with a carnival), deliberately placing objects so that they overlap, a person in white clothing in front of a white wall, an injured person on the road, a siren or horn (acoustic sensors can also be provided in driver assistance systems).

[0098] Thereby, preferably an editor of a type is used, which allows the first user 2 to configure the traffic situation 3. Furthermore, preferably, the operations of the first user 2 are recorded and stored in a control table. In that way it should be possible to determine which measures can be taken to create traffic situations 3 resulting in particularly complex scenarios.

[0099] The simulated traffic situation 3 is preferably modified in a ninth work step 109. This is done based on captured inputs of the first user 2, but also or alternatively an algorithm, in particular an evolutionary algorithm, can be used to modify the simulated traffic situation 3. Preferably, only existing parameters of the traffic situation 3 are changed during said modification, such as the color or speed of existing objects. However, the basic design of the traffic situation 3 is maintained.

[0100] Feasible parameters that can be influenced are the speed of the road user, in particular the initial speed, the direction of movement, in particular the path of the road user, the lighting conditions, the weather, the road surface, the temperature, the number and position of static and / or dynamic objects, the state and appearance of static and / or dynamic objects, the speed and direction of movement of dynamic objects, in particular the path, the state of the signal system, in particular the traffic light system, the traffic signs, the number of lanes, the acceleration or braking deceleration of the road user or object, signs of damage and / or deterioration of the road, the geographical direction of the traffic situation, etc. In particular, the color, surface pattern, shape and clothing of the objects and / or the position of the sun and the direction in which the sunlight shines on the traffic situation 3 can be changed.

[0101] Although the method 100 for testing a driver assistance system 7 has been described above with respect to one driver assistance system 7 under test and one user 2 controlling a first road user 4, it is also possible that the method 100 is used to test multiple vehicle driver assistance systems and / or there are multiple road users controlled by users. For example, the pedestrian 6 can also be controlled by a second user. It means that multiple vehicles can be controlled in a simulated traffic situation by the driver assistance system and / or multiple further road users can be controlled by further users.

[0102] FIG. 5 illustrates a system 10 for generating scenarios for testing driver assistance systems of a vehicle.

[0103] The system 10 preferably comprises means 11 for simulating a virtual traffic situation 3 with a plurality of virtual road users. In order to make the road users 4 controllable by a first user 2, the system further comprises at least one first user interface 12 and at least one second user interface 13.

[0104] The at least one first user interface, i.e. user interface 12, serves to output a virtual environment of the at least one first road user 1 to the first user 2. The virtual environment of the at least one first road user 4 is thereby determined based on the simulated virtual traffic situation 3. It is thereby substantially a representation of the virtual traffic situation 3 in the first scenario according to the perspective of the first road user 4 controlled by the first user 2.

[0105] As FIG. 5 shows, these user interfaces 12 are visual user interfaces, such as a screen, and audio interfaces, such as speakers, and possibly also devices capable of influencing the sense of balance of each user 2 .

[0106] The second user interface, i.e. user interface 13, is arranged to take inputs from each user 2. As shown in Fig. 4, these are preferably related to various operating elements, which, as already explained before, can be responsive to each road user 1 controlled by the user 2. If the road user 1 controlled by the first user 2 is a vehicle, the user interfaces 12, 13 are preferably arranged in the area of ​​a so-called seat box 19, which together with the user interfaces 12, 13 form a simulator, as shown in Fig. 4.

[0107] The system 10 preferably further comprises means 14 for operating the driver assistance system 2 in the virtual environment of the vehicle 1 based on the simulated traffic situation 3. The driver assistance system 2 drives the vehicle 1 through the traffic situation 3 and thereby exhibits a certain driving behavior. Furthermore, the system 10 preferably comprises means 15 for determining / detecting a scenario resulting from the driving behavior of the driver assistance system 2 in the virtual environment of the vehicle 1. Further preferably, the system 10 preferably comprises means 16 for determining a simulated quality factor of the obtained scenario, in particular a riskiness of the obtained driving situation of the obtained scenario, preferably depending on predefined criteria related to the obtained driving situation. The first or second user interface 12, 13, in particular a display, is further configured to output the quality factor to a user. Furthermore, a data interface may be provided, configured to output scenario data for further processing. Preferably, the means 11, 14, 15, 16, 17, 18 are part of a data processing device, preferably formed by a computer.

[0108] The means 14 for operating the driver assistance system 7 are again shown in detail in FIG.

[0109] Such means 14 preferably comprise a device 20 configured to simulate, based on the scenario data, a virtual environment of the vehicle 1. Furthermore, the means 20 is also configured to render this environment.

[0110] The interface 21 is configured to output, i.e. respectively emulate, the virtual environment of the driver assistance system 7. If the driver assistance system 7 comprises an optical camera, such an interface 21 can be a screen.

[0111] In the example shown in Figure 6, the sensor of the driver assistance system 7 is a radar sensor emitting a signal S. This signal S is acquired by a radar antenna 21 forming an interface.

[0112] Based on the acquired signals and the simulated environment, the simulating means 20 calculate a response signal S', which is then output via a radar antenna to the radar of the driver assistance system 7. By doing so, the functionality of the driver assistance system 7 can be tested. Depending on which components of the driver assistance system 7 are to be tested, the simulated virtual environment can be tested by emulating signals to the sensors of the driver assistance system 7, as shown in FIG. 6.

[0113] Alternatively, however, signals S' which are sent directly to the data processing unit 7 of the signals of the driver assistance system, as well as signals S' which are only processed by the software of the driver assistance system 7 can also be generated.

[0114] It should be noted that the exemplary embodiments are merely examples, which are in no way intended to limit the scope, field of application, and configuration of protection. Instead, the foregoing description provides a person skilled in the art with a guideline for implementing at least one exemplary embodiment, whereby various modifications can be made with respect to the function and configuration of the specifically described components without departing from the scope of protection obtained from the claims and equivalent combinations of features. [Explanation of symbols]

[0115] 1 Virtual road users and vehicles 2. First User 3. Virtual traffic situation 4 Virtual road user, motorcycle 5a, 5b, 5c, 5d Virtual Road Users 6 Virtual road users, pedestrians 7 Driver Assistance Systems 10. System 11 means 12 Primary User Interface 13 Secondary User Interface 14 means 15 means 16 means 17 means 18 means 19 Seat Box 20 Devices, means of simulation 21 Interface, radar antenna S, S' signals

Claims

1. A computer-implemented method (100) for testing a driver assistance system (2) of a vehicle (1), comprising the following operational steps: - simulating (101) a virtual traffic situation (3) comprising said vehicle (1) and at least one further road user (4, 5a, 5b, 5c, 5d, 6), a first road user (4) which can be controlled by a first user (2) and the further road user (5a, 5b, 5c, 5d, 6) which cannot be controlled by a user being controlled automatically, in particular by artificial intelligence or by logic-based control; - outputting (102) a virtual environment of at least one first road user (4) to said first user (2) via a first user interface (12), in particular an at least visual one, based on said virtual traffic situation (3); - a step (103) of capturing inputs of the first user (2) for controlling the at least one first road user (4) in the virtual environment of the first road user (4) via a second user interface (13), the captured inputs of the first user (2) and the resulting interaction of the at least one first road user (4) with its virtual environment being integrated into a simulation of the virtual traffic situation (3); operating (104) the driver assistance system (2) in the virtual environment of the vehicle (1) based on the simulated traffic situation (3), the driver assistance system (2) displaying driving behavior; - importing (105) scenarios resulting from driving behavior of the driver assistance system (2) in the virtual environment of the vehicle (1); - determining (106) a quality factor of said obtained scenario depending on predefined criteria, in particular the risk of said obtained scenario; outputting (107) said quality factor to said user via said first or second user interface, in particular a display; A computer-implemented method (100), comprising:

2. The method (100) of claim 1, further comprising the working step of taking (108) an input of the first user (2) via the first user interface (12) or the second user interface (13) for modifying the simulated traffic situation (3) preferably by means of an editor, the operations of the first user (2) being preferably recorded and stored in a control table.

3. 2. The method (100) of claim 1, further comprising a working step of modifying (109) the simulated scenario (3) based on the determined quality factor, in particular until an abort condition is reached.

4. The method (100) according to any one of claims 1 to 3, wherein a speed, in particular an initial speed, of the vehicle (1) and / or the at least one further first road user (4) and / or a route, in particular an initial route, of the vehicle (1) and / or the at least one further first road user (4) are predefined by the simulation.

5. The method (100) according to any one of claims 2 to 4, wherein when modifying the simulated traffic situation (3), only values ​​of existing parameters of the simulated traffic situation (3) are changed.

6. The parameters of said scenario (3) may be divided into the following groups depending on the type of driver assistance system (2) to be tested: speed of road users, in particular initial speed, direction of movement, in particular path of road users, lighting conditions, weather, road surface, temperature, number and position of static and / or dynamic objects, state and appearance of static and / or dynamic objects, speed and direction of movement, in particular path of dynamic objects, state of the signal system, in particular traffic light system, traffic signs, number of lanes, acceleration or braking deceleration of road users or objects, signs of damage and / or deterioration of the road, geographical direction of the traffic situation 6. The method (100) according to any one of claims 1 to 5, selected from:

7. The method (100) according to any one of the preceding claims, wherein the quality factor is characterized by a reward, in particular a notional reward, for the user.

8. The method (100) according to any one of claims 1 to 7, wherein the quality factor is higher the more dangerous the respectively obtained scenario (3), in particular the shorter the calculated duration until the time point of impact is.

9. The method (100) according to any one of claims 1 to 8, wherein the simulated scenario (3) is modified using an evolutionary algorithm.

10. 10. The method (100) according to any one of claims 1 to 9, wherein a utility function is approximated based on the determined quality factor specifying the quality factor value indicated by a particular simulated scenario (3).

11. The method (100) according to any one of the preceding claims, wherein the driver assistance system (2) is simulated.

12. The method (100) according to any one of claims 1 to 11, wherein historical data from previous test operations of the driver assistance system (2), in particular the driver assistance system (2), is incorporated into the initial simulation of the scenario.

13. 13. The method (100) according to any one of claims 1 to 12, wherein data relating to the virtual environment of the vehicle (1) is supplied to the driver assistance system (2) during operation of the driver assistance system (2) and / or its sensors, the driver assistance system (2) being stimulated based on the virtual environment of the vehicle (1).

14. A computer program comprising commands which, when run on a computer, cause the computer to carry out the steps of the method according to any one of claims 1 to 13.

15. A computer readable medium having stored thereon the computer program of claim 14.

16. A system (10) for generating scenario data for testing a driver assistance system of a vehicle (1), comprising: - means (11) for simulating a virtual traffic situation (3) including said vehicle and at least one further road user (1, 4, 5a, 5b, 5c, 5d, 6), where a first road user (4) can be controlled by a first user (2) and the further road user (5a, 5b, 5c, 5d, 6) which cannot be controlled by the user is controlled automatically, in particular by artificial intelligence or by logic-based control; a first user interface (12), in particular at least visual, for outputting a virtual environment of at least one first road user (4) to said first user (2) based on said virtual traffic situation (3); a second user interface (13) for capturing inputs of the first user (2) for controlling the at least one first road user (4) in the virtual environment of the first road user (4), the captured inputs of the first user (2) and the resulting interaction of the at least one first road user (4) with its virtual environment being integrated into a simulation of the virtual traffic situation (3); - means (14) for operating the driver assistance system (2) in the virtual environment of the vehicle (1) based on the simulated traffic situation (3), whereby the driver assistance system (2) displays a driving behavior; - means (15) for importing, in the virtual environment of the vehicle (1), scenarios resulting from the driving behavior of the driver assistance system (2); means (16) for determining a quality factor of said obtained scenario depending on predefined criteria, in particular the risk of said obtained scenario; Equipped with A system (10) wherein the first or second user interface (12, 13), in particular a display, is further configured to output the quality factor to the user (2).

Images