Method and system for generating scenario data for testing a vehicle driver assistance system - Patents.com

Abstract

The present invention relates to a system (10), and corresponding method, for generating scenario data for testing a driver assistance system of a vehicle, the system comprising: - means (11) for simulating a virtual traffic situation (3), in which at least one first road user (1) can be controlled by a first user (2) and in which simulation data are generated during the simulation; a first user interface (12) for outputting a virtual environment of at least one first road user (1) to a first user (2) based on the virtual traffic situation (3); a second user interface (13) for capturing inputs of a first user (2) for controlling at least one first road user (1) in the virtual environment of the first road user (1); means (14) for validating the generated simulation data against the occurrence of the scenario; means (15) for extracting scenario data relating to the scenario; A data storage device (16) for recording scenario data for testing the driver assistance system; Equipped with.

Description

[Technical field]

[0001] The present invention relates to a computer-implemented method for generating scenario data for testing a driver assistance system of a vehicle, and further to a corresponding system. [Background technology]

[0002] The penetration of driver assistance systems (Advanced Driver Assistance Systems - ADAS), which, if further advanced, would enable autonomous driving (Autonomous Driving - AD), continues to increase in both passenger and commercial vehicle sectors. Driver assistance systems make an important contribution to increasing actual road safety and serve to improve driving comfort.

[0003] In addition to systems which act in driving safety, such as ABS (Anti-lock Braking System) and ESP (Electronic Stability Program), a number of driver assistance systems are being promoted in the passenger car and commercial vehicle sector.

[0004] Driver assistance systems already in use to increase safety on real roads are parking assistance and adaptive automatic vehicle spacing control, also known as adaptive cruise control (ACC), which appropriately adjusts the desired speed selected by the driver to the distance from the vehicle driving in front. Further examples of such driver assistance systems are, in addition to ACC, ACC stop-and-go systems that automatically move the vehicle further in traffic jams or stationary traffic, lane departure warning or lane assistance systems that automatically keep the vehicle in its lane, and pre-crash systems that, for example, prepare or initiate braking in case of a possible collision in order to drain the vehicle's kinetic energy, as well as potentially initiating further measures if a collision is unavoidable.

[0005] These driver assistance systems increase safety on the roads, both by warning the driver of dangerous situations and by initiating automatic intervention to avoid or minimize accidents, for example by activating an emergency braking function. In addition, functions such as automatic parking, automatic lane keeping and automatic proximity control increase driving comfort.

[0006] The safety and comfort benefits of a driver assistance system are only positively perceived by vehicle occupants when the assistance provided by the driver assistance system is safe, reliable, and, where possible, convenient.

[0007] Furthermore, every driver assistance system, depending on its function, has to handle a given traffic scenario with maximum safety for the vehicle itself and without endangering other vehicles or other road users, respectively.

[0008] The degrees of automation of a vehicle are divided into so-called automation levels 1 to 5 (see, for example, the SAE J3016 standard). The present invention particularly relates to vehicles with driver assistance systems at automation levels 3 to 5, which are generally considered to be autonomous.

[0009] There are many and varied challenges in testing such systems. In particular, a balance needs to be found between the test effort and the test scope. The main challenge when testing ADAS / AD functions is to thereby show a guaranteed function of the driver assistance system in all possible situations, including particularly dangerous driving situations. Such dangerous driving situations contain a certain degree of danger, since a non-reaction or an incorrect reaction by the respective driver assistance system may lead to an accident.

[0010] Testing of driver assistance systems therefore needs to allow for a large number of driving situations that may occur in various scenarios. The range of possible scenarios therefore generally covers many aspects (e.g., different road characteristics, behavior of other road users, weather conditions, etc.). From this practically infinite and multi-dimensional parameter range, it is particularly important in testing driver assistance systems to derive these parameter configurations for dangerous scenarios that may lead to abnormal or dangerous driving situations.

[0011] As shown in FIG. 1, such dangerous scenarios have a much lower probability of occurrence relative to normal scenarios.

[0012] To validate the corresponding driver assistance systems, scientific publications consider that operating a vehicle with autonomous operation will only be statistically safer than a human-controlled vehicle when each driver assistance system has completed 275 million accident-free miles. Real test runs cannot actually achieve this, especially considering the very demanding development cycles and quality standards required in the automotive industry. For the reasons mentioned above, it is unlikely that a sufficient number of dangerous scenarios, or driving situations resulting from these scenarios, respectively, will be incorporated.

[0013] It is known from the prior art to use real test drive data from a real fleet of test vehicles to validate and verify driver assistance systems and to derive scenarios from recorded data. It is further known to use full factorial designs for validation and verification. [Prior art documents] [Non-patent literature]

[0014] [Non-Patent Document 1] “Metrics for assessing the criticality of traffic situations and scenarios” publication, P. Junietz et al., “11th Driver Assistance Systems and Automated Driving Workshop”, FAS 2017 Summary of the Invention [Problem to be solved by the invention]

[0015] One object of the present invention is to be able to test a driver assistance system, in particular an automated driving function, in a plurality of scenarios. In particular, an object of the present invention is to generate scenarios for testing the driver assistance system. [Means for solving the problem]

[0016] This problem is solved by the teaching of the independent claims. Advantageous embodiments can be found in the dependent claims.

[0017] A first aspect of the invention relates to a computer-implemented method for generating scenario data for testing a driver assistance system of a vehicle, the method comprising the operational steps of: The following sub-steps: - simulating a virtual traffic situation with a plurality of virtual road users, at least one first road user of the plurality of road users being controllable by the first user and road users that cannot be controlled by a user being controlled automatically, in particular by artificial intelligence or by logic-based control; - outputting a virtual environment of at least one first road user to the first user via a first, in particular at least visual, user interface based on the virtual traffic situation; a sub-step of capturing, via a second user interface, an input of a first road user for controlling at least one first road user in the virtual environment of the first road user, the captured input of the first user and a resultant interaction of the at least one first road user with the virtual environment being incorporated into a simulation of a virtual traffic situation; generating simulation data by - checking the generated simulation data for occurrences of scenarios resulting from an interaction of at least one first road user with the virtual environment, the occurrences of the scenarios being characterized by predefined configurations of simulated measurement variables, preferably corresponding to elementary maneuvers; Once occurrence of a scenario is determined, extracting scenario data relating to the scenario; recording scenario data for testing a driver assistance system; has.

[0018] Preferably, the extracted scenario data is output. Preferably, this occurs via a user interface or a data interface.

[0019] A user within the meaning of the present invention is a natural person, ie a human being.

[0020] A driver assistance system within the meaning of the present invention is preferably a driver assistance system configured to support the driver during the driving process or to at least partially control the vehicle, in particular at automation levels 3 to 5, or more particularly an automated driving function.

[0021] A road user within the meaning of the invention is preferably any object which performs an active part of the traffic, in particular a road user is a person, an animal or a vehicle.

[0022] Extraction within the meaning of the present invention preferably denotes separation or isolation.

[0023] In particular, the scenarios are separated or respectively isolated from the scenario data, and data ranges are thereby preferably selected within the scenario data.

[0024] Scenario data within the meaning of the present invention is preferably characterized by the positions and movements of road users and by the positions of static objects with respect to the scenario.

[0025] A scenario within the meaning of the invention is preferably formed from a series of, in particular static, scenes in a time series. Thereby, the scene shows the spatial configuration of at least one other object relative to its own object, for example the arrangement of road users. The scenario preferably incorporates dynamic and static content. A model for the systematic description of scenarios is preferably used here, which is more preferably the model of the PEGASUS project (https: / / www.pegasusprojekt.de), with six independent layers: 1. Roads (geometry,...), 2. Road facilities and rules (traffic signals,...), 3. Temporary changes and events (road works,...), 4. Moving objects (vehicles moving relative to the vehicle under test, objects related to traffic, such as pedestrians), 5. Environmental conditions (lighting situation, road weather,...), 6. Digital information (V2X, digital data / maps,...). The scenarios include, inter alia, driving situations in which the driver assistance system at least partially controls a vehicle equipped with the driver assistance system, referred to as the ego vehicle, which automatically performs at least one vehicle function of the ego vehicle.

[0026] A traffic situation within the meaning of the present invention preferably represents the totality of all road user environments in the traffic included in a defined spatial region and / or a defined time period or time point. These road user environments are preferably incorporated into the selection of applicable behavior patterns at a particular time point. Preferably, the traffic situation includes all relevant states, possibilities and determinants of actions. Although not required, the traffic situation can also be represented in terms of road users or objects.

[0027] The simulated measurement variables within the meaning of the present invention are preferably selected from the following group: the speed of the road user, in particular the initial speed, the direction of movement, in particular the path of the road user, the lighting conditions, the weather, the road surface, the temperature, the number and position of static and / or dynamic objects, the speed and direction of movement of dynamic objects, in particular the path, the state of the signal system, in particular a traffic light system, traffic signs, the number of lanes, acceleration or braking deceleration of the road user or object.

[0028] A predefined arrangement of measured variables within the meaning of the present invention is preferably an arrangement of values ​​of one or more measured variables, in particular a chronological sequence.

[0029] Labelling within the meaning of the present invention preferably means providing a classification designation.

[0030] The risk of a scenario within the meaning of the present invention preferably indicates a spatial or temporal proximity to a traffic situation without obtaining any possible accident-free outcome (by one's own efforts and taking into account significant uncertainties). The risk is at its maximum when an accident can no longer be avoided. Preferably, the risk is also called criticality. When considering the driving behavior or driving ability of the driver assistance system, the risk can characterize the probability of an accident and / or the calculated duration until the time point of impact. The maximum risk is preferably present when the calculated duration is 0 seconds and / or when the probability of an accident is P=1. The increasing accident probability can be triggered in particular by driving maneuvers, such as, for example, steering, braking, evasive reactions during acceleration, or significant gradient changes (so, for example, the vehicle swerves due to a sharp maneuver). The increasing accident probability can also force a violation of the driving task or its actual route (through evasive maneuvers), in particular with respect to other road users (in logical or AI-based guidance) and in dangerous driving situations, respectively. An increased accident probability may also result from external factors affecting the first road user or other road users, especially if the driver is, for example, not visible.

[0031] A quality factor within the meaning of the present invention preferably characterizes a simulated scenario. The quality factor is preferably understood as the quality or nature and / or relevance of the simulated scenario with respect to the danger of the driving situation for a particular driver assistance system.

[0032] Relevance within the meaning of the present invention is preferably understood as the frequency with which a scenario occurs in road traffic, for example a backlight scenario is more relevant than a scenario where an aircraft lands on a road, relevance also preferably depends on the region to which road traffic is relevant, for example a scenario may be relevant in Germany but irrelevant in China.

[0033] The vehicle environment within the meaning of the present invention is preferably formed at least by road users and other objects associated with the vehicle guidance provided by the driver assistance system. In particular, the vehicle environment includes the scenery and dynamic elements. The scenery preferably includes all static elements.

[0034] The present invention is based on an approach that uses real people to generate scenarios, but does not require any test drives in real traffic.

[0035] According to the invention, in each case at least one real driver drives a vehicle in the virtual environment. The invention allows a crowdsourcing approach to generate scenarios. One or more users can then navigate their selected road users through the virtual traffic situation in the simulator. When simulating a virtual traffic situation, as in real traffic, a virtually infinite number of different scenarios can arise due to the virtually infinite possibilities of navigating options for the road user(s) as well as other chance mechanisms. The invention uses predefined criteria to decide between the occurrence of a known scenario or the occurrence of a new scenario. To that end, the simulation process and especially the simulation data generated therefrom are continuously analyzed / monitored.

[0036] The crowdsourcing approach can thereby take advantage of people's natural behavioral instincts. The method of the present invention, or even a suitable system, can be provided to users in this way. These users can then drive the simulated traffic "for fun". Alternatively, a task can be set for the users, for example to get from point A to point B as quickly as possible while observing traffic rules or with the need to collect some objects. Furthermore, the users may not be able to concentrate when navigating through the simulated traffic, for example, they may need to provide certain voice inputs or the like.

[0037] The simulation physics correspond to reality, thereby generating the most realistic scenario data possible. This applies in particular to the physics of the road users and the physics of the environment. It is not possible to drive through objects or the like. In particular, preferentially, multiple users navigate multiple road users in a simulated traffic.

[0038] In one advantageous embodiment of the method, the scenario data generated thereby is already labeled, in particular the objects of the virtual traffic situation, since information about the properties of the objects is available in the simulation and thus information can be associated with the objects.

[0039] This is particularly advantageous for data from real test runs where all objects need to be labelled, which is generally very time-consuming as this can only be done by real people.

[0040] In a further advantageous embodiment of the method, the scenario data is preferably described during the extraction in such a way that it can be used to simulate the scenario using OpenSCENARIO® or as OSI data output, whereby the scenario data can still be used directly to simulate the scenario.

[0041] In a further advantageous embodiment of the method the user is requested to perform activities by means of various actions in the simulated virtual traffic environment. Such activities can for example be the simulated behaviour of other road users. In particular the other road users can behave in such a way that the user is forced to react. In a further advantageous embodiment the method further comprises the following working steps: Determining a quality factor for the extracted scenario data according to predefined criteria, whereby the quality factor is preferably characterized by the risk of the underlying scenario. The quality factor indicates the quality of the underlying scenario. Preferably, the extracted scenario data is output when the quality factor reaches an abort condition. Further preferably, the quality factor is output to a user via the first or second user interface, in particular a display. The abort condition can thereby preferably be a calculated duration of time to a time point of collision, or a probability of collision.

[0042] In a further advantageous embodiment of the invention, the higher the quality factor, the more dangerous the respective scenario, in particular the shorter the calculated duration until the time point of impact.

[0043] In a further advantageous embodiment of the method, the first user is granted a reward, in particular a conceptual reward, depending on the quality factor of the obtained scenario, which thereby motivates the user to create risky scenarios.

[0044] In a further advantageous embodiment of the method, a traffic flow model, in particular PTV-Vissim® or Eclipse SUMO, in particular version 1.8.0, is used to simulate the virtual traffic situation. The use of a traffic flow model allows the generation of a particularly realistic traffic situation.

[0045] The features and advantages explained above in relation to the first aspect of the invention also apply equally to the other aspects of the invention, and vice versa.

[0046] A second aspect of the invention relates to a computer-implemented method for testing a driver assistance system of a vehicle, comprising the following operational steps: providing scenario data characterising a scenario in which a vehicle is present and which has a plurality of other road users, the scenario data being generated by a method for generating scenario data according to a first aspect of the present invention; simulating a virtual environment for the vehicle from the provided scenario data; outputting the virtual environment to a driver assistance system via an interface; operating a driver assistance system in the vehicle virtual environment; It is.

[0047] In one advantageous embodiment of the method for testing a driver assistance system, the driver assistance system is simulated. This means that, according to the concept of "software in the loop", only the software or actual code of the driver assistance system, respectively, is taken into account or respectively implemented when simulating a virtual traffic situation. This thereby allows a purely simulation-based test of the driver assistance system.

[0048] In a further advantageous embodiment of the method for testing a driver assistance system, data relating to the vehicle environment is provided to the driver assistance system during operation of the driver assistance system and / or the driver assistance system, in particular its sensors, are stimulated based on the vehicle environment. Doing so allows testing of the driver assistance system, in particular its software or hardware as a whole, on a test bench. In particular, hardware-in-the-loop methods can be used for this purpose.

[0049] A third aspect of the present invention relates to a system for generating scenario data for testing a driver assistance system of a vehicle, the system comprising: Means for simulating a virtual traffic situation having a plurality of virtual road users, at least one first road user of the plurality of road users being controllable by the first user and road users that cannot be controlled by a user being controlled automatically, in particular by artificial intelligence or by logic-based control, wherein simulation data are generated during the simulation; a first, in particular at least visual, user interface for outputting a virtual environment of at least one first road user to a first user based on the virtual traffic situation; a second user interface capturing inputs of the first user for controlling at least one first road user in the virtual environment of the first road user, the simulating means being further configured to incorporate the captured inputs of the first user and the resulting interaction of the at least one first road user with the virtual environment into a simulation of a virtual traffic situation; - means for validating the generated scenario data against an occurrence of a scenario resulting from an interaction of at least one first road user with the rest of the environment, the occurrence of the scenario being characterized by a predefined configuration of simulated measurement variables, preferably corresponding to an elementary maneuver; means for extracting scenario data relating to the scenario when the occurrence of the scenario is determined by the means for verifying the generated simulation data; and a data storage device for recording scenario data for testing the driver assistance system.

[0050] Means within the meaning of this specification may be designed as hardware and / or software, in particular as a processing unit, in particular as a digital processing unit, in particular as a microprocessor unit (CPU), preferably in data or signal connection to a memory and / or a bus system and / or having one or more programs or program modules. The CPU may be configured to process commands implemented as programs stored in the memory system, to take input signals from the data bus and / or to send output signals to the data bus. The memory system may include one or more, in particular various, storage media, in particular optical, magnetic, solid-state and / or other non-volatile media. The programs may be designed to perform or be capable of performing the methods described herein, such that the CPU may execute the steps of such methods and then in particular generate the scenarios.

[0051] A fourth aspect of the present invention relates to a system for testing a driver assistance system of a vehicle, the system comprising: - a data store for providing scenario data characterising a scenario in which a vehicle is present and which has a plurality of other road users, the scenario data being generated by a method according to any one of claims 1 to 8; means for simulating a virtual environment for the vehicle based on the scenario data; an interface for outputting the virtual environment to a driver assistance system so that the driver assistance system can operate in the virtual environment of the vehicle based on a simulated scenario; Equipped with.

[0052] A further aspect of the invention relates to a computer program comprising instructions which, when run on a computer, cause the computer to carry out the method according to the first or second aspect of the invention.

[0053] Further features and advantages are shown in the following description of exemplary embodiments with reference to the figures, which are at least partially illustrated diagrammatically. [Brief description of the drawings]

[0054] [Figure 1] FIG. 1 is a diagram of the probability of a scenario occurring as a function of its criticality. [Diagram 2] FIG. 2 is a block diagram of an exemplary embodiment of a method for generating a scenario. [Figure 3a] 1 is a first example of a simulated virtual traffic situation. [Figure 3b] 2 is a second example of a simulated hypothetical traffic situation. [Figure 4] 1 is an exemplary embodiment of a system for generating scenario data for testing a driver assistance system of a vehicle. [Diagram 5] 1 is a block diagram of an exemplary embodiment of a method for testing a driver assistance system of a vehicle. [Figure 6] 1 is an example of a simulated scenario. [Figure 7] 1 is an exemplary embodiment of a system for testing a driver assistance system of a vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] Figure 1 shows the probability of occurrence of a scenario as a function of its criticality. The probability of occurrence is the probability that the scenario occurs in real road traffic.

[0056] It is noted in FIG. 1 that the majority of scenarios are of relatively low complexity and / or criticality, which also correspond to the normal life experience of a motor vehicle driver. The range of these scenarios is designated as "A" in FIG. 1. In contrast, scenarios of high complexity and / or criticality, the range of which is designated as "B" in FIG. 1, occur relatively less frequently. However, it is precisely these "B" scenarios of high complexity and / or criticality that are of major relevance for analyzing the functionality of a driver assistance system.

[0057] Therefore, obtaining a sufficient number and diversity of different scenarios of high "B" complexity during testing of a driver assistance system requires running a very large number of scenarios based on the distribution curves shown.

[0058] A method for generating a number of different scenarios for testing a driver assistance system is described below with reference to Figures 2 to 3b.

[0059] Simulation data is generated in a first work step 101. Preferably, the first work step 101 includes three sub-processes.

[0060] A virtual traffic situation 3 with a plurality of virtual road users 1, 4, 5a, 5b, 5c, 5d, 6 is simulated in the first of these processes 101-1. Preferably, at least one first road user 1 of the plurality of road users 1, 4, 5a, 5b, 5c, 5d, 6 can be controlled by a first user 2 (see FIG. 4) and these road users 4, 5a, 5b, 5c, 5d, 6 that cannot be controlled by a user are automatically controlled in this virtual traffic situation 3. Preferably, artificial intelligence, logic-based models or traffic flow models, in particular PTV-Vissim® or Eclipse SUMO, are thereby used. Preferably, in the simulated virtual traffic situation 3 there may be a plurality of road users controlled by users, i.e. humans.

[0061] There are essentially two approaches to simulating a virtual traffic situation: the simulation is based on data obtained in a real test drive. In this case, the parameters of the individual objects, such as for example the speeds of those road users, can be changed or used as captured during the real test drive. In an alternative embodiment of the method, the traffic situation 3 is generated based on a purely mathematical algorithm. Preferably, there can be a mixture of the two approaches.

[0062] An example of such a simulated traffic situation 3 is shown in Fig. 3a. In the traffic situation 3 shown in Fig. 3a, a pedestrian 6 is crossing the road. A first user-controlled vehicle 1 approaches the pedestrian 6 in the lane at his side. Other vehicles 5b, 5c, 5d are parked side by side along the lane, making the pedestrian 6 invisible or barely visible to the driver of the user-controlled vehicle 1. A further vehicle 5a at the same level as the pedestrian 6 is driving in the second lane as oncoming traffic. A motorcyclist 4 is approaching behind the other vehicle 5a and preparing to pass said other vehicle 5a. It cannot be deduced from Fig. 3a whether this motorcyclist 4 is visible to the driver of the vehicle controlled by the first user 2.

[0063] Other vehicles 5 a , 5 b , 5 c , 5 d , pedestrians 6 as well as a motorcyclist 4 form the virtual environment of the vehicle 1 controlled by the first user 2 in a traffic situation 3 .

[0064] Depending on how the first user 2 reacts or behaves in the initial scenario resulting from the traffic situation 3, i.e., which driving behavior the first user exhibits in the virtual environment of the vehicle 1 that he controls, dangerous or less dangerous driving situations or other scenarios are obtained. For example, if the first user 2 brakes to stop the vehicle 1, as shown in Fig. 3a by the bar in front of the movement arrow of the vehicle 1, the motorcyclist 4 can pass the oncoming vehicle 5a in the other lane without any trouble.

[0065] Figure 3b shows the same hypothetical traffic situation 3 as in Figure 3a, where the vehicle controlled by the first user 1 is in the same initial scenario as in Figure 3a. The first user continues to drive the vehicle 1 at an unreduced speed, as indicated by the movement arrow pointing away from the vehicle 1 controlled by the first user.

[0066] In such a case, the subsequent driving situation or scenario will likely develop into a collision of the motorbike driver 4 with the vehicle 1 driven by the first user 1. This is also shown in Fig. 3b. Such a driving situation / scenario will represent a very high risk.

[0067] In a second process 101 - 2 of the first work step 101 , a virtual traffic situation 3 is output to a first user 2 via a first user interface 12 .

[0068] Examples of possible user interfaces are shown in FIG. 4 and preferably include a visual user interface, in particular a screen, an audio user interface, in particular a speaker, and / or a user interface for stimulating the sense of balance of the first user 2.

[0069] In a third process 101-3 of the first working step 101, inputs of a first user 2 for controlling at least one road user in the virtual environment of the first road user 1 are captured via a second user interface 13.

[0070] A second user interface 13 is also shown in Figure 4. Preferably, these relate to the steering wheel, gear shift, hand brake, brake pedal, clutch, and / or accelerator, and other possible controls available to the driver in the vehicle.

[0071] However, depending on the type of road user 1, 4, 5a, 5b, 5c, 5d, 6 controlled by the user 2, other input means may also be provided as user interface 13, for example a type of joystick.

[0072] As explained above, the first road user 1 in Figures 3a and 3b is a black vehicle. The captured inputs of the first user 2 and the resulting interactions of the vehicle 1, i.e. the first road user, with its virtual environment are incorporated into a simulation of the traffic situation 3 shown in Figures 3a and 3b.

[0073] The interactions in the traffic situation 3 shown in Figures 3a and 3b are, for example, how the first user 2 reacts to the initial scenario. Depending on the reaction of the first user 2 to this initial scenario, other road users also react, in particular other oncoming vehicles 5a and the motorcyclist 4, as well as pedestrians 6. For example, when the motorcyclist 4 notices that the vehicle 1 controlled by the first user 2 does not reduce its speed, it is expected that the motorcyclist 4 will brake. These interactions then affect the evolution of the virtual traffic situation 3.

[0074] The operational step 101 of generating simulation data is therefore a continuous process, which continually operates in a loop, as shown in FIG. 2, thereby generating simulation data.

[0075] During the simulation, objects that are part of the virtual traffic situation 3 are already marked with meta-information. Individual labels are therefore not necessary. This concerns both static and dynamic objects. Thereby, the subsequent data that can be obtained from the simulation data contains so-called ground truth information.

[0076] When the scenario data is used, for example, to test a driver assistance system, it is possible to understand which objects the driver assistance system detected correctly and which ones it detected incorrectly. Examples of such labels are trees, pedestrians, cars, trucks, etc.

[0077] More preferably, an action to encourage activity from the first user is set in the driving situation 3. For example, in the driving situation 3 of Fig. 3a and Fig. 3b, this can be a vehicle driving behind the vehicle 1 driven by the first user 2 and forcing it to accelerate. An unexpected movement path of a pedestrian 6, such as running, can also be such an action.

[0078] In a second working step 102 of the method 100, the generated simulation data is checked against the occurrence of scenarios resulting from the interaction of at least one first road user 1, the black vehicle in Figures 3a and 3b, with the virtual environment. Both already known scenarios that have been performed before or predefined as templates, as well as scenarios that have not yet been predefined, can thereby be checked.

[0079] Both types of scenarios are preferably defined by predefined configurations of simulated measurement variables that can be determined from the virtual traffic situation 3. These predefined configurations form templates for the scenarios or correspond to elementary maneuvers from which the occurrence of the scenario can be inferred. This can be for example a strong braking deceleration of the vehicle 1 in Figures 3a and 3b, which is used as a trigger condition for the occurrence of a scenario that is not yet predefined.

[0080] Once it is determined that a scenario has occurred, scenario data relating to the scenario is extracted in a third operational step 103. In this context, extraction refers to isolating or isolating data ranges in the simulation data that relate specifically to the determined scenario. Preferably, the scenario data is visually represented during extraction as appropriate for simulating the scenario. Preferably, the data can be used with OpenSCENARIO® or OpenDrive®. More preferably, the data can be output as OSI data or OSI stream.

[0081] In a fourth operational step 104 of the method 100, scenario data for testing the driver assistance system is recorded. This data is then available for testing the driver assistance system. Such a testing method 200 is further described below with reference to FIG. 5.

[0082] In a fifth working step 105, a quality factor of the obtained scenario is preferably determined depending on predefined criteria, the quality factor being preferably characterized by one of the dangers of the scenario. Preferably, the more dangerous the obtained scenario is, the higher the quality factor. The danger is preferably determined by so-called "time to X" metrics, such as those described, for example, in "Metrics for assessing the criticality of traffic situations and scenarios" publication, P. Junietz et al., "11th Driver Assistance Systems and Automated Driving Workshop", FAS 2017. In particular, what can thereby be used as criteria are the duration of time to the impact point (time to impact), time to kickdown, time to maneuver, time to react, distance to the nearest encounter, time to the nearest encounter, worst case time to impact, etc. Furthermore, the danger is characterized by the probability of an accident.

[0083] Further preferably, the first user 2 is granted a reward, in particular a notional reward, depending on the quality factor of the presented scenario.

[0084] FIG. 4 illustrates a system 10 for generating scenarios for testing driver assistance systems of a vehicle.

[0085] The system 10 preferably comprises means 11 for simulating a virtual traffic situation 3 with a plurality of virtual road users. In order to make the road users 1 controllable by a first user 2, the system further comprises at least one first user interface 12 and at least one second user interface 13.

[0086] The at least one first user interface 12 serves to output a virtual environment of the at least one first road user 1 to the first user 2. The virtual environment of the at least one first road user 1 is thereby determined based on the simulated virtual traffic situation 3. It is thereby substantially a representation of the virtual traffic situation 3 in the first scenario according to the perspective of the first road user 1 controlled by the first user 2.

[0087] As FIG. 4 shows, these user interfaces 12 are visual user interfaces, such as a screen, and audio interfaces, such as speakers, and possibly also devices capable of influencing the sense of balance of each user 2 .

[0088] The second user interface, i.e. user interface 13, is arranged to take inputs from each user 2. As shown in Fig. 4, these are preferably related to various operating elements, which, as already explained before, can be responsive to each road user 1 controlled by the user 2. If the road user 1 controlled by the first user 2 is a vehicle, the user interfaces 12, 13 are preferably arranged in the area of ​​a so-called seat box 19, which together with the user interfaces 12, 13 form a simulator, as shown in Fig. 4.

[0089] The system 10 preferably further comprises means 14 for validating the generated simulation data for the occurrence of a scenario. Furthermore, the system 10 preferably comprises means 15 for extracting scenario data relating to the scenario as well as a data store 16 for recording the scenario data. Further preferably, the system 10 comprises means for determining a quality factor of the extracted scenario data, preferably depending on predefined criteria. Further preferably, the system 10 has a further interface 18, preferably configured as a user interface for outputting the quality factor to the user 2 and / or as a data interface for outputting the scenario data for further processing. Preferably, the means 11, 14, 15, 16, 17, 18 are part of a data processing device, preferably formed by a computer.

[0090] FIG. 5 illustrates a flow diagram of an exemplary embodiment of a method 200 for testing a driver assistance system 7 of a vehicle 8 illustrated in FIG.

[0091] A first operational step 201 of the method 200 involves simulating scenario data characterizing a scenario in which the vehicle 8 is present and which preferably includes a plurality of other road users 4', 5a', 5b', 5c', 5d', 6'. This scenario data is then preferably also based on a simulation extracted therefrom according to the method 100 described above.

[0092] Based on the scenario data a scenario is simulated in a second work step 202. The scenario comprises a vehicle 8 equipped with a driver assistance system 7 under test. Furthermore the scenario preferably includes a number of other road users or objects.

[0093] In the example scenario shown in Figure 6, these items, namely the parked vehicles 5b', 5c', 5d', the pedestrian 6', the oncoming vehicle 5a' in the other lane, as well as the motorcycle 4' also present in the said lane, are similar to the driving situation 3 shown in Figures 3a and 3b.

[0094] Based on the simulated scenario, a virtual environment of the vehicle 8 equipped with the driver assistance system 7 is generated and output in a third work step 203 .

[0095] The virtual environment is output to the driver assistance system 7 via the interface 23 in a third work step 203. Finally, the driver assistance system 7 is operated in the virtual environment of the vehicle 8 in a fourth work step 204.

[0096] The driving behavior of the driver assistance system 7 in a scenario or environment, respectively, can be further analyzed and evaluated. The driver assistance system 7 can be optimized based on such analysis or evaluation.

[0097] In the scenario shown in FIG. 6, the driver assistance system 7 of the vehicle 8 is equipped with a radar system for detecting objects located in the area surrounding the vehicle 8, in particular road users 4', 5a', 5b', 5c', 5d', 6'.

[0098] In the illustrated exemplary scenario, the driver assistance system 7 is integrated into a passenger car 8. However, the driver assistance system to be tested could equally well be integrated into a motorcycle 4'. For example, the motorcyclist is warned in advance by the sensors of the driver assistance system and therefore does not leave the flow of traffic. The driver assistance system of the motorcycle 4' therefore reacts and the black car can then continue on without crashing. A system 20 for testing driver assistance systems 7 suitable for carrying out the method 200 described with reference to figures 5 and 6 is shown in figure 7.

[0099] Such a system 20 comprises a data store 21 for providing scenario data characterizing a scenario in which the vehicle 8 is present. Means 22 are configured to simulate a virtual environment for the vehicle based on the scenario data. Furthermore, the means 22 are also configured to render the environment.

[0100] Finally, the interface 23 is configured to output the virtual environment of the driver assistance system 7. If the driver assistance system 7 comprises an optical camera, such an interface can be a screen. In the example shown in Fig. 7, the sensor of the driver assistance system is a radar sensor emitting a signal S. This signal S is acquired by the radar antenna 23.

[0101] Based on the acquired signals and the simulated environment, the simulating means 22 calculate a response signal S', which is then output to the radar of the driver assistance system. By doing so, the functionality of the driver assistance system 7 can be tested. Depending on which components of the driver assistance system 7 are to be tested, the simulated virtual environment can be tested by emulating the signals to the sensors of the driver assistance system 7, as shown in FIG. 7. However, alternatively, signals can also be generated that are sent directly to the data processing unit 7 of the signals of the driver assistance system, as well as signals S' that are only processed by the software of the driver assistance system 7.

[0102] The data storage device 21 and the simulating means 22 are preferably part of a data processing device.

[0103] It should be noted that the exemplary embodiments are merely examples, which are in no way intended to limit the scope, field of application, and configuration of protection. Instead, the foregoing description provides a person skilled in the art with a guideline for implementing at least one exemplary embodiment, whereby various modifications can be made with respect to the function and configuration of the specifically described components without departing from the scope of protection obtained from the claims and equivalent combinations of features. [Explanation of symbols]

[0104] 1 Road users, vehicles 2. First User 3. Virtual traffic situation 4. Road users, motorcyclists 4' Motorcycle 5a, 5b, 5c, 5d road users 5a', 5b', 5c', 5d' Road users, vehicles 6, 6' Road users, pedestrians 7 Driver Assistance Systems 8 Vehicles 10. System 11 means 12 Primary User Interface 13 Secondary User Interface 14 means 15 means 16 Means, data storage device 17 means 18 Means, Interface 19 Seat Box 20 Systems 21 Data storage device 22 Simulation methods 23 Interface, radar antenna S, S' signals

Claims

1. A computer-implemented method (100) for generating scenario data for testing a driver assistance system (7) of a vehicle (8), comprising the following operational steps: - a sub-step (101-1) of simulating a virtual traffic situation (3) with a plurality of virtual road users (1, 4, 5a, 5b, 5c, 5d, 6), of which at least one first road user (1) can be controlled by a first user (2) and the road users (4, 5a, 5b, 5c, 5d, 6) that cannot be controlled by a user are controlled automatically, in particular by artificial intelligence or by logic-based control; a sub-step (101-2) of outputting a virtual environment of the at least one first road user (1) to the first user (2) via a first, in particular at least visual, user interface (12) based on the virtual traffic situation (3); a sub-step (101-3) of capturing, via a second user interface (13), inputs of the first user (2) for controlling the at least one first road user (1) in the virtual environment of the first road user (1), the captured inputs of the first user (2) and the resulting interaction of the at least one first road user (1) with its virtual environment being incorporated into a simulation of the virtual traffic situation (3); A step (101) of generating simulation data by - checking (102) the generated simulation data for the occurrence of a scenario resulting from the interaction of the at least one first road user (1) with the virtual environment, the occurrence of the scenario being characterized by a predefined configuration of simulated measurement variables, preferably corresponding to an elementary maneuver; Once the occurrence of a scenario is determined, extracting (103) scenario data relating to said scenario; recording (104) said scenario data for testing said driver assistance system (7); A computer-implemented method (100) comprising:

2. The method (100) of claim 1, wherein objects of the virtual traffic situation (3) are labeled.

3. The method (100) according to claim 1 or 2, wherein the scenario data is preferably described during extraction so that it can be used to simulate the scenario using OpenSCENARIO® or as OSI data output.

4. The method (100) according to claim 1 or 2, wherein the first user (2) is prompted to perform an activity in the simulated virtual traffic environment by one or more actions.

5. The following work steps: determining (105) a quality factor for the extracted scenario data in response to predefined criteria, said quality factor being preferably characterized by the criticality of the underlying scenario; 5. The method (100) of any one of claims 1 to 4, further comprising:

6. The method (100) of claim 5, wherein the higher the quality factor, the more dangerous the respective scenario is, in particular the shorter the calculated duration until the time point of impact.

7. The method (100) according to any one of claims 1 to 6, wherein the first user (2) is granted a reward, in particular a notional reward, depending on a quality factor of the obtained scenario.

8. The method (100) according to any one of claims 1 to 7, wherein a traffic flow model, in particular PTV Vissim®, is used to simulate the virtual traffic situation (3).

9. A computer-implemented method (200) for testing a driver assistance system (7) of a first vehicle (8), comprising the following operational steps: - providing (201) scenario data characterising a scenario in which the first vehicle (8) is present and which has a plurality of other road users (4', 5a', 5b', 5c', 5d', 6'), the scenario data being generated by a method (100) according to any one of claims 1 to 8; simulating (202) a virtual environment for the first vehicle (8) from the provided scenario data; outputting (203) the virtual environment to the driver assistance system (7) via an interface (23); operating (204) the driver assistance system (7) in the virtual environment of the first vehicle (8); A computer-implemented method (200) comprising:

10. The method according to claim 9, wherein the driver assistance system (7) is simulated.

11. 11. The method according to claim 10, wherein data relating to the virtual environment of the first vehicle (8) is supplied to the driver assistance system (7) during the operation of the driver assistance system (7) and / or the driver assistance system (7), in particular its sensors, are stimulated based on the virtual environment of the first vehicle (8).

12. A computer program comprising commands which, when run on a computer, cause the computer to carry out the steps of the method according to any one of claims 1 to 11.

13. A computer readable medium having stored thereon the computer program of claim 12.

14. A system (10) for generating scenario data for testing a driver assistance system of a vehicle, comprising: - means (11) for simulating a virtual traffic situation (3) with a plurality of virtual road users (1, 4, 5a, 5b, 5c, 5d, 6), of which at least one first road user (1) can be controlled by a first user (2) and road users (4, 5a, 5b, 5c, 5d, 6) that cannot be controlled by a user are controlled automatically, in particular by artificial intelligence or by logic-based control, and wherein simulation data are generated during the simulation; a first, in particular at least visual, user interface (12) for outputting a virtual environment of at least one first road user (1) to said first user (2) based on said virtual traffic situation (3); a second user interface (13) capturing inputs of said first user (2) for controlling said at least one first road user (1) in the virtual environment of said first road user (1), said simulating means (11) being further configured to incorporate said captured inputs of said first user (2) and the resulting interaction of said at least one first road user (1) with its virtual environment into a simulation of said virtual traffic situation (3); - means (14) for validating said generated scenario data against an occurrence of a scenario resulting from said interaction of said at least one first road user (1) with the remainder of said virtual environment, said occurrence of a scenario being characterised by a predefined configuration of simulated measurement variables, preferably corresponding to an elementary manoeuvre; a means for extracting (15) scenario data relating to the scenario when the occurrence of the scenario is determined by the means for checking the generated simulation data; a data storage device (16) for recording the scenario data for testing the driver assistance system; A system (10) comprising:

15. A system (20) for testing a driver assistance system (7) of a first vehicle (8), comprising: - a data store (21) for providing scenario data characterising a scenario in which the first vehicle (8) is present and which has a plurality of other road users (4', 5a', 5b', 5c', 5d', 6'), said scenario data being generated by a method (100) according to any one of claims 1 to 8; means (22) for simulating a virtual environment of the first vehicle (8) based on the scenario data; an interface (23) for outputting the virtual environment to the driver assistance system (7) so that the driver assistance system (7) can operate in the virtual environment of the first vehicle (8) based on the simulated scenario; A system (20) comprising:

Images