Method and system for determining the critical system boundaries of an ADAS / ADS system

Abstract

The invention relates to a method for determining the critical system boundaries of an advanced driver assistance system (ADAS) and / or an automated driving system (ADS) with at least one driving function, in particular for testing, applying and verifying the ADAS / ADS system, comprising: - Generating a plurality of test cases (T i ) from a test module (300) with a test agent (350) for a parameterized scenario (SZp) i ); - Creating a test case space (430) with the generated test cases (T i ) by a parameter module (400); - Performing a simulation of the behavior of at least one driving function of the ADAS / ADS system in a test case (T i ) specific simulation environment from a simulation module (500), where the simulation is only for a subset of the test cases (T i ) is performed in the test case space (430); - Evaluating the simulation results so that a test case (T t ) either as a critical test case (T crit ) or as a non-critical test case (T right ) is classified; - Creating a scenario parameter space (440) for the tested test cases (T t ); - Determining a system boundary in the scenario parameter space (440) between the critical test cases (T crit ) and the non-critical test cases (T right ) using a search algorithm; - Generating new test cases (T new ) at the system boundary in the scenario parameter space (440).

Description

[0001] The invention relates to a method, a system and a computer program product for determining the critical system limits of an ADAS / ADS system, in particular for testing, applying and securing a driver assistance system (ADAS) and / or an automated driving system (ADS) with at least one driving function.

[0002] Modern vehicles are equipped with a variety of advanced driver assistance systems (ADAS) and automated driving systems (ADS) to support the driver while driving and increase safety. There are various types of driver assistance systems, such as adaptive cruise control, which automatically maintains a constant distance from the vehicle in front and adjusts the vehicle's speed; lane departure warning, which warns of lane departure; brake assist, which automatically brakes the vehicle when a critical situation is detected; and parking assistant, which assists the driver when entering and exiting parking spaces. Driver assistance systems are constantly being developed and are designed to increase road safety and prevent accidents.

[0003] The safe functioning of ADAS / ADS systems requires comprehensive validation strategies in software development, which entails correspondingly high integration and testing effort. Road users must be correctly identified and classified, and the multitude of possible traffic scenarios in a vehicle's environment must be correctly interpreted. Successful software development therefore depends heavily on the verification and validation of an ADAS / ADS system's software code. For this purpose, an ADAS / ADS system is specifically tested both in extreme and exceptional situations (corner cases) and in everyday traffic situations, particularly through test drives with test vehicles.

[0004] However, driving on real test tracks is associated with significant time and expense, so virtual simulation methods are increasingly being used to support the integration process. Traffic conditions are mapped using parameterized scenarios, and test cases are generated for a simulation environment. Similar to corner cases in real-world environments, critical scenarios are of particular interest for simulation, as the effects of software and dataset changes in an ADAS / ADS system can be efficiently quantified in these critical scenarios.

[0005] However, not every test case describes a critical traffic situation. Since simulations with a large number of non-critical test cases produce inefficient results, testing with a randomly selected pool of test cases is inefficient and costly. Therefore, it is desirable to be able to easily distinguish non-critical from critical test cases.

[0006] DE 10 2020 211 477 A1 discloses a method for evaluating the driving behavior of a motor vehicle that is driven at least partially automatically within an infrastructure on the basis of infrastructure area signals.

[0007] DE 10 2018 219 564 A1 discloses a method for controlling a driver assistance system, wherein the driver assistance system is only started when a detected driving environment and a determined driving state of the vehicle meet predetermined conditions.

[0008] The object underlying the invention is to provide possibilities for determining the critical system limits of an ADAS / ADS system in order to be able to carry out the testing, application and validation of an ADAS / ADS system with meaningful test cases and thus to create the basis for the release of the ADAS / ADS system on the basis of a secure data situation.

[0009] This object is achieved according to the invention with respect to a method by the features of patent claim 1, with respect to a system by the features of patent claim 11, with respect to an ADAS / ADS system by the features of patent claim 14, and with respect to a computer program product by the features of patent claim 15. The further claims relate to preferred embodiments of the invention.

[0010] The present invention enables the system boundary between critical and non-critical behavior of an ADAS / ADS system with respect to a specific scenario to be determined using an efficient search algorithm, enabling targeted testing at the system boundary of an ADAS / ADS system. This saves computing resources because the entire test case space no longer needs to be tested. The development, integration, and testing process of an ADAS / ADS system can be improved, thus increasing the overall safety of an ADAS / ADS system.

[0011] According to a first aspect, the invention provides a method for determining the critical system limits of a driver assistance system (ADAS) and / or an automated driving system (ADS) with at least one driving function, in particular for testing, applying, and validating the ADAS / ADS system. A scenario represents a traffic event in a temporal sequence and is defined by a selection of scenario parameters and associated scenario parameter values. The method comprises the following method steps: - Generating a plurality of test cases from a test module with a test agent for a parameterized scenario by selecting concrete scenario parameters and concrete scenario parameter values ​​from a scenario database for a simulation task; - Creating a test case space with the test cases through a parameter module; - performing a simulation of the behavior of at least one driving function of the ADAS / ADS system in a simulation environment determined by a test case by a simulation module to determine simulation results, wherein the simulation is performed only for a subset of the test cases in the test case space; - Evaluating the simulation results to determine a binary criticality indicator for the tested test case, so that a test case is classified either as a critical test case or as a non-critical test case; - Creating a scenario parameter space for the tested test cases, with each tested test case being assigned the corresponding criticality indicator; - Determining a system boundary in the scenario parameter space between the critical test cases and the non-critical test cases using a search algorithm; - Generating new test cases at the system boundary in the scenario parameter space.

[0012] In a further development, it is provided that a coarse-meshed network is placed over the test case space and the test cases arranged at least approximately at the nodes of the coarse-meshed network form the selected subset of the test cases.

[0013] In an advantageous embodiment, the search algorithm, starting from a starting position with a critical test case in the scenario parameter space, searches for uncritical positions in the scenario parameter space with uncritical test cases in a partial circle determined by a minimum standard distance and an addition factor.

[0014] In a further embodiment, it is provided that for each non-critical position found, a new position is generated halfway between the starting position and the non-critical position in question, and that a check is carried out to determine whether the new position is not already present in the scenario parameter space.

[0015] In particular, a new test case is created for each new position.

[0016] Advantageously, the coordinates of the new position in the scenario parameter space are passed to the test module to generate a new test case, whereby the behavior of the ADAS / ADS system in this new test case is simulated in the simulation module, whereby the behavior is evaluated with a criticality indicator and this is assigned to the vector representation of the new test case.

[0017] In a further embodiment, it is provided that the search algorithm performs a plurality of iterations, so that with each iteration new test cases are generated and the scenario parameter space is extended by new test cases at the system boundary between the critical test cases and the non-critical test cases.

[0018] In particular, it is intended that the determined system boundary between critical and non-critical behavior of the ADAS / ADS system in the scenario parameter space is used for software development of the ADAS / ADS system with Continuous Integration (CI).

[0019] In a further development, it is planned that the determined system boundary between critical and non-critical behavior of the ADAS / ADS system in the scenario parameter space is used for a safety argument for the release of the ADAS / ADS system.

[0020] For this purpose, the determined system boundary is advantageously compared with information about the probability of occurrence of specific scenarios in real test drives in real traffic.

[0021] With a second aspect, the invention provides a system for determining the critical system limits of a driver assistance system (ADAS) and / or an automated driving system (ADS) with at least one driving function, in particular for testing, applying, and validating the ADAS / ADS system. A scenario represents a traffic event in a temporal sequence and is defined by a selection of scenario parameters and associated scenario parameter values. The system comprises a scenario database, a test case module with a test agent, a parameter module, and a simulation module. The test module is designed to generate a plurality of test cases for a parameterized scenario by selecting specific scenario parameters and specific scenario parameter values ​​from a scenario database for a simulation task. The parameter module is designed to create a test case space with the test cases.The simulation module is configured to simulate the behavior of at least one driving function of the ADAS / ADS system in a simulation environment defined by a test case to determine simulation results. The simulation is performed only for a subset of the test cases in the test case space. The simulation module is configured to evaluate the simulation results to determine a binary criticality indicator for the tested test case, so that a test case is classified either as a critical test case or as a non-critical test case. The parameter module is configured to create a scenario parameter space for the tested test cases, with the corresponding criticality indicator being assigned to each tested test case.The parameter module is also designed to determine a system boundary in the scenario parameter space between the critical test cases and the non-critical test cases using a search algorithm and to generate new test cases in the area at the system boundary in the scenario parameter space.

[0022] In a further development, it is provided that a coarse-meshed network is placed over the test case space and the test cases arranged at least approximately at the nodes of the coarse-meshed network form the selected subset of the test cases.

[0023] In an advantageous embodiment, the search algorithm, starting from a starting position with a critical test case in the scenario parameter space, searches for non-critical positions in the scenario parameter space with non-critical test cases in a partial circle determined by a minimum standard distance and an addition factor, wherein for each non-critical position found, a new position is generated halfway between the starting position and the relevant non-critical position.

[0024] According to a third aspect, the invention provides an ADAS / ADS system for a motor vehicle, which uses the method according to the first aspect for testing, applying and securing and / or for verifying, calibrating and validating the ADAS / ADS system.

[0025] According to a fourth aspect, the invention provides a computer program product comprising executable program code configured to carry out the method according to the first aspect when executed.

[0026] The invention is explained in more detail below with reference to embodiments shown in the drawing.

[0027] It shows: Fig. 1 is a block diagram illustrating an embodiment of a system according to the invention; Fig. 2a a schematic representation to explain a first method step according to the invention in a scenario parameter space; Fig. 2b shows a schematic representation to explain a second method step according to the invention in the scenario parameter space; Fig. 2c shows a schematic representation to explain a third method step according to the invention in the scenario parameter space; Fig. 3 a flow chart to explain the individual method steps of a method according to the invention; Fig. 4 is a block diagram of a computer program product according to an embodiment of the third aspect of the invention.

[0028] Further features, aspects and advantages of the invention and its embodiments are explained in the following description in conjunction with the patent claims.

[0029] Simulated traffic scenarios generated through programming are increasingly being used to test, calibrate, and validate advanced driver assistance systems (ADAS) and automated driving systems (AFS). A scenario describes a traffic event in a chronological sequence, such as driving over a motorway bridge, turning into a turning lane, driving through a tunnel, entering a roundabout, or stopping at a pedestrian crossing. An overtaking maneuver, also known as a cut-in scenario, can be described, for example, as a scenario in which a first vehicle initially drives behind another vehicle, then changes into the other lane and increases speed to overtake the other vehicle.

[0030] The prerequisite for the release of a specific ADAS / ADS system is its successful verification, calibration, and validation. Calibration serves to adapt the functions to the respective vehicle type and to the desired behavior of a driving function without changing the software code. For this purpose, the calibration parameters are modified and made available to the ADAS / ADS system in a data set. The goal of validation is to comprehensively test the data set obtained during calibration and to demonstrate the reliability and robustness of an ADAS / ADS system across the entire Operational Design Domain (ODD). The evaluation of the performance of an ADAS / ADS system depends in particular on its behavior in critical scenarios, also known as worst-case scenarios.Therefore, for the release of an ADAS / ADS system, it is important to know the boundary between critical and non-critical test cases in order to ensure that the release of an ADAS / ADS system actually only takes place for the non-critical test cases.

[0031] Fig. Figure 1 shows a system 100 according to the invention for determining the critical system limits of an ADAS / ADS system, in particular for testing, applying, and validating the ADAS / ADS system with at least one driving function. Since different ADAS / ADS systems differ in their specifications, the system limits also vary from system to system.

[0032] The system 100 according to the invention comprises a scenario database 200, a test case module 300 with a test agent 350, a parameter module 400, and a simulation module 500. Furthermore, a scenario identification module 700 and an output module 800 may be provided. The scenario database 200, the test case module 300, the parameter module 400, the simulation module 500, the scenario identification module 700, and the output module 800 may each be equipped with a processor and / or a memory unit, but may also be integrated into a common processing unit.

[0033] In the context of the invention, a "module" can be understood, for example, as a processor and / or a processor unit and / or a memory unit for storing program instructions. The processor is specifically configured to execute the program instructions in order to carry out or implement the method according to the invention or a step of the method according to the invention. In particular, a module can be integrated into a cloud computing infrastructure 220. However, it can also be provided that a computing unit, such as a processor, comprises multiple modules.

[0034] In the context of the invention, a "processor" can be understood, for example, as a machine or an electronic circuit. A processor can be, in particular, a central processing unit (CPU), a microprocessor, or a microcontroller, e.g., an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions, etc. A processor can also be a virtualized processor, a virtual machine, or a soft CPU.

[0035] In the context of the invention, a "storage unit" or "storage module" can be understood as, for example, a volatile memory in the form of random access memory (RAM), a permanent memory such as a hard drive or a data storage device, or, for example, a replaceable memory module. The memory module can also be designed as a cloud storage solution.

[0036] In the context of the invention, “data” refers to both raw data and already processed data from measurement results from sensors and other data sources.

[0037] The test case module 300 comprises a software application for creating a test case space 430 for a parameterized scenario SZp i . The test case space 430 contains a set of test cases T i . For another parameterized scenario SZp j another test case space 430 is created, which is then filled with the test cases T jfor this parameterized scenario SZp j is filled.

[0038] The creation of test cases T i for testing, applying and securing a specific ADAS / ADS system is therefore carried out on the basis of parameterized scenarios SZp i , which are also called logical scenarios. As a parameterized scenario SZp i In the context of the invention, a scenario is referred to, which is written in particular in machine-readable code. A parameterized scenario SZp i is determined by various scenario parameters P1, P2, ..., P n from a set of possible scenario parameters P i and associated scenario parameter values ​​PV1, PV2, ..., PV n from a set of possible scenario parameter values ​​PV i defined, where scenario parameter values ​​PV i the value range of a scenario parameter P i Scenario parameter P ithus characterize, within the scope of the present invention, properties and features of a parameterized scenario SZp i .

[0039] An example of a scenario parameter P i a parameterized scenario SZp i is the speed of an ego vehicle. For this scenario parameter “Speed”, the value range of the corresponding scenario parameter value PV i cover the range from 100 km / h to 180 km / h. For another parameterized scenario SZp k However, the range of the parameter value PV i for the scenario parameter “speed” range from 40 km / h to 70 km / h.

[0040] Other scenario parameters P iare, for example, a physical quantity, a chemical quantity, a torque, a rotational speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a motorway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographical structure such as a gradient, a time of day, a temperature, a precipitation value, and / or a season.

[0041] Within the scope of the present invention, a distinction is made between a parameterized scenario SZp and a concrete scenario SZc. A parameterized scenario SZp is defined within the scope of this invention as a scenario in which both the scenario parameters P ias well as the associated scenario parameter values ​​PV i not all are specified. A concrete scenario SZc is a scenario SZ in which the concrete scenario parameters Pc i and the corresponding concrete scenario parameter values ​​PVc i or value ranges of the concrete scenario parameter values ​​PVc i are defined. Both scenarios, the parameterized scenario SZp and the concrete scenario SZc, are scenarios that are specifically written in machine-readable code.

[0042] For the creation of parameterized scenarios SZp iVarious sources can be used, such as requirement specifications, expert knowledge, and / or measurements taken on public roads or on test sites using sensors. The sensors used can include, in particular, radar systems with one or more radar sensors, LIDAR systems for optical distance and speed measurement, image-capturing 2D / 3D cameras in the visible, IR, and / or UV ranges, and / or GPS systems.

[0043] These parameterized scenarios SZp generated from various data sources k are stored in the scenario database 200. The scenario database 200 also contains scenario parameters P i , Scenario parameter values ​​PV i , concrete scenarios SZc i , concrete scenario parameters Pc i , concrete scenario parameter values ​​PVc i and, if necessary, further information is stored.

[0044] The test agent 350 of the test case module 300 comprises a test strategy with a software application. The test strategy specifies how the test agent 350 selects suitable parameterized scenarios SZp k and specific scenario parameters Pc i and specific scenario parameter values ​​PVc i and, if necessary, further information, a plurality of test cases T i for a simulation task for testing, calibrating, and validating the ADAS / ADS system. The test strategy used is determined by the purpose of the simulation, i.e., what insights into the behavior of the ADAS / ADS system or a specific driving function are to be gained during the simulation. An example simulation task is the simulation of an overtaking maneuver on a highway (cut-in scenario).

[0045] Thus, the concrete scenario parameters Pc to be varied are iselected depending on the application. Other specific scenario parameters Pc i are kept constant, e.g. scenario parameters Pc i that describe the vehicle if only a vehicle derivative is considered for testing. Additionally, a calibration data set can be defined.

[0046] The test cases T determined in this way i The test agent 350 stores the data in the test database 370. The data for a test case T i can contain a test identification number (Test ID), a user name, a creation time and the name of the selected parameterized scenario Szp i In addition, the specific scenario parameters Pc i stored that the parameterized scenario Szp i into a concrete scenario Szc i and thus into a test case T i transform.

[0047] The test cases T stored in the test database 370i are passed on to the parameter module 400. The parameter module 400 contains a first software application 420, which is generated from the test cases T i creates a test case space 430 which is defined by the scenario parameters P i of the respective parameterized scenario SZ.

[0048] This test case space 430 is now evaluated by simulation. For this purpose, the behavior of the ADAS / ADS system is evaluated in the various simulation environments created by the test cases T i defined in the test case space 430, are tested by the simulation module 500 using a software application 550. Since, due to the large number of possible scenario parameter combinations, a large number of possible test cases T ican be generated, a detailed simulation of the entire test case space 430 would be too time-consuming and computationally intensive. In order to keep the number of simulations as low as possible, the invention provides for a very coarse-meshed investigation of the test case space 430. For this purpose, a coarse-meshed network is placed over the test case space 430 and the test cases T arranged at least approximately at the nodes of the coarse-meshed network k form the selected subset of test cases T i for which a simulation is carried out.

[0049] The tested test cases T t are evaluated as critical and non-critical test cases using a binary criticality indicator. Critical behavior is defined as exceeding certain safety thresholds. This criticality indicator is linked to the tested test case T t In particular, with a vector representation of the tested test case T tThe criticality indicator can be attached to the vector. These tested and evaluated test cases T t are now mapped in a coarse-meshed scenario parameter space 440.

[0050] Fig. Figure 2a shows an exemplary scenario parameter space 440, which is spanned by two scenario parameters P1 and P2. For the example of a cut-in scenario, the first parameter P1 is, for example, the distance d cutin between an ego vehicle and a target vehicle and the second parameter P2 is the speed of the ego vehicle v ego Depending on the two parameters P1, P2, the tested test case T t a binary criticality indicator is assigned, which includes the two categories "critical" and "non-critical". As an evaluation criterion, whether a tested test case T t is assessed as critical or non-critical, for example, the minimum time interval t minto a vehicle ahead. As a critical test case T crit A test case is then classified in which the minimum time interval cannot be maintained for a duration of more than one second: tmin>1s

[0051] In this case, the ego vehicle is in the vicinity of the target vehicle for more than one second. Test case T t is therefore classified as critical.

[0052] A tested test case T t , where the minimum time interval t min less than one second, is considered as non-critical test case T right rated.

[0053] For reasons of clarity, Fig. 2a only two scenario parameters P1 and P2 are shown. However, any number of scenario parameters P ibe chosen so that the dimension of the scenario parameter space 440 is arbitrarily large. In particular, the scenario parameter space 440 can also represent a latent vector space in which the test cases T t are vector representations.

[0054] According to the invention, a system boundary between a critical and a non-critical behavior of the ADAS / ADS system is now determined in the coarse-meshed scenario parameter space 440 by means of a second software application 450. The second software application 450 comprises a search algorithm that performs search iterations in order to determine the system boundaries for the behavior of the ADAS / ADS system to be examined in the considered parameterized scenario SZ i The prerequisite is that there are 440 critical test cases T in the scenario parameter space. critIn each search iteration, three steps are performed successively for all positions in the scenario parameter space 440 with a positive relevance indicator. The positions in the scenario parameter space 440 correspond to the tested and evaluated test cases T j .

[0055] As in the Fig. As shown in Figure 2a, in a first step, starting from a critical starting position S, a minimum standardized distance d min to other critical positions within the scenario parameter space 440.

[0056] As in Fig. 2b, in a second step all relevant non-critical positions are searched for that are within the minimum standard distance d mindefined pitch circle starting from the investigated starting position S. It is assumed that the system boundary between critical and non-critical behavior of the ADAS / ADS system lies between these positions. The minimum standard distance d min an addition factor a dist added to increase the radius of the reference circle in which relevant non-critical positions are searched. This allows the investigation area to be enlarged. If no relevant non-critical positions are found for the critical starting position S of the iteration being investigated, this starting position is no longer considered for subsequent iterations.

[0057] As in Fig. As shown in Figure 2c, in a third step, for each relevant non-critical position found, a new position is generated halfway between the critical starting position being examined and the relevant non-critical position. To ensure that no positions are considered twice, an additional check is performed to determine whether the new position already exists in the scenario parameter space 440.

[0058] From these new positions, new test cases T new For this purpose, the coordinates of the positions in the scenario parameter space 440 are transferred to the test module 350, which then creates the new test cases T new generated. In the simulation module 400, the behavior of the ADAS / ADS system in these new test cases T new simulated. The behavior is again evaluated with a criticality indicator and this is compared with the vector representation of the new test cases T new . linked.

[0059] The coarse-meshed scenario parameter space 440 is thus expanded during the execution of a plurality of k iterations by new test cases T new .at the assumed system boundary between critical and non-critical test cases. With each iteration, the density of test cases T increases. new + T j in the area of ​​the assumed system boundary. This means that by specifying the number k of iterations and the addition factor a dist of the radius can be used to determine how precisely and how wide the system boundary is examined. With a small value of k and a large value of a dist However, this also leads to a less precise determination of the system boundary. Overall, the method according to the invention makes it possible to determine the number of test cases T to be simulated for an initially large test case space 430. i to reduce.

[0060] The data from the scenario parameter space 440 and further evaluation results across the identified system boundaries can be forwarded to the output module 800. The output module 800 is embodied, in particular, as a computer, tablet, or mobile phone. The output module 800 comprises a software application 850 for generating output data 870 from the evaluation results, in particular in the form of a report. The report can contain, for example, graphics, diagrams, text messages, a PowerPoint presentation, video sequences, etc. The output data 870 can then be displayed on a user interface of the output module 800, embodied, in particular, as a screen.

[0061] In a further development, it can be provided that a plurality of simulation modules 500 are used for the simultaneous evaluation of a plurality of test case spaces 430, so that a large number of test cases T ican be processed in parallel. The simulation modules 500 can be integrated into a cluster of computing units. This can significantly reduce computing time.

[0062] The inventive localization of the position of the system boundary between critical and non-critical behavior of the ADAS / ADS system in the scenario parameter space 440 enables a holistic comparison of software and hardware changes, regardless of the phase of the development process in which these changes are implemented. For example, newly trained neutral networks of perception functions can be tested. An improvement in the system behavior of the ADAS / ADS system then becomes visible as a reduction in the critical behavior range in the scenario parameter space 440. Furthermore, changes to the calibration parameters can be validated for the entire scenario parameter space 440. Furthermore, various vehicle derivatives can be tested in order to specifically select and test the vehicle with the largest critical range in the parameter space 370.

[0063] Furthermore, the inventive determination of the system boundary in the scenario parameter space 440 allows for the application of holistic continuous integration (CI) in software development. With CI, code changes from multiple developers are continuously merged to identify and resolve integration issues as quickly as possible, thus developing high-quality software as efficiently as possible. With the help of CI, errors can be identified early in the development process of an ADAS / ADS system, reducing the time and cost required for error correction. Furthermore, it ensures that code changes are correctly integrated, reducing the risk of conflicts and other integration issues.

[0064] Furthermore, knowledge of the system boundary between critical and non-critical behavior in the scenario parameter space 440 can be used for the safety argumentation for the release of the ADAS / ADS system. However, this requires information about the probability of occurrence of specific scenarios SZp i required in real test drives in real traffic.

[0065] To obtain these, 700 concrete scenarios SZc can be created in the scenario identification module. i identified by test drives and their concrete scenario parameter values ​​PVc i This allows the completeness of the parameterized scenarios SZp i used for the simulation can be verified. Furthermore, the behavior of the ADAS / ADS system in real driving tests can be compared with the behavior in the simulation. For this purpose, it is investigated how many specific scenarios SZc ifrom the real test drive lie in the critical range of scenario parameter space 440. From this, parameters can be derived for the probability of occurrence of a critical system behavior of the ADAS / ADS system during a standardized time or distance interval. These parameters can, in turn, form the basis for a safety argument for the approval of the ADAS / ADS system.

[0066] To obtain these, 700 concrete scenarios SZc can be created in the scenario identification module. i identified from test drives and their concrete scenario parameter values ​​PVc i This allows the completeness of the parameterized scenarios SZp used for the simulation to be determined. i Furthermore, the behavior of the ADAS / ADS system in real driving tests can be compared with the behavior of the ADAS / ADS system tested in the simulation. For this purpose, it is investigated how many concrete scenarios SZc ifrom the real test drive lie in the critical range of scenario parameter space 440. From this, characteristic values ​​for the probability of occurrence of a critical system behavior of the ADAS / ADS system during a standardized time or distance interval can be derived. These characteristic values, in turn, can serve as the basis for a safety argument for the approval of the ADAS / ADS system.

[0067] The characteristic values ​​for the probability of occurrence of a critical system behavior can also be transferred to the output module 800. The output module 800 then creates a report for the release of the ADAS / ADS system.

[0068] In Fig. 3 shows the method steps for determining the critical system limits of a driver assistance system (ADAS) and / or an automated driving system (ADS) with at least one driving function, in particular for testing, applying and securing the ADAS / ADS system.

[0069] In a step S10, a plurality of test cases T i from a test module 300 with a test agent 350 for a parameterized scenario SZp i by selecting specific scenario parameters Pc i and concrete scenario parameter values ​​PVc i generated from a scenario database 200 for a simulation task.

[0070] In a step S20, a test case space 430 with the test cases T i created by a parameter module 400.

[0071] In a step S30, a simulation of the behavior of at least one driving function of the ADAS / ADS system is carried out in a test case T i specific simulation environment by a simulation module 500 to determine simulation results, wherein the simulation is only for a subset of the test cases T i carried out in test case room 430.

[0072] In a step S40, the simulation results are evaluated to determine a binary criticality indicator for the tested test case T t to determine so that a test case T t either as critical test case T crit or as a non-critical test case T right is classified.

[0073] In a step S50, a scenario parameter space 440 for the tested test cases T t created, with each tested test case T t the corresponding criticality indicator is assigned.

[0074] In a step S60, a system boundary in the scenario parameter space 440 between the critical test cases T crit and the non-critical test cases T right determined using a search algorithm.

[0075] In a step S70, new test cases T new generated at the system boundary in the scenario parameter space 440.

[0076] Fig.4 schematically illustrates a computer program product 900 comprising executable program code 950 configured to perform the method according to the first aspect of the present invention.

[0077] The present invention enables the system boundary between critical and non-critical behavior of an ADAS / ADS system with respect to a specific scenario to be determined using an efficient search algorithm, enabling targeted testing at the system boundary of an ADAS / ADS system. This saves computing resources because the entire test case space no longer needs to be tested. The development, integration, and testing process of an ADAS / ADS system can be improved, thus increasing the overall safety of an ADAS / ADS system. Reference symbol 100 systems 200 scenario database 220 Cloud Computing Infrastructure 300 test modules 350 test agents 370 test database 400 parameter module 420 Software application 430 test case room 440 Scenario parameter space 450 software application 500 simulation module 550 software application 700 Scenario Identification Module 800 output module 850 Report 900 Computer program product 950 program code QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2020 211 477 A1

[0006] DE 10 2018 219 564 A1

[0007]

Claims

[1] Method for determining the critical system limits of a driver assistance system (ADAS) and / or an automated driving system (ADS) with at least one driving function, in particular for testing, applying and securing the ADAS / ADS system, wherein a scenario (SZ i ) represents a traffic event in a temporal sequence and is determined by a selection of scenario parameters (P i ) and associated scenario parameter values ​​(PV i ) is defined as comprising: - Generating (S10) a plurality of test cases (T i ) by a test module (300) with a test agent (350) for a parameterized scenario (SZp i ) by selecting concrete scenario parameters (Pc i ) and concrete scenario parameter values ​​(PVc i ) from a scenario database (200) for a simulation task; - Creating (S20) a test case space (430) with the test cases (T i ) by a parameter module (400); - Carrying out (S30) a simulation of the behavior of at least one driving function of the ADAS / ADS system in a test case (T i ) specific simulation environment by a simulation module (500) to determine simulation results, wherein the simulation is only carried out for a subset of the test cases (T i ) is carried out in the test case space (430); - Evaluate (S40) the simulation results to determine a binary criticality indicator for the tested test case (T t ) so that a test case (T t ) either as a critical test case (T crit ) or as a non-critical test case (T right ) is classified; - Creating (S50) a scenario parameter space (440) for the tested test cases (T t ), where each tested test case (T t ) the corresponding criticality indicator is assigned; - Determining (S60) a system boundary in the scenario parameter space (440) between the critical test cases (T crit ) and the non-critical test cases (T right ) with a search algorithm; - Generation (S70) of new test cases (T new ) at the system boundary in the scenario parameter space (440). [2] Method according to claim 1, wherein a coarse-meshed network is placed over the test case space (430) and the test cases (T k ) the selected subset of test cases (T i ) form. [3] Method according to claim 1 or 2, wherein the search algorithm starts from a starting position (S) with a critical test case (T crit ) in the scenario parameter space (440) in a range defined by a minimum standard distance (d min ) and an addition factor (a dist) determined partial circle according to uncritical positions in the scenario parameter space (440) with uncritical test cases (T crit ) seeks. [4] Method according to claim 3, wherein for each non-critical position found, a new position is generated halfway between the starting position and the non-critical position in question, and wherein it is checked whether the new position is not already present in the scenario parameter space (440). [5] Method according to claim 4, wherein for each new position a new test case (T new ) is generated. [6] Method according to claim 5, wherein the coordinates of the new position in the scenario parameter space (440) are sent to the test module (350) for generating a new test case (T new ), whereby in the simulation module (400) the behavior of the ADAS / ADS system in this new test case (T new) is simulated, whereby the behavior is evaluated with a criticality indicator and this is assigned to the vector representation of the new test case (T new ) is assigned. [7] Method according to one of claims 1 to 6, wherein the search algorithm performs a plurality of iterations, so that with each iteration new test cases (T new ) and the scenario parameter space (440) is expanded to include new test cases (T new ) at the system boundary between the critical test cases (T crit ) and the non-critical test cases (T right ) is expanded. [8] Method according to one of claims 1 to 7, wherein the determined system boundary between critical and non-critical behavior of the ADAS / ADS system in the scenario parameter space (440) is used for software development of the ADAS / ADS system with continuous integration (CI). [9] Method according to one of claims 1 to 8, wherein the determined system boundary between critical and non-critical behavior of the ADAS / ADS system in the scenario parameter space (440) is used for a safety argumentation for the release of the ADAS / ADS system. [10] Method according to claim 9, wherein, for a release, the determined system boundary is supplemented with information about the probability of occurrence of specific scenarios (SZp i ) in real test drives in real traffic. [11] System (100) Method for determining the critical system limits of a driver assistance system (ADAS) and / or an automated driving system (ADS) with at least one driving function, in particular for testing, applying and securing the ADAS / ADS system, wherein a scenario (SZ i ) represents a traffic event in a temporal sequence and is determined by a selection of scenario parameters (P i ) and associated scenario parameter values ​​(PVi ), comprising a scenario database (200), a test case module (300) with a test agent (350), a parameter module (400) and a simulation module (500), wherein the test module (300) is designed to process a plurality of test cases (T i ) by a test module (300) for a parameterized scenario (SZp i ) by selecting concrete scenario parameters (Pc i ) and concrete scenario parameter values ​​(PVc i ) from a scenario database (200) for a simulation task; wherein the parameter module (400) is designed to generate a test case space (430) with the test cases (T i ); wherein the simulation module (500) is designed to create a simulation of the behavior of at least one driving function of the ADAS / ADS system in a test case (T i) specific simulation environment to determine simulation results, and wherein the simulation is only carried out for a subset of the test cases (T i ) in the test case space (430); wherein the simulation module (500) is designed to evaluate the simulation results in order to determine a binary criticality indicator for the tested test case (T t ) so that a test case (T t ) either as a critical test case (T crit ) or as a non-critical test case (T right ) is classified; wherein the parameter module (400) is designed to create a scenario parameter space (440) for the tested test cases (T t ), where each tested test case (T t ) the associated criticality indicator is assigned; and wherein the parameter module (400) is designed to define a system boundary in the scenario parameter space (440) between the critical test cases (T crit ) and the non-critical test cases (Tright ) using a search algorithm and, together with the test module (300), to generate new test cases (T new ) at the system boundary in the scenario parameter space (440). [12] System (100) according to claim 11, wherein a coarse-meshed network is placed over the test case space (430) and the test cases (T k ) the selected subset of test cases (T i ) form. [13] System (100) according to claim 11 or 12, wherein the search algorithm starts from a starting position (S) with a critical test case (T crit ) in the scenario parameter space (440) in a range defined by a minimum standard distance (d min ) and an addition factor (a dist ) determined partial circle according to uncritical positions in the scenario parameter space (440) with uncritical test cases (T crit) and for each non-critical position found, a new position is created halfway between the start position and the non-critical position in question. [14] ADAS / ADS system for a motor vehicle, characterized by that the method according to one of claims 1 to 10 is used for testing, applying and securing and / or for verifying, calibrating and validating the ADAS / ADS system. [15] A computer program product (900) comprising an executable program code (950) configured to carry out the method according to any one of claims 1 to 10 when executed.

Images