Methods for optimizing tests of control systems for automated vehicle dynamics systems

Abstract

Methods for optimizing tests of control systems for automated vehicle dynamics systems comprising the following steps: - Defining relevant parameters and relevant system responses of the control system to be tested, - Generating quasi-random parameter combinations of the relevant parameters, - Generating system responses of the control system under test in response to the quasi-randomly generated parameter combinations, - Creating and training a prediction model based on the quasi-randomly generated parameter combinations and the system responses of the control system under test, - Generating and analyzing the relevant system responses of the prediction model depending on the quasi-randomly generated parameter combinations, - Limiting the parameter space to at least a sub-area depending on the analyzed system responses of the prediction model, - quasi-random generation of new parameter combinations in at least one sub-area with higher density, - Repeating the previous three steps: generating and analyzing the relevant system responses of the prediction model depending on the quasi-randomly generated parameter combinations, narrowing down the parameter space to at least one sub-area depending on the analyzed system responses of the prediction model, and quasi-randomly generating new parameter combinations in the at least one sub-area with greater density. - Determining the confidence level of the prediction model after each repetition of the step of quasi-randomly generating new parameter combinations with greater density for each sub-area narrowed down in this repetition, - Ending the repetition of the three steps: generating and analyzing the relevant system responses of the prediction model depending on the quasi-randomly generated parameter combinations, narrowing down the parameter space to at least one sub-area depending on the analyzed system responses of the prediction model, and quasi-randomly generating new parameter combinations in the at least one sub-area with greater density for the at least one narrowed-down sub-area whose determined confidence level of the prediction model reaches a predetermined confidence level. - Obtaining optimized parameter combinations for testing control systems for automated vehicle dynamics systems.

Description

[0001] The invention relates to a method for optimizing tests of control systems for automated vehicle dynamics systems, in particular for optimizing the selection of test parameters and performing tests more efficiently over large parameter spaces. State of the art

[0002] In the development of complex driver assistance systems, the number of parameters and test scenarios to be investigated is constantly increasing. Even modern methods and technologies, such as computer-aided simulations, cannot reliably provide the necessary resources for a comprehensive investigation of systems. Currently, there are various approaches to testing these systems, hereinafter referred to as test systems.

[0003] Before testing the test system, the parameters to be included are typically defined. These describe the circumstances, influencing factors, and framework conditions for a test. Both internal and external system parameters can be selected for testing. Checking the test system for compliance with the defined requirements can be done through qualification, verification, or validation.

[0004] Based on the identified parameters, a parameter space is defined whose dimensions correspond to the number of selected parameters. Within this parameter space, various parameter settings or values ​​are tested. Different parameter combinations are used, in which several or all parameters are varied and / or combined with different values.

[0005] The tests generate test data based on the parameter combinations tested. The tests themselves can be conducted in various ways. They can be performed on virtual test systems, such as simulations, and on real test systems, such as vehicles. Numerous tests exist in between, where parts of the test system are virtual and other parts are real. The generated test data provides insights into the test system under test, which, combined with relevant expertise, lead to optimizations.

[0006] The validity of the test data, and therefore of the tests, depends on how the parameter combinations used for testing are generated. Factors such as the number, values, combinations, and distribution of the parameters within the parameter space play a role.

[0007] In full factorial design of experiments, every possible parameter combination is tested. For continuous parameters, fixed parameter values ​​must be selected as a subset of all possible settings. The total number of parameter combinations to be tested increases with the number of parameter values. The parameter space is tested along a grid. The resulting test data can then be used directly for analysis and to gain insights. Depending on the number of parameters and the grid density, the number of possible combinations can become so large that it is impossible to perform all tests. For this reason, the parameter spaces are sometimes reduced, or specific subsets of the full factorial parameter combinations are selected. Their selection or restriction is arbitrary.Due to the complex interrelationships, critical areas cannot be identified with sufficient certainty and may therefore be overlooked.

[0008] In the statistical modeling of parameter combinations, probability distributions, such as a normal distribution, are used as the basis for these combinations, and it is assumed that a particular parameter combination occurs most frequently. The remaining tests are then distributed around the parameter combination with the highest probability of occurrence, with decreasing density. The resulting test data are then used directly for the corresponding analyses.

[0009] Simulation methods with feedback are also known. Based on the result of a test, a decision is made as to which parameter combination will be simulated next. The goal is to uncover local and / or global maxima or minima, as well as system boundaries. This approach has several disadvantages. Especially with the full factorial simulation approach, the number of trials increases exponentially with the number of parameters. A successful simulation can only be achieved with a significant investment of resources and time. If system responses are disregarded, the simulation is carried out according to a predefined plan. This can lead to insufficient measurement data being acquired in critical regions of the parameter space.While simulations with feedback employ dynamic experimental design, simple simulations typically fail to detect multiple minima or maxima, as they are usually designed to find a single extreme value. More complex simulations with feedback may be capable of locating multiple extrema simultaneously, but this comes at the cost of high resource consumption.

[0010] WO 2015 067 649 A1 concerns virtual test optimization for driver assistance systems (DAS). A test scenario, defined by test parameters, is run through in a real-world driving test and / or virtually. To create a second test, the first test is modified to shift the test parameter within a critical range. A disadvantage is that the parameter variation leading to critical driving behavior is arbitrarily, at least a priori, determined or arises randomly from a test. There is no systematic delimitation of critical ranges in which the activation of an DAS is triggered. Object of the invention

[0011] In contrast, the object of the invention is to improve the generation and execution of tests for control systems for automated vehicle dynamics systems. In particular, tests with globally consistent confidence and locally higher confidence should require less overall resources than in the prior art and enable a reduction in test time and effort. Description and advantages of the invention

[0012] The problem is solved by a method for optimizing tests of control systems for automated vehicle dynamics systems according to the measures of independent claim 1.

[0013] Automated vehicle dynamics systems are systems for controlling vehicle components, such as drive, deceleration, or steering, which are not directly operated by the driver but are activated and actuated automatically or autonomously by dedicated control units. These vehicle dynamics systems can be designed as driver assistance systems for specific driving situations. Examples of such driver assistance systems include the anti-lock braking system (ABS), electronic stability program (ESP), adaptive cruise control (ACC), automatic emergency braking (AEB), and lane keeping assist (LKA). These functions are considered automation levels 1 or 2, which support vehicle operation or take over partial tasks of vehicle control. Automated vehicle dynamics systems can also be classified as automation level 3 (operational automation).Conditional automation) and higher levels, where the vehicle temporarily or permanently takes over tasks independently. The latter involves the integration of multiple systems, each performing subtasks, which together drive the vehicle in a highly automated or even autonomous manner.

[0014] In the first step of the method according to the invention, relevant parameters and system responses of the control system to be tested are defined. The control system to be tested is the test system. The relevant parameters constitute the parameter space, the dimensions of which depend on the number of parameters. The variable parameters and the system responses to be recorded, or specific observations, also called Key Performance Indicators (KPIs), depend primarily on the design of the test system itself, i.e., its hardware and software design, its functional scope, i.e., the actual control objective of the vehicle dynamics system, and the environment in which the control system is to be used. Furthermore, they also depend on the test objectives, the degree of abstraction of the control system to be tested, and its system boundaries.Defining the parameters and system responses, along with other specifications, is part of the initial experimental design. The KPIs can be classic accident-related metrics, such as time to impact, or comfort-related responses from the system, such as maximum lateral accelerations.

[0015] In the next step, which can correspond to the beginning of a first iteration, quasi-random parameter combinations of the relevant parameters are generated over the parameter space. Each defined parameter can be of different types, for example, as a Boolean variable that can only assume a few states, via discrete nominal or ordinal scaled variables, to continuous distributions. These include various types of probability distributions, such as the discrete uniform distribution or the normal distribution. A combination of both types for individual parameters can also be useful. The choice of distribution type is made in relation to the test objective or the desired representativeness of reality. For example, the discrete uniform distribution is suitable for investigations across the entire range of values ​​of the parameters. Among other things, the performance of a function can be tested across the entire application domain, such as the required environment or...Scenarios can be analyzed and evaluated. This also allows for the assessment of various functional and system configurations or limits of system performance, such as sensor tolerances or physical limitations. Finally, realistic probability distributions for the parameters can be determined based on actual driving data and measurement recordings.

[0016] Quasi-random means that the distribution of parameter values ​​across the parameter space is not determined by a true random mechanism, such as a fixed probability distribution. The number and density of test points can be adjusted depending on the system complexity and test objective, possibly using established Design of Experiments (DoE) methods. After calculating the quasi-random parameter combinations, their confidence or performance can be assessed using various metrics. Compared to a full factorial design, these combinations have a significantly reduced scope, yet still cover the entire parameter space.

[0017] The next step involves generating system responses from the control system under test, i.e., the test system itself, in response to the quasi-randomly generated parameter combinations. The test system can represent any configuration of a control system for automated driving dynamics systems throughout the entire development process, ranging from a modeled concept idea to software under test, a single component under test, an assembly under test, or even a complete vehicle under test. The test system can be modeled using established concept-in-the-loop (CiL), software-in-the-loop (SiL), hardware-in-the-loop (HiL), or model-in-the-loop (MiL) methods, or it can be the complete system under test. If modeling is used, it should be as simple as possible to ensure efficient generation of the system responses.Simple, preferably existing (partial) models can be used, as well as simple deterministic software code that omits the underlying physical principles. In the case of a complete system under test, the system responses can also be generated through test drives on a test bench or in the real world. Finally, combinations of modeled and actually generated system responses from the test system are also conceivable.

[0018] In the next step of the method according to the invention, a prediction model is created and trained based on the quasi-randomly generated parameter combinations and the corresponding system responses of the control system under test, i.e., the test system. The prediction model can be a statistical model built using machine learning algorithms, for example, in the form of an artificial neural network. Advantageously, this allows the system behavior to be represented in a trained model, and the system responses for arbitrary, unknown parameter combinations can be estimated. Control mechanisms can be used during the creation of the prediction model to calculate its confidence.

[0019] In the next step, the relevant system responses of the prediction model are generated and analyzed based on the quasi-randomly generated parameter combinations. These parameter combinations are distributed across the entire parameter space with the original density. The predictions of the model, i.e., the system responses or KPIs, are evaluated for their criticality, for example, by comparing them with predefined thresholds, by identifying high gradients between neighboring system responses, or by detecting unexpected system behavior such as violations of functional or test objectives. Inferences are drawn about the corresponding parameters, and at least one critical area, also called a Region of Interest (ROI), is identified within the parameter space.

[0020] The next step involves narrowing down the parameter space to at least a sub-region based on the analyzed system responses of the prediction model. The parameter values ​​are limited to a range around the identified critical region. This is done individually for each parameter of the parameter combination. The result is a limited sub-region of the parameter space. The area around the critical region can be defined in advance of the experimental design using a percentage or absolute deviation for all parameters or individually for each parameter. Theoretically, the accuracy of the prediction model could already meet the required standard across the entire parameter space from the outset. It is also theoretically conceivable that no critical regions exist. In practice, however, these cases are unlikely to occur given the current level of complexity of vehicle dynamics systems.The inventive method could then be terminated at this point. These cases will not be considered further.

[0021] In the next step, new parameter combinations are generated quasi-randomly in at least one sub-area with a higher density compared to the original distribution density. Within this sub-area, the same or different criteria can be used to generate the parameter combination as were used to generate the parameter combinations across the entire parameter space.

[0022] The next step involves repeating the previous three steps: generating and analyzing the relevant system responses of the prediction model based on the quasi-randomly generated parameter combinations; narrowing down the parameter space to at least one sub-area based on the analyzed system responses of the prediction model; and quasi-randomly generating new parameter combinations with higher density in at least one sub-area. With each iteration, the process narrows down to at least one further sub-area and subsequently increases the density of parameter combinations in these critical sub-areas—that is, only where testing or adjustments are actually needed. The starting point for each subsequent iteration is the previously narrowed sub-area(s), not the entire parameter space before the iteration.

[0023] Next, the confidence level of the prediction model is determined after each iteration of the step of quasi-randomly generating new parameter combinations with higher density for each subset defined in that iteration. Determining the confidence level of a trained model is generally known. For example, cross-validation can be performed using known input and output combinations. With appropriate design, some models, such as certain neural networks, can output their own goodness of fit, thus estimating how well the model reproduces the relationship between input variables and the responses based on them.

[0024] Next, the repetition of the three steps is completed: generating and analyzing the relevant system responses of the prediction model depending on the quasi-randomly generated parameter combinations; narrowing down the parameter space to at least one sub-area depending on the analyzed system responses of the prediction model; and quasi-randomly generating new parameter combinations in the at least one sub-area with higher density for the at least one narrowed-down sub-area whose determined confidence level of the prediction model reaches or exceeds a predefined confidence level. Determining the confidence level after each narrowing of the parameter space to a sub-area makes it possible to calculate several different local confidence levels.The process involves estimating the quality factors and thereby raising the local density of parameter combinations within the critical sub-areas to a corresponding level, i.e., iterating until the required confidence level is reached for each sub-area. This advantageously further improves the adaptability of the method according to the invention to the respective development stage, since different confidence levels can be specified depending on the development stage. In other words, the steps are repeated or iterated until all critical areas are determined with sufficient accuracy. The criterion for sufficient accuracy is the confidence level of the prediction model in the respective defined sub-area.

[0025] Finally, optimized parameter combinations are obtained for testing control systems for automated driving dynamics systems. Optimized, or more precisely, resource-optimized, means that locally densely populated areas contrast with a globally low density, reducing overall resource consumption by orders of magnitude compared to the state of the art. As a result, parameter combinations are available that are sufficiently densely distributed at critical points. These can then be used in (further) tests of the automated driving dynamics systems, e.g., in test drives or complex hardware-in-the-loop (HiL) tests, but also in software-in-the-loop (SiL) or model-in-the-loop (MiL) tests. With the increasing development of driver assistance and driving dynamics systems, more complex parameter combinations and thus more complex tests are required, also due to the increased complexity of the parameters involved (sensor inputs, situation variables, functions involved).The method according to the invention offers the advantage of providing optimized parameter combinations for each development stage. Advantageously, the simulation of system responses at a higher development stage could be performed, at least partially, by the prediction model of the preceding stage. Furthermore, critically defined sub-areas can be identified, even without subsequent testing, particularly in early development or functional design phases, revealing critical weaknesses in the automated driving dynamics system. Addressing these weaknesses early on saves significant resources that would otherwise be required for later modifications.

[0026] A key advantage of the method according to the invention is the efficient mapping of large parameter spaces with a small number of parameter combinations to be tested, which can exhibit a high local density. Particularly advantageous is the ability to identify multiple extremes, i.e., multiple critical regions, which can be analyzed individually or in parallel. A further advantage is the ease with which initial system responses of the test system to the parameter combinations can be generated due to the quasi-random distribution of these combinations. This, precisely because of the low initial density, enables the rapid and resource-efficient generation of test data for the prediction model. Furthermore, the test system is, in most cases, already available or can be assembled from model components.This allows for the generation of a sufficiently accurate initial combination of system inputs and outputs, enabling the training of the prediction model to estimate system responses for any unknown parameter combinations. In particular, the combination of training a prediction model with easily generated data and the subsequent iterative focus on critical areas results in a significant increase in efficiency in test case optimization, while simultaneously ensuring applicability across the entire development chain, from concept creation to series production validation.

[0027] In an advantageous embodiment of the method according to the invention, after each quasi-random generation of new parameter combinations in at least one sub-area with higher density, the relevant system responses of the prediction model for these new parameter combinations are generated and used with the new parameter combinations for further training of the prediction model. Depending on the application or validation goal, this advantageously enables further training that can be more precise and / or targeted to specific areas of the parameter space.

[0028] In an advantageous embodiment of the method according to the invention, the confidence level of the prediction model is determined for the entire parameter space after each training step, and the training of the prediction model is terminated upon reaching a predetermined confidence level. This advantageously further improves the adaptability of the method according to the invention to the respective development stage, the validation goal, and / or the specific application, since different confidence levels may be required in each case. Furthermore, computing resources can be saved for further training beyond an unnecessary accuracy level.

[0029] In an advantageous embodiment of the method according to the invention, the test system is modeled based on a functional model, a system model, and an environment model of the control system for automated driving dynamics systems. The functional model models the function of the automated driving dynamics system, i.e., its actual purpose. The system model models how this purpose is to be implemented, including the sensors, processors, and actuators involved, as well as the communication between these elements. The environment model models where this is to be implemented, i.e., in which scenarios. A particularly advantageous aspect is that these building blocks can also be adapted independently to the driving dynamics system under test. This allows the same functions to be tested in different systems without creating a new functional model. The individual building blocks can therefore be reused.The same applies to different environments and scenarios. Furthermore, the division into these three building blocks is both very easy to model and particularly realistic. This approach is therefore especially resource-efficient.

[0030] The intended function can be simplified, for example, using scene-based function development with scenarios, which allows part of the environment to be defined in advance. Furthermore, relevant system properties and environmental variables must be defined. In addition to general requirements, such as weather conditions, the environment also includes identified scenarios that the overall system should be able to handle. The scenarios are defined by road and infrastructure elements as well as objects with specific starting situations and various actions over time. The complexity of the three elements to be modeled can vary considerably depending on the objective. For example, in system modeling, perfect system behavior, such as that of an ideal sensor, can be represented, provided that the influences of these three elements are explicitly excluded from the analysis.Alternatively, complex sensor characteristics can be modeled if the focus is on their influence. The initial generation method can also be created and executed using real vehicle data and / or simple deterministic software code, omitting the underlying physical principles. The complexity of the test system model can depend on the respective development phase and typically increases as the concept phase progresses through various series development phases to series release and monitoring, both for individual functions and the vehicle as a whole. This advantageously allows for the quick and easy creation of a model of the control system under test that is appropriate to the current development stage.Functional and system characteristics, as well as scenario specifications, can be described by parameters such as sensor tolerances or the speeds of other objects within the scenarios. Varying and combining these individual parameters leads to specific scenarios with defined functional behavior and specific system performance.

[0031] In an advantageous embodiment of the method according to the invention, the relevant system responses of the prediction model are analyzed by comparing the system responses of the prediction model with predefined thresholds, by comparing the system responses of the prediction model with predefined system responses that lie outside the system boundaries or the test objective, and / or by comparing a confidence level determined based on the system responses of the prediction model with a predefined confidence level, whereby a conclusion is drawn about the causal parameter combination from the system responses of the prediction model identified in this way. System responses of the prediction model can be expressed as quantitative quantities, such as distance to the vehicle ahead or braking distance, as qualitative quantities, such as whether a rear-end collision has occurred, and can be counted and / or scaled.Accordingly, these system responses can be compared with predefined numerical ranges or qualitative statements, and conclusions can be drawn about the presence of a critical response. The same conclusion can be drawn if the system response of the prediction model does not reach its confidence level, or if the system boundaries are exceeded, for example, the defined function, or if the test objectives are violated. In these cases, the parameter combinations responsible for the respective system responses of the prediction model are identified. In the next step of the method according to the invention, the search is then narrowed down to a sub-range around these identified parameter combinations.

[0032] One aspect of the present invention relates to a device for optimizing tests of control systems for automated vehicle dynamics systems, wherein the device is configured as a computing unit to perform all steps of a method according to any one of claims 1 to 5. It is understood that each step of the method according to the invention can be performed on the same or on different computing units, such as computers, that are interconnected. These computing units can be arranged locally or globally distributed. The computing unit can be part of a stationary or portable computer. Each computing unit has its own memory or memory shared with other computing units, as well as at least one processor.

[0033] One aspect of the invention relates to a computer program for optimizing tests of control systems for automated vehicle dynamics systems, wherein the computer program causes a computing unit to execute all steps of a method according to any one of claims 1 to 5 when it is executed on the computing unit. The computing unit has its own memory or memory shared with other computing units, as well as at least one processor. One of the specified methods is stored in the memory in the form of the computer program, and the processor is provided for executing the method when the computer program is loaded from memory into the processor.

[0034] The invention further relates to a computer-readable storage medium on which a computer program according to the present invention is stored. Computer-readable storage media, also referred to as machine-readable storage media, are known per se and can be designed as magnetic storage media (floppy disks), optical storage media (CDs), flash memory (USB sticks), read-only memory (ROM), or many other types.

[0035] The invention further relates to a program code with processing instructions for creating a computer program executable on a computer according to the present invention, wherein the program code yields the computer program when the program code is converted into an executable computer program according to the processing instructions.

[0036] The invention further relates to a computer program product, wherein the computer program product comprises a computer-readable storage medium according to the present invention and a computer program stored on the computer-readable storage medium according to the present invention, comprising a program code according to the present invention, wherein the program code is suitable for executing a method according to the present invention when the computer program is executed on a computer. Example of implementation

[0037] Further features, applications, and advantages of the invention will become apparent from the following description of exemplary embodiments of the invention with reference to the schematic drawings. These serve only to illustrate the invention and have no limiting effect on the subject matter of the invention as set forth in the patent claims.

[0038] This shows: Fig. 1a-c a comparison of different density distributions of parameter combinations of a two-dimensional parameter space; Fig. 2a-c a sequence of iterations to narrow down sub-areas of the parameter space; Fig. 3 a predicted minimum time to collision (Time To Collision - TTC) depending on the initial distance (time gap) and the deceleration of the target vehicle.

[0039] Fig. 1a shows, using a purely exemplary two-dimensional representation, an equidistant and Fig. 1b a quasi-random distribution of parameter combinations over the entire parameter space. Both distributions shown do not reflect the true ratio of the respective numbers of parameter combinations, but merely serve to illustrate that... Fig. 1b to not be evenly or equidistantly distributed, as well as to be in comparison to Fig. 1a significantly smaller number, and therefore a significantly lower density of parameter combinations. In Fig. Figure 1a shows a (rough) boundary 11 that separates the critical parameter combinations 10 lying within it from those lying outside. Whether a parameter combination is critical results from the system response corresponding to that parameter combination. If this response is critical, for example due to falling below a critical threshold, such as the time to impact, or due to a negative system response, such as hitting an obstacle, then this results in a critical parameter combination. In comparison, the parameter combinations lying outside the boundary 11 are non-critical because their corresponding system responses are non-critical.

[0040] The density of parameter combinations in Fig. 1b is insufficient to make a sufficiently accurate statement, based on model prediction, about which specific parameters, and in what values, are responsible for the criticality of the system response. Therefore, no meaningful corrective measures can yet be derived. Consequently, the parameter space is restricted around the critical parameters, and additional, quasi-random parameter combinations with higher density are generated within this space. These additional parameter combinations include 14 ( Fig. 1c) The system response, which can be critical or non-critical, is predicted and reflected back to the corresponding parameter combination. In this way, each iteration step provides a more detailed distinction between critical and non-critical parameter combinations. This allows for concrete statements about the causes of critical system responses. As a result, different critical sub-areas can be identified. The density of parameter combinations present in these areas is greater than the original density and can even be locally higher than the equidistant distribution according to [reference missing]. Fig. 1a. Because the non-critical areas 13, which are significantly larger than the critical sub-areas, have a very low density of parameter combinations to be tested, the testing effort using the method according to the invention is orders of magnitude lower than with conventional methods with the same confidence level (e.g. full factorial experiments).

[0041] Fig. Figures 2a-c illustrate the local density of the parameter combinations. Fig. Figure 2a shows a quasi-random distribution of parameter combinations 20 after the first iteration (circles), which includes not only the originally quasi-randomly generated parameter combinations but also the system responses subsequently generated by the test system. No local concentrations are yet visible. These are shown in Fig. Figure 2b illustrates this. The parameter combinations 21 (triangles) shown there after the second iteration, i.e., after the generation of system responses by the prediction model, demonstrate a densification of the parameter combinations in the (not shown) critical areas with locally increased density. After the third iteration, a further narrowing of the locally increased density can be seen based on the parameter combinations 22 (stars).

[0042] The invention will now be explained using the specific example of a control system for the longitudinal distance of an ego vehicle to the vehicle ahead, also called the target vehicle. In this exemplary scenario, the ego vehicle follows the target vehicle at a distance expressed as a time gap Δt. The target vehicle begins to decelerate with a (negative) acceleration aco to a final speed. The chosen scenario illustrates a typical application on a highway where the traffic flow is disrupted by roadworks, approaching the end of a traffic jam, or an accident. The ego vehicle is equipped with a number of sensors for object detection and the longitudinal distance control system, which combines adaptive cruise control (ACC) with a maximum deceleration of -3 m / s². 2 and automatic emergency braking system (AEB) with a maximum deceleration of -8 m / s 2The ego vehicle reacts to the dynamics of the target vehicle to maintain a safe distance. The system and the function of the ego vehicle's longitudinal distance control system were simulated based on conceptual models. The relevant system response or KPI (Key Performance Indicator) is defined as the Time To Collision (TTC). A TTC of 2 seconds was chosen as the safety-critical threshold.

[0043] The parameter space has five dimensions due to five relevant parameters, namely the ego velocity v. Ego , the initial (scenario start) time gap Δt, the (negative) acceleration of the target vehicle aco, the initial (scenario start) velocity of the target vehicle v CO,start as well as the final (scenario end) speed of the target vehicle a CO,finalFrom this parameter space, for example, 1920 quasi-random parameter combinations were generated using methods of statistical design of experiments (DoE). These were fed into the simulation of the test system, resulting in a corresponding time-to-impact (TTC) for each parameter combination. Subsequently, these simulated TTCs and the corresponding input parameters were used to train a model to predict the time to impact. Fig. Figure 3 now shows an example 2D representation of the 5D-TTC prediction model. In contrast to the Fig. 1a-c and Fig. Figures 2a-c do not represent the parameter space, i.e., not the input variables of the prediction model, but rather the system responses, i.e., the output variables. The vertical axis represents the initial time gap Δt in seconds, and the horizontal axis represents the (negative) acceleration of the target vehicle aco in m / s². 2The diagram area shows the distribution of the predicted TTC in s as level sets, i.e., the sets of all predicted TTCs to which the same TTC is assigned. The level sets are represented by isolines or level curves 30. In the present scenario, both vehicles travel at an initial speed of 120 km / h with the vertically plotted initial time gaps, when the target vehicle decelerates to a final speed of 20 km / h with the horizontally plotted decelerations aco.

[0044] Critical areas were defined based on the safety-critical threshold of 2 s plus a safety margin of 0.5 s. The safety margin (offset) is derived from the mean absolute error of the prediction model. Initially, two critical areas, 33 and 34, were identified, delineated by the dashed critical level lines 31 and 32 and shown hatched. If the initial time gap Δt is small and the deceleration of the target vehicle is very high, a critical time until impact is quite to be expected (critical area 33). In contrast, critical area 34 would probably only have been detectable with a full factorial experimental design. In retrospect, the transition between ACC and AEB could possibly be the cause of this second critical area 34.

[0045] The error probability of the prediction across the entire parameter space, i.e., the confidence level of the prediction model, is below the predefined target. The first iteration, which includes not only the originally quasi-randomly generated parameter combinations but also the system responses of the simulated test system, is sufficiently accurate to identify the critical regions of the TTC (Time-to-Calculation) and, based on these, to define sub-regions (ROIs) of the underlying parameter combinations. Subsequently, quasi-random parameter combinations are generated again in these sub-regions with an increased density compared to the original parameter combinations.

[0046] Elaborate experiments with 38,400 equidistant parameter combinations distributed across the parameter space, closely resembling a full factorial design of experiments, yielded a coefficient of determination of the first iteration of the prediction model alone of R. 2This results in a confidence level of 95%. Therefore, a reduced dataset of 5% of the nearly full factorial dataset is sufficient to achieve adequate accuracy for training the prediction model in this application. Depending on the use case or test objective, a higher or lower confidence level may be required. Restricting the sub-regions of the parameter space allows for further local increases in prediction confidence without processing very large and computationally intensive datasets. Reference symbol list 10 critical parameter combinations 11. Rough boundary of critical sub-area 12 non-critical area 13 non-critical area 14 additional parameter combinations 15. Detailed delimitation of critical sub-area 20 parameter combinations after the first iteration 21 parameter combinations after the second iteration 22 parameter combinations after the third iteration 30 Level line TTC 31 critical level line TTC 32 critical level line TTC 33 critical area 34 critical area

Claims

[1] Method for optimising tests of control systems for automated driving dynamics systems, comprising the following steps: - Defining relevant parameters and relevant system responses of the control system to be tested, - Generating quasi-random parameter combinations of the relevant parameters, - Generation of system responses of the control system to be tested in response to the quasi-randomly generated parameter combinations, - Creating and training a prediction model depending on the quasi-randomly generated parameter combinations and the system responses of the control system to be tested, - Generating and analyzing the relevant system responses of the prediction model depending on the quasi-randomly generated parameter combinations, - Restricting the parameter space to at least one sub-area depending on the analyzed system responses of the prediction model, - quasi-random generation of new parameter combinations in the at least one sub-area with greater density, - Repeating the previous three steps: generating and analyzing the relevant system responses of the prediction model depending on the quasi-randomly generated parameter combinations, limiting the parameter space to at least one sub-area depending on the analyzed system responses of the prediction model and quasi-randomly generating new parameter combinations in the at least one sub-area with greater density, - Determining the confidence level of the prediction model after each repetition of the step of quasi-randomly generating new parameter combinations with greater density for each sub-area defined in this repetition, - Terminating the repetition of the three steps of generating and analyzing the relevant system responses of the prediction model depending on the quasi-randomly generated parameter combinations, limiting the parameter space to at least one sub-area depending on the analyzed system responses of the prediction model and quasi-randomly generating new parameter combinations in the at least one sub-area with greater density for the at least one limited sub-area whose determined confidence level of the prediction model reaches a predetermined confidence level and - Obtaining optimized parameter combinations for testing control systems for automated driving dynamics systems. [2] Method for optimizing tests of control systems for automated driving dynamics systems according to claim 1, wherein after each quasi-random generation of new parameter combinations in the at least one partial area with greater density, the relevant system responses of the prediction model of these new parameter combinations are generated and used with the new parameter combinations for further training of the prediction model. [3] Method for optimizing tests of control systems for automated driving dynamics systems according to one of the preceding claims, wherein the confidence level of the prediction model is determined after each training step for the entire parameter space and the training of the prediction model is terminated when a predetermined confidence level is reached. [4] Method for optimizing tests of control systems for automated driving dynamics systems according to one of the preceding claims, wherein the modeling of the test system is carried out on the basis of a functional model, a system model and an environmental model of the control system for automated driving dynamics systems. [5] Method for optimising tests of control systems for automated driving dynamics systems according to one of the preceding claims, wherein the analysis of the relevant system responses of the prediction model is carried out by comparing the system responses with predetermined threshold values, by comparing the system responses with predefined system responses which lie outside the system limits or the validation target and / or by comparing a confidence level determined on the basis of the system responses with a predetermined confidence level, wherein a conclusion is drawn about the causal parameter combination from the system responses thus identified. [6] Device for optimising tests of control systems for automated driving dynamics systems, wherein the device is designed as a computing unit in order to carry out all steps of a method according to one of claims 1 to 5. [7] Computer program for optimizing tests of control systems for automated driving dynamics systems, wherein the computer program causes a computing unit to carry out all steps of a method according to one of claims 1 to 5 when it is executed on the computing unit. [8] Computer-readable storage medium, characterized by that a computer program according to claim 7 is stored on the computer-readable storage medium. [9] Program code with processing instructions for creating a computer program that can be run on a computer, characterized by that the program code results in the computer program according to claim 7 when the program code is converted into an executable computer program according to the processing instructions. [10] Computer program product, characterized bythat the computer program product comprises a computer-readable storage medium according to claim 8 and a computer program according to claim 7 stored on the computer-readable storage medium with a program code according to claim 9, wherein the program code is suitable for carrying out a method according to one of claims 1 to 5 when the computer program is executed on a computer.

Images