COMPUTER-IMPLEMENTED PROCEDURE FOR SCENARIO-BASED TESTING AND / OR HOMOLOGATION OF AT LEAST PARTIALLY AUTONOMOUS DRIVING FUNCTIONS TO BE TESTED USING KEY PERFORMANCE INDICATORS (KPI)

Description

[0001] The present invention relates to a computer-implemented method for scenario-based testing and / or homologation of at least partially autonomous driving functions to be tested by means of key performance indicators (KPI).

[0002] The present invention further relates to a test unit for scenario-based testing and / or homologation of at least partially autonomous driving functions to be tested using key performance indicators (KPIs). The present invention further relates to a computer program and a computer-readable data carrier. State of the art

[0003] Driver assistance systems such as adaptive cruise control and / or functions for highly automated or autonomous driving can be verified or validated using various testing methods. In particular, hardware-in-the-loop methods, software-in-the-loop methods, simulations, and / or test drives can be used.

[0004] The effort, in particular the time and / or cost required to test such vehicle functions using the above-mentioned testing methods is typically very high, since a large number of potentially possible driving situations must be tested.

[0005] Testing an at least partially autonomous vehicle exclusively on the road, covering billions of kilometers, is not feasible due to time and cost constraints. Furthermore, many redundant test kilometers would be generated, while critical and unusual situations relevant to the capabilities of the at least partially autonomous vehicle would not arise.

[0006] This can lead to significant effort, particularly for test drives and simulations. DE 10 2017 200 180 A1 specifies a method for verifying and / or validating a vehicle function intended to guide a vehicle autonomously in the longitudinal and / or transverse directions.

[0007] The method comprises determining, on the basis of environmental data relating to an environment of the vehicle, a test control instruction of the vehicle function to an actuator of the vehicle, wherein the test control instruction is not implemented by the actuator.

[0008] The method further comprises simulating, on the basis of environmental data and using a road user model with respect to at least one road user in the vicinity of the vehicle, a fictitious traffic situation that would exist if the test control instruction had been implemented.

[0009] The method further comprises providing test data relating to the fictitious traffic situation. The vehicle function is operated passively in the vehicle to determine the test control command.

[0010] The disadvantage of this method is that in order to verify and / or validate the vehicle function, actual operation of the vehicle is required to determine the required data.

[0011] Christian Berger: Automating acceptance tests for sensor- and actuator-based systems on the example of autonomous vehicles discloses a method for automating acceptance tests using the example of an autonomous vehicle.

[0012] The conceptual theory presented consists in deriving a formal, machine-processable specification of the system context of a system under development from the requirements and acceptance criteria in order to generate and execute interactive, unsupervised, and automatable simulations for the system under development.

[0013] Manufacturers of at least partially autonomous vehicles require a parallelized, simulation-based solution. This is where the "scenario-based testing" method comes in. This requires intelligent test follow-up control that evaluates simulation results and, if necessary, adjusts the parameters.

[0014] It is therefore an object of the invention to provide a method, a test unit, a computer program, and a computer-readable data carrier that provide an evaluation of simulations and / or test cases in scenario-based testing and / or homologation of at least partially autonomous driving functions to be tested using key performance indicators (KPIs). The method according to the invention comprises mapping KPIs using KPI plug-ins; Dynamic and reusable selection of KPI plug-ins for simulations and / or test cases; Selection of at least one KPI plug-in by a KPI plug-in mechanism during simulation and / or test definition; Automatic execution of the selected KPI plug-in by the KPI plug-in mechanism during execution of the KPI plug-in mechanism; Determining a KPI value by passing a simulation result to a KPI script of the KPI plug-in; Transmitting the simulation result and the KPI value to a test creation and configuration and generating optimized test cases and / or an optimized parameter configuration; and Providing an at least partially autonomous driving function optimized using the optimized test cases and / or the optimized parameter configuration.

[0015] Furthermore, the test unit for evaluating simulations and / or test cases in scenario-based testing and / or homologation of at least partially autonomous driving functions to be tested by means of key performance indicators (KPIs) comprises means for mapping KPIs using KPI plug-ins; dynamic and reusable selection of KPI plug-ins for simulations and / or test cases; means for selecting at least one KPI plug-in using a KPI plug-in mechanism during simulation and / or test definition; means for automatically executing the selected KPI plug-in using the KPI plug-in mechanism during execution of the KPI plug-in mechanism; means for determining a KPI value by passing a simulation result to a KPI script of the KPI plug-in; Means for transmitting the simulation result and the KPI value to a test creation and configuration and generating optimized test cases and / or an optimized parameter configuration;and means for providing an at least partially autonomous driving function optimized using the optimized test cases and / or the optimized parameter configuration; Disclosure of the invention

[0016] The object is achieved according to the invention by a computer-implemented method for evaluating simulations and / or test cases in scenario-based testing and / or homologation of at least partially autonomous driving functions to be tested by means of key performance indicators (KPI) according to patent claim 1, a test unit according to patent claim 12, a computer program according to patent claim 14 and a computer-readable data carrier according to patent claim 15.

[0017] An autonomous vehicle contains numerous control units. Each individual control unit and their interconnectedness must be extensively tested during development and homologation. Scenario-based testing is used to ensure the faultless functioning of the control units in every traffic situation. Scenario-based testing analyzes the vehicle's driving behavior in a traffic situation that is as realistic as possible. The aspects of the traffic situation to be analyzed and their evaluation depend on the system being tested. For this purpose, scenario-based testing of systems and system components for the autonomous control of a motor vehicle defines scenarios that can be described as an abstraction of a traffic situation. Test cases can then be executed for each scenario.A logical scenario is the abstraction of a traffic situation, including the road, driving behavior, and surrounding traffic, without specifying specific parameter values. By selecting specific parameter values, the logical scenario becomes a concrete scenario. Such a concrete scenario corresponds to a specific traffic situation.

[0018] For the fundamental differentiation of traffic scenarios in scenario-based testing, not only static parameters such as, but not exclusively, the surroundings, buildings, or roadway width are used, but also, in particular, the driving behavior of individual road users. The movements of road users and thus the driving behavior are described by trajectories. Trajectories describe a path in both spatial and temporal directions. Parameters such as speed can be used to differentiate between the movements of road users.

[0019] An autonomous driving function is implemented by a system, such as a control unit. The control unit is typically tested in a real vehicle in real traffic situations, validated through hardware-in-the-loop tests, or alternatively, through fully virtual tests.

[0020] A simulation can also be used for this purpose.

[0021] Using this method, for example, a so-called cut-in scenario can be distinguished from other scenarios. A cut-in scenario can be described as a traffic situation in which a highly automated or autonomous vehicle is driving in a given lane and another vehicle, traveling at a lower speed than the ego vehicle, merges from another lane into the lane of the ego vehicle at a certain distance. The ego vehicle here refers to the vehicle under test (SUT).

[0022] The speeds of the ego vehicle and the other vehicle, also called the fellow vehicle, are constant. Since the speed of the ego vehicle is higher than that of the fellow, the ego vehicle must be decelerated to avoid a collision between the two vehicles.

[0023] But a cut-in scenario can also occur in various forms, such as a distinction in the speed of road users.

[0024] Furthermore, a passing-by scenario is possible, in which the ego vehicle represents the overtaking vehicle. The ego vehicle is traveling in a given lane and overtaking another vehicle at a lower speed than the ego vehicle. The ego vehicle then changes lanes and passes the fellow vehicle at a higher speed. The speed of the ego vehicle does not have to be constant in this scenario. After passing the fellow vehicle, the ego vehicle returns to the previous lane.

[0025] Such scenarios can be simulated with different parameter values. To maximize the use of simulation and computing time, test follow-up is required to achieve high coverage (homologation). This requires an appropriate evaluation of simulation results, and if necessary, of each simulation step.

[0026] In the method according to the invention, the term Key Performance Indicator (KPI) refers to KPI values that can be used to measure and / or determine the progress or degree of fulfillment of important objectives or critical success factors after or during a simulation of at least partially autonomous vehicles. KPIs and / or KPI values allow an evaluation of the simulation and / or the simulation step for test adjustment, allowing for more targeted, resource-efficient, and time-efficient testing.

[0027] For this purpose, KPIs are implemented as KPI plug-ins according to the invention. A KPI plug-in comprises an executable script or a KPI script and a configuration description. The executable script records how the KPI value is determined, and the configuration description specifies at least which input data the script requires, how it must be executed, and, if necessary, additional metadata that describes the scenarios in which the KPI plug-in is applicable. Metadata that indicates the applicability of the KPI plug-in in a scenario can, for example, be a condition for scenery parameters, such as a specific lane width, or driving situation parameters, such as the number of road users or speed restrictions. This allows a KPI plug-in to be reused in different scenarios and / or tests. It is not necessary to adapt the script to be executed and / or the configuration file for use in different scenarios.A KPI plug-in can thus be dynamically selected during test and simulation creation and / or simulation execution and can be integrated into tests and / or simulations and can thus be used in various simulations and / or tests in different scenarios. According to the invention, the selection can be made before and / or after test and / or simulation creation. The selection of a plug-in can vary depending on the embodiment of the invention. In a preferred embodiment, a selection of the KPI plug-in in a user interface is possible by a test / simulation creator. In further embodiments, a programmable integration of the KPI plug-in is also possible. Further embodiments are intended to be encompassed by the computer-implemented method according to the invention. The computer-implemented method according to the invention shortens execution times of a simulation, including simulation creation.

[0028] Another object of the invention is to automate the execution of at least one selected KPI plug-in. For this purpose, a KPI plug-in mechanism is set up. Upon test definition and / or simulation creation, the KPI plug-in mechanism receives the KPI plug-in. The KPI plug-in mechanism then holds the KPI plug-in and executes it at the designated location. Executing the KPI plug-in provides the KPI value, which the KPI plug-in mechanism saves with the test and / or simulation results after execution.

[0029] The KPI plug-in mechanism is designed to execute KPI plug-ins in two different ways. The execution variant is specified in the configuration file.

[0030] A first execution variant takes place online and thus simultaneously with the simulation. In this case, an evaluation takes place in each simulation step, and the KPI value is available immediately after the simulation. The evaluation is based on the simulation parameters available during execution, such as scenery parameters and / or driving situation parameters.

[0031] The second execution variant is downstream of the current simulation, i.e., offline. This means that parameters are not observed in real time during the simulation and used to determine the KPI value; instead, the KPI value is determined after the simulation. Simulation results are passed to the KPI script to determine the KPI value. In principle, the KPI plug-in mechanism can contain at least one KPI plug-in, but also multiple KPI plug-ins. All KPI plug-ins registered with the KPI plug-in mechanism for a test execution and / or simulation are executed.

[0032] For better reusability, all KPI plug-ins are managed in a KPI plug-in pool.

[0033] In contrast to existing methods for integrating assessment into scenario-based testing, the KPI plug-in mechanism manages the KPI plug-ins and their execution. This means that the KPI plug-ins are not rigidly linked to the simulation and do not need to be redefined for each scenario or SUT. With existing methods, linking assessment and scenario is necessary. In addition to reusability, the automatic execution of KPI plug-ins is a key feature of the method according to the invention. At runtime, the KPI plug-in mechanism automatically loads, checks, and executes the selected KPI plug-ins.

[0034] The KPI plug-in mechanism requires a standardized KPI configuration and standardized interfaces for KPI scripts. These standards allow KPI development to be treated separately from scenario development.

[0035] Further embodiments of the present invention are the subject of the further subclaims and the following description with reference to the figures.

[0036] The test unit includes means for evaluating simulations and / or test cases in scenario-based testing and / or homologation of at least partially autonomous driving functions to be tested through Key Performance Indicators (KPIs).

[0037] According to a further aspect of the invention, a computer program with program code is provided for carrying out the method according to the invention when the computer program is executed on a computer. According to a further aspect of the invention, a data carrier with program code of a computer program is provided for carrying out the method according to the invention when the computer program is executed on a computer.

[0038] The features of the computer-implemented method described herein can be used to evaluate simulations and / or test cases in scenario-based testing and / or homologation of at least partially autonomous driving functions to be tested using key performance indicators (KPIs). The test unit according to the invention is also suitable for performing test adjustments of a variety of different devices or control units of, for example, automobiles, utility vehicles and / or commercial vehicles, ships, or aircraft by evaluating simulation results and / or simulation steps of different scenarios or driving situations. Short description of the drawings

[0039] For a better understanding of the present invention and its advantages, reference is now made to the following description in conjunction with the accompanying drawings. The invention is explained in more detail below using exemplary embodiments shown in the schematic illustrations of the drawings.

[0040] They show: Figure 1 shows a schematic representation for the inventive differentiation of scenarios. Figure 2 shows a schematic view indicating a boundary between critical and non-critical test results. Figure 3 shows an inventive representation of a KPI plug-in. Figure 4 shows a schematic representation for the inventive description of the KPI plug-in mechanism and the KPI plug-in pool. Figure 5 shows a schematic representation for the inventive description of the use of KPI plug-ins. Figure 6 shows a schematic representation for the inventive description of the use of KPI plug-ins. Figure 7 shows a schematic representation for the inventive determination of a KPI value. Figure 8 shows the process of the inventive offline use of the KPI plug-in mechanism. Figure 9 shows the process of the inventive online use of the KPI plug-in mechanism. Detailed description of the embodiments

[0041] Figure 1Describes two different scenarios S1 and S2. Each depicts an intersection area. In both scenarios S1 and S2, an ego vehicle (Ego) is depicted. In S1, the ego vehicle (Ego) performs a turning maneuver. The ego vehicle is also the "Subject Under Test" (SUT). Four fellow vehicles (F1 to F4) are involved. In S2, the ego vehicle follows the driving path straight ahead without the involvement of fellow vehicles. Therefore, there are differences in the scenery parameters as well as driving situation parameters. The goal of the scenarios can be, for example, the testing and simulation of an adaptive cruise control system. During the test definition, the KPI plug-in mechanism receives the KPI plug-in for evaluating the adaptive cruise control system, consisting of the script and the configuration description. During and / or after the simulation, the defined KPI script is executed, and the resulting KPI value is saved.It is possible to use a consistent KPI plug-in for both scenarios.

[0042] In another version, adaptive cruise control can be tested in a self-driving vehicle (ego) in scenarios on the highway and / or in the city center. A cut-in scenario may be of particular interest. The scenarios differ, particularly in the scenery parameters.

[0043] First, the highway scenario is selected for the test configuration. In this scenario, it is relevant for the ego vehicle (ego) whether a collision has occurred. In addition, the impact speed should be used to determine the severity of the collision. If no collision occurs, the minimum distance between the merging vehicle and the ego vehicle (ego) with the adaptive cruise control is of interest. A KPI plug-in with specific value ranges can be defined for these specifications. The actual implementation of the KPI plug-in takes place in a KPI script. A configuration description is created for the script, which describes the evaluation and lists the required simulation information. This KPI plug-in can be implemented as an online or offline KPI plug-in. During the test definition, the KPI plug-in mechanism receives the KPI plug-in, consisting of the script and the configuration description.During and / or after the simulation, the defined KPI script is executed and the resulting KPI value is saved. After the test execution of the highway scenario, the ego vehicle (Ego) can also be tested in a second scenario, e.g. in the city center. In this case, the simulation information required for the KPI plug-in is available, so that this KPI plug-in can be selected again when defining the test. This reusability is provided by the use of the KPI plug-in mechanism. In addition to the described KPI plug-in for testing adaptive cruise control, other KPI plug-ins can be used, for example to evaluate the comfort or fuel consumption of the vehicle during the test drive and / or simulation.

[0044] Figure 2shows a function that defines a boundary between critical and non-critical test results. The points shown are simulated test results. Alternatively, they may also be approximated test results.

[0045] The function shown is the safety objective function, which in a preferred embodiment has a numerical value that has a minimum value at a safety distance between the ego vehicle (ego) and the other motor vehicle, the fellow vehicle, of ≥ VFELLOW x 0.55, a maximum value in the event of a collision between the ego vehicle (ego) and the other motor vehicle, and a numerical value that is greater than the minimum value at a safety distance between the motor vehicle and the other motor vehicle of ≤ VFELLOW x 0.55. Such a safety objective function can be part of a KPI script and used to determine a KPI value.

[0046] As an alternative to the safety objective function, a comfort objective function or an efficiency objective function can be simulated and / or approximated, for example. This objective function has a numerical value that has a minimum value in the event of no change in the acceleration of the motor vehicle, a maximum value in the event of a collision between the ego vehicle (ego) and the other motor vehicle, and a numerical value between the minimum value and the maximum value in the event of a change in the acceleration of the ego vehicle (ego), depending on the magnitude of the change in acceleration. The majority of driving situation parameters, in particular the speed VEGO of the ego vehicle (ego) and the speed VFELLOW of the other motor vehicle, the fellow vehicle, are generated within the specified definition range, e.g., by simulation.

[0047] For evaluation, such objective functions can be integrated into a KPI plug-in and selected in the test and / or simulation definition. The KPI plug-in can be executed directly in the simulation, allowing a KPI value to be determined.

[0048] Figure 3 shows an inventive representation of a KPI plug-in (KPI-PI). KPI plug-ins (KPI-PI) have an interface to the KPI plug-in mechanism. Furthermore, a KPI plug-in (KPI-PI) consists of a configuration file (C) and an executable script (S).

[0049] The configuration file (C) provides information on how the KPI plug-in is used, i.e., in which type of scenarios an evaluation is possible. This information can include, for example, scenery parameters and / or driving situation parameters, such as the road width in the scenario and / or the number of road users and / or speed restrictions. This list is not exhaustive; it merely serves to clarify the type of information. The configuration file (C) also specifies whether the KPI plug-in (KPI-PI) can be used online, i.e., during the simulation, or offline, after the simulation.

[0050] The executable script (S) contains the information for determining a KPI value that is used to evaluate the simulation and / or a simulation step.

[0051] Figure 4shows a schematic representation for the inventive description of the KPI plug-in mechanism (KPI-PIM) and the KPI plug-in pool (KPI-PIP).

[0052] The KPI Plug-In Pool (KPI-PIP) collects and manages all available KPI plug-ins (KPI-PIs). This includes both currently used KPI plug-ins (KPI-PIs) and unused KPI plug-ins (KPI-PIs). All known KPI plug-ins (KPI-PIs) can be viewed via the KPI Plug-In Pool (KPI-PIP).

[0053] The KPI plug-in mechanism (KPI-PIM) controls the automatic execution of the selected KPI plug-ins (KPI-PIs). In contrast to common methods of integrating an assessment into scenario-based testing and thus, for example, linking it to the scenario itself, the KPI plug-in mechanism (KPI-PIM) manages the KPI plug-ins (KPI-PIs) and their execution. This means that the KPIs are not rigidly linked to the simulation and do not need to be redefined for each scenario or SUT. Instead, they are managed as KPI plug-ins (KPI-PIs), administered in the KPI plug-in pool (KPI-PIP), and executed by the KPI plug-in mechanism (KPI-PIM). This ensures the reusability of the KPI plug-ins (KPI-PI), and the automatic execution of the KPI plug-ins (KPI-PI) represents a significant saving in time and resources. At runtime, the KPI plug-in mechanism (KPI-PIM) automatically loads, checks, and executes the selected KPI plug-ins.

[0054] Figure 5 shows a schematic representation for the inventive description of the use of KPI plug-ins (KPI-PI) in a concrete scenario and thus in a defined simulation / a defined test.

[0055] For this purpose, Figure 5 The KPI plug-in mechanism (KPI-PIM) is shown with two KPI plug-ins (KPI-PI), KPI-P-I1 and KPI-P-I2. The KPI plug-in mechanism (KPI-PIM) is shown in conjunction with a test T1 in a scenario S1.

[0056] The KPI plug-ins (KPI-PI) KPI-P-I1 and KPI-P-I2 are not directly linked to the scenario. Their generic and dynamic definition allows KPI plug-ins (KPI-PI) to be reused for different scenarios and tests. Therefore, a new KPI plug-in mechanism (KPI-PIM) or KPI plug-in (KPI-PI) is not created for each new scenario.

[0057] Also Figure 6shows a schematic representation for the inventive description of the use of KPI plug-ins (KPI-PI) in a concrete scenario and thus in a defined simulation / test.

[0058] Here, the KPI plug-in mechanism (KPI-PIM) is shown in conjunction with a test T2 in a scenario S2. In Figure 6 The KPI plug-in mechanism (KPI-PIM) with the KPI plug-ins (KPI-PI) KPI-P-I1 and KPI-P-I3 is shown. The KPI plug-ins (KPI-PI) are not directly linked to the scenario or the test. This illustrates that the KPI plug-in (KPI-PI) KPI-P-I1 is used both in S1 and through Figure 5 shown, and in S2, as in Figure 6 shown. This reuse occurs without adapting the corresponding KPI plug-in and the associated KPI plug-in script.

[0059] Figure 7shows a schematic representation for determining a KPI value according to the invention. If the occurrence of a collision (VC) is relevant for a scenario and for a specific SUT therein, a KPI plug-in (KPI-PI) must be selected for this purpose, such as one that evaluates the safety of the driving situation (Safety KPI). The KPI plug-in (KPI-PI) can, in particular, determine an impact speed (IV) in the event of a collision (VC) or, should no collision (VC) occur, specify a minimum distance (Min D) between the ego vehicle (Ego) and a fellow vehicle. These results are particularly relevant for evaluating the scenario in a cut-in scenario, in which, for example, an adaptive cruise control system is to be tested. The determined KPI values can be used as a basis for re-parameterizing the simulation.

[0060] Figure 8 shows the process of the inventive offline use of the KPI plug-in mechanism (KPI-PIM).

[0061] The KPIs in the form of KPI plug-ins (KPI-PIs) can be executed online or offline using the KPI plug-in mechanism (KPI-PIM). In the offline case, the KPI execution, i.e., the execution of KPI plug-ins (KPI-PIs), can be performed directly after the simulation using the KPI plug-in mechanism (KPI-PIM). KPI execution at a later time is also possible.

[0062] First, a configuration (CJ) of jobs or test cases is created. This configuration is used in the test execution (EJ) and serves as input for the simulation (Sim). The simulation (Sim) generates test results (TR), which are made available to the KPI plug-in mechanism (KPI-PIM) and used when executing the selected KPI plug-ins (KPI-PI).

[0063] In a favorable implementation, results from the KPI plug-in mechanism (KPI-PIM) in the form of KPI values can be passed back to the test configuration (CJ) and lead to optimized test creation. One possibility is to use the KPI value for better parameter configuration, thus achieving increased test coverage.

[0064] Figure 9 shows the process of the inventive online use of the KPI plug-in mechanism (KPI-PIM).

[0065] For this purpose, in a preferred embodiment, information from the simulation (Sim) is transferred to the KPI plug-in mechanism (KPI-PIM) after each simulation step. The KPI plug-in mechanism (KPI-PIM) executes the selected KPI plug-ins (KPI-PI) and transfers the KPI value directly back to the simulation (Sim). Results (TR), including a KPI value, are transmitted together to the test creation and configuration (CJ) and can preferably be used to generate optimized test cases and / or an optimized parameter configuration.

[0066] The KPI plug-in (KPI-PI) can also be used as a termination condition for the simulation (Sim). In another preferred embodiment, if a KPI value generated by the KPI plug-in mechanism (KPI-PIM) exceeds a threshold defined in the configuration, the simulation can be terminated by transmitting the KPI value in the next simulation step. The threshold can be contained in the test definition and / or in the configuration file (C) of the KPI plug-in (KPI-PI).

Claims

1. A computer-implemented method for evaluating simulations and / or test cases in scenario-based testing and / or homologation of at least partially autonomous driving functions to be tested by means of key performance indicators (KPI), comprising the steps: mapping KPIs using KPI plug-ins; dynamically and reusably selecting KPI plug-ins for simulations and / or test cases; selecting at least one KPI plug-in by a KPI plug-in mechanism in the simulation and / or test definition; automatically executing the selected KPI plug-in by the KPI plug-in mechanism during the execution of the KPI plug-in mechanism; determining a KPI value by passing a simulation result to a KPI script of the KPI plug-in; transmitting the simulation result and the KPI value to a test creation and configuration and generating optimized test cases and / or an optimized parameter configuration; and providing an at least partially autonomous driving function optimized using the optimized test cases and / or the optimized parameter configuration.

2. The computer-implemented method according to claim 1, characterized in that at least one scenario comprises at least one test of a device for at least partially autonomously guiding a means of transport and / or road user.

3. The computer-implemented method according to claim 2, characterized in that at least one test is determined by at least one parameter interrogated at simulation runtime and / or at the end of the simulation, wherein parameters include in particular: - scenery parameters comprising at least one of the features: the number of and / or the width of a lane and / or curves and / or road restrictions and / or the ambient temperature, - driving situation parameters describing the number and properties of moving objects in the scenario, comprising at least one of the features: the number of road users and / or the number of lane changes in a traffic situation and / or the speed of the road users and / or means of transport.

4. The computer-implemented method according to claim 2, characterized in that means of transport and / or road users comprise at least ego vehicles and / or fellow vehicles, wherein an ego vehicle is the vehicle having the system under test (SUT) and a fellow vehicle is any other vehicle in the scenario.

5. The computer-implemented method according to claim 1, characterized in that a KPI plug-in consists of a configuration description and a script executable by a computer.

6. The computer-implemented method according to any one of the preceding claims, characterized in that online KPI plug-ins can be executed at simulation runtime and / or offline after the simulation.

7. The computer-implemented method according to claim 6, characterized in that for KPI plug-ins executed online, the executable script is started automatically at simulation runtime and an evaluation of at least one parameter takes place in each simulation step, so that the evaluation is based on simulation parameters available at runtime and a KPI value is determined therefrom.

8. The computer-implemented method according to claim 6, characterized in that for KPI plug-ins executed offline, the executable script is started automatically after the end of the simulation, so that the KPI value is determined on the basis of the simulation results.

9. The computer-implemented method according to any one of the preceding claims, characterized in that the currently determined KPI value indicates an evaluation of the current simulation and / or of at least one simulation step.

10. The computer-implemented method according to any one of the preceding claims, characterized in that the KPI plug-in mechanism includes at least one KPI plug-in and all KPI plug-ins selected in the KPI plug-in mechanism for simulation and / or test execution are automatically executed.

11. The computer-implemented method according to any one of the preceding claims, characterized in that KPI plug-ins are managed in a KPI plug-in pool.

12. A test unit for evaluating simulations and / or test cases in scenario-based testing and / or homologation of at least partially autonomous driving functions to be tested by key performance indicators (KPI), comprising: means for mapping KPIs using KPI plug-ins; dynamically and reusably selecting KPI plug-ins for simulations and / or test cases; means for selecting at least one KPI plug-in by a KPI plug-in mechanism in the simulation and / or test definition; means for automatically executing the selected KPI plug-in by the KPI plug-in mechanism during the execution of the KPI plug-in mechanism; means for determining a KPI value by passing a simulation result to a KPI script of the KPI plug-in; means for transmitting the simulation result and the KPI value to a test creation and configuration and generating optimized test cases and / or an optimized parameter configuration; and means for providing an at least partially autonomous driving function optimized using the optimized test cases and / or the optimized parameter configuration.

13. The test unit according to claim 12, characterized in that the test unit is implemented by a control device for which at least one scenario for virtual and / or real tests is determined.

14. A computer program having program code for performing the method according to any one of the claims 1 through 11 when the computer program is executed on a computer.

15. A computer-readable data storage medium having program code of a computer program for performing the method according to any one of the claims 1 through 11 when the computer program is executed on a computer.