Method, system and computer program product for the objective evaluation of the performance of an ADAS / ADS system

The method and system provide a framework for objectively evaluating ADAS/ADS systems by generating scenario-based evaluation indicators, enabling precise comparison and improvement of their performance and functionality across different designs.

DE102022119715B4Active Publication Date: 2025-01-09DR ING H C F PORSCHE AG

Patent Information

Application Number
DE102022119715
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-08-05
Publication Date
2025-01-09
Estimated Expiration
2042-08-05

AI Technical Summary

Technical Problem

Existing ADAS/ADS systems lack a uniform metric for evaluating performance, leading to inaccurate comparisons and limited objective assessment due to differences in components, software versions, and application data sets, making it difficult to assess their capability and performance objectively.

Method used

A method and system that generates evaluation indicators for real and simulated scenarios using scenario parameters, employing artificial intelligence algorithms and calculation methods to create objective metrics for comparing the performance of differently designed ADAS/ADS systems.

Benefits of technology

Enables objective, precise, and comprehensive evaluation of ADAS/ADS systems by comparing evaluation indicators across various scenarios, allowing for improved assessment of safety and functionality, and facilitating targeted improvements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Method for objectively assessing the performance of an ADAS / ADS system (210) of a vehicle (200) for a defined driving task in at least one selected scenario (SZ) i ), in particular for testing and training at least one driving function of the Advanced Driver Assistance System (ADAS) and / or the Automated Driving System (ADS), wherein a scenario (SZ i ) represents a traffic event in a temporal sequence and is defined by a selection of scenario parameters (P1, P2, ..., P n ) and associated scenario parameter values ​​(PV1, PV2, ..., PV n ) is defined as encompassing: - Identifying (S10) real-world scenarios (SZr) i ) from real-time sensor data (220) while driving the vehicle on a test track (200) and / or from stored data from a scenario identification module (300); - Generating (S20) simulated scenarios (SZs i) from a simulation module (400); - Transmitting (S30) the real scenarios (SZr i ) and the simulated scenarios (SZs i ) to an evaluation module (500); - Calculate (S40) an evaluation indicator (570) for at least one real scenario (SZr) i ) and an evaluation indicator (570) for at least one simulated scenario (SZs i ) from the evaluation module (500), wherein the evaluation indicator (570) represents the performance of the ADAS / ADS system (210) for the specified driving task, and wherein the evaluation indicators (570) are configured as key performance indicators (KPIs); generating (S50) evaluation results (750) from an evaluation module (700), wherein an evaluation result (750) for a real-world scenario (SZr) i ) and a simulated scenario (SZs i ) each an assignment between the respective determined assessment indicator (570) and the scenario parameter value (PVc) i) of a selected scenario parameter (Pc i ) of the respective real-world scenario (SZr i ) or of the respective simulated scenario (SZs i ) includes, wherein the evaluation results (750) are presented as KPI diagrams, in which the values ​​of a KPI are shown as a function of the parameter values ​​(PV) in a KPI diagram i ) of a scenario parameter (P1) for a specific real-world scenario (SZr) i ) or a simulated scenario (SZs i ) are represented, and the KPI diagrams are designed as histograms with segments for cluster analysis.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method, a system and a computer program product for the objective evaluation of the performance of an ADAS / ADS system.

[0002] Modern vehicles are equipped with a variety of driver assistance systems and automated driving assistance functions to support the driver and increase safety. These systems support features such as speed and distance control, including lane keeping and lane change functions. A specific maximum speed can be set, which will not be exceeded as long as the speed limiter function is activated. For distance control, which maintains a specific distance, particularly to a vehicle ahead, radar sensors and camera systems are used. This allows the distance to vehicles ahead, as well as to vehicles to the side, to be monitored. This results in improved driving comfort and greater safety, especially when driving on highways and during overtaking maneuvers.

[0003] This trend towards advanced driver assistance systems (ADAS) and automated driving systems (ADS) in motor vehicles, but also in aircraft, watercraft, and other moving objects, necessitates comprehensive safety strategies. This is because the responsibility for vehicle control no longer rests solely with the driver, but rather is taken over by computer units within the vehicle. Therefore, it is essential to ensure that fully or partially autonomous systems exhibit a significantly lower error rate in their driving behavior. To guarantee the safe operation of an ADAS / ADS system, the detection and classification of other objects and the interpretation of traffic scenarios in the vicinity of a moving object, especially a vehicle, are crucial prerequisites.This requires targeted testing and training of driver assistance systems and automated driving systems in both extreme and exceptional situations (corner cases) as well as in everyday situations. Such extreme situations arise from a particular combination of various factors. Examples include infrastructural features such as the type of road, the buildings along the roadside, the quality of the markings, and environmental conditions such as weather conditions, time of day, and season. Furthermore, the behavior of other road users, the geographical topography, and weather conditions play a significant role.

[0004] To evaluate the performance and functionality of one or more functions, or the overall performance of an ADAS / ADS system, key performance indicators (KPIs) are used. KPIs are primarily numerical values, for example, on a scale of 1 to 100. KPIs serve to describe the performance of an ADAS / ADS system, and different KPIs can be defined for various evaluation categories such as comfort, safety, driving dynamics, and efficiency. Additional KPIs can also be implemented to verify the correct functionality of an ADAS / ADS system. An example of a KPI is the evaluation of a minimum distance to another vehicle or average acceleration during a deceleration scenario.

[0005] Since ADAS / ADS systems from different manufacturers are typically developed independently, they differ, for example, in the components used, the software versions, and the application datasets. The calculated KPIs are therefore context-dependent, as they depend on both the performance of a specific ADAS / ADS system and the scenario under consideration. Consequently, KPIs for differently configured ADAS / ADS systems can only be compared to a limited extent, since it is not always clear whether the ADAS / ADS systems were tested with the same scenarios and from which data sources the scenarios were generated. The significance of a KPI value therefore often only relates to a specific ADAS / ADS system, making the comparison of KPIs for differently configured ADAS / ADS systems difficult and leading to inaccurate results.This means that the capability and performance of an ADAS / ADS system often cannot be objectively assessed, as there is a lack of a uniform metric for evaluation.

[0006] DE10 2019 209554 A1 discloses a method for testing the robustness of a simulable technical system, wherein the technical system is simulated depending on a test scenario and a test configuration and the simulation result is evaluated.

[0007] DE 10 2017 221971 A1 discloses a method for adapting a vehicle control system to a changed vehicle state, wherein the control performance of the vehicle control system is assessed taking into account at least one control criterion. The control criterion is determined for a specific driving scenario by means of a simulation.

[0008] German patent DE 10 2020 120141A1 discloses a method for optimizing tests of control systems for automated vehicle dynamics systems, in which relevant parameters and relevant system responses are defined and quasi-random parameter combinations of the relevant parameters and system responses of the test system are generated. A probabilistic system model is created and trained depending on the generated parameter combinations and the system responses, whereby the system responses generated by the system model are evaluated.

[0009] DE 10 2018 215351 A1 discloses a method for generating a collection of driving scenario information, wherein the driving scenario data is generated by sensor data from at least one vehicle.

[0010] WO 2021 / 189011 A1 discloses the generation of scenarios using a declarative description for testing an autonomous or semi-autonomous machine in a virtual environment. The system is capable of calculating the coverage of various aspects based on the generated virtual scenarios, thus providing a comprehensive framework for generating scenarios in virtual environments, testing and validating the scenarios themselves, and / or testing and validating the underlying autonomous or semi-autonomous systems of the machine, all based on a declarative description.

[0011] German patent DE 11 2020 002 166 T5 discloses the use of real-world sensor data to generate transformed sensor data for testing one or more vehicle functions. The sensor data recorded by the sensors can be augmented, transformed, or otherwise updated to generate a test dataset of sensor data containing condition information defined by a simulation test profile. This profile includes additional or alternative instances of sensor data to test a vehicle in a variety of test scenarios, including those that could be dangerous to test under real-world conditions.

[0012] The object underlying the invention is to provide possibilities for the analysis and comparison of key performance indicators (KPIs) for different test drives and differently designed driver assistance systems (ADAS) and / or automated driving systems (ADS), which differ from each other in particular with regard to the software versions and application data sets used, in order to create the conditions for an objective comparability of the performance and capabilities of differently designed ADAS / ADS systems and / or driving functions.

[0013] This problem is solved according to the invention with respect to a method by the features of claim 1, with respect to a system by the features of claim 7, with respect to an ADAS / ADS system by the features of claim 9, and with respect to a computer program product by the features of claim 10. The further claims relate to preferred embodiments of the invention.

[0014] The present invention enables the objective evaluation and comparison of test results from differently configured ADAS / ADS systems by generating evaluation indicators for scenarios. These scenarios include real-world scenarios measured in real time or created from historical data, as well as simulated scenarios generated by appropriate simulation software. According to the invention, for each scenario under consideration, a calculated evaluation indicator is related to a scenario parameter value of a selected scenario parameter. This results in an objective metric for a calculated evaluation indicator, allowing for the comparison of evaluation indicators from different ADAS / ADS systems.In a further development, it is possible to compare the simulated scenarios generated by simulation with those created from measurements, from which the quality of a simulation method can be derived. The present invention thus enables the objective, precise, and reproducible evaluation of the performance and capabilities of an ADAS / ADS system, allowing for a comparison of differently configured ADAS / ADS systems and the development of potential improvement measures. Overall, this significantly improves the assessment of the safety and functionality of one or more driving functions of an ADAS / ADS system for a specific driving task.

[0015] According to a first aspect, the invention provides a method for objectively evaluating the performance of a vehicle's ADAS / ADS system for a defined driving task in at least one selected scenario, in particular for testing and training at least one driving function of the driver assistance system (ADAS) and / or the automated driving system (ADS). A scenario represents a traffic event in a temporal sequence and is defined by a selection of scenario parameters and associated scenario parameter values. The method comprises the following steps: - Identifying specific real-world scenarios from real-time sensor data recorded while driving the vehicle on a test track or from stored data from a scenario identification module; - Generating simulated scenarios from a simulation module; - Transmitting the real scenarios and the simulated scenarios to an evaluation module; - Calculating an evaluation indicator for at least one real scenario and an evaluation indicator for at least one simulated scenario from the evaluation module, wherein the evaluation indicator represents the performance of the ADAS / ADS system for the specified driving task, and wherein the evaluation indicators are designed as key performance indicators (KPIs); - Generating evaluation results from an evaluation module, wherein an evaluation result for a real scenario and a simulated scenario each includes an assignment between the respective determined evaluation indicator and the scenario parameter value of a selected scenario parameter of the respective real scenario or the respective simulated scenario, wherein the evaluation results are designed as KPI diagrams, wherein in a KPI diagram the values ​​of a KPI are displayed as a function of the parameter values ​​of a scenario parameter for a specific real scenario or a simulated scenario, and wherein the KPI diagrams are designed as histograms with segments for cluster analysis.

[0016] In a further development, it is planned that the scenario parameter value of the selected scenario parameter of the respective real scenario and / or the respective simulated scenario will be extracted by the evaluation module from the data set of the real scenario and / or the simulated scenario.

[0017] In particular, the KPI diagrams for differently trained ADAS / ADS systems and / or the KPI diagrams for different real-world scenarios and / or for simulated scenarios and real-world scenarios for an ADAS / ADS system are compared.

[0018] In a training course, it is stipulated that the scenario identification module includes a software application that uses calculation methods and / or algorithms of artificial intelligence; that the simulation module includes a software application that uses calculation methods and / or algorithms of artificial intelligence; that the evaluation module includes a software application that uses calculation methods and / or algorithms of artificial intelligence; and that the analysis module includes a software application that uses calculation methods and / or algorithms of artificial intelligence.

[0019] In particular, it is intended that the calculation methods and / or algorithms of the artificial intelligence are designed as means, minimum and maximum values, lookup tables, models of expected values, linear regression methods, Gaussian processes, fast Fourier transforms, integral and differential calculus, Markov methods, probability methods such as Monte Carlo methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, feedback neural networks, and / or convolutional neural networks.

[0020] Advantageously, a scenario parameter represents a physical quantity, a chemical quantity, a torque, a rotational speed, a voltage, a current, a velocity, an acceleration, a jerk, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a road sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographical structure such as a gradient, a time of day, a temperature, a precipitation value, a weather condition and / or a season.

[0021] According to a second aspect, the invention provides a system for objectively evaluating the performance of a vehicle's ADAS / ADS system for a defined driving task in at least one selected scenario, particularly for testing and training at least one driving function of the driver assistance system (ADAS) and / or the automated driving system (ADS). A scenario represents a traffic event in a temporal sequence and is defined by a selection of scenario parameters and their associated values. The system comprises sensors connected to the vehicle, a scenario identification module, a simulation module, an evaluation module, and a data analysis module. The sensors are configured to record data in real time while the vehicle is driving along a test track.The scenario identification module is designed to identify real-world scenarios from real-time sensor data or stored data. The simulation module is designed to generate simulated scenarios. The evaluation module is designed to calculate an evaluation indicator for at least one real-world scenario and an evaluation indicator for at least one simulated scenario, where the evaluation indicator represents the performance of the ADAS / ADS system for the defined driving task, and where the evaluation indicators are configured as key performance indicators (KPIs). The evaluation module is designed to generate evaluation results, where each evaluation result for a real-world scenario and a simulated scenario represents a mapping between the respective evaluation indicator and the scenario parameter value of a selected scenario parameter of the respective real-world scenario.of the respective simulated scenario. The evaluation results are presented as KPI diagrams, where a KPI diagram displays the values ​​of a KPI as a function of the parameter values ​​of a scenario parameter for a specific real-world or simulated scenario, and where the KPI diagrams are designed as histograms with segments for cluster analysis.

[0022] In particular, it is planned that the KPI diagrams for differently trained ADAS / ADS systems and / or the KPI diagrams for different real-world scenarios and / or for simulated scenarios and real-world scenarios for an ADAS / ADS system will be compared with each other.

[0023] According to a third aspect, the invention provides an ADAS / ADS system for a vehicle which uses the method according to the first aspect for testing and training and / or for calibrating and validating the ADAS / ADS system.

[0024] According to a fourth aspect, the invention provides a computer program product comprising an executable program code configured to perform the method according to the first aspect when executed.

[0025] The invention will now be explained in more detail with reference to exemplary embodiments shown in the drawing.

[0026] This shows: Fig. 1 a block diagram to illustrate an embodiment of a system according to the invention; Fig. 2 a schematic representation of a KPI diagram; Fig. 3 a schematic representation of a KPI diagram for a cut-in scenario; Fig. 4 a flowchart to explain the individual process steps of a process according to the invention; Fig. 5 a block diagram of a computer program product according to an embodiment of the third aspect of the invention.

[0027] Additional features, aspects and advantages of the invention or its embodiments are explained in the following description in conjunction with the claims.

[0028] For testing, training, and validating advanced driver assistance systems (ADAS) and automated driving systems (ADS), real and increasingly also simulated traffic scenarios created through programming are used. Within the scope of the invention, a scenario is defined as a traffic event in a temporal sequence. Examples of scenarios include driving over a highway bridge, turning in a turning lane, driving through a tunnel, entering a roundabout, or stopping at a pedestrian crossing. Furthermore, specific visibility conditions, for example, due to twilight or high sunlight, as well as environmental conditions such as weather and season, traffic volume, and specific geographical topographical features can influence a scenario.For example, an overtaking maneuver can be described as a scenario in which a first vehicle is initially behind another vehicle, then changes lanes to the other lane and increases its speed to overtake the other vehicle. Such a scenario is also known as a cut-in scenario.

[0029] Fig. Figure 1 shows a system 100 according to the invention for the objective evaluation of the performance of an advanced driver assistance system (ADAS) and / or an automated driving system (ADS). The system 100 according to the invention comprises a vehicle 200 with an ADAS / ADS system 210 having one or more driving functions. The vehicle 100 is connected to sensors 220 that record data in the form of measurement data and images of the vehicle 200's surroundings and the behavior of the ADAS / ADS system 210 along a test track. The sensor data 220 can be transmitted directly to a scenario identification module 300 or stored in a database 240. The database 240 is also connected to the scenario identification module 300 for transmitting stored data. Furthermore, a simulation module 400 is provided, which can, for example, be connected to a test scenario database 410 for creating simulated scenarios.Furthermore, an input module 250 is provided, which is connected to the scenario identification module 300 and the simulation module 400. The input module 250 has a user interface, in particular in the form of a touchscreen, a microphone, a camera, etc. The real-world scenarios identified by the scenario identification module 300 SZr. i and the scenarios simulated by the simulation module 400 SZs i The data is passed on to an evaluation module 500. The evaluation results generated by the evaluation module 500 are evaluated by an evaluation module 700. The input module 250, the scenario identification module 300, the simulation module 400, the evaluation module 500, and the evaluation module 700 can each be equipped with a processor and / or a memory unit.

[0030] In the context of the invention, a "module" can be understood to mean, for example, a processor and / or a processor unit and / or a memory unit for storing program instructions. The processor is specifically configured to execute the program instructions in such a way as to implement or realize the method according to the invention or a step thereof. In particular, a module can be integrated into a cloud computing infrastructure 800.

[0031] In the context of the invention, a "processor" can be understood to mean, for example, a machine or an electronic circuit. In particular, a processor can be a central processing unit (CPU), a microprocessor, or a microcontroller, such as an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions, etc. A processor can also be understood to be a virtualized processor, a virtual machine, or a soft CPU.It can, for example, also be a programmable processor that is equipped with configuration steps for executing the aforementioned method according to the invention, or is configured with configuration steps such that the programmable processor realizes the features of the method, the system, the modules, or other aspects and / or partial aspects of the invention according to the invention. In particular, the processor can contain highly parallel computing units and powerful graphics modules.

[0032] In the context of the invention, a "storage unit" or "storage module" can refer, for example, to volatile memory in the form of random-access memory (RAM), persistent storage such as a hard drive or data carrier, or, for example, a removable storage module. The storage module can also be a cloud-based storage solution.

[0033] The term "database" refers to both a storage algorithm and the hardware in the form of a storage unit. In particular, a database can be part of a cloud computing infrastructure.

[0034] In the context of the invention, "data" includes both raw data and processed data from sensor measurement results and other data sources.

[0035] The objective evaluation of the performance of an ADS / ADS system 210 and / or a driving function is carried out according to the invention on the basis of real scenarios SZr i , which occur during the testing and trialing of the ADAS / ADS system 210 along a driving route, and / or simulated scenarios SZs i . Both a real scenario SZr i as well as a simulated scenario SZs i is determined by various scenario parameters P1, P2, ..., P n from a set of possible scenario parameters Pi and associated scenario parameter values ​​PV1, PV2, ..., PV n from a set of possible scenario parameter values ​​PV i defined, where scenario parameter values ​​PV i the range of values ​​of a scenario parameter P i define a scenario parameter P iFor example, it represents a physical quantity, a chemical quantity, a torque, a rotational speed, a voltage, a current, an acceleration, a velocity, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person, or a cyclist, a stationary object such as a building or a tree, a road configuration such as a highway, a road sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographical feature such as a gradient, a time of day, a temperature, a precipitation value, weather conditions, and / or a season. Scenario parameter P i Thus, within the scope of the present invention, they characterize properties and features of a scenario SZ. i An example of a scenario parameter P i of a scenario SZ iThis is the speed of an ego vehicle. For this scenario parameter "speed", the value range of the associated scenario parameter value PV can be set. i The range from 100 km / h to 180 km / h. For another scenario, SZ k However, the range of values ​​for the parameter value PV can change. i for the scenario parameter “speed” from 40 km / h to 70 km / h.

[0036] The real-world scenarios SZr, based on actual tests and measurements i, which arise or occur when the vehicle 200 equipped with the ADAS / ADS system 210 is driven along a test track, are identified by the scenario identification module 300 using the data recorded in real time by the sensors 220 and / or using data stored in the database 240 and, if necessary, extended by additional requirement specifications, which can be entered in particular by means of the input module 250, by means of a software application 320.

[0037] The sensors 220 can be configured in particular as radar systems with one or more radar sensors, LiDAR systems for optical distance and speed measurement, 2D / 3D image-capturing cameras in the visible, IR and / or UV range, and / or GPS systems. Furthermore, the sensors 220 can be configured as accelerometers, velocity sensors, capacitive sensors, inductive sensors, voltage sensors, torque sensors, speed sensors, precipitation sensors and / or temperature sensors, etc.

[0038] Database 240 can store data in various forms, including images, graphs, time series, key performance indicators, and more. For example, target values ​​and parameters defining safety standards can be stored in Database 240. Furthermore, it can contain data on the road network, including road specifications such as lanes and bridges, road infrastructure such as pavement, roadside buildings, road alignment, country-specific characteristics, weather conditions, and so on. Database 240 can also be integrated into Cloud Computing Infrastructure 800.

[0039] From the data recorded in real time by the sensors 220 at a specific geographical location and / or from the data stored in the database 240, the software application 320 of the scenario identification module 300 generates a suitable real-world scenario SZr. iderived. The software application 320 uses SZr to identify the real-world scenarios. i especially artificial intelligence algorithms. These artificial intelligence algorithms can include, in particular, encoders and decoders with neural networks.

[0040] A neural network consists of neurons arranged in multiple layers and interconnected in various ways. A neuron can receive information at its input from outside or from another neuron, evaluate the information in a specific way, and then forward it in a modified form to another neuron at its output or output it as the final result. Hidden neurons are located between the input and output neurons. Depending on the network type, there can be several layers of hidden neurons. They are responsible for the transmission and processing of information. Output neurons ultimately deliver a result and transmit it to the outside world. Neural networks can be trained through unsupervised or supervised learning.

[0041] Different arrangements and connections of neurons result in various types of neural networks, such as feedforward neural networks (FNNs), recurrent neural networks (RNNs), and convolutional neural networks (CNNs). A convolutional neural network has multiple layers and is well-suited for machine learning and applications in pattern recognition and image recognition. Since a large portion of the data captured by sensors is in the form of images, convolutional neural networks (CNNs) are particularly common.

[0042] The identified real-world scenarios SZr i can be stored, in particular, in a scenario database 340. The data for an identified real-world scenario SZr i can use the designation for the selected real-world scenario SZr iIncluded are the important scenario parameters P. i .of scenario SZr i saved.

[0043] The simulation module 400 also uses a software application 420 to simulate scenarios SZs i to generate. The software application 420 can also use artificial intelligence algorithms for this purpose. In particular, it can be provided that the simulation module 400 is connected to a test scenario database 410 in which test scenarios generated from various data sources are stored. i , Scenario parameter P i , Scenario parameter values ​​PV i as well as other information. The test scenario database 410 can also be integrated into the cloud computing infrastructure 800. In addition, the simulation module 400 can be connected to other databases, in particular database 240, which stores additional information for carrying out a simulation.

[0044] For example, the software application 420 of the simulation module 400 can be used by selecting suitable test scenarios SZt i and scenario parameters P i as well as scenario parameter values ​​PV i and, if applicable, further information, a plurality of test cases T i Create a simulation for one or more driving tasks of the ADAS / ADS system under test (210). The specific driving task is defined before the simulation begins, for example, by an expert such as an engineer. Alternatively, the driving task may be defined by a software application. An example of a driving task is an overtaking maneuver on a highway, also known as a cut-in maneuver.

[0045] Such a cut-in scenario can be achieved through the speed v ego of the Ego vehicle 200, the speed v target of a target vehicle, the spatial distance d cutinbetween the two vehicles and the spatial length l cutin The overtaking maneuver can be described. The speed of one or both vehicles can also change during the overtaking maneuver due to acceleration or braking. This also applies to the spatial distance d. cutin between the two vehicles.

[0046] The following table lists possible scenario parameters P i and scenario parameter values ​​PV i depicted in a cut-in scenario: Scenario parameter P i Scenario parameter value PV i v ego 10 - 250 km / h v target 10 - 250 km / h d cutin 10 - 100 m l cutin 10 - 100 m

[0047] In particular, T can be used for the selection and design of test cases. i The simulation should include a test strategy that specifies how the test cases T are calculated. i to be created. Various calculation methods and algorithms, especially artificial intelligence algorithms, can be used to define the test strategy.

[0048] After selecting a test case Ti creates software application 420 for this test case T i a simulated scenario SZs i , which reflects the simulation of the behavior of the ADAS / ADS system 210 or a driving function of the ADAS / ADS system 210 for a defined driving task.

[0049] In order to evaluate the performance of an ADAS / ADS system 210, the simulated scenarios SZs created by the simulation are used. i forwarded to the evaluation module 500 for the calculation of evaluation indicators 570, particularly in the form of KPLs. Evaluation module 500 includes a software application 520 for this purpose, which simulates the scenarios SZs. iThe performance and functionality of one or more functions, or the overall performance of the ADAS / ADS system under consideration, are evaluated, particularly in the form of key performance indicators (KPIs). The software application uses artificial intelligence calculation methods and / or algorithms, such as neural networks, to generate the evaluation indicators.

[0050] Key Performance Indicators (KPIs) are used to describe the performance of the ADAS / ADS system 210 under test. Different KPIs are defined for various evaluation categories such as comfort, safety, driving dynamics, and efficiency. Additional KPIs can be implemented to verify the correct functionality of the ADAS / ADS system 210. Examples of KPIs include the minimum distance to another vehicle or average acceleration during a deceleration scenario. KPIs can be represented by numerical values ​​or Boolean values. For example, a scale from 1 to 100 can be used for KPIs.

[0051] Furthermore, the real-world scenarios identified by the scenario identification module 300 are SZr iforwarded to the evaluation module 500 for the determination of evaluation indicators 570, in particular in the form of KPIs, in order to evaluate the quality of the ADAS / ADS system 210 used.

[0052] To determine the specific KPIs of different real-world scenarios SZr i or the KPIs of simulated scenarios SZs i with real-world scenarios SZr i In order to be able to compare them, the scenario parameters P are determined according to the invention. i and the scenario parameter values ​​PV i of the respective scenario under consideration SZ i They are used as evaluation metrics. For this purpose, the simulated scenarios SZs are used. i and the real-world scenarios SZr i and the specified KPIs are passed on to the evaluation module 700. The evaluation module 700 comprises a software application 720, which generates evaluation results 750 from the transmitted information, particularly in the form of KPI diagrams.

[0053] In Fig. Figure 2 shows an example KPI diagram where the values ​​of a KPI depend on the parameter values ​​PV. i The diagram shows a scenario parameter P1. Each point in the diagram represents a real scenario SZr. i or a simulated scenario SZs i To enable a comparison of the KPIs for a driving situation tested with the ADAS / ADS system 210, for example, an initial KPI diagram with real-world scenarios SZr is presented. i and a second KPI diagram with simulated scenarios SZs i created.

[0054] For the example of the cut-in scenario already considered, the first parameter P1 could be the relative velocity v. rel The interaction is between the ego vehicle 200 and a target vehicle, and the KPI is the braking jerk, i.e., the rate of change of acceleration of vehicle 200 during the cut-in scenario. Such a KPI diagram is in Fig. Figure 3 is shown. Since the KPI diagram displays the results from a large number of simulations or measured scenarios, the calculated KPIs for a parameter value PV can be i The KPI chart varies. Furthermore, it is divided into segments to visualize clustering within each segment. It is conceivable that, depending on a clustering value, the segments could be marked differently, for example with different colors, to simplify the comparison between two KPI charts.

[0055] From the in Fig. The KPI diagram shown in Figure 3 shows that a low relative speed between the two vehicles results in a low KPI value. The higher the relative speed between the two vehicles, the higher the KPI value. Deviations from this rule may indicate that the associated real-world scenario SZr i or simulated scenario SZs i to examine more closely.

[0056] In particular, the KPI diagrams of differently trained ADAS / ADS systems 210 for a driving task are compared with each other in order to be able to evaluate the performance of an ADAS / ADS system 210 objectively, precisely and comprehensibly.

[0057] In Fig. 4 are the procedural steps for the objective evaluation of the performance of an ADAS / ADS system 210 of a vehicle 200 for a defined driving task in at least one selected scenario SZ i depicted.

[0058] In step S10, real scenarios SZr are presented. i identified from real-time recorded data from sensors 220 while driving on a test track with the vehicle 200 or from stored data from a scenario identification module 300.

[0059] In step S20, simulated scenarios SZs are used. i generated by a simulation module 400.

[0060] In step S30, the real-world scenarios SZr are presented. i and the simulated scenarios of SZ i transmitted to an evaluation module 500.

[0061] In step S40, the evaluation module 500 calculates an evaluation indicator 570 for at least one real scenario SZr. iand an evaluation indicator 570 for at least one simulated scenario SZs i , where the rating indicator 570 represents the performance of the ADAS / ADS system 210 for the specified driving task.

[0062] In step S50, evaluation results 750 are generated by an evaluation module 700, where one evaluation result 750 is for a real scenario SZr i and a simulated scenario SZs i each assignment between the respective assessment indicator 570 and the scenario parameter value PVc i a selected scenario parameter PC i of the respective real-world scenario SZsr i or the respective simulated scenario SZs i contains.

[0063] Fig.Figure 5 schematically represents a computer program product 1000 comprising an executable program code 1050 configured to execute the method according to the first aspect of the present invention.

[0064] The present invention enables the objective evaluation and comparison of test results from differently configured ADAS / ADS systems by calculating evaluation indicators for scenarios derived from measurements and simulated scenarios. This is achieved by relating a calculated evaluation indicator for a real or simulated scenario to a scenario parameter value of a selected scenario parameter, particularly in the form of KPI diagrams. This results in an objective metric, allowing for the comparison of evaluation indicators from differently configured ADAS / ADS systems. The present invention thus enables the objective, precise, and verifiable evaluation of the performance and capabilities of an ADAS / ADS system, enabling the development of potential improvement measures.Overall, this significantly improves the assessment of the safety and functionality of one or more driving functions of an ADAS / ADS system for a specific driving task. This saves resources, as the need to drive actual test tracks with standard traffic situations and specific corner cases can be reduced. Reference sign 100 System 200 vehicles 210 ADAS / ADS system 220 sensors 240 database 250 Input module 300 Scenario Identification Module 320 Software application 340 scenario database 400 Simulation module 410 Test Scenario Database 420 Software application 500 rating module 520 Software application 570 evaluation indicators 700 Evaluation module 720 Software application 750 evaluation results 800 cloud computing infrastructure 1000 computer program product 1050 program code

Claims

[1] Method for objectively evaluating the performance of an ADAS / ADS system (210) of a vehicle (200) for a defined driving task in at least one selected scenario (SZ i ), in particular for testing and training at least one driving function of the driver assistance system (ADAS) and / or the automated driving system (ADS), wherein a scenario (SZ i ) represents a traffic event in a temporal sequence and is determined by a selection of scenario parameters (P1, P2, ..., P n ) and associated scenario parameter values ​​(PV1, PV2, ..., PV n ) is defined as comprising: - Identifying (S10) real scenarios (SZr i ) from data recorded in real time by sensors (220) while driving along a test track with the vehicle (200) and / or from stored data from a scenario identification module (300); - Generation (S20) of simulated scenarios (SZs i) by a simulation module (400); - Transmission (S30) of the real scenarios (SZr i ) and the simulated scenarios (SZs i ) to an evaluation module (500); - Calculating (S40) an evaluation indicator (570) for at least one real scenario (SZr i ) and an evaluation indicator (570) for at least one simulated scenario (SZs i ) by the evaluation module (500), wherein the evaluation indicator (570) represents the performance of the ADAS / ADS system (210) for the defined driving task, and wherein the evaluation indicators (570) are designed as key performance indicators (KPIs); generating (S50) evaluation results (750) by an evaluation module (700), wherein an evaluation result (750) for a real scenario (SZr i ) and a simulated scenario (SZs i ) an assignment between the respective evaluation indicator (570) and the scenario parameter value (PVc i) of a selected scenario parameter (Pc i ) of the respective real scenario (SZr i ) or of the respective simulated scenario (SZs i ), wherein the evaluation results (750) are designed as KPI diagrams, wherein in a KPI diagram the values ​​of a KPI are shown as a function of the parameter values ​​(PV i ) of a scenario parameter (P1) for a specific real scenario (SZr i ) or a simulated scenario (SZs i ), and the KPI diagrams are designed as histograms with segments for cluster analysis. [2] Method according to claim 1, wherein the scenario parameter value (PV i ) of the selected scenario parameter (P i ) of the respective real scenario (SZr i ) and / or the respective simulated scenario (SZs i ) by the evaluation module (500) from the data set of the real scenario (SZr i) and / or the simulated scenario (SZs i ) is extracted. [3] Method according to claim 1 or 2, wherein the KPI diagrams for differently designed ADAS / ADS systems (210) and / or the KPI diagrams for different real scenarios (SZr i ) and / or for simulated scenarios (SZs i ) and real scenarios (SZr i ) for an ADAS / ADS system (210) are compared. [4] Method according to one of claims 1 to 3, wherein the scenario identification module (300) comprises a software application (320) that uses calculation methods and / or artificial intelligence algorithms; wherein the simulation module (400) comprises a software application (420) that uses calculation methods and / or artificial intelligence algorithms; wherein the evaluation module (500) comprises a software application (520) that uses calculation methods and / or artificial intelligence algorithms; and wherein the evaluation module (700) comprises a software application (720) that uses calculation methods and / or artificial intelligence algorithms. [5] Method according to claim 4, wherein the calculation methods and / or algorithms of artificial intelligence are designed as mean values, minimum and maximum values, lookup tables, models of expected values, linear regression methods, Gaussian processes, fast Fourier transformations, integral and differential calculus, Markov methods, probability methods such as Monte Carlo methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, feedback neural networks, and / or convolved neural networks. [6] Method according to one of claims 1 to 5, wherein a scenario parameter (P i) represents a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, a speed, an acceleration, a jerk, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a motorway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographical feature such as a gradient, a time of day, a temperature, a precipitation value, a weather condition and / or a season. [7] System (100) for objectively evaluating the performance of an ADAS / ADS system (210) of a vehicle (200) for a defined driving task in at least one selected scenario (SZ i), in particular for testing and training at least one driving function of the driver assistance system (ADAS) and / or the automated driving system (ADS), wherein a scenario (SZ i ) represents a traffic event in a temporal sequence and is determined by a selection of scenario parameters (P1, P2, ..., P n ) and associated scenario parameter values ​​(PV1, PV2, ..., PV n ), comprising sensors (220) connected to the vehicle (200), a scenario identification module (300), a simulation module (400), an assessment module (500) and an evaluation module (700); wherein the sensors (220) are designed to record data in real time while driving on a test track with the vehicle (200); wherein the scenario identification module (300) is designed to identify real scenarios (SZr i) from the data recorded in real time by the sensors (220) or from stored data; wherein the simulation module (400) is designed to generate simulated scenarios (SZs i ); wherein the evaluation module (500) is designed to generate an evaluation indicator (570) for at least one real scenario (SZr i ) and an evaluation indicator (570) for at least one simulated scenario (SZs i ), wherein the evaluation indicator (570) represents the performance of the ADAS / ADS system (210) for the defined driving task, and wherein the evaluation indicators (570) are designed as key performance indicators (KPIs); and wherein the evaluation module (700) is designed to generate evaluation results (750), wherein an evaluation result (750) for a real scenario (rSZs i ) and a simulated scenario (SZs i) an assignment between the respective evaluation indicator (570) and the scenario parameter value (PV i ) of a selected scenario parameter (P i ) of the respective real scenario (SZr i ) or the respective simulated scenario (SZs i ), wherein the evaluation results (750) are designed as KPI diagrams, wherein in a KPI diagram the values ​​of a KPI are shown as a function of the parameter values ​​(PV i ) of a scenario parameter (P1) for a specific real scenario (SZr i ) or a simulated scenario (SZs i ), and the KPI diagrams are designed as histograms with segments for cluster analysis. [8] System (100) according to claim 7, wherein the KPI diagrams for differently designed ADAS / ADS systems (210) and / or the KPI diagrams for different real scenarios (SZr i ) and / or for simulated scenarios (SZs i) and real scenarios (SZr i ) for an ADAS / ADS system (210) are compared. [9] ADAS / ADS system (210) for a vehicle (200), characterized by that the method according to one of claims 1 to 6 is used for testing and training and / or for calibrating and validating the ADAS / ADS system (210). [10] A computer program product (1000) comprising an executable program code (1050) configured to carry out the method according to any one of claims 1 to 6 when executed.

Citation Information

Patent Citations

  • Procedure for adapting a vehicle control system

    DE102017221971A1

  • Method for generating a collection of information on driving scenarios of at least one vehicle, as well as vehicle, arrangement and system consisting thereof.

    DE102018215351A1

  • Method for testing the robustness of a simulateable technical system, computer program, machine-readable storage medium and device, as well as a technical system

    DE102019209554A1

  • Methods for optimizing tests of control systems for automated vehicle dynamics systems using probabilistically predicted system responses

    DE102020120141A1

  • SIMULATION OF REALISTIC TEST DATA FROM TRANSFORMED REAL-WORLD SENSOR DATA FOR AUTONOMOUS MACHINE APPLICATIONS

    DE112020002166T5

Cited By

  • Segment-based driver analysis and individualized driver assistance

    US12509113B2

  • Segment-based driver analysis and individualized driver assistance

    US20240140481A1