Designing an automatic driving control system in massively parallel simulations
Patent Information
- Application Number
- EP2023721888
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-06-22
- Filing Date
- 2023-04-24
- Publication Date
- 2025-04-30
AI Technical Summary
Developing and validating automated driving control systems for vehicles is challenging due to the need for extensive real-world testing, which is impractical and unsafe, especially for critical scenarios, and existing methods struggle to ensure safety and realism in simulation-based approaches.
A method involving the creation of a hybrid simulation environment with real and synthetic data, using parallelized long-term simulations with a time scale factor greater than one, to generate and validate training and validation data sets for critical and safety-relevant situations, allowing for the adjustment and validation of automatic driving control systems.
This approach enables efficient and effective design and validation of automatic driving control systems by combining real data validity with simulation flexibility and security, allowing for the simulation of complex and critical scenarios that would be impossible to test in real-world conditions, thereby ensuring the safety and reliability of automated vehicles.
Smart Images
Figure 1.1
Abstract
Description
[0001] DESIGN OF AN AUTOMATIC DRIVE CONTROL SYSTEM IN MASSIVE
[0002] PARALLEL SIMULATIONS
[0003] The invention relates to a method for designing and validating an automatic driving control system of a vehicle.
[0004] Automated driving control functions can, on the one hand, support a human driver in the sense of a driver assistance system, but in the future they will become significantly more important for highly automated and autonomous vehicles. Depending on the degree of authority, such automated driving control functions intervene more or less in the driving control of a vehicle and, particularly in fully automated vehicles, take over complete vehicle control. This leads to demanding safety requirements, as vehicles participating in road traffic pose a fundamental risk to other road users in the event of incorrect behavior. Driving control functions receive data from corresponding sensor units to use perception processes to detect the environment, in particular other road users and objects in the surrounding area, including traffic signs.Since highly complex software components are required to execute the automated driving control functions, take over vehicle control, and make appropriate decisions in traffic, demonstrating the required safety is correspondingly difficult and complex. However, testing a sufficient number of real-world scenarios for the development and final validation of automated driving control systems in practice would require an unacceptably large number of kilometers driven and correspondingly many hours of driving by a vehicle equipped with an automated driving control system. Covering billions of kilometers, for example, and the correspondingly delayed evaluation of the scenarios experienced there is disproportionately long for the development and validation of an automated driving control system and cannot be carried out in practice.A purely route-based, statistical proof of the safety of the driving function before the market launch of such a vehicle is therefore technically impractical. This is also explained in the following publication: "W. Wachenfeld and H. Winner, “The release of autonomous vehicles,” in Autonomous Driving: Technical, Legal and Social Aspects, M. Maurer, J.C. Gerdes, B. Lenz, and H. Winner, Eds. Springer, 2016, pp. 425-449." Future homologation, e.g., according to NCAP, will also require an increasingly complex test procedure that can only be successfully implemented with the help of realistic simulation environments. Furthermore, given the risk to third parties, it is not possible to test potentially critical scenarios in real traffic. However, it is precisely these potentially critical scenarios that provide the data needed to develop a vehicle's automatic driving control system, as these are precisely the ones that lead to accidents.These critical scenarios are also important for the validation of a fully developed automated driving control system. Therefore, in the development and validation of automated driving control systems, simulation-based methods are used and tested, particularly in operating areas that are critical for the automated driving control function, such as when approaching the end of a traffic jam or in high-frequency urban traffic areas. For example, scenario-based development and testing can be used, even during early development phases, to assess how safely the future real vehicle will operate in the open world (possibly within a limited operating range).
[0005] However, the simulation-based development and testing of automated vehicles with an automatic driving control system, in particular the validation of a driving control function of an automatic driving control system for the automatic operation of a vehicle, is only possible with the help of simulation if the respective simulation modules demonstrably produce realistic results. This also applies, among other things, to the modeling of traffic dynamics, i.e., the simulation of the behavior of other road users. A realistic representation of the environmental dynamics is therefore a key component for identifying potentially critical scenarios for an automated vehicle.
[0006] For the reliable use of these simulation-based methods, it is essential to model all entities relevant to the respective scenario with sufficient validity. In the context of simulation-based validation of automated driving functions, the realistic representation of the entire system, consisting of the digital twin and the environment, is becoming increasingly important, as the automated vehicle assumes responsibility for the entire driving task within a defined operating range. The valid representation of all relevant input variables for the ego vehicle's environment detection (system under test) is essential, with the representation of realistic traffic dynamics (position, behavior, and resulting movement of surrounding road users) playing a key role, for example, in the ego vehicle's decision-making process.It is known in the art to train neural networks to generate currently valid output values based on sensor data during subsequent operation of such an artificial neural network, based on known relationships between input and output values. Neural networks are often used to operate a control system of a technical system.
[0007] WO 2021 / 204983 A1 relates to a method for configuring a control device for a technical system, wherein a) a simulation module is provided for physically simulating the technical system based on physical variables of the technical system fed in as simulation input data, b) a temporal sequence of control signals and a temporal sequence of measured state data of the technical system are recorded, c) the temporal sequence of the control signals is fed into a neural network, d) output data of the neural network are fed into the simulation module as simulation input data, e) a physical simulation is carried out by the simulation module based on the output data of the neural network and simulated state data of the technical system are determined, f) the neural network is trained thereon,to reduce a gap between the measured state data and the simulated state data, g) a hybrid simulator for simulating the technical system based on control signals of the technical system is formed by the neural network trained in this way and the simulation module, h) a plurality of different further control signals is generated, i) the further control signals are fed into the hybrid simulator and into a further neural network, j) by means of the hybrid simulator, associated simulated reactions of the technical system are determined therefrom in the form of simulated state data and further output data are determined by the further neural network, k) the further neural network is trained to reduce a gap between the simulated state data determined by the hybrid simulator and the further output data, i) the control device is configured thereby,that it includes the further neural network trained in this way.
[0008] The object of the invention is to improve the design and validation of an automatic driving control system of a vehicle.
[0009] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.
[0010] A first aspect of the invention relates to a method for designing and validating an automatic driving control system of a vehicle, comprising the steps:
[0011] - Providing data from road users observed in reality and assigning it to the prevailing scenario, including the trajectories executed by the road users,
[0012] - Preparing the data of real road users to ensure the completeness of the data, and modelling the real road users as traffic agents,
[0013] - Creation of a hybrid simulation environment based on the processed data and the modelled traffic agents, adding synthetic data of virtual road users as the initial database,
[0014] - Conducting long-term simulations with the hybrid simulation environment using the initial database in multiple parallel simulations and with a temporal scale factor greater than one to real time, integrating the automatic driving control system with simulation data instead of sensor data, whereby variations of the initial database are carried out in the simulations and the response of the automatic driving control system to the hybrid simulation environment is determined,
[0015] - Generation of a training data set and a validation data set from the varied initial database depending on simulation results for critical and safety-relevant situations, and
[0016] - Performing an adjustment process of parameters of a driving control function of the automatic driving control system based on the training data set, and after completion of all adjustment processes of the driving control function:
[0017] - performing a validation of the automatic driving control system on the basis of the validation data set, wherein at least one algorithm of the driving control function of the automatic driving control system is validated.
[0018] The data of road users observed in reality can be recorded using stationary sensors, particularly in areas of particular interest such as urban intersections, but can also be collected from the advantageous bird's eye view, for example using a balloon, a quadcopter, a small-scale fixed-wing aircraft, or another unmanned aircraft. This means that the real behavior of real road users is known and can form a realistic approach to the initial database for the simulations. The respective associated scenario is recorded accordingly, for example, to provide the background scenario, such as intersection data, for the simulations in addition to the real road users. This scenario is advantageously included in the respective simulation in order to create a simulation that is as true to reality as possible.The respective associated scenario may also include weather data, such as the prevailing light intensity, wind strength, precipitation, fog, etc.; the trajectories of the real observing road users include a path curve and associated time information, so that the temporally relative movement of the road users among each other can be reproduced in the agent models of the simulations.
[0019] Preparing the data from real road users to ensure data completeness ensures that the entirety of the traffic dynamics relevant to the automated vehicle can be observed within the identified traffic sections. With the real data basis of the relevant traffic dynamics of real road users available, the simulation environment can then be calibrated and validated. This environment is capable of sufficiently validly modeling and simulating the dynamics of all relevant road users for the corresponding traffic space. The real road users are then modeled as traffic agents so that the actually observed behavior of the real road users can be represented in the respective simulation, and the respective simulation can depict dynamic scenarios.The calibrated and validated simulation environment forms the basis for the subsequent process step of massively parallel long-term simulation, in which the modeled traffic agents are used as the initial database with the addition of synthetic data from virtual road users.
[0020] Based on the validated simulation environment, long-term simulations are carried out using the source database in multiple parallelization of the simulations and with a temporal scale factor greater than one to real time, integrating the automatic driving control system with simulation data instead of sensor data. The fact that a temporal scale factor greater than one to real time means that the simulations are executed considerably faster than real time, allowing enormous amounts of data from very long simulated driving times to be processed in simulations in a relatively short time – hence the term “long-term simulation.” Multiple parallelization also contributes to this, i.e., multiple simulations are executed in parallel. Preferably, each individual simulation executes a respective variation of the source database or a respective set of variations.However, multiple variations of each simulation can also be executed in chronological order. The multiple parallelization, together with the compression of real time into a significantly faster process, enables massively parallel long-term simulation, allowing advantageous simulation of road user scenarios and thus the design and validation of the automatic driving control system, a number of which could realistically no longer be tested based on existing scenarios. However, the appropriate scaling in the simulation processes allows a significantly high number of scenarios, real and virtual road users, and variations in environmental variables to be utilized to design and validate the automatic driving control system. This allows, in particular, critical and safety-relevant scenarios and the behavior of other road users to be tested.This massively parallel long-term simulation is carried out, for example, on a cloud platform. This step offers the opportunity to significantly expand the existing database in terms of scope, complexity, and future mixed traffic scenarios. For example, probability distributions of relevant influencing parameters of traffic dynamics can be specifically varied. Examples of influencing parameters here include the accepted time gap for individual road user types (aggressive / defensive behavior) or the reaction time (human limitations). Furthermore, within the long-term simulation, a future mixed traffic situation (in which automated road users and human road users encounter each other) that is not observable in current road traffic can be simulated by using agent models for automated vehicles.The sum of the exemplary influencing parameters, along with other meta-parameters such as the parallelization factor of the simulations and the simulation duration / route to be simulated, form the hyperparameters of the long-term simulation. This approach allows for the targeted generation of sufficiently large amounts of synthetic data regarding the traffic dynamics relevant to the driving function, for example, by generating a higher and more aggressive traffic volume compared to reality.
[0021] This is followed by the generation of a training dataset and a validation dataset from the varied initial dataset, depending on simulation results for critical and safety-relevant situations. Both the training dataset and the validation dataset advantageously consist of both observed data from real road users and synthetic data from virtual road users. Within this process step, the real and synthetic datasets are combined into a single database, with the proportion of synthetic data typically significantly exceeding the proportion of real data.
[0022] Using the training dataset, the automated driving control system can learn a safe response within a vast number of realistic traffic situations in which the automated vehicle is expected to act during future operation. For example, end-to-end training strategies for situation understanding, decision-making, control, and actuators of the automated driving control system from the field of supervised machine learning can be advantageously implemented.
[0023] In the next process step, the validation dataset is used to verify whether the automated driving control system operates safely across a very large number of realistic scenarios. The generated validation dataset defines all the surrounding traffic dynamics relevant to the automated vehicle. While this methodology is particularly suitable for validating an algorithm for the driving control function of the automated driving control system and thus for safeguarding the situational understanding, decision-making, control, and actuators of the automated driving control system, the database can also be used to support the safeguarding of the automated vehicle's environmental sensing / perception. This can be achieved, for example, by combining the validation dataset with a realistic visualization simulator.The modeling of, for example, material properties and reflections of the relevant static and dynamic objects in the environment of the automated driving function is taken over by the visualization simulator, while the varied initial database defines the relevant traffic dynamics (type and manner of movement of the dynamic objects) in the corresponding scenario.
[0024] After this process step, a final check is preferably carried out to determine whether the previously defined validity criteria are met and whether the automated driving control system is considered validated for the relevant traffic areas. If this is not the case, either the simulation hyperparameters must be adjusted or additional real-world observations must be conducted. The subsequent process steps can then be repeated for the respective case.
[0025] A beneficial effect of the invention is that it combines the validity of real data with the flexibility, efficiency, and reliability of a simulation. This makes it possible to train and validate an automated driving control system with a data volume several orders of magnitude larger than with a purely real-data-based approach. This allows for further design and validation options, such as the possibility of investigating the mixed traffic scenario involving human and automated road users mentioned below.While maintaining sufficient validity of the entire data volume, a purely real-data-based test approach does not provide the opportunity to investigate future mixed-traffic scenarios in which the vehicle under consideration must be able to operate safely among human road users during its operation, with varying degrees of automated vehicle penetration. The sensible combination of real-world observation and synthetic data generation leads to significant increases in efficiency and effectiveness in the development and testing of automated driving functions. Recording trajectory data from real road users also offers an effective and efficient option compared to real-world driving with a prototype – information about the trajectories of different road users involved in a scenario is often sufficient for the design and validation of various modules of a driving function.A calibrated and validated traffic flow simulation also offers the opportunity to significantly expand the real-world data base in terms of scope, complexity, and future mixed-traffic scenarios. Very complex / critical traffic conditions can be reliably represented, as well as (as yet) unobservable conditions, for testing the automatic driving control system.
[0026] According to an advantageous embodiment, the synthetic data of the virtual road users are generated by modifying the data of the real road users.
[0027] According to a further advantageous embodiment, the synthetic data of virtual road users are generated by completely synthetic generation without reference to the real road users.
[0028] According to a further advantageous embodiment, the synthetic data of the virtual road users are generated partly by modifying the data of the real road users and partly by completely synthetic generation without reference to the real road users.
[0029] According to a further advantageous embodiment, the variations of the initial database comprise a varying ratio between modeled human traffic agents and virtual road users.
[0030] Varying the ratio between the virtual road users and the modeled human traffic agents allows a sensitivity analysis regarding the degree of penetration of automated road users, especially automated vehicles. Freely varying this ratio is not possible in a physically real situation. This embodiment can advantageously be used to determine the sensitivity of the automatic driving control system to the proportion of automatically controlled vehicles in the traffic area around the host vehicle. This is of great interest because automatically controlled vehicles exhibit different behavior than human-controlled vehicles. The proportion of automatically controlled vehicles relative to all road users is also referred to as the degree of penetration of automated vehicles.
[0031] According to a further advantageous embodiment, the number of virtual road users in the initial database is greater than the number of real road users used.
[0032] According to a further advantageous embodiment, the proportion of virtual road users is larger in the training data set than in the validation data set.
[0033] According to a further advantageous embodiment, strategic and tactical, or strategic, tactical and operational simulation elements for the vehicle are carried out in the simulations carried out.
[0034] The hybrid, varied initial database is primarily used for training at the strategic, tactical, or operational levels. An example of the strategic level is the sensible route selection based on a defined start and destination location. An example of the tactical level is the execution of the maneuver prediction module in the form of the future trajectories of the road users surrounding the automated vehicle in a specific traffic situation. In this case, the behavior / trajectories of real road users during safe passage of the respective scenario can be used as ground truth. The operational level particularly concerns a specific vehicle guidance function.Another obvious use case is the training of a reinforcement learning agent that is exposed to the corresponding traffic dynamics and is rewarded or penalized for its behavior depending on its reaction, thus enabling it to learn safe behavior across the relevant traffic scenarios.
[0035] According to a further advantageous embodiment, in order to carry out the adaptation process of parameters of a driving control function of the automatic driving control system, a sensitivity analysis is determined on the effect of variations in the initial database on the respective reaction of the automatic driving control system, and the result of the sensitivity analysis is used for a targeted adaptation process of parameters of a driving control function of the automatic driving control system.
[0036] This sensitivity analysis not only considers a single data point in the simulation space, but also varies the initial database around a data point to gain a sense of the impact of these changes on the response of the automated driving control system in multiple directions. This particularly applies to the number of modeled automated road users relative to the total number of other road users simulated in the simulation. This degree of penetration is of particular interest in sensitivity analysis.
[0037] According to a further advantageous embodiment, during the validation of the automatic driving control system, a sensor unit of the automatic driving control system is validated by performing a new simulation and output data of the simulation forming input data for the sensor unit.
[0038] In addition to the algorithm of a driving control function of the automatic driving control system, this embodiment also validates the environment detection / perception of the automated vehicle. For this purpose, a sensor unit with a connected perception module is preferably considered, with artificial data from the simulator being fed to the sensor unit instead of real data in order to validate the environment detection and perception of the automated vehicle. This particularly applies to image recognition software and environment detection algorithms based on other sensor units such as radar and / or lidar. Advantageously, material properties and reflections of the relevant static and dynamic objects in the environment of the automated driving function can also be applied by a visualization module in the simulation to generate realistic states of detectable sensor data.According to a further advantageous embodiment, optical material properties and / or reflections of the vehicle's surroundings are therefore simulated in the simulation.
[0039] According to a further advantageous embodiment, the variations in the initial database are achieved by imposing modified probability distributions of traffic agents. The probability distributions (such as normal distribution, binomial distribution, or Weibull distribution) are varied accordingly to create different scenarios. For example, the value of a standard deviation of random variables such as pedestrian behavior can be increased to obtain more differentiated scenarios.
[0040] Further advantages, features, and details will become apparent from the following description, which – where appropriate with reference to the drawings – describes at least one embodiment in detail. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.
[0041] It shows:
[0042] Fig. 1 : A method for designing and validating an automatic driving control system of a vehicle according to an embodiment of the invention.
[0043] Fig. 1 shows a method for designing and validating an automatic driving control system for an automated vehicle. In a first step, data from road users (pedestrians, cyclists, automated and non-automated passenger vehicles, etc.) observed in reality with the aid of a drone are provided SO and assigned to the respective prevailing scenario in which the road users were observed, in particular an intersection, a country road, a motorway journey, etc. The data from the road users observed by the drone also include the trajectories of the road users, i.e., their path curves with time information. In addition, the current weather conditions in the respective scenario under consideration are recorded by the drone in order to reproduce them in subsequent simulations.In the following step S1, the data of real road users recorded by the drone are processed to ensure the completeness of the data, so that the data it can record is actually made available to the vehicle's automatic driving control system, including its sensor units, in the simulation environment, without incurring data gaps caused by the drone's recording. The real road users recorded by the drone are modeled as traffic agents for the simulation environment. During this creation S2 of the hybrid simulation environment, in addition to the processed data from the observed reality with the modeled traffic agents, synthetic data of virtual road users are added, whereby the synthetic data advantageously takes up more space than the real data, i.e.While the real-world observed road users, with their trajectories and behavior, form the basis of the simulations, appropriate duplications and variations of the duplications enable the considerable size of the simulations with an enormous variety of simulated scenarios. This hybrid database of modeled real road users and fully synthetic road users (which may, however, be based on the data of real road users and merely further modified, for example, to provoke even more aggressive behavior from other road users) is used as the initial database.In the next step S3, several simulations are run using the generated hybrid simulation environment with the initial database, multiple times parallelized and with a temporal scale factor greater than one to real time, integrating the automatic driving control system with simulation data instead of sensor data. This means that a large number of simulations are executed simultaneously in parallel to generate an artificial environment for the automatic driving control system, for example, on a computing cluster with a large number of CPU cores and / or a network of graphics cards.The simulation speed is significantly faster than real-time, allowing a vast number of road users, scenarios, and kilometers driven by the automated vehicle to be simulated with the automatic driving control system for designing and / or validating the driving control system, working with a dataset that would be practically unattainable through practical tests (i.e., a real vehicle journey). In the simulations, variations of the initial database are performed to determine the respective response of the automatic driving control system to variations in the hybrid simulation environment using a sensitivity analysis. Based on these variations and the results, a training dataset and a validation dataset are generated (S4) to identify particularly critical and safety-relevant situations.Thus, an adaptation process S5 of parameters of a driving control function of the automatic driving control system can be carried out based on the training data set. This particularly applies to parameters of an artificial neural network, which can be adapted multiple times using supervised learning methods. After all adaptation processes of the driving control function have been completed, a validation of the automatic driving control system is carried out S6 based on the validation data set, whereby at least one algorithm of the driving control function of the automatic driving control system is validated.
[0044] Although the invention has been illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned by way of example are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.
[0045] List of reference symbols
[0046] 50 Provide
[0047] 51 Processing
[0048] 52 Create
[0049] 53 Implementation
[0050] 54 Generation
[0051] 55 Implementation
[0052] 56 Execute
Claims
Patent claims 1. A method for designing and validating an automatic vehicle control system, comprising the steps of: - Providing (SO) data from road users observed in reality, assigning them to the prevailing scenario, including the trajectories executed by the road users, - Processing (S1) the data of real road users to ensure the completeness of the data, and modeling the real road users as traffic agents, - Creating (S2) a hybrid simulation environment based on the processed data and the modeled traffic agents, adding synthetic data of virtual road users as an input data basis, - Execution (S3) of long-term simulations with the hybrid simulation environment with the input data basis in multiple parallelization of the simulations and with a time scale factor greater than one to real time, incorporating the automatic driving control system with simulation data instead of sensor data, whereby variations of the input data basis are carried out in the simulations and the reaction of the automatic driving control system to the hybrid simulation environment is determined, - Generation (S4) of a training data set and a validation data set from the varied input data basis depending on simulation results for critical and safety-relevant situations, - Performing (S5) an adaptation process of parameters of a driving control function of the automatic driving control system based on the training data set, and after completion of all adaptation processes of the driving control function: - Performing (S6) a validation of the automatic driving control system based on the validation data set, wherein at least one algorithm of the driving control function of the automatic driving control system is validated.
2. Method according to claim 1, wherein the synthetic data of the virtual road users are partly generated by modifying the data of the real road users and partly generated by completely synthetic generation without reference to the real road users.
3. Method according to any one of the preceding claims, where the variations in the source data include a varying ratio between modeled human traffic agents and virtual road users.
4. Method according to one of the preceding claims, wherein the number of virtual road users in the initial database is greater than the number of real road users used.
5. Method according to one of the preceding claims, wherein the proportion of virtual road users in the training data set is greater than in the validation data set.
6. Method according to one of the preceding claims, wherein strategic, tactical and operational simulation elements for the vehicle are performed in the simulations carried out.
7. Method according to one of the preceding claims, wherein, in order to carry out the adaptation process of parameters of a driving control function of the automatic driving control system, a sensitivity analysis is determined on the effect of variations of the output data basis on the respective reaction of the automatic driving control system, and the result of the sensitivity analysis is used for a targeted adaptation process of parameters of a driving control function of the automatic driving control system.
8. Method according to one of the preceding claims, wherein, in the validation of the automatic driving control system, a sensor unit of the automatic driving control system is validated by performing a re-simulation and the output data of the simulation form input data for the sensor unit.
9. Method according to claim 8, wherein optical material properties and / or reflections of the vehicle's environment are simulated in the simulation.
10. Method according to one of the preceding claims, wherein the variations of the initial data basis are effected by imprinting modified probability distributions of traffic agents.