Adaptive real-data-based simulation of a centrally coordinated traffic area

Abstract

The invention relates to a method comprising the steps of: providing (S1), from a scenario library, a data set containing information relating to a real traffic scenario from a traffic area such as a multi-storey car park with a central coordination unit for guiding automated vehicles and its own sensor network, carrying out (S2) a simulation with virtual counterparts of guided automated vehicles and non-coordinated road users with a simulated virtual test vehicle, determining (S3) a dissimilarity metric between the data set and trajectories determined in the simulation, and, if the dissimilarity metric is greater than a limit value: repeating (S4) the simulation while replacing the real movement data and / or behavioural models of non-coordinated road users with virtual agent models; adapting (S5), if necessary, the scenario library using synthetic sensor information from the repeated simulation.

Description

[0001] ADAPTIVE REAL DATA-BASED SIMULATION OF A CENTRALIZED

[0002] COORDINATED TRAFFIC SPACE

[0003] The invention relates to a method for carrying out a traffic simulation for testing a vehicle component.

[0004] In the development and validation of automated driving control systems, simulation-based methods are used and tested, particularly with regard to the operating ranges critical for the automated driving control function. For example, scenario-based development and testing can be used to assess how safely the future real vehicle will operate in the open world (possibly within a limited operating range) even during early development phases by using the digital twin of the automated vehicle.

[0005] This generally applies to (highly) automated vehicles that are approved and designed for general and public road traffic. Individual vehicles typically have their own implementations and designs of automated driving control systems (albeit adhering to standardized protocols and standards). Another technical implementation for highly automated driving of a vehicle, for which the same considerations apply, is the application of simulations with a very wide variety of parameter variations, hardware-in-the-loop tests, behavioral simulations of other road users, etc.can in principle take place in locally demarcated traffic areas, such as parking garages, industrial production areas, logistics areas and the like, where automated vehicles are used, but these do not rely on their own sensors and driving control systems as in general free road traffic, but a networked system of sensors transmits information to a central coordination unit, so that the coordination unit is able to collectively coordinate automated vehicles by means of a central driving control system and specify their movements.For example, centrally predefined parking space allocations in parking garages can be used to improve the use of parking spaces and ensure optimized parking, re-parking, and re-exiting procedures for each vehicle. Furthermore, in logistics areas, this can not only optimize goods management but also optimize the path guidance of automated logistics vehicles. Since uncoordinated road users, such as pedestrians, uncoordinated vehicles, and the like, are typically to be expected in such locally defined traffic areas with special sensor technology and a central coordination unit,, as well as technical uncertainties may exist, such as the latency of the calculations for specifying the respective trajectories or different software versions that need to be checked, a simulation with a large number of parameter variations within a maximum expected parameter space is recommended in order to be able to guarantee the safety and appropriateness of the journeys of the automated vehicles.

[0006] So-called "adaptive replay-to-sim" methods are known in the state of the art. In such methods, real-life scenarios are recorded and information about them is made available for simulation, allowing tests of driving control systems (or their sensors) to be carried out based on realistic data. The advantage of transferring the data to a simulation is the possibility of safe virtual testing, as well as the possibility of deliberately modifying the real-life data.

[0007] DE 10 2019 206 908 B4 relates to a method for training at least one algorithm for a control unit of a motor vehicle, wherein the control unit is provided for implementing an automated or autonomous driving function by intervening in assemblies of the motor vehicle on the basis of input data using the at least one algorithm, wherein the algorithm is trained by a self-learning neural network, comprising the following steps: a) providing a computer program product module for the automated or autonomous driving function, wherein the computer program product module contains the algorithm to be trained and the self-learning neural network, b) embedding the trained computer program product module in the control unit of the motor vehicle, c) driving the motor vehicle in a real traffic environment by a human driver, wherein the journey determines a driven trajectory,d) feeding data from an environmental sensor system and a motor vehicle sensor system to the control unit and calculating a virtual trajectory using the algorithm, e) deriving a metric from a comparison of the driven trajectory and the virtual trajectory and storing the data from the environmental sensor system and the motor vehicle sensor system in a memory if certain metric criteria are met for a traffic situation, f) providing information regarding the traffic situation to a traffic simulation, g) analyzing the traffic situation using the traffic simulation, whereby a virtual image of the traffic situation is created using the map data, the environmental sensor system and the motor vehicle sensor system, whereby the data of the traffic situation are varied using the traffic simulation,and h) training the algorithm by varying the traffic situation. The traffic simulation is carried out as a simulation of the traffic situation from the perspective of the traffic environment. Behavioral and technological models are provided for each individual road user in the traffic simulation, so that a future traffic situation is simulated based on a variation of the provided behavior and technological models. Furthermore, past environmental and vehicle data are also varied in the traffic simulation.

[0008] The publication "A Needle in a Haystack - How to Derive Relevant Scenarios for Testing Automated Driving Systems in Urban Areas" by Nico Weber, Dr.-lng. Christoph Thiem, and Prof. Dr.-lng. Ulrich Konigorski, a preprint for the "30th Aachen Colloquium Sustainable Mobility 2021," also demonstrates the use of rigid scenarios for traffic simulation and their flexibility. Based on an "adaptive replay-to-sim" approach, it presents a simulation-based toolchain for the development and testing of vehicles equipped with automated driving control systems in urban environments. Multimodal interactions between different road users are considered and, among other things, provided for feeding back into a scenario database.

[0009] For testing and / or validating driver assistance systems that support manual driving of vehicles such as passenger cars or trucks, as well as automation systems for highly automated and even autonomous vehicles (collectively referred to as "automated driving control functions"; English: "Advanced Driver Assistance Systems" or "Autonomous Driving Systems," collectively abbreviated to ADAS / AD), the use of scenario libraries is also known. These ideally contain descriptions of a multitude of (ideally as widely as possible) varying scenarios, so that a vehicle does not encounter fundamentally untested situations during subsequent normal operation. In such scenario libraries, scenario descriptions are typically stored in a machine-readable and automatically storable form.Scenario description languages, which employ specialized documentation and programming languages, are typically used for this purpose. One example is the established scenario description language of the "ASAM eV" association called 'OpenSCENARIO®' for use in scenario libraries. There are also scenario libraries with data in machine-readable formats that also provide human-understandable context, such as "ADScene" and "Safety Pool."

[0010] In order to be able to investigate the behavior of a driving control system or its sensor unit in a reactive manner to the environment of a road user under consideration, in particular an automated vehicle (typically called "vehicle under test"), with the help of such a scenario library, the journey of this selected real road user under consideration can be simulated, while the behavior of the other real road users as well as the environmental parameters can be taken from the scenario library and, in this case, are based on real data.

[0011] However, because the data on the behavior of other real road users and on the environmental parameters reflect facts that have come about as a result of the other road users adapting their reactions to the behavior of the real road user under consideration (which is replaced in the simulation by the "vehicle under test" as a virtual counterpart), the behavior of the other road users recorded in the simulation according to their stored data is only valid if the original behavior of the real road user under consideration and that of its virtual counterpart in the simulation (the simulated "vehicle under test") are essentially the same.If the simulated behavior of the road user as a virtual "vehicle under test" deviates too much from the behavior of the real road user under consideration, the integrity of the degree of realism of the simulation environment is limited and, in some cases, is therefore unsuitable for testing a driving control system or its sensors or a sensor unit for the simulated and thus virtual "vehicle under test".

[0012] The object of the invention is to provide the most efficient possible simulation of a locally delimited traffic area with, in particular, a stationary sensor network and a central coordination unit for coordinating automated vehicles in the presence of non-centrally coordinated road users.

[0013] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.

[0014] A first aspect of the invention relates to a method, in particular a computer-implemented method, for carrying out a traffic simulation, comprising the steps of: - Providing from a scenario library: a data set with information about a real traffic scenario from a locally delimited traffic area with a central coordination unit for coordinating automated vehicles by specifying trajectories for the automated vehicles based on sensor information from a stationary sensor network of the traffic area, wherein the data set comprises trajectory information of the automated vehicles guided by the coordination unit in the traffic scenario recorded by the sensor network and / or determined by the coordination unit, as well as real movement data and / or behavior models of non-coordinated road users in the traffic scenario recorded by the sensor network,

[0015] - Carrying out a first run of a simulation with virtual counterparts of the guided automated vehicles and the non-coordinated road users in such a way that all trajectory information and movement data of the virtual counterparts are the same as those from the data set, with the exception of at least one of the guided automated vehicles to be replaced, which is simulated as a virtual test vehicle, wherein at least one boundary condition is changed in the simulation compared to the data set,

[0016] - Determining a dissimilarity metric between the respective trajectory information of the data set and the new trajectories of one or more of the guided automated vehicles, in particular the respective virtual test vehicle, determined in the simulation and comparing the dissimilarity metric with a predetermined limit value, and if the dissimilarity metric is greater than the limit value:

[0017] - Repeating the simulation by replacing the real movement data and / or behavior models of non-coordinated road users with virtual agent models and generating additional synthetic sensor information, or synthetic sensor information that completely replaces the real sensor information, for the sensor network depicted in the simulation on the basis of the determined behavior of the agent models, so that the coordination unit carries out the coordination of at least some, in particular all, of the automated vehicles in the repeated simulation depending on the synthetic sensor information; and

[0018] - if necessary, adapt the scenario library with the synthetic sensor information from the repeated simulation.

[0019] In one step of the process, the vehicle component to be tested is tested by executing the traffic simulation. The vehicle component to be tested includes, in particular, a corresponding sensor and / or a trajectory planner such as a driving control computer. The vehicle component to be tested is either integrated into the simulation as a physical device using a hardware-in-the-loop method, or is virtually represented and also simulated. This results in improved testing of the vehicle component to be tested compared to conventional testing methods. The scenario library provides a data set with information about a locally defined traffic area.Such a locally delimited traffic area, for example a parking garage optimized for autonomous vehicles or a logistics area with a large number of work machines, has the advantage over a free traffic area of ​​the general public traffic area of ​​being able to carry out central coordination and specification of paths to be followed by vehicles in order to coordinate the road users with each other, as well as to specify optimal trajectories for the individual road users, for example an optimal parking process in a parking garage.

[0020] In such a locally defined traffic area, the central coordination unit assumes the function of a driving control computer. However, unlike the driving control systems of individual automated vehicles, the coordination unit synchronously controls a large number of automated vehicles. This means that the control electronics and sensors are no longer necessarily located in the respective vehicle itself. This is particularly true for the real-world counterpart to the virtual test vehicle from the real traffic scenario. These are, in particular, vehicles with external intelligence and perception that move in the same controlled environment (parking garage / logistics area). However, it can be assumed that in such a locally defined traffic area, not all road users are automated vehicles controlled by the coordination unit.Rather, it also includes non-coordinated road users, such as pedestrians, vehicles controlled by people directly or remotely, or other vehicles not controlled by the central coordination unit, i.e., non-coordinated vehicles.

[0021] However, due to the design and structure of the sensor network, the scenario library dataset contains both trajectory information about the movements of the automated vehicles controlled by the central coordination unit, as well as movement data and / or behavior models of non-coordinated road users, such as the pedestrians mentioned above. In an adaptive replay-to-sim process, the movement data from the dataset is thus suitable for transferring to a simulation in order to recreate realistic scenarios from the locally defined traffic space. However, one or more of the automated vehicles controlled by the central coordination unit are replaced in the simulation by a respective virtual test vehicle. Automated vehicles controlled by the coordination unit that are not replaced become a specific subtype of non-coordinated road user.The behavioral models can be determined from the data set in order to use machine learning methods to build catalogs or database scenarios, as well as to classify the scenarios that have occurred and to establish statistical distributions for the behavior of the non-coordinated road users.

[0022] The virtual counterpart to the real vehicle to be replaced is simulated under changed boundary conditions, which may lead to the simulated coordination unit determining a changed planned trajectory of the virtual vehicle. This forms the basis for determining the dissimilarity metric, which is used to check whether the trajectory of the virtual test vehicle in the simulation is sufficiently similar to the trajectory of the underlying real vehicle. Such a deviation between the respective trajectory information can arise from the fact that changed boundary conditions are simulated in the first run of the simulation, such as changed behavior of the non-coordinated road users or changed latencies of the coordination unit.

[0023] Once the dissimilarity metric has been determined, its value is compared to a threshold. If the threshold is exceeded by the dissimilarity metric, there are already non-negligible deviations between the trajectories of at least one virtual test vehicle and the corresponding real vehicle in the previous trip in the real traffic scenario.

[0024] Therefore, the other non-coordinated road users, identified by their real-world recorded trajectories, are replaced by agent models in the simulation. Such agent models are capable of actively reacting to the behavior of the virtual test vehicle during the simulation, which is not possible when using the recorded trajectories of the real road users, because the latter are used statically in the simulation, and only the virtual test vehicle would then be able to execute reactions in the simulation. However, if agent models are used, a reciprocal reaction occurs between the virtual test vehicle and the other road users implemented as agent models.

[0025] The number of agent models to replace the real-world recorded trajectories of other road users in the simulation is best decided based on how close or potentially reactive the other road users are to the virtual test vehicle in the current traffic situation. The simplest decision to do this is to replace all road users surrounding the virtual test vehicle in the simulation with agent models, but this comes at the expense of the computing power required to run the traffic simulation.

[0026] The agent-based trajectories are used in particular to generate new synthetic data for the sensor network or directly generate new trajectory data for the central coordination unit. If the agent-based trajectories are used to generate new synthetic data for one or more sensors of the sensor network to be tested or for their respective perception channels, real-time 3D graphic data is preferably used. This data is generated, in particular, by a 3D engine and fed into the respective perception channel, i.e., as sensor information.

[0027] Whether the scenario library is adjusted depends in particular on whether the dissimilarity metric was smaller than the threshold or not.

[0028] An advantageous effect of the invention is that X-in-the-loop tests of the perception channel become more realistic and adaptive by using original trajectories and the resulting consistent sensor information (as opposed to completely synthetic ones) as long as possible. Agent-based simulation, in turn, advantageously supports the simulation-based approach with its advantages: the enormous effort (personnel / time / financial) required by employees of an automobile manufacturer, for example, to run scenarios for testing the perception channels can be reduced by the method according to the invention if existing scenarios can be made more flexible in the proposed form and utilized for testing the perception channel. Therefore, there is no danger to test drivers or other road users in the event of a problem with the perception technology being developed or tested or the associated perception channel.Simple automated parameter variation and test evaluation are thus possible.

[0029] According to an advantageous embodiment, the boundary conditions changed in the first run of the simulation comprise at least one of the following: changed algorithms of the coordination unit for specifying trajectories for the guided automated vehicles, changed behavior or behavior patterns or other parameterization of the behavior models of the non-coordinated road users, changed latencies in the transmission of the commands transmitted from the coordination unit to the guided automated vehicles, changed latencies regarding the calculation of the commands determined by the coordination unit for the guided automated vehicles, changed perception by the stationary sensor network.

[0030] According to a further advantageous embodiment, the dissimilarity metric comprises a geometric distance between trajectories from the trajectory information, wherein the dissimilarity metric comprises a sum of respective geometric distances when the dissimilarity metric is related to a plurality of the guided automated vehicles.

[0031] Such a geometric distance can be determined, for example, by the area between the paths of the virtual test vehicle (the simulated "vehicle under test") and the real counterpart as a measure of the geometric distance to each other.

[0032] According to a further advantageous embodiment, the locally delimited traffic space is one of the following: a parking lot, a parking garage, a logistics zone.

[0033] According to a further advantageous embodiment, the sensor information includes all trajectory information of the real automated vehicles guided by the coordination unit as well as all real movement data of the non-coordinated road users, each from a bird's eye view.

[0034] For this purpose, the dataset preferably already contains corresponding, comprehensive information from a bird's-eye view. This is particularly the case if the sensor network monitors the locally defined traffic area, so that the data is either available directly from a bird's-eye view or can be reconstructed into one. For example, the sensor network in the locally defined traffic area may have ceiling-mounted cameras or a so-called "indoor GPS" to track the temporal position sequence and thus the trajectories of at least the guided automated vehicles and, ideally, also of the uncoordinated road users.

[0035] According to a further advantageous embodiment, the sensor information comprises at least some trajectory information of the real automated vehicles guided by the coordination unit and / or at least some real movement data of the non-coordinated road users, each from a first-person perspective. For example, if a parking garage is the demarcated traffic space under consideration, it can also be assumed that vehicles authorized and intended for general public transport are driving in this traffic space. However, depending on their intended level of automation, such vehicles already have a multitude of environmental sensors themselves, the information from which is typically acquired from a first-person perspective, but which can be used to support coordination by the coordination unit.

[0036] According to a further advantageous embodiment, the sensor information comprises camera data.

[0037] According to a further advantageous embodiment, the synthetic sensor information for the sensor network, which additionally or completely replaces the real sensor information, comprises synthetic real-time graphic data which is generated by a 3D engine and fed into the respective associated perception channel of the sensor network.

[0038] According to a further advantageous embodiment, the non-coordinated road users include at least one pedestrian.

[0039] According to a further advantageous embodiment, one of the following maneuvers is simulated in the simulation for the virtual test vehicle: parking, reversing, vehicle handover on a logistics area.

[0040] A further aspect of the invention relates to a data processing device comprising means for carrying out the steps of the method as described above and below.

[0041] A further aspect of the invention relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method as described above and below.

[0042] Advantages and preferred developments of the proposed device or the proposed computer program product arise from an analogous and analogous application of the statements made above in connection with the proposed method. Further advantages, features, and details emerge from the following description, in which at least one exemplary embodiment is described in detail—possibly with reference to the drawings. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.

[0043] It shows:

[0044] Fig. 1 : A method according to an embodiment of the invention.

[0045] The representations in the figure are schematic and not to scale.

[0046] Fig. 1 shows a method for performing a traffic simulation of a parking garage with a central coordination unit for coordinating automated vehicles. First, a data set containing information about a real traffic scenario from the parking garage with a central coordination unit for coordinating automated vehicles is provided S1 from a scenario library by specifying trajectories for the automated vehicles based on sensor information from a stationary camera sensor network of the parking garage. The data set contains trajectory information of the automated vehicles guided by the coordination unit in the traffic scenario, recorded by the sensor network and determined by the coordination unit, as well as real movement data and / or behavior models of non-coordinated road users such as pedestrians in the traffic scenario, recorded by the sensor network.Furthermore, the execution S2 of a first run of a simulation with virtual counterparts of the non-coordinated road users is carried out in such a way that all movement data of the virtual counterparts are identical to those from the data set. The guided automated vehicles are simulated as respective virtual test vehicles. In the simulation, at least one boundary condition is changed compared to the data set, wherein the boundary conditions changed in the first run of the simulation include at least one of the following: changed algorithms of the coordination unit, changed behavior or behavior patterns or other parameterization of the behavior models of the non-coordinated road users, changed latencies in the transmission of the commands transmitted from the coordination unit to the guided automated vehicles,Changed latencies regarding the calculation of the commands determined by the coordination unit for the guided automated vehicles; changed perception by the stationary sensor network. Furthermore, S3 determines a dissimilarity metric between the respective trajectory information of the data set and the new trajectories of one or more of the guided automated vehicles, in particular the respective virtual test vehicle, determined in the simulation, and compares the dissimilarity metric with a specified threshold. If the dissimilarity metric is greater than the threshold,Repeating S4 the simulation by replacing the real movement data and / or behavior models of non-coordinated road users with virtual agent models and generating additional synthetic sensor information for the sensor network represented in the simulation or synthetic sensor information that completely replaces the real sensor information based on the determined behavior of the agent models, so that the coordination unit coordinates at least some, in particular all, of the automated vehicles in the repeated simulation depending on the synthetic sensor information. If the dissimilarity metric has exceeded the limit and new synthetic data has thus been generated for the sensor network, the scenario library is adapted S5 with the synthetic sensor information from the repeated simulation.

[0047] Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of variations exist. It is also clear that exemplary embodiments are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms. With knowledge of the disclosed inventive concept, the person skilled in the art can make various changes, for example, with regard to the function or arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description. List of reference symbols.

[0048] 51 Provision

[0049] 52 Execute

[0050] 53 Investigate

[0051] 54 Repeat

[0052] 55 Adjust

Claims

Patent claims 1. A method for performing a traffic simulation for testing a vehicle component, comprising the steps: - Providing (S1) from a scenario library: a data set with information about a real traffic scenario from a locally defined traffic area with a central coordination unit for coordinating automated vehicles by specifying trajectories for the automated vehicles based on sensor information from a stationary sensor network of the traffic area, wherein the data set comprises trajectory information of the automated vehicles controlled by the coordination unit in the traffic scenario recorded by the sensor network and / or specified by the coordination unit, as well as real movement data and / or behavior models of non-coordinated road users in the traffic scenario recorded by the sensor network, - Carrying out (S2) a first run of a simulation with virtual counterparts of the guided automated vehicles and the non-coordinated road users in such a way that all trajectory information and movement data of the virtual counterparts are the same as those from the data set, with the exception of at least one of the guided automated vehicles to be replaced, which is simulated as a virtual test vehicle with a vehicle component to be tested, wherein at least one boundary condition is changed in the simulation compared to the data set, - Determining (S3) a dissimilarity metric between the respective trajectory information of the data set and the new trajectories of one or more of the guided automated vehicles, in particular the respective virtual test vehicle, determined in the simulation and comparing the dissimilarity metric with a predetermined limit value, and if the dissimilarity metric is greater than the limit value: - Repeating (S4) the simulation by replacing the real movement data and / or behavior models of non-coordinated road users with virtual agent models and generating additional synthetic sensor information for the sensor network depicted in the simulation or synthetic sensor information that completely replaces the real sensor information on the basis of the determined behavior of the agent models, so that the coordination unit in the repeated simulation carries out the coordination of at least some, in particular all, of the automated vehicles depending on the synthetic sensor information; and - if necessary, adapt (S5) the scenario library with the synthetic sensor information of the repeated simulation, and test the vehicle component to be tested.

2. The method according to claim 1, wherein the boundary conditions changed in the first run of the simulation comprise at least one of the following: changed algorithms of the coordination unit, changed behavior or behavior patterns or other parameterization of the behavior models of the non-coordinated road users, changed latencies in the transmission of the commands transmitted from the coordination unit to the guided automated vehicles, changed latencies with regard to the calculation of the commands determined by the coordination unit for the guided automated vehicles, changed perception by the stationary sensor network.

3. The method according to any one of the preceding claims, wherein the dissimilarity metric comprises a geometric distance between trajectories from the trajectory information, wherein the dissimilarity metric comprises a sum of respective geometric distances when the dissimilarity metric is related to a plurality of the guided automated vehicles.

4. Method according to one of the preceding claims, wherein the locally delimited traffic space is one of the following: a parking lot, a parking garage, a logistics zone.

5. Method according to one of the preceding claims, wherein the sensor information comprises all trajectory information of the real automated vehicles guided by the coordination unit as well as all real movement data of the non-coordinated road users, each from a bird's eye view.

6. The method according to claim 5, wherein the sensor information comprises at least some trajectory information of the real automated vehicles guided by the coordination unit and / or at least some real movement data of the non-coordinated road users, each from a first-person perspective.

7. Method according to one of the preceding claims, wherein the sensor information comprises camera data.

8. The method according to claim 7, wherein the synthetic sensor information for the sensor network, which supplements or completely replaces the real sensor information, comprises synthetic real-time graphics data generated by a 3D engine and fed into the respective associated perception channel of the sensor network.

9. The method according to one of the preceding claims, wherein the non-coordinated road users comprise at least one pedestrian.

10. Method according to one of the preceding claims, wherein in the simulation one of the following maneuvers is simulated for the virtual test vehicle: parking, reversing, vehicle handover in a logistics zone.

Images