Computer-implemented method and system for training planning model

Abstract

A computer-implemented training method is proposed for a planning model that should provide future behavior of participants of a given traffic scene based on scene-specific information. In the scope of a training method, for at least one training scene and at least one participant of the training scene, the following steps are carried out in each case in successive simulation steps: predicting a future behavior of the participant by means of a planning model to be trained, the future development of the training scene is simulated by means of a given simulation model and taking into account the predicted behavior of the participant, and the predicted behavior of the participant is compared with the actual behavior (ground live) of the participant in the temporal development of the training scene. According to the invention, at least one set of potential features representative of the state of the training scene simulated in that simulation step is generated in each simulation step. The set of potential features is then used as a basis for the prediction of the behavior of the participant in a subsequent simulation step.

Description

Technical Field

[0001] The invention relates to a computer-implemented training method for a planning model, which provides the future behavior of at least one participant of a given traffic scenario based on scenario-specific information. Furthermore, the invention relates to a computer-implemented system for performing such a training method and a computer-implemented method for predicting and / or planning the future behavior of at least one participant of a given traffic scenario by means of a planning model trained according to the invention. Background Art

[0002] The training method discussed here provides that, in successive simulation steps, for at least one training scenario and at least one participant of said training scenario, respectively

[0003] - predicting the future behavior of the participants with the help of the planning model to be trained,

[0004] - simulating the future development of the training scenario with the aid of a given simulation model and taking into account the predicted behavior of the participants, and

[0005] - Compare the predicted behavior of the participant with the actual behavior of the participant during the temporal development of the training scenario (the Ground Truth).

[0006] Learning-based methods, in particular deep learning (DL), enable the development of planning models for automated driving functions (Automated Driving AD) that scale over many scenarios in the real world. In the case of learning-based methods, a distinction is made between reinforcement learning (RL), in which the agent learns by experimenting in simulation, and imitation learning (IL), in which the agent learns from demonstrations (e.g. from trajectories traveled by humans).

[0007] In the case of imitation learning, the planning model to be trained usually generates a predicted behavior in the form of a trajectory. Most of the time, only the first section of the trajectory is traveled or compared with the trajectory actually traveled, because over-planning with a new trajectory is performed in regular time steps. It has been shown that the performance sought by the planning model is usually not achieved in the case of pure IL, the so-called behavior cloning, during application, i.e. closed-loop behavior. Because in this case, learning is only based on the trajectory predicted in each time step, and the over-planning aspect is completely ignored. In the case of behavior cloning, small errors may accumulate, and the planning model has never seen the small errors in the data during training and therefore does not learn to reduce the small errors. In order to combat this, differentiable simulations are increasingly being implemented, which are described, for example, in Howell et al. "Dojo: A Differentiable Simulator for Robotics", https: / / arxiv.org / abs / 2203.00806. In the case of differentiable simulations, closed-loop behavior can already be taken into account during the learning process. The following is combined with Figure 1a and Figure 1b Elaborate on the ideas behind using differentiable simulations in more detail.

[0008] exist Figure 1a and 1b , a time axis denoted by t is shown in each case with a plurality of successive time steps of a predetermined duration and the time course of a trajectory 2 actually travelled by the vehicle 1 , which is also referred to as ground truth.

[0009] Figure 1a The diagram illustrates how the classic Imitation Learning (IL) method works. Thus, Figure 1a A trajectory 10 is shown which is predicted by a planning model for vehicle 1 and extends over five time steps. Predicted trajectory 10 is evaluated by comparison with ground truth 2 by determining the deviation between predicted trajectory 10 and ground truth 2 at a specific point in time as loss 3 .

[0010] In contrast, through Figure 1b In the case of a differentiable simulation as illustrated, deviations accumulate over time. To this end, the future development of the training scenario is simulated in successive time steps with the aid of a given simulation model and taking into account the predicted behavior of the participants. Furthermore, in each time step, an overplanning is performed on the trajectory 10, 11, 12, 13 predicted in the previous time step - the trajectory 11, 12, 13, 14, taking into account the results of the simulation. The results of this step-by-step simulation are shown in Figure 1b It is shown in the form of a simulated trajectory 4. Figure 1bThe predictions in the individual time steps - trajectories 11, 12, 13, 14 - are shown to be based on the starting point on the simulated trajectory 4. Figure 1a As in the case of , the evaluation of each track 11, 12, 13, 14 is performed by comparing with the ground truth 2. Here, Figure 1a Similarly, the resulting losses are shown by a double arrow that is not shown in more detail. The simulated trajectory 4 is passed as a differentiable learning signal via the simulation model to the planning model to be trained. As a result, the planning model sees more of its influence on the system than the ground truth 2, can learn from it, and thus shows greater stability and higher performance in driving behavior or closed-loop operation.

[0011] Independently of the above-described IL approach for training AD planning models, the use of latent features from DL-based perception and environment modeling has been established in the context of planning and prediction for AD applications, since it has been shown that a relatively large information content can be provided with the aid of these latent features, which for example also includes uncertainties from the environment perception, which has a positive impact on the quality of prediction and planning. Thus, when objects are not completely recognized and / or classified by the perception network, the planning component can use the uncertainty information of the latent features from the perception network to prompt appropriate, cautious maneuvers. This would not be possible if a separation was made between the perception module and the planning and prediction components and the planning and prediction components were provided essentially only with object information of the current traffic scene (such as object lists, maps, occupancy grids, etc.) without additional information about the reliability of the object information. Summary of the invention

[0012] Advantageous improvements to imitation learning with differentiable simulation are proposed. In particular, measures are proposed that enable the use of differentiable simulation when training a planning model that uses latent features from perception as input data.

[0013] To this end, according to the invention, at least one set of potential features representing the state of the training scenario simulated in the simulation step is generated in each simulation step. This set of potential features is then used as the basis for predicting the behavior of the participants in the subsequent simulation step.

[0014] The invention is based on the idea of ​​using imitation learning together with differentiable simulation to also train a planning model which generates a behavior plan for the individual participants in the traffic scene based on latent features representative of the scene.

[0015] This is in contrast to the fact that only potential features representing a given state of the training scenario can be generated based on the training data, but potential features representing later updated states of the training scenario cannot be generated. Therefore, the potential features provided to the planning model to be trained are only used for initial predictions in the first simulation step of the differentiable simulation. In subsequent simulation steps, these potential features can no longer be used meaningfully for predictions because they do not take into account the further development of the training scenario. According to the present invention, this problem is solved in the following way, that in addition to simulating the temporal development of the training scenario, at least one set of potential features representing the respectively simulated states of the training scenario is also generated in each simulation step. As a result, the potential features representing the respectively simulated states of the training scenario can be provided to the planning model to be trained in each subsequent simulation step.

[0016] The training data provided for the training method according to the invention should include at least a description of at least one training scenario and a description of the actual behavior of the participants during the temporal development of the training scenario, which is also referred to below as ground truth.

[0017] The description of the training scenario preferably includes scenario-specific information that has been aggregated at a given point in time. The scenario-specific information of the training scenario can come not only from infrastructure-based information sources of the vehicle itself but also from infrastructure-based information sources outside the vehicle. Typically, the scenario-specific information is data that has been recorded by a camera, lidar and / or radar sensor. In addition to this, the scenario-specific information can also be detected with the aid of inertial sensors of traffic participants. The scenario-specific information is often also supplemented by GPS data and environmental information (such as weather data and road condition data). The scenario-specific information is aggregated at a given point in time and accordingly includes sensor data and other data that are current at this point in time. However, the scenario-specific information can also include sensor data and other data that have been collected at a predetermined time interval up to a given point in time.

[0018] Alternatively or in addition to the scene-specific information, the training data may also include an environment model as a description of the training scene, which is derived from the scene-specific information aggregated at a given point in time. Thus, by evaluating the scene-specific information, objects can be identified and classified in the training scene in order to create object lists for the training scene. These object lists can be enriched with information about the state and / or state changes of the objects. An occupancy grid can be generated for the training scene and refined by combining with map information. Such an occupancy grid can furthermore be supplemented by information about the infrastructure and / or road topography of the training scene.

[0019] The behavior of the participants in the traffic scene is often described in the form of trajectory data, in particular by a time series of position information and / or movement information of the participants. In addition, the trajectory data can also include further information, such as information about the orientation of the participants. In principle, however, the behavior of the participants can also be described in other forms. Advantageously, for the description of the actual behavior of the participants during the time development of the training scene, i.e. for the description of the ground truth, a description that is identical to the predicted behavior of the participants is selected.

[0020] In an advantageous variant of the training method according to the invention, an initial environment model is provided for a given training scenario and is then updated over time in successive simulation steps. If the training data do not already include a description of the training scenario in the form of an environment model, but rather the description of the training scenario is present in the form of scenario-specific information, the initial environment model can be generated simply using a DL perception module, for example.

[0021] It has also proven to be advantageous to generate at least one initial set of potential features as a representative of a given training scenario based on the training data in order to use this initial set of potential features as a basis for predictions of the behavior of the participants in a first simulation step. If the training data include scenario-specific information as a description of the training scenario, the initial set of potential features can be generated with the aid of a DL perception module. However, it is also possible to use a feature generator specially provided for this purpose, which generates an initial set of potential features based on an environmental model of the training scenario. It is important here to provide the planning model to be trained with potential features for predictions in the first simulation step. In the subsequent simulation step, according to the invention, the planning module to be trained uses the potential features generated in the previous simulation step to predict the behavior of the participants.

[0022] Advantageously, the behavior simulated for the participant in the previous simulation step is also taken into account in the prediction in the further simulation step. If the behavior is predicted in the form of trajectory data, the position of the participant determined in the previous simulation step can be advantageously used as the starting point of the new predicted trajectory at the time of the new behavior prediction.

[0023] According to the invention, in each simulation step of the training method according to the invention, at least one set of potential features representing the states of the training scenario simulated in this simulation step is generated.

[0024] In a variant of the training method according to the invention, latent features are predicted in a latent space, i.e. based on latent features representing the corresponding states of the training scene. Thus, an initial set of latent features generated based on the training data is used as the basis for the prediction of latent features in a first simulation step. Then, the sets of latent features of the previous simulation steps are used as the basis for the prediction of latent features in further simulation steps. In an advantageous improvement of this variant, the state of the environment model of the training scene in the corresponding simulation step is also taken into account when predicting the latent features.

[0025] In a further variant of the training method according to the invention, the potential features in the individual simulation steps are generated solely on the basis of the current state of the training scene or an environment model of the training scene, respectively.

[0026] If the behavior of the participant is predicted in the form of trajectory data, ie as a sequence of prediction time points of a predefined prediction interval, the predicted behavior of the participant can be continuously interpolated between the prediction time points for comparison with the ground truth. It has proven to be particularly advantageous if the simulation steps and the prediction intervals are not of the same length.

[0027] The planning model to be trained usually includes adaptive parameters that are optimized within the scope of the training method. For this purpose, the behavior of the participants predicted in the individual simulation steps is respectively compared with the ground truth, and the losses are determined based on the determined deviations. The parameters of the planning model are then modified so that the losses are gradually reduced. It is particularly advantageous to take into account the comparison results from a plurality of simulation steps in each case. This ensures that the planning model is not only learned by comparison with the ground truth, but also from its influence on the system.

[0028] In addition to the above-described training method, a corresponding computer-implemented system for training a planning model and a method for predicting and / or planning the future behavior of at least one participant in a given traffic scenario are also claimed, wherein the planning model trained in this way is used. The system according to the invention and the prediction method / planning method according to the invention are explained in more detail below with reference to the figures. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above has been based on Figure 1a and 1b The prior art is discussed.

[0030] Figure 1a Illustration of how classic Imitation Learning (IL) methods work.

[0031] Figure 1b Illustration of the working mode of the IL method with differentiable simulation.

[0032] According to the following Figures 2 to 6The measures according to the invention as well as preferred embodiments and developments of the invention are explained in more detail.

[0033] Figure 2 The diagram illustrates the mode of operation of the training method according to the invention and the interaction of the basic components of the system according to the invention for training a planning model.

[0034] Figure 3 A first method variant for generating latent features in the individual simulation steps of the training method according to the invention is illustrated.

[0035] Figure 4 A further method variant for generating latent features in the individual simulation steps of the training method according to the invention is illustrated.

[0036] Figure 5 Illustration of method variations for generating an initial set of latent features representative for a given training scenario.

[0037] Figure 6 The use of a planning model trained according to the present invention is illustrated to predict and / or plan the future behavior of at least one participant of a given traffic scenario. DETAILED DESCRIPTION

[0038] Figure 2 The block diagram in the upper part of shows the components of the computer-implemented system according to the invention in the time course of the training of a planning model 200, which includes at least one neural network. The weights of this neural network are to be optimized within the scope of the training method. Figure 2 The time course of the training is illustrated by the time axis t in the lower half of . Starting from an initial time step 0, a plurality of successive time steps of a predetermined duration are plotted on the time axis t. On the time axis t a ground truth trajectory 2 is recorded which reproduces the time course of the trajectory 2 actually traveled by the vehicle in a given training scenario. The planning model 200 should be trained based on the behavior of this vehicle.

[0039] The ground truth trajectory 2 is part of a training sample which further comprises training data for describing a training scenario at a given point in time. In the embodiment described here, scenario-specific information 5 is provided for describing a training scenario, which information has been aggregated at a given point in time.

[0040] At the beginning of training, i.e. in an initial time step 0, the scene-specific information 5 is fed to the perception module 21 where it is mapped with the aid of a backbone network component to at least one initial set of latent features 60. Thus, the perception module 21 functions here as a feature generator that generates at least one initial set of latent features 60 representing a given training scene. Furthermore, the perception module 21 here comprises a DL component that generates an initial environment model 70 of a given training scene based on the scene-specific information 5 and / or the latent features 60. This can be, for example, an object-level representation, a 3D or 2D occupancy grid, a map or can be a visibility grid.

[0041] Figure 2 The block diagram of FIG. 20 illustrates that the planning model 200 to be trained is integrated into the training system via a corresponding interface. The initial set of potential features 60 is thus provided to the planning model 200 via a corresponding interface. In addition, the initial set of potential features 60 is fed to the simulation module 22 together with the data of the initial environment model 70.

[0042] In a first time step after the initial time step 0, the planning model 200 predicts the behavior of the vehicle, here in the form of a trajectory, using the initial set of latent features 60. The result 11 of this first prediction is provided to the simulation module 22, which then simulates the future development of the training scenario in a first simulation step taking into account the predicted behavior 11 of the vehicle. The result of this simulation is an updated environment model 71 (starting from the initial environment model 70).

[0043] According to the invention, the simulation module 22 also generates at least one set of potential features 61 representing the states of the training scene simulated in the first simulation step. To this end, in the embodiment of the invention shown here, the simulation module uses not only the potential features 60 or 61 in each simulation step but also the information from the environment model 70 or 71 of the previous simulation step. The set of potential features 61 generated by the simulation module 22 is then fed back to the planning model 200 via the corresponding interface of the system. As a result, the potential features 61 representing the currently simulated states of the training scene are available to the planning model 200 for prediction in the next simulation step.

[0044] Here, only the first and second simulation steps with the prediction results 11 and 12 are shown. However, since the simulation module 22 generates at least one set of potential features representing the state of the training scenario simulated in the simulation step in each simulation step according to the invention and uses the generated set of potential features as the basis for the prediction of the behavior of the participants in the subsequent simulation steps, the above method can be repeated for any number of simulation steps.

[0045] The prediction results 11 and 12 of the individual simulation steps are compared at least section by section with the ground truth trajectory 2 by means of the comparison module 23, which is indicated here by arrows 231 and 232. The comparison result is determined in each case. The weights of the planning model 200 are modified depending on the result of the comparison between the predicted behavior of the participant 11 or 12 and the ground truth 2. Figure 2 The link symbol 233 in indicates that comparison results from a plurality of simulation steps are taken into account here.

[0046] Combined with the above Figure 2 The differentiable simulation for planning and forecasting described is constructed as follows:

[0047] 1. For a driving situation or traffic scenario given in the form of training data, the IL planning model or DL ​​prediction model to be trained generates outputs, such as control commands for the vehicle or trajectories for a trajectory regulator, and is generated based on latent features.

[0048] 2. In the simulation step, the states for the participants in the scenario are updated, also with the aid of the simulation model and taking into account the output of the planning / forecasting model. In addition, latent features are generated that represent the simulated driving situation.

[0049] 3. Assume the simulated driving situation as the current state, and jump to step 1 if the desired time length of the simulation has not yet been reached.

[0050] exist Figure 3 In the case of the method variant shown in FIG. 1 , the latent features are generated in individual simulation steps of the training method according to the invention by direct prediction in the latent space.

[0051] with Figure 2 As in the case of the embodiment shown in , in the case of this method variant, potential features 600 are provided to the simulation module in each simulation step - in the first simulation step, an initial set of potential features and in subsequent simulation steps, a set of potential features generated in the previous simulation step. In addition, the simulation module is equipped with a DL component 221, which predicts new potential features 601 for subsequent simulation steps based on these potential features 600. For this purpose, for example, a transformer with a corresponding configuration of a CNN decoder can be used. In addition, here in each simulation step, information about the state of the environment model 700, 701 of the training scene and map information 800, 801 are provided to the simulation module from the previous simulation step. This information 700, 800 or 701, 801 is processed here with the aid of the graphic encoder 222 of the simulation module, so that this information can be taken into account when predicting the potential characteristics with the aid of the DL module 221.

[0052] Therefore, in the case of this method variant, a DL model is used that predicts the potential features. As input for the model, the potential features from the previous simulation step and further data (such as information about the state and / or state changes of the driving situation) are provided. The information about the driving scene can also contain information about the road topology in addition to the simulated objects. It is even conceivable that the information about the driving scene includes sensor-oriented information, such as simulated lidar reflections. The most suitable DL topology (feedforward NN, convolutional NN, graph NN, transformer NN) is used to predict the potential features. Figure 3 An encoder-decoder example architecture is shown.

[0053] Figure 4 Another possibility for generating latent features in the individual simulation steps of the training method according to the invention is shown. In this embodiment, latent features 600 are generated from the current system state, here from the state of the environment model 700 in the corresponding simulation step, and map information 800, by means of an encoder-decoder architecture 223. In contrast to the previous method, the latent features are not predicted here. Such generation of latent features 600 can advantageously be trained directly from the recorded real data, for example as a GAN (Generative Adversarial Network) or a VAE (Variational Autoencoder).

[0054] exist Figure 2 In the embodiment of the invention shown in , an initial set of potential features 60 is generated based on scene-specific training data 5 by means of a backbone network component of a perception module 21. However, when, for example, scene-specific information and / or a backbone network are not available for generating "real" potential features, an initial set of potential features 60 can also be generated based on an initial environment model 70 of the training scene by means of a feature generator 50 specially provided for this purpose. Figure 5 Shown in.

[0055] Figure 6The diagram illustrates the use of a planning model 200 trained according to the invention for predicting and / or planning the future behavior of at least one participant in a given traffic scenario. To this end, scenario-specific information 105 is aggregated at the measurement time. The scenario-specific information 105 is collected and preprocessed in a preparation phase 110. With the aid of a downstream perception module 120, latent features 150 are then generated as a representative of the current traffic scenario. These latent features 150 are used here on the one hand to generate an environment model 170 and on the other hand as input to the planning model 200. The planning model 200 then generates a prediction 300 based on the latent features 150, which can either be used directly to control the actuator systems of the vehicle or can also serve as a basis for further planning steps.

Claims

1. A computer-implemented training method for a planning model (200) providing a future behavior of at least one participant of a given traffic scenario based on scenario-specific information, In the context of the training method, for at least one training scenario and at least one participant in the training scenario, in successive simulation steps, a. predicting the future behavior of the participants (11, 12) by means of the planning model to be trained (200), b. simulating the future development of the training scenario by means of a given simulation model (22) and taking into account the predicted behavior (11, 12) of the participants, and c. comparing the predicted behavior of the participant (11, 12) with the actual behavior of the participant in the temporal development of the training scenario (ground truth) (2); It is characterized in that In each simulation step, at least one set of latent features (61) representing the states of the training scenario simulated in the simulation step is generated, and this set of latent features (61) is used as the basis for predicting the behavior of the participants in the subsequent simulation step.

2. The training method according to claim 1, characterized in that: Providing training data, wherein the training data at least includes: a description of the at least one training scenario in the form of scenario-specific information (5) aggregated at a given point in time and / or in the form of an environment model derived from the scenario-specific information aggregated at a given point in time, and - A description of the actual behavior of the participants (ground truth) (2) during the temporal development of the training scenario.

3. The training method according to claim 2, characterized in that: - providing an initial environment model (70) for the given training scenario based on the training data (5), and - Updating the initial environment model (70) in successive simulation steps.

4. The training method according to any one of claims 2 or 3, characterized in that: generating at least one initial set of potential features (60) based on the training data (5) as representative of the given training scenario, and An initial set of latent features (60) is used as the basis for predictions of the participant's behavior in a first simulation step.

5. The training method according to any one of claims 1 to 4, characterized in that: When predicting the behavior of the participant in a further simulation step, the behavior simulated for the participant in the previous simulation step is respectively taken into account.

6. The training method according to any one of claims 4 or 5, characterized in that: The initial set of latent features (60) is used as a basis for generating latent features (61) in a first simulation step, and the sets of latent features of the previous simulation steps are used as a basis for generating latent features in further simulation steps, respectively.

7. The training method according to any one of claims 3 to 6, characterized in that: The potential features are generated in the individual simulation steps based on the state of the environment model of the training scenario in the respective simulation step and / or taking into account the state of the environment model of the training scenario in the respective simulation step.

8. The training method according to any one of claims 1 to 7, characterized in that: The behavior of the participant is predicted for a sequence of prediction times of a predetermined prediction interval, and the predicted behavior of the participant is continuously interpolated between the prediction times for comparison with ground truth.

9. The training method according to claim 8, characterized in that: The behavior of the participants is predicted in the form of trajectory data.

10. The training method according to any one of claims 1 to 9, wherein the planning model to be trained comprises adaptive parameters which are optimized within the scope of the training method, characterized in that Parameters of the planning model are modified depending on the results of a comparison between the predicted behavior of the user and ground truth, wherein the comparison results from a plurality of simulation steps are respectively taken into account.

11. A computer-implemented system for training a planning model (200), in particular for training a planning model (200) according to the method according to any one of claims 1 to 10, wherein the planning model predicts the future behavior of at least one participant of a given traffic scenario based on scenario-specific information, the system comprising at least: a. training data (5, 2), which describes at least one training scenario with at least one participant at a given point in time and describes the actual behavior of the participant during the temporal development of the training scenario (ground truth), b. a feature generator (21; 50) for generating at least an initial set of potential features (60) representing a given training scenario, c. an interface for integrating the planning model to be trained (200), d. a simulation module (22) for simulating the future development of the training scenario in successive simulation steps, wherein in each simulation step the behavior of the participants (11, 12) predicted by the planning model (200) to be trained is taken into account, and at least one set of potential features (61) representing the state of the training scenario simulated in the simulation step is generated, and e. A comparison module (23, 233) for comparing the predicted behavior (11, 12) of the participant with the ground truth (2), and modifying the parameters of the planning model (200) according to the comparison result.

12. A computer-implemented method for predicting and / or planning the future behavior of at least one participant in a given traffic scenario, wherein the scenario-specific information (105) is aggregated at the measurement time point to generate at least one set of potential features (150) as a representative of the given traffic scenario based on the scenario-specific information (105), and The behavior of the at least one participant is predicted using a planning model (200) trained using a method according to any one of claims 1 to 10.

Images