Computer-implemented method for generating at least one training example for training a behaviour planner, use of at least one training example for training a behaviour planner, behaviour planner

Abstract

The invention relates to a computer-implemented method for generating at least one training example (1) for training a behaviour planner for an at least partially automated vehicle, wherein a training example (1) comprises input data (2) and a predefined target behaviour (3). According to the invention, the method comprises the following steps: a) removing a scene (4) from a previously provided scene pool (5), wherein a scene (4) comprises a scenery description (6) and a model behaviour (7), corresponding in time to thereto, of an agent (8) acting therein, b) defining a starting time within the scene (4), c) simulating an agent behaviour (10) as of the starting time on the basis of the scenery description (6) with the aid of a test behaviour planner, d) terminating the simulation when a termination condition is reached, and e) constructing at least one training example (1) by e1) selecting a reference time between the starting time and the end of the simulation, e2) generating input data (2) on the basis of the scenery description (6) and / or the agent behaviour (10) from a first time interval up until the reference time, and e3) generating a predefined target behaviour (3) on the basis of the model behaviour (7) of the agent (8) acting in the scene (5) in a second time interval as of the reference time. The invention further relates to the use of at least one new training example generated according to a method according to the invention for training a behaviour planner, to a behaviour planner, to an automatically driving vehicle, and to a robot.

Description

[0001] R.413661

[0002] - 1 -

[0003] Description

[0004] Title:

[0005] Computer-implemented method for generating at least one training example for training a behavior planner, use of at least one training example for training a behavior planner, behavior planner

[0006] The present invention relates to a computer-implemented method for generating at least one training example for training a behavior planner, having the features of the preamble of claim 1. The invention further relates to the use of at least one new training example generated according to a method of the invention for training a behavior planner. In addition, the invention relates to a corresponding behavior planner, an automated driving vehicle, and a robot.

[0007] State of the art

[0008] Autonomous driving is a significant development in modern mobility and encompasses a variety of technologies that enable vehicles to navigate without human intervention. A key aspect of this technology is behavior planning, which deals with deciding how a vehicle should act in different traffic situations. This takes into account factors such as traffic rules, the behavior of other road users, and environmental conditions. More specifically, trajectory planning focuses on the concrete calculation of the optimal driving corridor, or lane, called the trajectory, that the vehicle should follow to reach its destination safely and efficiently.

[0009] These planning processes typically utilize neural networks capable of recognizing and reacting to complex traffic patterns. Typical architectures include convolutional neural networks (CNNs) for processing image data captured by cameras (R.413661).

[0010] - 2 - as well as recurrent neural networks (RNNs), which can analyze temporal sequences of events. Transformer architectures are also becoming increasingly common for a wide variety of input modalities. These neural networks are typically trained using supervised learning methods, which employ large amounts of annotated data to prepare the network for various traffic scenarios. The requirement that all training data be labeled with a model solution or predefined target behavior becomes problematic when human experts are needed, which in itself is time-consuming and resource-intensive.

[0011] In the field of supervised learning, imitation learning plays a crucial role. Here, the algorithm is confronted with demonstrations of real-world driving behavior, typically demonstrated by experts, and learns to replicate it. This involves logging and recording a very large number of driving hours with an expert. The recordings include both the vehicle's sensor perception of the environment, such as radar, lidar, and video camera image data, and data about the vehicle's state at a given time, including position (including trajectory), speed, acceleration, steering angle, and much more. All of this data is presented, in whole or in part, to the behavior planner during training. The planner then designs a future trajectory and compares this design with the predefined target behavior derived from the expert's patterned driving behavior.

[0012] Imitation learning is particularly advantageous for training a behavior planner in automated driving for several reasons. The first reason is its highly efficient use of data. With each training example presented, the behavior planner learns the proven behavior of experienced drivers, instead of having to independently deduce effective driving behavior through lengthy trial and error, testing the traffic consequences of random actions. This also means a faster learning curve and thus shorter training sessions with less data. Furthermore, imitation learning can be flexibly adapted to different driving styles and conditions by using different demonstrations.

[0013] - 3 - can be used. It is thus adaptable to different regions, traffic patterns and local requirements - a trivial example being the implementation of left-hand traffic, which is widespread in the Commonwealth.

[0014] However, the example described above also illustrates that a significant disadvantage of imitation learning lies in the large number of expert driving hours required. Generating training data is very time-consuming and resource-intensive. Another disadvantage is that expert driving behavior is generally anticipatory and thus has a safety-promoting effect on the surrounding environment. As a result, dangerous situations rarely arise. Consequently, dangerous situations are underrepresented when training the behavior planner. The level of confidence with which the behavior planner can increasingly handle dangerous situations is therefore likely to be too low for real-world use.

[0015] Imitation learning also has its most significant weakness in its lack of robustness. This stems from the fact that a learned behavior plan, while ideally close to expert behavior, cannot perfectly replicate it. Thus, the behavior planned by the learner, such as a trajectory to be followed, typically deviates slightly from the expert's trajectory. These deviations accumulate over time, making it easy for the vehicle to maneuver into situations unfamiliar from the training. This can be illustrated with a brief example: when driving around a curve that the expert can easily follow, the vehicle might briefly deviate significantly further from the center of the lane than the expert ever did during training.The resulting overall situation deviates so significantly from the situations experienced during training that the behavior planner can no longer link it to a behavior learned in training. This leads to algorithmic instability, behavioral planning and driving decisions with alarmingly low confidence levels, erratic driving behavior, and even total failure, dangerous behavior, and collisions. The effect of the mathematical distribution of training characteristics differing from the distribution of characteristics identifiable in real-world use is addressed in the field of R.413661.

[0016] - 4 - in machine learning is called covariate shift and is very pronounced in imitation learning.

[0017] Several approaches exist to address the problem of covariate shift in imitation learning. One example is data augmentation, where noise is introduced into the training data. However, this only leads to local variations and, in the case of the behavioral planner, does not result in the desired learning of appropriate behavior when there are large deviations from situations covered by the expert in the training set. Other models exist that attempt to circumvent the covariate shift problem using different methods, all of which achieve this circumvention by accepting other disadvantages, including significant implementation effort, high computational costs, or the resource-intensive requirement of manual models.

[0018] The invention therefore addresses the problem of reducing covariate shift during the training of a behavior planner using imitation learning for an at least partially automated vehicle. To solve this problem, the method according to claim 1 is proposed. Further preferred embodiments are described in the dependent claims. Furthermore, the use of at least one training example for training a behavior planner and a behavior planner itself are described.

[0019] Disclosure of the invention

[0020] A computer-implemented method is proposed for generating at least one training example for training a behavior planner for an at least partially automated vehicle, wherein a training example comprises input data and a predefined target behavior. According to the invention, the following steps are performed: a) Extracting a scene from a pre-provided scene pool, wherein a scene comprises a scene description and a temporally corresponding pattern behavior of an agent acting therein, R.413661

[0021] - 5 - b) Defining an initial time within the scene, c) Simulating agent behavior from the initial time based on the scene description using a test behavior planner, d) Terminating the simulation upon reaching a termination condition, and e) Constructing at least one training example (1) by e1) Selecting a reference time between the initial time and the end of the simulation, e2) Generating input data (2) based on the scene description (6) and / or the agent behavior (10) from a first time interval up to the reference time, and e3) Generating a predefined target behavior (3) based on the pattern behavior (7) of the agent (8) acting in the scene (5) in a second time interval from the reference time.

[0022] The proposed method supplements real-world data with variations to increase the amount of information available for training. Conversely, this reduces the number of recorded scenes with patterned behavior, such as those exhibited by experts, required for reliable training. This, in turn, significantly reduces the time and resources needed to create a training dataset. Furthermore, covariate shift is reduced because a larger number of diverse training examples are generated from a single scene.

[0023] Normally, a training dataset used to train a behavior planner is generated directly from the individual scenes in the scene pool. As a rule, the scene description is processed, in whole or in part, possibly in conjunction with other available state information of the agent, into input data and provided with the pattern behavior as the predefined target behavior. Usually, at least one training example is available for each scene in the scene pool. With a method according to the invention, additional new R.413661

[0024] - 6 -

[0025] Training examples are generated that, while utilizing a scene description already present in the existing training dataset, define a different initial state for the agent than one caused by the pattern behavior. Such a training example confronts the behavior planner with a previously unknown overall situation during training.

[0026] A scene is understood as the temporal development of an overall situation, such as a traffic situation. This essentially comprises the temporal development of the surrounding environment and an agent acting within the scene. A large number of scenes are collected in advance in a scene pool. The temporal duration of the individual scenes can vary. The individual scenes in the scene pool are data from environmental data collection, such as real-world situations. They initially include a scene description, which refers to scene-specific information about the temporal development of the surrounding environment. For example, this information can be represented by sensor data from the agent, which might be a vehicle driven by an expert.Scenes also include pattern behavior, which refers to the agent's state information and actions within the evolving environment that correspond to the scene description in time. Crucially, the scene description and the associated pattern behavior must be temporally synchronized. This means that each point in time in the scene description can be uniquely assigned to a corresponding point in time in the pattern behavior.

[0027] The simulation in step c) is preferably performed on a main processor (CPU). The scene description, starting from the initial point in time, serves as the environment within which a test behavior planner designs a behavior. The test behavior planner is not necessarily the behavior planner being trained. Nor does the test behavior planner need to be AI-based; on the contrary, the use of a purely rule-based test behavior planner is equally conceivable. Within the simulation environment, which is populated with information from the scene description, the test behavior planner's design strategy is deployed. Provided that R.413661

[0028] - 7 -

[0029] If the test behavior planner relies on a definition of a boundary condition regarding a start state within the scene description, the agent state present at the start time from the scene, a start state based on predefined rules, or a generic start state of a different kind can be passed as a boundary condition.

[0030] The simulation continues until a termination condition is met. This termination condition can be, among other things, a simulation time limit, a deviation from the expected behavior, or the occurrence of a specific event in the scene description, such as a pedestrian entering the roadway.

[0031] In step e), at least one new training example is constructed in a three-part process comprising sub-step e1) and sub-steps e2) and e3), which can be executed in any order. In sub-step e1), a reference time point is chosen between the start time and the termination time of the simulation. In the temporal dimension of the scene, the reference time point marks a first and a second time interval, with the first time interval ending at the reference time point and the second time interval beginning at the reference time point.

[0032] In substep e2), information intended for training the behavior planner to be trained and belonging to the first time interval is extracted from the scene description and the simulated agent behavior complementing it, and then converted into input data compatible with the behavior planner to be trained. In substep e3), a corresponding predefined target behavior is generated from the pattern behavior of the agent acting in the scene in the second time interval. The predefined target behavior can, for example, simply be equated with the agent's pattern behavior for terminating the simulation. In another example, a transition behavior converging with the pattern behavior within a certain time can be calculated and used as the predefined target behavior. R.413661

[0033] - 8 -

[0034] The creation of a new training example is illustrated by the following example. A behavioral planner to be trained is to plan a driving trajectory based on detected objects. Within a simulation, a test behavioral planner, trained to plan a driving trajectory using radar, video, and speed data, designs a driving trajectory by deploying its planning strategy. The simulation is terminated if the simulated driving trajectory deviates from the pattern behavior (also a driving trajectory, henceforth the pattern trajectory) by a predefined amount. The simulated position at the time of simulation termination and the object information from the scene description up to the point of simulation termination are converted into input data, to which the pattern trajectory from the scene is assigned as the predefined target behavior.

[0035] In a further development of the invention, it is proposed that in step c) the behavior planner to be trained is used as the test behavior planner. This preferred embodiment has the advantage that the test examples generated by a method according to the invention cover situations that arise directly from an application of the planning strategy learned so far by the behavior planner to be trained. Thus, situations into which the behavior planner itself maneuvers are included in the training. The result is greater robustness in real-world use.

[0036] Furthermore, it is proposed that the behavior planner be implemented as a neural network, preferably a recurrent neural network. The advantage of this preferred embodiment lies in its ability to process highly complex relationships between various input features. Recurrent neural networks, in particular, are advantageous because they are capable of considering past event sequences of varying lengths. This allows behavior planning to be based not only on a specific situation but also on its evolution. This preferred embodiment thus enables more reliable and robust behavior planning.

[0037] Furthermore, it is proposed that the termination condition in step d) be a maximum tolerated deviation of the planned behavior from the pattern behavior. This preferred embodiment results in R.413661.

[0038] - 9 - A new training example is generated when the planned behavior deviates from a tolerance corridor around the pattern behavior. This directly serves the goal of detecting deviations from the pattern behavior early on, instead of only considering them as erroneous when a fatal consequence arises. The result is a broader representation of traffic situations not triggered by expert behavior and thus a further reduction of covariate shift.

[0039] Furthermore, it is proposed that a maximum tolerated number of simulation steps or a maximum tolerated simulation time be defined to achieve the termination condition in step d). A new training example is thus only generated if the termination condition is reached within this number of simulation steps or simulation time. This preferred embodiment has the advantage that situations that do not lead to the desired termination condition, and therefore would not generate a new training example that differs sufficiently from an existing training example, are not used to generate a new training example. This avoids artificial duplication of existing training data and a systematic shift of the training data towards certain situations (so-called bias).

[0040] It is also proposed that in step e.3), a transition behavior that successively approximates the pattern behavior is preferably generated computationally by means of interpolation or modeling and used as the predefined target behavior. This preferred embodiment has the advantage that, starting from the initial training example, the predefined target behavior to be learned is applied without discontinuities. Illustrated by a concrete example of trajectory planning, this means that if a training example specifies a vehicle position at the edge of the lane, the predefined target behavior specifies an actually feasible path from the edge of the lane back to the center, instead of assuming an unrealizable jump to the center of the lane. Such jumps would have two potential undesirable disadvantages. In the worst case, they cause very short-term, abrupt driving reactions of the vehicle following the trajectory.But even if such events do not occur, the behavior planner R.413661.

[0041] - 10 - If, for example, a driver plans a smooth, objectively desired route back to the center of the lane, they will experience a brief high loss function value during training, despite choosing the objectively "correct" behavior. Generally, the higher the loss function value, the greater the adjustment of weights that a behavior planner makes in response to the training example. However, if a current weight setting leads to the desired behavior, a change should be avoided. Objectively correct behavior should therefore generally be associated with low loss function values. The preferred embodiment ensures this.

[0042] Furthermore, it is proposed that after step e3), in step f), at least one new training example is added to a training dataset. With this preferred embodiment, a training dataset already undergoing training can be continuously supplemented.

[0043] Building on this, it is proposed that steps a) to f) be repeated until the training dataset has been extended by a predetermined number of training examples. This preferred embodiment combines a method according to the invention for generating a desired number of new training examples. When the behavioral planner to be trained is used as the test behavioral planner, new training examples are generated from the artificially created training examples. Each iteration generates situations into which a previously learned planning strategy of the behavioral planner leads independently. This reduces the covariate shift.

[0044] It is also proposed that scenes contained in the scene pool be weighted so that, in step a), scenes with increasingly lower probability are selected as the starting point the more simulation steps or simulation time they require in step c) to fulfill the termination condition. This preferred embodiment is advantageous because it allows situations that lead the test behavior planner away from the pattern behavior particularly quickly to be generated more frequently and thus trained more often. In this way, it enables consideration of different training needs with different situations. At the same time, the generation of new training examples is facilitated.

[0045] - 11 - statistically actively directed towards situations that bring about a high training effect.

[0046] In a further development of the invention, it is proposed that the process steps be carried out during the training of a behavioral planner. This preferred embodiment has the advantage that the generation of new training data can be made dependent on a need identified during training. In particular, if the behavioral planner to be trained is used as the test behavioral planner, training examples are generated only for those situations into which the behavioral planner to be trained independently maneuvers itself by deploying its planning strategy.

[0047] Building on this, it is further proposed that the process steps be performed between two training epochs. With this preferred embodiment, the training dataset is supplemented during the ongoing training. For example, the training can be briefly interrupted after each training epoch to supplement the training dataset. If the behavioral planner to be trained is used as a test behavioral planner, training examples are always generated for a subsequent training epoch that necessitate the use of the planning strategy trained up to that point. In this way, the training process can be very precisely tailored to the specific training needs of the behavioral planner.

[0048] It is further proposed that the process steps be performed in parallel with an ongoing training epoch. With this preferred embodiment, the training is not interrupted after a training epoch, but rather the data set used in that training epoch is supplemented with new training examples while the next training epoch is already running. New training data generated in a batch of a training epoch is incorporated into the training no earlier than the second training epoch following it. This preferred embodiment accelerates the training because there is no interruption to supplement the training data set. Preferably, the training is performed on a graphics processing unit (GPU), and the simulation to generate a new training example is performed on a main processing unit (CPU). R.413661

[0049] - 12 -

[0050] Furthermore, it is proposed that at least one new training example generated according to a method of the invention be used to train a behavioral planner. Training using at least one, preferably several, such generated training examples has the aforementioned advantages, in particular a reduced covariate shift.

[0051] In a further development of the invention, a behavior planner for an at least partially automated vehicle and / or a robot is proposed, which is trained using a method according to the invention. Such a behavior planner has the aforementioned advantages.

[0052] Furthermore, it is proposed that the behavior planner be a trajectory planner. A trajectory planner is a behavior planner for designing a lane to be driven in a specific traffic situation, also called a trajectory. This preferred embodiment is particularly advantageous because trajectory planning is frequently confronted with real-world situations that do not occur in expert behavior and is therefore particularly sensitive to instability and a lack of robustness in imitation learning. Conversely, trajectory planning benefits particularly from the aforementioned advantages of a behavior planner according to the invention.

[0053] In a further development of the invention, an automated driving vehicle is proposed, comprising a behavior planner according to the invention. Such a vehicle has the aforementioned advantages.

[0054] A robot comprising a behavior planner according to the invention is also proposed. Such a robot possesses the aforementioned advantages.

[0055] The invention is explained in more detail below with reference to figures. These show

[0056] Figure 1 shows a schematic representation of a process sequence according to the invention, R.413661

[0057] - 13 -

[0058] Figure 2 shows a schematic comparison of two possible sequences of a training method according to the invention.

[0059] Character description

[0060] Fig. 1 shows a schematic representation of a process flow according to the invention based on a simple trajectory planning scenario. Real-world and recorded traffic situations, so-called scenes 4, are collected in a scene pool 5 for use in imitation learning. A scene 4 comprises a scene description 6 and an agent 8 acting within it. The action of the agent 8, in this example a driving trajectory, is a pattern behavior 7 that corresponds temporally to the scene description, typically an ideal behavior initiated in reality by an expert and objective with respect to the scene 4. As a rule, a training example 1a is directly generated from the information of a scene 4 in the scene pool 5, whereby the scene description 6 is converted into input data 2a and the pattern behavior 7 into a predefined target behavior 3a. Such a training example 1a is used to train a behavior planner.Using a method according to the invention, the set of training examples 1a generated in this way is supplemented by a new training example 1.

[0061] According to the invention, a scene 4 is first selected from the scene pool 5. Scene 4 is then transferred to a simulation computer 9, on which a test behavior planner is running. Within scene 4, an initial time is defined from which scene 4 is to be considered. A simulation is then performed on the simulation computer 9, in which the test behavior planner plans agent behavior 10 based on the given scene description 6. This agent behavior 10 extends from the initial time until the time at which a termination condition is met. In the present example, this termination condition is met when the agent behavior 10 deviates from the pattern behavior 7 by a maximum tolerated amount. Subsequently, a transition behavior 11, which successively approximates the pattern behavior 7, is calculated by means of interpolation. Based on R.413661

[0062] - 14 -

[0063] Scene description 6, possibly including agent behavior 10, generates new input data 2. In this case, the input data 2 represents a situation in which agent 8 is in the position reached at the point where the simulation terminates. The transition behavior 11 is transformed into a new predefined target behavior 3 corresponding to the new input data 2. The new input data 2 and the new predefined target behavior 3 together form a new training example 1.

[0064] Fig. 2 shows a schematic comparison of two possible sequences of a training method according to the invention. In sequential sequence A, a training data set D is used to train a behavioral planner. This training data set D is first fed into a first training epoch A1. After this is completed, in a first extension A2, the training data set D is supplemented by at least one new training example 1 using a method according to the invention. It is then fed into a second training epoch A3. Once this is completed, the training data set D is again supplemented by at least one new training example 1 in a second extension A4 before being fed into a third training epoch A5. This sequential chaining is carried out until the training is completed.

[0065] In contrast to the sequential process A, the parallel process B involves a division of tasks. The training of the behavioral planner is performed on a graphics processing unit (GPU) R1, while a method according to the invention for generating training examples 1 is carried out on a main processing unit (CPU) R2. For this purpose, a training data set D is fed into a first training step, equivalent to a first training epoch B1.1. After completion of the first training epoch B1.1, a second training step B2 immediately follows. This involves performing a second training epoch B2.1 on the same data on the graphics processing unit R1. Simultaneously, the training data set D is transferred to the main processing unit R2, where a first extension B2.2 of the same data set is performed using a method according to the invention.The training data set D1, thus extended, is then passed to the graphical processing unit R1 for a second training step B3 and there in a third R.413661.

[0066] - 15 -

[0067] Training epoch B3.1 is used for training. In parallel, on the main processing unit R2, the training data set D used in the second training epoch B2.1 is itself extended in a second extension B3.2 by at least one new training example 1. This process is carried out until the training is complete.

Claims

R.413661 - 16 - Claims 1. Computer-implemented method for generating at least one training example (1) for training a behavior planner for at least a partially automated vehicle, wherein a training example (1) comprises input data (2) and a predefined target behavior (3), characterized by the following steps: a) extracting a scene (4) from a pre-provided scene pool (5), wherein a scene (4) comprises a scene description (6) and a temporally corresponding pattern behavior (7) of an agent (8) acting therein, b) defining an output time within the scene (4), c) simulating agent behavior (10) from the output time based on the scene description (6) using a test behavior planner, d) terminating the simulation upon reaching a termination condition, and e) constructing at least one training example (1) by e1) selecting a reference time between the output time and the end of the simulation.e2) Generating input data (2) based on the scene description (6) and / or the agent behavior (10) from a first time interval up to the reference time, and e3) generating a predefined target behavior (3) based on the pattern behavior (7) of the agent (8) acting in the scene (5) in a second time interval from the reference time.

2. Method according to claim 1, characterized in that in step c) the behavior planner to be trained is used as the test behavior planner. R.413661 - 17 - 3. Method according to claim 1 or 2, characterized in that the behavior planner is implemented as a neural network, preferably as a recurrent neural network.

4. Method according to one of the preceding claims, characterized in that a maximum tolerated deviation of the agent behavior from the pattern behavior is chosen as the termination condition in step d).

5. Method according to one of the preceding claims, characterized in that, in order to achieve the termination condition in step d), a maximum tolerated number of simulation steps or a maximum tolerated simulation time is specified.

6. Method according to one of the preceding claims 1 to 5, characterized in that in step e3) a transition behavior (11) which successively approximates the pattern behavior (7) is preferably generated computationally by means of interpolation or a model calculation and is used as a predetermined target behavior (3).

7. Method according to one of the preceding claims, characterized in that after step e3) in a step f) the at least one new training example (1) is added to a training data set (D).

8. Method according to claim 7, characterized in that steps a) to f) are repeated until the training data set (D) has been extended by a predetermined number of training examples (1).

9. Method according to one of the preceding claims, characterized in that scenes (4) contained in the scene pool (5) are weighted so that in step a) scenes (4) are selected as the starting point with an increasingly lower probability the more simulation steps or simulation time they need in step c) to meet the termination condition. R.413661 - 18 - 10. Method according to one of the preceding claims, characterized in that the method steps are carried out during the training of a behavior planner.

11. Method according to claim 10, characterized in that the Procedure steps are carried out between two training periods.

12. The method of claim 10, characterized in that the Procedure steps are carried out in parallel with an ongoing training period.

13. Use of at least one new training example (1) generated according to a method according to claims 1 to 12 for training a behavior planner.

14. Behavior planner for a vehicle that is at least partially automated and / or a robot that is trained using a method according to claims 1 to 12.

15. Behavior planner according to claim 14, characterized in that the behavior planner is a trajectory planner.

16. Automated driving vehicle comprising a behavior planner according to claim 14 or 15.

17. Robot comprising a behavior planner according to claim 14 or 15.

Images