Method for generating weighted training samples, use of weighted training samples, behavioral planner, vehicle and robot

Abstract

The invention relates to a computer-implemented method for generating at least one weighted training sample for training a behavioral planner for an at least partially automated driving vehicle or robot, the method comprises the following steps: extracting a state from a pre-provided initial training sample as an initial state; deducing a planning strategy of the behavior planner within a simulation range from the initial state to generate a behavior obtained by simulation; interrupting simulation when an interruption condition is met; evaluating the starting state and / or at least one state experienced in the simulation by a behavior resulting from the simulation; at least one training sample is weighted as a function of an evaluation of a state of the training sample and / or of an evaluation of at least one state experienced by the behavior resulting from the simulation after the training sample. The invention further relates to the use of at least one weighted training sample for training a behavioral planner, to a behavioral planner, to an automated vehicle and to a robot.

Description

Technical Field

[0001] This invention relates to a computer-implemented method for generating at least one weighted training sample for training a behavior planner. The invention also relates to the use of the at least one weighted training sample for training a behavior planner generated according to the method of the invention. Furthermore, the invention relates to a behavior planner, an automated driving vehicle, and a robot. Background Technology

[0002] The task of autonomous driving is to control the vehicle based on sensor data (such as radar, lidar, and RGB cameras) to reach the destination as quickly, comfortably, and safely as possible, i.e., without causing collisions or violating traffic rules. This driving task can be broken down into sub-tasks: perception, prediction, planning, and adaptation. The perception task involves extracting relevant information from sensor data, such as the location of objects (e.g., other vehicles or road users), identifying lane markings, and recognizing traffic signs. Since the identified objects are often dynamic obstacles, their future positions must then be predicted (the prediction phase) to avoid collisions. Based on this, the planning task is to generate a trajectory, which is then adjusted in the adaptation phase.

[0003] In the planning phase, trainable artificial intelligence models are frequently used. These models create plans based on a set of rules learned autonomously, known as planning strategies. The simplest way to train such planning strategies is through so-called "open-loop training," such as the "behavior cloning" method. Here, a dataset pre-recorded by experts is used. Each training sample includes a state and an action selected by the expert. During training, the AI ​​model is forced to face the state of the training sample and is required to plan an appropriate action. The action selected by the AI ​​based on its learned planning strategy is then compared with the expert's action. This trains the planning strategy to mimic the expert-selected action as well as possible. However, this method is susceptible to distribution shifts. This is because the states introduced by the expert in the training dataset differ systematically from the states experienced by the planning strategy in reality, since the occurrence of a given state depends, in principle, on the action taken in response to a previous state. When selecting actions, the planning strategy often arrives at results different from the expert's, leading to the experience of states in reality that were not present in the training dataset. The greater the deviation of these states from the states in the training dataset, the stronger the transferability required by the planning strategy, and the lower the confidence of the planning strategy in its action selection. Then, a point is quickly reached where the AI ​​model can no longer plan meaningful behavior. The consequences are collisions, uncomfortable behavior, and unexpected traffic incidents. Overcoming this problem becomes difficult because, due to the existence of state chains, it is only very difficult to causally attribute the occurrence of collisions, unexpected events, or uncomfortable behavior to specific incorrect decisions; this effect is known as the "sequence decision problem."

[0004] One possible solution to this problem is so-called "closed-loop training." Here, the AI ​​model is trained in a simulation. The actions selected by the planning policy are simulated, and the subsequent states are derived. The planning policy is then reapplied to the subsequent states obtained from the simulation. The states experienced by the planning policy in the simulation are used for training, for example, by generating new training samples based on these states. Since the states used for training are the result of applying the planning policy, they significantly better map the states encountered by the AI ​​model in reality compared to, for example, an open-loop approach. Furthermore, it significantly models sequential decision problems compared to an open-loop approach: the subsequent steps can simply determine which initial state led to the unfavorable chain of events.

[0005] The following example illustrates the advantages of closed-loop training: If a planning strategy is trained to control a vehicle, actions can be described by steering angle and acceleration, and the state can be described by the vehicle's current dynamic state, including environmental information (map information, observed vehicles, etc.). In open-loop training, for each training sample, the expert action is compared with the action selected by the planning strategy, for example, by calculating the difference in the planned steering angle. It's important to note that only the magnitude of the deviation is considered here; deviations in one direction are treated equally with deviations in another. Therefore, open-loop training completely disregards subsequent states. What doesn't differentiate the training effect is whether the response leads to decreased comfort or a collision. During training, small deviations in steering angle are penalized less than large deviations, without considering whether the small deviation might cause fatal consequences immediately or in the near future.

[0006] In closed-loop training, the steering angle planned by the planning strategy is simulated. This explicitly determines the consequences of the deviation between the expert action and the chosen action. Therefore, large errors, such as collisions or unexpected events, can in individual cases be attributed to small initial errors with a causal relationship (e.g., a slightly excessive steering angle). Thus, while existing techniques offer the possibility of predicting the consequences of a planning strategy's decisions based on its initial state through closed-loop training—allowing for consideration of penalties during training where actions that should typically be slightly penalized indirectly have adverse consequences—they lack the technical possibility of detecting consequences not only in their existence but also in their concrete content in a numerical and machine-understandable form, and making these consequences available for use in the training process. In particular, they lack the possibility of comparing the consequences of decisions with each other.

[0007] Therefore, the planning strategy of an automated driving algorithm might require the vehicle to stop 2.5 meters behind the vehicle in front of it. An erroneous action by the automated algorithm could then result in stopping too far or too short. Both are erroneous actions and, in principle, predictable in closed-loop methods, allowing for the general mitigation of such erroneous actions. However, "too short," especially a negative distance, is significantly more intolerable in practice than "too far." The driving algorithm is neither aware of this situation nor able to consider it appropriately during training because, unlike open-loop methods, while closed-loop methods continue to identify future erroneous actions, they cannot qualitatively distinguish between them. For completeness, it should be mentioned here that the concepts of reinforcement learning and inverse reinforcement learning present exceptions, as these concepts explicitly encode expected behavior in the form of reward functions; however, these concepts have other drawbacks and are therefore not always applicable. Summary of the Invention

[0008] Therefore, the objective of this invention is to provide information for the training process regarding the severity of the consequences of decisions made by the planning strategy.

[0009] To address this task, a method according to the present invention for generating at least one weighted training sample for training a behavior planner is proposed. Other preferred embodiments can be found in the extended technical solutions. Furthermore, uses of at least one method product, a behavior planner, a vehicle, and a robot are also provided.

[0010] A computer-implemented method is proposed for generating at least one weighted training sample for training a behavior planner for at least partially automated driving of a vehicle or robot, wherein the training sample is considered as a tuple consisting of states and corresponding pre-given target behaviors, and the following steps are performed: a) Extract the state from the pre-provided initial training samples as the initial state; b) Starting from the initial state, derive the planning strategy of the behavior planner within the scope of the simulation to generate the simulated behavior; c) Interrupt the simulation when the interruption condition is met; d) Evaluate the initial state and / or at least one state experienced by the behavior derived from the simulation in the simulation; e) Weight at least one training sample based on an evaluation of the state of the training sample and / or an evaluation of at least one state experienced by the behavior derived from the simulation after the training sample, wherein the training sample is preferably the initial training sample and / or a newly generated training sample based on the experienced state.

[0011] Extrapolating planning strategies in simulations allows for the identification of potential subsequent states that the planning strategy might experience starting from a given state. If a state with severe consequences, such as a collision, occurs, these states are causally related to previously experienced states through simulation. Therefore, all previous states, including the original training samples, are supplemented with abstract information about possible consequences, encoded as weights on their respective training samples, and can then be automatically considered during training. For example, the training dataset can be sorted according to the severity of increased or decreased consequences (i.e., according to weights), or training samples with particularly severe consequences can be trained more frequently than other training samples during training.

[0012] If simulation is used to generate new training samples in an automated manner, then in step e), the new training sample or these new training samples are weighted, taking into account their own evaluation and / or individual evaluations of their (at least some) subsequent states.

[0013] Furthermore, it is proposed that the maximum tolerable deviation between the simulated behavior and the expected behavior, and / or the occurrence of unexpected events, be selected as the interruption condition in step c). The deviation between the simulated behavior and the expected behavior is also referred to erroneous behavior below. Setting the simulation duration "when erroneous behavior and / or unexpected events occur, rather than, for example, at a certain simulation time" ensures that: on the one hand, the weighting is persuasive; on the other hand, resources are not wasted on meaninglessly continuing to simulate critical events that are already urgently needing to be avoided. Therefore, the simulation process is kept within a meaningful timeframe.

[0014] Unintended events are traffic incidents that need to be avoided and have at least one indirect causal relationship with the behavior planner's decision. These incidents may be collisions, posing a danger or obstruction to other road users, or simply violating traffic rules such as stopping at an intersection. Unintended events can be identified in the simulation by monitoring the overlap of object bounding boxes, the so-called "bounding boxes." For example, a collision can be identified if the bounding box of a vehicle guided by the behavior planner overlaps with the bounding boxes of other vehicles.

[0015] Furthermore, it is proposed that, in order to achieve the interruption condition in step c), the maximum permissible number of simulation steps or the maximum permissible simulation time be determined. This preferred implementation primarily saves computational power. If no unexpected events and / or erroneous behaviors occur within a pre-given time, the training sample is weighted in exactly the same manner as training samples that can be successfully handled by the planned strategy. Here, it is based on the assumption that from a certain distance between the unexpected events and / or erroneous behaviors and the state under consideration, the causal relationship can be simply assumed to be non-existent without significant adverse effects.

[0016] In an extended technical solution of the present invention, it is proposed that: in step d), an evaluation score is used to perform the evaluation. The utilization of the evaluation score provides the possibility of proportionally incorporating multiple aspects into the evaluation. In an example case, a certain state may lead to a traffic violation and a collision. By utilizing the evaluation score, this state can be given a different evaluation than other states that do not lead to a traffic violation but do lead to a collision. This possibility of differentiation is advantageous and improves the final utilization of the weighted training samples.

[0017] Based on this, it is proposed that evaluation scores be assigned to the occurrence of erroneous behaviors and / or unexpected events from a pre-created selection consisting of erroneous behaviors and / or unexpected events. Through this preferred implementation, erroneous behaviors and / or unexpected events are qualitatively graded one-time. This grading can then be considered in all future evaluations. To illustrate with a concrete example, this means that events such as "collision," "rule violation," and "dangerous" were originally evaluated as "unexpected" using binary quantification, but now they can be examined differentially: different evaluation scores are assigned to each event according to needs through the selection, mapping, for example, that "collision" is more serious than "dangerous," and "dangerous" is more serious than "rule violation." Additional events can be continuously added, and individual evaluations can be subsequently adjusted without repeating the entire evaluation and weighting process, including simulation. Erroneous behaviors, by definition, initially reflect deviations from expected behaviors, and in the case of erroneous behaviors, can be distinguished between deviations in one direction or deviations in another direction within the selection. In the case of examples already described in the prior art section, in the selected set, "the distance from the vehicle ahead is less than the required distance" may be evaluated as more serious than "the distance exceeds the required distance".

[0018] Furthermore, it is proposed that an evaluation score be assigned to the deviation between the simulated behavior and the expected behavior, with larger deviations receiving more evaluation scores. Therefore, the evaluation considers not only the "occurrence" or "absence" of identified erroneous behaviors (potentially in one or another direction), but also the magnitude of the deviation from the expected behavior. This erroneous behavior, while not leading to a specific unexpected event, still does not conform to the expected behavior and should be treated differently in the final weighting compared to objectively "correct" behavior due to the magnitude of the deviation. Subsequently, training samples based on states that initially lead to a sustained, numerically significant deviation from the expected behavior and subsequently result in unexpected events and / or erroneous behaviors can be weighted differently from training samples where the state initially produces behavior close to the expected behavior, but ultimately ends with unexpected events and / or erroneous behaviors.

[0019] Furthermore, it is proposed that steps a) to e) be performed frequently until a quality standard is met, preferably until a predetermined number of weighted training samples are generated. This preferred implementation enables the implementation of a mechanical chain according to the method of the invention; thereby, the entire training dataset can be weighted, or weights can be assigned to newly generated training samples using conventional methods.

[0020] It is also proposed that steps a) through e) of the method be performed during the training of the behavior planner. This preferred implementation allows for the benefit of weighting during training. Furthermore, the existing training dataset can be re-weighted each time the behavior planner's planning strategy is further developed based on training, ensuring that the weights always match the model's current training level.

[0021] To this end, it is proposed to select and / or rank training samples for at least one training cycle based on a weighted average of the training samples, and to use this selection and / or ranking as the basis for the at least one training cycle. Through this preferred implementation, the weighting of training samples is used to improve training performance. Therefore, the training process can be constructed starting from the determined difficulty or fallibility of the training samples; difficult (i.e., correspondingly weighted) training samples can be presented more frequently, and preliminary work can be performed progressively from easy to difficult, following a model of course learning already validated in machine learning, in a weighted ranking manner.

[0022] Furthermore, it is proposed that the behavior planner be trained using "differentiable simulation." Differentiable simulation allows for the causal attribution of events to previous decisions. Here, differentiable simulation can be used simultaneously to perform steps b) and c) of the method according to the invention. During training via differentiable simulation, the future is calculated over a fixed time interval, and the simulated behavior, or the state experienced in the simulation, is compared with the desired state and / or desired behavior. From the result of this comparison, backpropagation is performed, traversing all experienced states. Here, unlike, for example, in behavior cloning, the decisions prior to the final decision are adjusted by changing the weights. In this preferred embodiment of the invention, the advantage of training via differentiable simulation is obtained, and by weighting the training samples leading to a particular consequence, the consequence can be qualitatively detected and provided to the training process.

[0023] To utilize this situation, an extended technical solution of the present invention proposes that, before performing backpropagation, the loss function value be multiplied by a weighting factor, which depends on the weighting of the training samples that led to the loss function value. This preferred implementation allows the severity of events or errors occurring in the training effect to be considered during backpropagation, even when training via differentiable simulation. Therefore, for example, critical events, such as collisions, produce a stronger training effect than less severe events. A loss function is a function that quantifies the deviation between actual behavior and / or actual state and expected behavior and / or (i.e.) expected state. Loss functions are typically used in the training of neural networks to quantify the deviation between the outcome predicted by the neural network and the expected outcome. The loss function value forms the basis for the degree of weight adjustment made within the network layers in response to the outcome.

[0024] The invention also proposes the use of at least one weighted training sample generated according to the invention for training a behavior planner. The aforementioned advantages are present in training using at least one, preferably multiple, weighted training samples.

[0025] Furthermore, a behavior planner for at least partially automated driving vehicles and / or robots is proposed, which is trained using the method according to the invention. This behavior planner has the advantages described above.

[0026] It is also proposed that the behavior planner is a trajectory planner. A trajectory planner is a behavior planner used to design a driving trajectory in a given traffic scenario. This preferred implementation is particularly advantageous because precisely the planning of motion—which includes creating driving trajectories and / or control commands—is often forced to confront real-world situations not encountered in expert behavior, and is therefore particularly sensitive to the instability and lack of robustness of conventional methods, such as behavior cloning methods. Conversely, trajectory planners particularly benefit from the aforementioned advantages of the behavior planner according to the invention.

[0027] An automated driving vehicle is also proposed, which includes a behavior planner according to the invention. This vehicle has the advantages described above.

[0028] A robot is also proposed, which includes a behavior planner according to the invention. This robot has the advantages described above. Attached Figure Description

[0029] The invention will be explained in more detail below with the aid of the accompanying drawings. These drawings illustrate schematic diagrams of the method flow according to the invention.

[0030] Figure 1 A schematic diagram of the method flow according to the present invention. Detailed Implementation

[0031] Appendix Figure 1 A schematic diagram illustrating an exemplary method flow according to the present invention is shown. In a first method step S1, an arbitrary initial training sample is provided from the provided training dataset. In method step S2, starting from the state of this initial training sample, the planning strategy of the behavior planner is deduced within the scope of the simulation until an interruption criterion, such as an unexpected event like a collision, occurs. In a subsequent method step S3, the states experienced are collected and temporarily stored. In the next method step S4, each state is evaluated. Here, in sub-step S4.a, the evaluation is based on the deviation of the behavior from the desired behavior. In sub-step S4.b, the evaluation is based on the occurrence of unexpected events and / or erroneous behaviors from a pre-created selection set. Here, these evaluations are summarized in a state-specific evaluation score. Subsequently, in method step S5, the initial training sample is weighted based on the final total evaluation derived from each state.

[0032] Steps S1 through S5 are repeated until the entire training dataset is weighted. The planning policy is then retrained, taking into account the weights that will now take effect. This consideration is reflected, for example, in ranking the training samples according to their weights, and / or in more frequent training of the correspondingly weighted (i.e., "harder") training samples. After training, steps S1 through S5 are re-executed based on the newly trained planning policy, and the weights of the training samples are thus adjusted according to the planning policy's capability profile.

Claims

1. A computer-implemented method for generating at least one weighted training sample, said weighted training sample being used to train a behavior planner for at least partially automated driving of a vehicle or robot, wherein, Training samples are considered as tuples consisting of states and corresponding pre-given target behaviors, wherein the method includes the following steps: a) Extract the state from the pre-provided initial training samples as the initial state; b) Starting from the initial state, deduce the planning strategy of the behavior planner within the scope of the simulation to generate the behavior derived from the simulation; c) Interrupt the simulation when the interruption condition is met; d) Evaluate the initial state and / or at least one state experienced by the behavior derived from the simulation in the simulation; e) Weight at least one training sample based on an evaluation of the state of the training sample and / or an evaluation of at least one state experienced by the behavior derived from the simulation after the training sample, wherein the training sample is preferably the initial training sample and / or a newly generated training sample based on the experienced state.

2. The method according to claim 1, characterized in that: The maximum tolerable deviation between the simulated behavior and the expected behavior, and / or the occurrence of unexpected events, are selected as the interruption conditions in step c).

3. The method according to any one of the preceding claims, characterized in that: To achieve the interruption condition in step c), determine the maximum permissible number of simulation steps or the maximum permissible simulation time.

4. The method according to any one of the preceding claims, characterized in that: In step d), the evaluation is performed using an evaluation score.

5. The method according to claim 4, characterized in that: An evaluation score is given for the occurrence of erroneous behaviors and / or unexpected events from a pre-created selection consisting of erroneous behaviors and / or unexpected events.

6. The method according to claim 4 or 5, characterized in that: The deviation between the simulated behavior and the expected behavior is given an evaluation score, with larger deviations receiving more evaluation scores.

7. The method according to any one of the preceding claims, characterized in that: Steps a) through e) are performed frequently until the quality standard is met, preferably until a predetermined number of weighted training samples are generated.

8. The method according to any one of the preceding claims, characterized in that: Method steps a) to e) are performed during the training of the behavior planner.

9. The method according to claim 8, characterized in that: Training samples are selected and / or sorted according to a weighted average of the training samples for at least one training period, and the selection and / or sorting is used as the basis for the at least one training period.

10. The method according to claim 8 or 9, characterized in that: The behavior planner is trained based on a differentiable simulation.

11. The method according to any one of the preceding claims, characterized in that: Before performing backpropagation, the loss function value is multiplied by a weight factor that depends on the weighting of the training samples that resulted in the loss function value.

12. One use of at least one weighted training sample generated by the method according to any one of claims 1 to 11 for training a behavior planner.

13. A behavior planner for at least partially automated driving of a vehicle and / or robot, the behavior planner being trained using the method according to any one of claims 1 to 11.

14. The behavior planner according to claim 13, characterized in that: The behavior planner is a trajectory planner.

15. An automated driving vehicle comprising a behavior planner as described in claim 13 or 14.

16. A robot comprising a behavior planner as described in claim 13 or 14.

Images