Training neural network system to predict behavior of interacting actor

By introducing auxiliary tasks and helper networks in GNN training, forcing them to consider interaction relationships, the problem of GNN ignoring interactions when predicting the behavior of multiple interactive actors is solved, and the prediction quality and accuracy is improved, especially when the interaction is weak In the case of

CN118940799APending Publication Date: 2024-11-12ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410574565.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-11
Filing Date
2024-05-10
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Graph neural networks (GNNs) are difficult to force their dependence on interactions when predicting the behavior of multiple interactions, and may ignore interactions and obtain suboptimal solutions, especially when interactions are weak, it is difficult to ignore interactions without considering interactions. A satisfactory prediction is drawn.

Method used

By introducing auxiliary tasks during training, the helper network is used to reconstruct the original representation from the masked action representation, forcing the GNN to consider the interaction, using the loss function to evaluate the deviation between the reconstruction and predicted behavior, and optimizing the parameters of the GNN and the helper network to Improve interactive considerations.

Benefits of technology

Improve prediction quality, especially in applications where interaction is weak but important, ensuring that GNN provides more accurate solutions without ignoring interaction, improving the safety and accuracy of autonomous driving and driving assistance systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940799A_ABST
    Figure CN118940799A_ABST
Patent Text Reader

Abstract

A method for training a neural network system to predict behavior of interacting actuators includes: providing a training record of input data for each actuator; generating, by an encoder, an actor representation from each training record; processing, by the GNN to be trained and a decoder, the actor representations into predicted behavior data for each actor; determining masked actor representations from actor representations by modifying only a respective strict subset of values of each actor representation in actor representations for at least two selected actor representations; the to-be-trained GNN processes the masked actor representation into interaction representation; determining, by the helper network to be trained, a reconstruction of the actor representation from the interactive representation; rating deviations of the predicted behavioral data and the reconstruction from the actor representation by means of a predetermined loss function; and optimizing the parameters characterizing the behavior of the GNN and the parameters characterizing the behavior of the helper network towards an objective of improving the rating made by the loss function when processing additional training records.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the training of neural network systems to predict the behavior of multiple interacting actors, such as vehicles or robots in traffic situations. Background Art

[0002] In autonomous driving and driver assistance applications, a major task is to predict the behavior of different drivers. For a given vehicle, this prediction requires a probabilistic inference framework that takes a set of measurements (speed, relative position with respect to the lane, etc.). The framework then solves a problem and outputs a prediction for each driver in the future time step. Finally, the prediction output is used in downstream autonomous driving or driver assistance functions (such as adaptive cruise control) to improve driving comfort.

[0003] When machine learning models are trained to perform such tasks, they are expected to learn natural behaviors and traffic rules only from the recorded training data. For example, the future trajectory of a vehicle should follow the marked lanes. In addition, the future trajectories of two vehicles should not intersect, which would mean a collision. That is, good predictions also depend on the interactions between traffic participants. Graph Neural Networks (GNNs) are well suited to modeling such interactions.

[0004] However, it is difficult to force GNNs to rely on interactions in practice. When faced only with an optimization objective related to the task to be solved, GNNs can ignore interactions as long as it can derive a satisfactory solution without considering interactions. So-called regularization methods modify the optimization objective so that it encourages consideration of interactions. Summary of the invention

[0005] The present invention provides a method for training a neural network system to predict the behavior of a set of interacting actors.

[0006] The neural network system includes an encoder configured to convert input data about each actor into a one-dimensional actor representation having values ​​representing the characteristics of the actor. For example, for each of the A actors, a record x of the input data i Can include T in F over discrete time steps in The history of features makes the input data record x i is a tensor To allow analysis of interactions without having to consider the time dimension, the encoder can convert this tensor x i Mapped to the actor representation tensor The actor represents a tensor Contains only feature vectors of dimension d for each actor. For example, the encoder can be an already trained neural network, such as a Transformer Network or a Temporal Convolutional Network. In particular, the encoder can be kept frozen during the present training method.

[0007] The neural network system also includes a graph neural network GNN, which is configured to predict a complete graph of modified actor representations based on a complete graph of actor representations. That is, the actor representations are combined into a complete graph G = (V, E), where the set of nodes V represents actors and the set of edges E represents connections between edges. "Complete" means that each pair of nodes is connected by an edge, i.e., E = {(v, u) | v, u∈V}, and each actor is allowed to interact with all other actors.

[0008] The neural network system further includes a decoder configured to convert the modified actor representation into predicted behavior data for each actor. For example, the predicted behavior data Can be a tensor The tensor For each of the A actors, including T out F of future time steps out In this paper, T out Can be different from T in , and F out Can be different from F in Like the encoder, the decoder can be a neural network that can remain frozen during the present training method.

[0009] During the training method, a training record x of the input data for each actor is provided. i * These training records x i * May or may not be annotated with ground truth y i * . That is, training can be supervised or unsupervised.

[0010] From each training record x i * In the example, the encoder generates the actor representation F i The trained GNN and decoder represent these actors as F i Processed into predicted behavior data for each actor In other words, the predicted behavior data It can be written as:

[0011]

[0012] Output GNN(F i ) is the predicted behavior data we are looking for The latent representation of , and the decoder transforms it into the predicted behavior data

[0013] In order to give GNN an auxiliary task that encourages interaction consideration, we can i , determine the masked actor representation F i '. To this end, in the actor representation F for at least two selected actors i In the masked actor representation F, only a corresponding strict subset of the values ​​of each actor representation is modified. That is, in each actor representation, one or more values ​​remain unmodified. The auxiliary task is to extract the masked actor representation F from the masked actor representation F. i 'Reconstruct the original actor representation F i That is, the actor represents F i The information in that has been obscured by the mask will be recovered by exploiting the interaction between the actors.

[0014] This auxiliary task is handled by the GNN to be trained in conjunction with the helper network. The GNN is able to actually consider the interactions between actors and thus represent the masked actor F i 'Processed into interactive representation G i =GNN(F i ). The helper network is constructed from the interaction representation G i Determine the actor representation F i Reconstruction of F i # In particular, the helper network can be configured to represent the interaction representation G associated with each actor individually. i That is, if there are A actors, the helper network can be called A times to process the interaction representation G associated with actors 1,…,A. i part.

[0015] In the following, reference symbol F i 、F i ' and F i # Both for the complete tensor of (original / modified / reconstructed) actor representations, and for the individual actor representations associated with one specific actor, to save another layer of indexing.

[0016] The predetermined loss function rates the following:

[0017] On the one hand, the predicted behavior data That is, the performance of the GNN in solving the original task, and

[0018] On the other hand, reconstruction F i # With the actor F i , that is, the performance of the collaboration between GNN and the helper network when solving auxiliary tasks.

[0019] For example, the loss function may include a sum of terms related to these individual objectives. For the reconstruction term, one example of a loss function that may be used is the Huber loss.

[0020] The parameters characterizing the behavior of the GNN and the parameters characterizing the behavior of the helper network are optimized towards the following goal: when processing additional training records x i * In particular, the mentioned parameters can all be combined into a single parameter vector and this parameter can be optimized. For this purpose, for example, a gradient descent method with respect to the rating made by the loss function can be used.

[0021] It was found that adding auxiliary tasks to the training process improved the final predictions The quality of the GNN is improved, especially in applications where the interactions between actors are weak but still important. If the interactions between actors are very strong, then it is almost impossible to come up with a good solution to the original task without considering the interactions. But if the interactions are weak, then the GNN can "limp" to a feasible but certainly suboptimal solution to the original task without considering the interactions.

[0022] An example of an application in which the interaction can be considered weak is predicting the behavior of traffic participants in a traffic situation. In particular, in this application, the interaction can be weak since it is only intermittent. For example, when the ego vehicle is traveling along a free-to-use marked lane, it is of utmost importance that the ego vehicle stays in the lane. As long as no other vehicle changes lane towards the lane in which the ego vehicle is traveling, the interaction with other vehicles does not necessitate a change in the behavior of the autonomous vehicle. However, if such a lane change occurs, or a vehicle queue begins to form in front of a red traffic light, the interaction suddenly becomes relevant.

[0023] In another example, the driver of the ego vehicle may accelerate towards another vehicle in a lane. Eventually, the driver of the ego vehicle may decide to overtake the other vehicle. In this example, the moment when the driver decides to overtake the other vehicle marks the beginning of the interaction with the other vehicle, because the behavior of the other vehicle (here: slower speed) influences the driver's decision to overtake.

[0024] A particular advantage of modifying only a strict subset of the values ​​of each actor representation (i.e., leaving at least some values ​​unchanged) is that loss of information and resulting ambiguity in the solution to the auxiliary task is avoided. For example, if all values ​​in the actor representations for two actors P and Q are set to 0, then the two actors become indistinguishable from each other because the graph is complete and every actor is connected to every other actor. Therefore, the GNN and the helper network cannot know that the first actor should have been P and the second actor should have been Q, and not the other way around. But if P and Q are swapped in the reconstruction, the loss function will penalize it.

[0025] Therefore, in a particularly advantageous embodiment, the actor represents F i The strict subset of modified values ​​of is selected so that at most the masked actor representations F for at least two selected actors i ' is not that different. Therefore, how likely the strict subset of modified values ​​may be depends on the specific actor representation. For example, if all values ​​except one are modified, then it is modified if and only if this one value was in the original actor representation F i At different times, the masked actor indicates F i 'Keep it distinguishable.

[0026] In another particularly advantageous embodiment, the actor represents F i The strict subset of modified values ​​of is chosen to be at least such that the original value cannot be isolated from the corresponding masked actor representation F i This ensures that the interaction between actors is used to recover the masked actor representation F i ' is the only available source of masked information. For some actors, there may be an actor representation F i The interdependence of the values ​​in , such that if one value is known, then the other is also known. For example, if the vehicle is currently turning right, it cannot be turning left at the same time.

[0027] Advantageously, the actor represents F i The modification of the values ​​of comprises overwriting these values ​​with predetermined values. For example, the predetermined value may be 0. In this way, the previous information in the values ​​is effectively removed.

[0028] In another advantageous embodiment, the modified actor representation F i is randomly selected so that each actor represents F i is modified with a predetermined probability τ. In this way, the probability τ becomes a hyperparameter that determines the amount of auxiliary tasks. The unmodified actor represents F i Keep it as it is. The actor indicates F iThe more modified, the more other actors represent F i The less information is available in the scene, and the more thoroughly the interactions with the remaining actors must be investigated for a successful reconstruction.

[0029] In another advantageous embodiment, in each actor representation F to be modified i In , the modified values ​​are randomly selected so that each value is modified with a predetermined probability σ. In this way, this probability σ becomes another hyperparameter that determines the difficulty of the reconstruction task.

[0030] In another advantageous embodiment, a multilayer perceptron MLP is selected as the helper network. The output of this network can be in the same space as the input, namely the actor representation F for a specific actor. i .

[0031] In a further particularly advantageous embodiment, the input data x i Time series data including the position, trajectory and / or behavior of an actor. This can then be a logical extension of the time series. In particular, the new forecast It can always be made based on some sliding history of the time series data.

[0032] In another particularly advantageous embodiment, the time series data of the position, trajectory and / or behavior of the actor is segmented into training records x i * The previous part and the basic truth value y i * The latter part of the training data is used for predicting the position, trajectory and / or behavior by the neural network system based on the training records. In this way, there is no need to obtain or manually annotate dedicated ground truth data.

[0033] As discussed previously, in a particularly advantageous embodiment, the actors may be selected to be traffic participants in a traffic situation. As explained above, the interactions between these actors may be weak and intermittent, but it is very important to consider them when they do occur. Furthermore, the improved accuracy that can be brought about by the present training method is a safety improvement.

[0034] In a further particularly advantageous embodiment, the measurement data relating to the plurality of actors are acquired by at least one sensor. The measurement data are used as input data x i is provided to the trained neural network system so that the neural network system outputs the predicted behavior data for each actor Given the improved training, these predictions More accurate, especially in situations where the correct behavior depends heavily on the interaction between actors.

[0035] In another particularly advantageous embodiment, based on the predicted behavior data An actuation signal is determined. The vehicle, the robot and / or the driver assistance system is actuated with the actuation signal. In this way, the probability that the reaction of the respective actuated system to the actuation signal is appropriate in the situation described by the measurement data is improved.

[0036] The method may be computer-implemented in whole or in part and embodied in software. Therefore, the present invention also relates to a computer program having machine-readable instructions, which, when executed by one or more computers and / or computing instances, cause one or more computers and / or computing instances to perform the method. In this document, control units for vehicles or robots and other embedded systems capable of executing machine-readable instructions are also considered computers. Computing instances include virtual machines, containers, or other execution environments that allow the execution of machine-readable instructions in the cloud.

[0037] The non-transitory storage medium and / or download product may include the computer program. The download product is an electronic product that can be sold online and transmitted over a network for immediate fulfillment. One or more computers and / or computing instances may be equipped with the computer program and / or the non-transitory storage medium and / or download product.

[0038] Hereinafter, the present invention will be described using the accompanying drawings without any intention of limiting the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings show:

[0040] Figure 1a and 1b An exemplary embodiment of a method 100 for training a neural network system 1;

[0041] Figure 2 Illustration of GNN 3 training for both the main task and auxiliary tasks;

[0042] Figure 3 From the masked actor representation F i 'Reconstruct the actor representation F i # Illustration of the auxiliary task. DETAILED DESCRIPTION

[0043] Figure 1a and 1b is a schematic flow chart of an embodiment of a method 100 for training a neural network system 1 to predict the behavior of a set of interacting actors 5a-5e.

[0044] exist Figure 1a and 1b In the example shown in , in step 105 , actors 5 a - 5 e are selected as traffic participants interacting in the traffic situation.

[0045] In step 110, a training record x of input data for each actor is provided. i * .

[0046] According to block 111, the input data x i Time series data of the positions, trajectories and / or behaviors of the actors 5a - 5e may be included.

[0047] According to block 111a, the time series data of the positions, trajectories and / or behaviors of the actors 5a-5e may be segmented into training records x i * The previous part and the basic truth value y i * The latter part is used by the neural network system 1 based on the training record x i * to predict location, trajectory, and / or behavior.

[0048] In step 120, from each training record x i * In the example, the encoder 2 of the neural network system 1 generates the actor representation F i .

[0049] In step 130, the GNN 3 to be trained and the decoder 4 of the neural network system 1 represent the actor F i Processed into predicted behavior data for each actor 5a-5e

[0050] In step 140, from the actor representation F i Determine the masked actor representation F i '. To this end, in the actor representation F for at least two selected actors 5a-5e i In the above example, only each actor representation F is modified. i The remaining values ​​not in the strict subset remain unchanged.

[0051] According to block 141, the actor indicates F i The strict subset of modified values ​​of may be selected so that at most the masked actor representations F for at least two selected actors i 'Not that different.

[0052] According to block 142, the actor indicates F i The strict subset of modified values ​​of may be chosen to at least make the original value indiscernible from the corresponding masked actor representation F i 'Export so large.

[0053] According to block 143, the actor representation F is modified. i The values ​​of include overwriting these values ​​with predetermined values. In particular, the predetermined value may be 0.

[0054] According to block 144, the modified actor representation F i can be randomly drawn so that each actor represents F i is modified with a predetermined probability τ.

[0055] According to block 145, at each actor representation F to be modified i In , the modified values ​​are randomly selected so that each value is modified with a predetermined probability σ.

[0056] In step 150, the GNN 3 to be trained represents the masked actor F i 'Processed into interactive representation G i .

[0057] In step 160, the helper network 6 to be trained is trained from the interaction representation G i Determine the actor representation F i Reconstruction of F i # .

[0058] According to block 161 , a multilayer perceptron (MLP) may be selected as a helper network.

[0059] In step 170, the predetermined loss function 7 is used to calculate the predicted behavior data. and rebuild F i # With the actor F i The rating is marked with reference symbol 7a.

[0060] In step 180, parameters 3a characterizing the behavior of GNN 3 and parameters 6a characterizing the behavior of helper network 6 are optimized towards the goal of: when processing additional training records x i * The ranking 7a made by the loss function 7 is improved while the optimization parameters 3a, 6a are marked with reference symbols 3a* and 6a*, respectively. The parameters 3a* also define the final trained state 1* of the neural network system 1 including the encoder 2, the GNN 3 and the decoder 4. The helper network 6 is only used during training and does not form part of the final neural network system 1.

[0061] In step 190 , measurement data related to the plurality of actors 5 a - 5 e are acquired by at least one sensor 8 .

[0062] In step 200, the measured data is used as input data x i is provided to the trained neural network system 1* so that the neural network system 1* outputs the predicted behavior data about each actor 5a-5e

[0063] In step 210, based on the predicted behavior data An actuation signal 210a is determined.

[0064] In step 220 , the vehicle 50 , the robot 60 , and / or the driver assistance system 70 are actuated using the actuation signal 210 a .

[0065] Figure 2 Diagram showing how to train records based on input data x i * The same dataset is used to train the same GNN 3.

[0066] In one aspect, for obtaining the predicted behavior data The main tasks, and

[0067] On the other hand, for the masked actor representation F i 'Reconstruct the actor representation F i # auxiliary tasks.

[0068] Encoder 2 will train the record x i * Processed into an actor representation F i .

[0069] For the main task, these actors represent F i is fed directly into GNN 3, and the resulting work product GNN (F i ) is decoded by decoder 4 into the predicted behavior data sought

[0070] For the auxiliary task, step 140 of method 100 represents the actor F i Processed into a masked actor representation F i '. These masked actors represent F i 'Then processed by GNN 3 into the interaction representation G i . From these interactions, it is shown that G i , the helper network 6 determines the actor representation F i Reconstruction of F i# .

[0071] exist Figure 2 In the example shown in and the corresponding training record x i * The basic truth value y i * The performance on the main task is measured by comparing the reconstruction F i # With the original actor F i The comparison is made to measure the performance on the auxiliary task. The results of both comparisons are aggregated in a loss function 7, which delivers a rating 7a. This rating 7a is feedback for training the GNN 3 and the helper network 6 used only during training.

[0072] Figure 3 The auxiliary tasks are illustrated in slightly more detail. Figure 3 In the example shown in , the training record x i * The training records x include time series data of different aspects of the behavior of the actors 5a-5e. As discussed previously, in step 120 of the method 100, these training records x i * Encoded as the actor representation F associated with the actors 5a-5e i , and can be arranged in a complete graph.

[0073] exist Figure 3 In the example shown in FIG. 1 , in step 140, the actor representations F for the selected actors 5a and 5c are i Set some but not all values ​​to 0 to modify these actors to represent F i This produces a masked actor representation F i ', where the values ​​associated with the unselected actors 5b, 5d and 5e remain intact.

[0074] In step 150, the masked actor representation F i ' is processed into an interactive representation G i . From these interactions, it is shown that G i In step 160, the helper network 6 determines the reconstruction F i # The goal of the auxiliary task is to reconstruct F i # Matches the original actor representation F i .

Claims

1. A method (100) for training a neural network system (1) to predict the behavior of a set of interacting actors (5a-5e), the neural network system (1) comprising: The encoder (2) is configured to convert the input data x about each actor (5a-5e) i Converted into a one-dimensional actor representation having values ​​representing the actor's features; a graph neural network GNN (3) configured to predict a complete graph of the modified actor representation based on the complete graph of the actor representation; and a decoder (4) configured to convert the modified actor representation into predicted behavior data for each actor (5a-5e) The method (100) comprises the following steps: Provide (110) training records x of input data for each actor i * ; The encoder (2) extracts from each training record x i * The (120) actor representation F is generated in i ; The actor representation F is represented by the GNN to be trained (3) and the decoder (4) i Processed (130) into predicted behavior data for each actor (5a-5e) By selecting an actor representation F for at least two selected actors (5a-5e) i Only modify each actor representation F i The corresponding strict subset of the values ​​of , from the actor representation F i Determine (140) the masked actor representation F i '; The masked actor representation F is represented by the GNN to be trained (3) i 'Process (150) into interactive representation G i ; The helper network (6) to be trained is trained from the interaction representation G i Determine (160) the actor representation F i Reconstruction of F i # ; ·Use the predetermined loss function (7) to predict the behavior data And the reconstruction F i # With the actor F i The deviations are rated (170); and Optimize (180) the parameters (3a) characterizing the behavior of the GNN (3) and the parameters (6a) characterizing the behavior of the helper network (6) towards the goal of: when processing additional training records x i * , while improving the rating (7a) made by the loss function (7).

2. The method (100) according to claim 1, wherein the actor represents F i A strict subset of modified values ​​of is selected (141) such that at most the masked actor representations F for the at least two selected actors (5a-5e) are i 'Not that different.

3. The method (100) according to any one of claims 1 to 2, wherein the actor represents F i The strict subset of modified values ​​of is selected (142) to at least make the original value not individually discernible from the corresponding masked actor representation F i 'Export so large.

4. The method (100) according to any one of claims 1 to 3, wherein the modification of the actor representation F i The values ​​of include overwriting (143) the values ​​with predetermined values.

5. The method (100) according to any one of claims 1 to 4, wherein the modified actor representation F i is randomly selected (144) so ​​that each actor represents F i is modified with a predetermined probability τ.

6. The method (100) according to any one of claims 1 to 5, wherein in each actor representation F to be modified i In , the values ​​to be modified are randomly drawn (145) so that each value is modified with a predetermined probability σ.

7. The method (100) according to any one of claims 1 to 6, wherein a multilayer perceptron (MLP) is selected (161) as the helper network.

8. The method (100) according to any one of claims 1 to 7, wherein the input data x i Includes (111) time series data of the position, trajectory and / or behavior of the actor (5a-5e).

9. The method (100) according to claim 8, wherein the time series data of the positions, trajectories and / or behaviors of the actors (5a-5e) are segmented (111a) into training records x i * The previous part and the basic truth value y i * The latter part is used by the neural network system (1) based on the training record x i * to predict the location, trajectory and / or behavior.

10. The method (100) according to any one of claims 1 to 9, wherein actors (5a-5e) are selected (105) as traffic participants interacting in the traffic situation.

11. The method (100) according to any one of claims 1 to 10, further comprising: Acquiring (190) measurement data associated with a plurality of actors (5a-5e) by at least one sensor (8); as well as Take the measured data as input data x i Provide (200) to the trained neural network system (1*), so that the neural network system (1*) outputs the predicted behavior data about each actor (5a-5e) 12. The method (100) according to claim 11, further comprising: Based on predicted behavior data to determine (210) an actuation signal (210a); as well as Actuating (220) the vehicle (50), the robot (60) and / or the driver assistance system (70) using the actuation signal (210a).

13. A computer program comprising machine-readable instructions which, when executed by one or more computers and / or computing instances, cause the one or more computers and / or computing instances to perform the method (100) of any one of claims 1 to 12.

14. A non-transitory machine-readable data carrier and / or a download product having a computer program according to claim 13.

15. One or more computers and / or computing instances having a computer program according to claim 13 and / or having a machine-readable data carrier and / or a download product according to claim 14.