Transform-based networked autonomous vehicle trajectory prediction method

Abstract

The invention, which relates to the automatic driving field, discloses a Transform-based networked automatic driving vehicle trajectory prediction method comprising the following steps: obtaining vehicle driving data and preprocessing the vehicle driving data to obtain time sequence information of each vehicle; a context vector of each vehicle is obtained by using a Transform encoder; inputting the context vector of each vehicle into a graph neural network to obtain an overall feature context vector I; obtaining a driving intention probability matrix of the current vehicle by using a random forest algorithm; splicing the driving intention probability matrix of the current vehicle and the overall feature context vector I, inputting the spliced driving intention probability matrix and the spliced overall feature context vector I into a decoder, and finally outputting the trajectory distribution probability of the vehicle; through combination of Transformer and GNN, not only can the dynamic state of a single vehicle be understood, but also the relative position and motion relation between vehicles can be understood, so that the vehicle trajectory can be predicted more accurately.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a Transformer-based method for predicting the trajectory of a networked autonomous driving vehicle. Background Art

[0002] Traffic safety has always been a key concern of cities. In recent years, my country has taken a series of strong measures to prevent traffic accidents, and the accident rate has gradually decreased. However, due to my country's large population base and a large proportion of motorized travel, the number of casualties in road traffic accidents is still larger than that in developed countries. Every year, the number of casualties and property losses caused by traffic accidents is countless. How to improve traffic safety issues in urban road operations has always been a difficult problem faced by many large cities.

[0003] In recent years, the technology of intelligent connected vehicles has developed rapidly. In an intelligent connected environment, vehicles can exchange information with each other, with roadside facilities and with traffic participants. Autonomous driving based on an intelligent connected environment has been proven to effectively improve traffic safety. Trajectory prediction is an essential part of autonomous driving. Whether the prediction result is accurate affects vehicle route planning, so vehicle trajectory prediction is a good research direction.

[0004] Most current trajectory prediction models use a framework based on long short-term memory (LSTM) and gated recurrent neural network (GRU), and use a self-loop structure to memorize the features of historical data. Although LSTM and GRU can achieve long-term sequence dependency, LSTM still does not work well for sequences that are too long.

[0005] Based on this, the present invention aims to provide a new vehicle trajectory prediction method, which needs to solve the problem that the trajectory prediction model is inconvenient to extract features for too long sequences. Summary of the invention

[0006] In order to make up for the shortcomings of the prior art problems, the purpose of the present invention is to provide a Transformer-based networked autonomous driving vehicle trajectory prediction method, which uses Transformer to extract vehicle context information, and GNN is used to build interaction relationships between vehicles and update vehicle context vectors. The self-attention mechanism of the Transformer model can process sequence data without being restricted by sequence length, and effectively capture long-distance dependencies. Combining it with GNN can not only understand the dynamics of a single vehicle, but also understand the relative positions and motion relationships between vehicles, thereby more accurately predicting vehicle trajectories.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A method for predicting the trajectory of a connected autonomous driving vehicle based on a Transformer, characterized by comprising the following steps:

[0009] (1) Acquiring vehicle driving data, which includes vehicle location information, motion information, and operating condition information;

[0010] (2) Preprocessing the acquired vehicle driving data to obtain the time series information H of each vehicle;

[0011] (3) Input the time series information H of each vehicle into the Transformer encoder to obtain the context vector of each vehicle;

[0012] (4) Input the context vector of each vehicle into the graph neural network, update the context vector of the vehicle, and combine the context vectors of all vehicles into the overall feature context vector I;

[0013] (5) Using the random forest algorithm, the driving intention probability matrix of the current vehicle is obtained according to the updated context vector of the vehicle;

[0014] (6) The driving intention probability matrix of the current vehicle is concatenated with the overall feature context vector I and input into the decoder, which finally outputs the trajectory distribution probability of the vehicle.

[0015] The preprocessing in step (2) of the present invention is to convert the vehicle position information, motion information and working condition information in step (1) into the following representation:

[0016]

[0017]

[0018]

[0019] in, is the characteristic information of the rth vehicle at time t; is the motion information of the rth vehicle at time t; is the working condition information of the rth vehicle at time t; is the horizontal and vertical position coordinates of the vehicle, is the absolute speed of the rth vehicle at time t; is the acceleration of the rth vehicle at time t; is the steering angle of the rth vehicle at time t; is the throttle position of the rth vehicle at time t; is the clutch position of the rth vehicle at time t; The right turn signal flag at time t, which is 1 if the car turns on the right turn signal, otherwise it is 0; The left turn signal flag at time t is 1 if the car turns on the left turn signal, otherwise it is 0;

[0020] The data are then processed in sequence through a word embedding layer, a normalization algorithm layer, and a data whitening layer to unify the data to a specified mean and standard deviation.

[0021] In the present invention, the Transformer encoder includes a multi-head self-attention layer, a fully connected layer, and a position encoding layer; after the time series information of the vehicle is sequentially input into the multi-head self-attention layer, the fully connected layer, and the position encoding layer of the encoder, the context vector of each vehicle is obtained.

[0022] In the present invention, the overall feature context vector I is obtained by using a graph neural network, and the specific process is as follows:

[0023] Construct the graph structure: define vehicles as nodes of the graph; construct edges based on the distance between vehicles and the relationship between relative speeds; for vehicles r and s, calculate the distance: Calculate the relative velocity: v rs =|v r -v s |; Set the distance threshold to d threshold , the relative velocity threshold is v threshold , traverse all vehicle pairs (r, s), and determine whether to establish an edge based on the threshold. If d rs <d threshold And v rs <v threshold , then an edge is established between vehicle r and vehicle s; based on the above judgment, the adjacency matrix A is constructed:

[0024]

[0025] Consider each car establishing an edge with itself, that is, A rr =1 to ensure that the node can retain its own information;

[0026] Graph convolution: Apply graph convolution operation to each node. Assuming there are k layers of graph convolution, in each layer, the new feature representation of node v is calculated by the weighted sum of the features of its neighboring nodes:

[0027]

[0028] in, is the feature representation of node v in the kth layer; N(v) is the set of neighbor nodes of node v; d u is the degree of neighbor node u, which refers to the number of edges directly connected to node u; refers to the feature representation of the neighbor node u at the k-1 layer; W (k)is the weight matrix used in the kth layer; b (k) is the bias vector at the kth layer; σ is the activation function ReLU;

[0029] Feature aggregation: After k layers of graph convolution, a new feature representation of each node is obtained; the node features are aggregated using global average pooling in the last layer to obtain the overall representation of the graph;

[0030] Context vector update: The final feature representation of each node Combined with the features of other nodes, we get the updated context vector: n is the total number of vehicles;

[0031] The updated context vector As the final context representation of each vehicle at time step t; the context vectors C of all vehicles are combined into the overall feature context vector I: I = [C1, C2, ..., C n ].

[0032] In the present invention, the updated context vector As input features, the labels of driving intentions were used as targets, and a random forest classifier was trained using Python’s scikit-learn library;

[0033] The probability of driving intention is predicted by the trained random forest classifier; the output driving intention probability matrix is ​​M = {ω1, ω2, ω3, ω4, ω5, ω6}, where ω1→[1, 0, 0, 0, 0], ω2→[0, 1, 0, 0, 0], ω3→[0, 0, 1, 0, 0, 0], ω4→[0, 0, 0, 1, 0, 0], ω5→[0, 0, 0, 0, 1, 0], ω6→[0, 0, 0, 0, 0, 1], ω1~ω6 represent changing lanes to the left, going straight, changing lanes to the right, changing lanes to the left under deceleration, going straight under deceleration, and changing lanes to the right under deceleration, respectively; in M, only the value with the highest probability of 6 intentions is 1, and the rest are 0, indicating the predicted intention of the current driving behavior.

[0034] In the present invention, the decoder includes three sub-layers, namely, the encoding-decoding attention layer, the multi-head self-attention layer and the fully connected layer;

[0035] The encoder-decoder attention layer calculates the attention weights through the context vector I and the driving intention probability matrix M. Among them, Q is the query matrix, which comes from the decoder; K is the key matrix, which comes from the encoder (context vector); V is the value matrix, which comes from the encoder (context vector); dk is the dimension of the key; the specific parameter learning is obtained through back propagation during the training process;

[0036] The multi-head self-attention layer allows the model to focus on different parts of the input at the same time, increasing the expressive power of the model:

[0037] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O , h=8

[0038] head i =Attention(QW i Q , KW i K , VW i V )

[0039] Among them, W i Q , W i K , W i V is the weight matrix of each head; W O is the output weight matrix after the final concatenation; the specific weight matrix is ​​obtained by continuous updating through back propagation;

[0040] The fully connected layer maps the attention output to the target space and generates the final trajectory distribution probability P(O|I,M):

[0041] P(O|I,M)=softmax(W fc (output)+b fc ),

[0042] Among them, output is the output of the multi-head self-attention layer; W fc is the weight of the fully connected layer; b fc is the bias of the fully connected layer; weights and biases are continuously updated through back propagation during training until the model converges.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. The Transformer model used in the present invention has a self-attention mechanism, which enables the model to process sequence data without being restricted by sequence length and effectively capture long-distance dependencies. This is crucial for trajectory prediction models because the movement trajectory of vehicles often involves long-term sequence data. Traditional LSTM and GRU models may encounter the problem of gradient vanishing or exploding when processing long sequences, while Transformer effectively solves this problem through the self-attention mechanism, improving the accuracy and efficiency of long sequence feature extraction.

[0045] 2. In the present invention, Transformer is used to extract the context information of the vehicle, while GNN is used to build the interaction relationship between vehicles and update the context vector of the vehicle. This combination enables the model to understand not only the dynamics of a single vehicle, but also the relative position and motion relationship between vehicles, thereby more accurately predicting vehicle trajectories. The introduction of GNN also helps the model's performance in processing non-Euclidean structured data, such as the relative position and motion state of vehicles on the road, which are difficult to capture with traditional sequence models.

[0046] 3. In the present invention, random forest is used to predict the driving intention of the vehicle, which helps the model understand the possible future action direction of the vehicle, such as changing lanes, going straight or slowing down, etc. The driving intention probability matrix M is concatenated with the overall feature context vector I and input into the decoder, so that the model can take into account the driving intention of the vehicle when predicting the trajectory, thereby improving the accuracy and reliability of trajectory prediction.

[0047] 4. The multi-model fusion method of the present invention makes full use of the advantages of each model. The combination of Transformer's ability to process sequence data, GNN's ability to process spatial relationships, and random forest's ability to process classification tasks enhances the trajectory prediction model in multiple dimensions. This combination also improves the generalization ability of the model because it can capture and learn the characteristics of the data from different angles, making the model more robust in the face of complex and changing traffic environments. In addition, this fusion method also helps to improve the interpretability of the model because each model can provide specific information about its predictions, which helps to understand the model's decision-making process, thereby better adjusting and optimizing the model in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0050] like Figure 1 As shown, the present invention discloses a Transformer-based networked autonomous driving vehicle trajectory prediction method, comprising the following steps:

[0051] Step 1: Obtain the vehicle driving data, including vehicle location information, motion information and working condition information.

[0052] Step 2: Preprocess the acquired vehicle driving data to obtain the time series information H of each vehicle.

[0053] Convert the acquired vehicle position information, motion information and working condition information into a representation that can be input into the neural network:

[0054]

[0055]

[0056]

[0057] in, is the characteristic information of the rth vehicle at time t; is the motion information of the rth vehicle at time t; is the operating condition information of the rth vehicle at time t. is the horizontal and vertical position coordinates of the vehicle, is the absolute speed of the rth vehicle at time t; is the acceleration of the rth vehicle at time t; is the steering angle of the rth vehicle at time t; is the throttle position of the rth vehicle at time t; is the clutch position of the rth vehicle at time t; The right turn signal flag at time t, which is 1 if the car turns on the right turn signal, otherwise it is 0; The left turn signal flag at time t is 1 if the car turns on the left turn signal, otherwise it is 0;

[0058] The data are then processed in sequence through a word embedding layer, a normalization algorithm layer, and a data whitening layer to unify the data to a specified mean and standard deviation.

[0059] Formula (1) defines the input of the trajectory prediction model First, use the embedding layer to convert the input into a floating-point vector s', and use the zero-phase component analysis data whitening layer to perform data whitening to obtain s" (t) .

[0060] The process of solving ZCA whitening is: zero mean each dimension of s', and the result is recorded as The covariance matrix of the data is Among them, m is the number of samples.

[0061] Perform eigenvalue decomposition on the covariance matrix C, that is, where C = UΛU T , U is the eigenvector matrix of C, and Λ is the eigenvalue matrix of C.

[0062] The definition of PCA whitening is In the formula, is the data after PCA whitening processing, It means that the data on each principal axis is divided by the square root of the corresponding eigenvalue. ZCA whitening is to multiply the PCA whitened data by the eigenvector matrix U and transform it back to the original space, that is,

[0063] In summary, feature information After the embedding layer and the whitening layer, we get s" (t) .

[0064] Step 3: Input the time series information H of each vehicle into the Transformer encoder to obtain the context vector of each vehicle.

[0065] The Transformer encoder consists of a multi-head self-attention layer, a fully connected layer, and a position encoding layer. The time series information of the vehicle is sequentially input into the multi-head self-attention layer, the fully connected layer, and the position encoding layer of the encoder to obtain the context vector of each vehicle.

[0066] Step 4: Input the context vector of each vehicle into the graph neural network, update the vehicle's context vector, and combine the context vectors of all vehicles into an overall feature context vector I.

[0067] Construct the graph structure: define vehicles as nodes of the graph; construct edges based on the distance between vehicles and the relationship between relative speeds; for vehicles r and s, calculate the distance: Calculate the relative velocity: v rs =|v r -v s |; Set the distance threshold to d threshold , the relative velocity threshold is v threshold , traverse all vehicle pairs (r, s), and determine whether to establish an edge based on the threshold. If d rs <d threshold And v rs <v threshold , then an edge is established between vehicle r and vehicle s; based on the above judgment, the adjacency matrix A is constructed:

[0068]

[0069] Consider each car establishing an edge with itself, that is, A rr =1 to ensure that the node can retain its own information;

[0070] Graph convolution: Apply graph convolution operation to each node. Assuming there are k layers of graph convolution, in each layer, the new feature representation of node v is calculated by the weighted sum of the features of its neighboring nodes:

[0071]

[0072] in, is the feature representation of node v in the kth layer; N(v) is the set of neighbor nodes of node v; d u is the degree of neighbor node u, which refers to the number of edges directly connected to node u; refers to the feature representation of the neighbor node u at the k-1 layer; W (k) is the weight matrix used in the kth layer; b (k) is the bias vector at the kth layer; σ is the activation function ReLU;

[0073] Feature aggregation: After k layers of graph convolution, a new feature representation of each node is obtained; the node features are aggregated using global average pooling in the last layer to obtain the overall representation of the graph;

[0074] Context vector update: The final feature representation of each node Combined with the features of other nodes, we get the updated context vector: n is the total number of vehicles;

[0075] The updated context vector As the final context representation of each vehicle at time step t; the context vectors C of all vehicles are combined into the overall feature context vector I: I = [C1, C2, …, C n ].

[0076] Step 5: Use the random forest algorithm to obtain the driving intention probability matrix M of the current vehicle based on the updated context vector of the vehicle.

[0077] The updated context vector As input features, the labels of driving intentions were used as targets, and a random forest classifier was trained using Python’s scikit-learn library;

[0078] The probability of driving intention is predicted by the trained random forest classifier; the output driving intention probability matrix is ​​M = {ω1, ω2, ω3, ω4, ω5, ω6}, where ω1→[1, 0, 0, 0, 0], ω2→[0, 1, 0, 0, 0], ω3→[0, 0, 1, 0, 0, 0], ω4→[0, 0, 0, 1, 0, 0], ω5→[0, 0, 0, 0, 1, 0], ω6→[0, 0, 0, 0, 0, 1], ω1~ω6 represent changing lanes to the left, going straight, changing lanes to the right, changing lanes to the left under deceleration, going straight under deceleration, and changing lanes to the right under deceleration, respectively; in M, only the value with the highest probability of 6 intentions is 1, and the rest are 0, indicating the predicted intention of the current driving behavior.

[0079] Step 6: Concatenate the current vehicle's driving intention probability matrix with the overall feature context vector I and input it into the decoder to finally output the vehicle's trajectory distribution probability.

[0080] The decoder consists of three sub-layers, namely the encoding-decoding attention layer, the multi-head self-attention layer and the fully connected layer;

[0081] The encoder-decoder attention layer calculates the attention weights through the context vector I and the driving intention probability matrix M. Among them, Q is the query matrix, which comes from the decoder; K is the key matrix, which comes from the encoder (context vector); V is the value matrix, which comes from the encoder (context vector); dk is the dimension of the key; the specific parameter learning is obtained through back propagation during the training process;

[0082] The multi-head self-attention layer allows the model to focus on different parts of the input at the same time, increasing the expressive power of the model:

[0083] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O , h=8

[0084] head i =Attention(QW i Q , KW i K , VW i V )

[0085] Among them, W i Q , W i K , W i V is the weight matrix of each head; W O is the output weight matrix after the final concatenation; the specific weight matrix is ​​obtained by continuously updating through back propagation;

[0086] The fully connected layer maps the attention output to the target space and generates the final trajectory distribution probability P(O|I,M):

[0087] P(O|I,M)=softmax(W fc (output)+b fc ),

[0088] Among them, output is the output of the multi-head self-attention layer; W fc is the weight of the fully connected layer; b fcis the bias of the fully connected layer; weights and biases are continuously updated through back propagation during training until the model converges.

[0089] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A Transformer-based connected autonomous vehicle trajectory prediction method, characterized in that: The following steps are involved: (1) Acquiring vehicle driving data, which includes vehicle location information, motion information, and operating condition information; (2) Preprocessing the acquired vehicle driving data to obtain the time series information H of each vehicle; (3) Input the time series information H of each vehicle into the Transformer encoder to obtain the context vector of each vehicle; (4) Input the context vector of each vehicle into the graph neural network, update the context vector of the vehicle, and combine the context vectors of all vehicles into the overall feature context vector I; (5) Using the random forest algorithm, the driving intention probability matrix of the current vehicle is obtained according to the updated context vector of the vehicle; (6) The driving intention probability matrix of the current vehicle is concatenated with the overall feature context vector I and input into the decoder, which finally outputs the trajectory distribution probability of the vehicle.

2. The method for predicting the trajectory of a connected autonomous driving vehicle based on Transformer according to claim 1, characterized in that: The preprocessing in step (2) is to convert the vehicle position information, motion information and working condition information in step (1) into the following representation: in, is the characteristic information of the rth vehicle at time t; is the motion information of the rth vehicle at time t; is the working condition information of the rth vehicle at time t; is the horizontal and vertical position coordinates of the vehicle, is the absolute speed of the rth vehicle at time t; is the acceleration of the rth vehicle at time t; is the steering angle of the rth vehicle at time t; is the throttle position of the rth vehicle at time t; is the clutch position of the rth vehicle at time t; The right turn signal flag at time t, which is 1 if the car turns on the right turn signal, otherwise it is 0; The left turn signal flag at time t is 1 if the car turns on the left turn signal, otherwise it is 0; The data are then processed in sequence through a word embedding layer, a normalization algorithm layer, and a data whitening layer to unify the data to a specified mean and standard deviation.

3. The method for predicting the trajectory of a connected autonomous driving vehicle based on Transformer according to claim 1, characterized in that: The Transformer encoder includes a multi-head self-attention layer, a fully connected layer, and a position encoding layer; the time series information of the vehicle is sequentially input into the multi-head self-attention layer, the fully connected layer, and the position encoding layer of the encoder to obtain the context vector of each vehicle.

4. The method for predicting the trajectory of a connected autonomous driving vehicle based on Transformer according to claim 1, characterized in that: The overall feature context vector is obtained using the graph neural network. The specific process is as follows: Construct the graph structure: define vehicles as nodes of the graph; construct edges based on the distance between vehicles and the relationship between relative speeds. For vehicles r and s, calculate the distance: Calculate the relative velocity: v rs =|v r -v s |; Set the distance threshold to d threshold , the relative velocity threshold is v threshold , traverse all vehicle pairs (r, s), and determine whether to establish an edge based on the threshold. If d rs <d threshold And v rs <v threshold , then an edge is established between vehicle r and vehicle s; based on the above judgment, the adjacency matrix A is constructed: Consider each car establishing an edge with itself, that is, A rr =1 to ensure that the node can retain its own information; Graph convolution: Apply graph convolution operation to each node. Assuming there are k layers of graph convolution, in each layer, the new feature representation of node v is calculated by the weighted sum of the features of its neighboring nodes: in, is the feature representation of node v in the kth layer; N(v) is the set of neighbor nodes of node v; d u is the degree of neighbor node u, which refers to the number of edges directly connected to node u; refers to the feature representation of the neighbor node u at the k-1 layer; W (k) is the weight matrix used in the kth layer; b (k) is the bias vector at the kth layer; σ is the activation function ReLU; Feature aggregation: After k layers of graph convolution, a new feature representation of each node is obtained; the node features are aggregated using global average pooling in the last layer to obtain the overall representation of the graph; Context vector update: The final feature representation of each node Combined with the features of other nodes, we get the updated context vector: n is the total number of vehicles; The updated context vector As the final context representation of each vehicle at time step t; the context vectors C of all vehicles are combined into the overall feature context vector I: I = [C1, C2, …, C n ].

5. The method for predicting the trajectory of a connected autonomous driving vehicle based on Transformer according to claim 4, characterized in that: The updated context vector As input features, the labels of driving intentions were used as targets, and a random forest classifier was trained using Python’s scikit-learn library; The probability of driving intention is predicted by the trained random forest classifier, and the output driving intention probability matrix is ​​M = {ω1,ω2,ω3,ω4,ω5,ω6}, where ω1→[1,0,0,0,0,0], ω2→[0,1,0,0,0,0], ω3→[0,0,1,0,0,0], ω4→[0,0,0,1,0,0], ω5→[0,0,0,0,1,0], ω6→[0,0,0,0,0,1], ω1~ω6 represent changing lanes to the left, going straight, changing lanes to the right, changing lanes to the left under deceleration, going straight under deceleration, and changing lanes to the right under deceleration, respectively; in M, only the value with the highest probability of 6 intentions is 1, and the rest are 0, indicating the predicted intention of the current driving behavior.

6. The method for predicting the trajectory of a connected autonomous driving vehicle based on Transformer according to claim 1, characterized in that: The decoder consists of three sub-layers, namely the encoding-decoding attention layer, the multi-head self-attention layer and the fully connected layer; The encoder-decoder attention layer calculates the attention weights through the context vector I and the driving intention probability matrix M. Among them, Q is the query matrix, which comes from the decoder; K is the key matrix; V is the value matrix; d k is the dimension of the key; the specific parameter learning is obtained through back propagation during the training process; The multi-head self-attention layer allows the model to focus on different parts of the input at the same time, increasing the expressive power of the model: MultiHead(Q,K,V)=Concat(head1,head2,…,head h )w o ,h=8 Among them, W i Q , W i K , W i V is the weight matrix of each head; W O is the output weight matrix after the final concatenation; the specific weight matrix is ​​obtained by continuous updating through back propagation; The fully connected layer maps the attention output to the target space and generates the final trajectory distribution probability P(O|I,M): P(OI,M)=softmax(W fc (output)+b fc ), Among them, output is the output of the multi-head self-attention layer; W fc is the weight of the fully connected layer; b fc is the bias of the fully connected layer; weights and biases are continuously updated through back propagation during training until the model converges.

Images