Trajectory prediction method for networked automatic driving vehicle based on transformer
By combining Transformer, GNN and Random Forest models, the problem of inconvenient long sequence feature extraction in existing trajectory prediction models is solved, and more accurate and robust vehicle trajectory prediction is achieved.
Patent Information
- Application Number
- CN202411647092.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-11-28
- Estimated Expiration
- 2044-11-18
AI Technical Summary
Existing trajectory prediction models are inconvenient for feature extraction when dealing with long sequences, especially the LSTM model, which performs poorly in long sequences and is difficult to accurately predict vehicle trajectories.
The Transformer model is used to extract vehicle context information, and GNN is combined to construct vehicle interaction relationships. The random forest algorithm is used to predict driving intentions, and the accuracy of trajectory prediction is improved by multi-model fusion.
Transformer's self-attention mechanism effectively captures long-distance dependencies, GNN understands the relationships between vehicles, random forest predicts driving intentions, and multi-model fusion improves the accuracy and robustness of trajectory prediction.
Smart Images
Figure CN119598316B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of automatic driving, and in particular to a trajectory prediction method for networked automatic driving vehicles based on a Transformer. BACKGROUND
[0002] Traffic safety has always been a major concern for cities, and in recent years China has taken a series of strong measures to prevent traffic accidents, and the accident rate has gradually decreased. However, due to the large population base and high proportion of motorized travel in China, the number of road traffic accident casualties is still large compared with developed countries, and the personal injury and property loss caused by traffic accidents every year is incalculable. How to improve the traffic safety problem in urban road operation has always been a difficult problem faced by many big cities.
[0003] In recent years, intelligent networked vehicle technology has developed rapidly, and in the intelligent networked environment, vehicles, roadside facilities and traffic participants can realize information interaction, and automatic driving relying on the intelligent networked environment has been proven to effectively improve traffic safety. Trajectory prediction is an essential part of realizing automatic driving, and the accuracy of the prediction result affects vehicle route planning, so vehicle trajectory prediction is a good research direction.
[0004] Most current trajectory prediction models use a framework based on long short-term memory (LSTM) and gated recurrent neural networks (GRU), and use self-loop structures to remember historical data features. Although LSTM and GRU can achieve long-term dependencies of sequences, for too long sequences, the effect of LSTM is still not good.
[0005] Therefore, the present application aims to provide a new vehicle trajectory prediction method, which solves the problem that trajectory prediction models are not convenient for extracting features of too long sequences. SUMMARY
[0006] In order to make up for the shortcomings of the prior art, the present application aims to provide a trajectory prediction method for networked automatic driving vehicles based on a Transformer, which uses a Transformer to extract the context information of a vehicle, and a GNN to construct the interaction relationship between vehicles and update the context vector of the vehicle. The self-attention mechanism of the Transformer model can process sequence data without being limited by the length of the sequence, effectively capturing long-distance dependencies, and combining it with the GNN can not only understand the dynamics of a single vehicle, but also understand the relative position and motion relationship between vehicles, thereby more accurately predicting the trajectory of the vehicle.
[0007] To achieve the above-mentioned purpose, the present application provides the following technical solutions:
[0008] A trajectory prediction method for networked automatic driving vehicles based on a Transformer, characterized by the following steps:
[0009] (1) Obtain vehicle driving data, which includes vehicle position information, motion information and working condition information;
[0010] (2) Preprocess the obtained vehicle driving data to obtain the time series information H of each vehicle;
[0011] (3) Input the time series information H of each vehicle into the Transformer encoder to obtain the context vector of each vehicle;
[0012] (4) Input the context vector of each vehicle into the graph neural network to update the context vector of the vehicle, and combine the context vectors of all vehicles into an overall feature context vector I;
[0013] (5) Use the random forest algorithm to obtain the driving intention probability matrix of the current vehicle according to the updated context vector of the vehicle;
[0014] (6) Concatenate the driving intention probability matrix of the current vehicle with the overall feature context vector I, and input it into the decoder to finally output the trajectory distribution probability of the vehicle.
[0015] The preprocessing in step (2) of the present application is to convert the vehicle position information, motion information and working condition information in step (1) into the following representation:
[0016]
[0017]
[0018]
[0019] wherein, is the feature information of the rth vehicle at time t; is the motion information of the rth vehicle at time t; is the working condition information of the rth vehicle at time t; is the lateral and longitudinal position coordinates of the vehicle, is the absolute speed of the rth vehicle at time t; is the acceleration of the rth vehicle at time t; is the steering angle of the rth vehicle at time t; is the throttle position of the rth vehicle at time t; is the clutch position of the rth vehicle at time t; is the right turn signal flag of the rth vehicle at time t, which is 1 if the vehicle turns on the right turn signal, otherwise it is 0; For t time left turn light flag bit, if the car turns left, it is 1, otherwise it is 0;
[0020] Then, the data is uniformly processed to a specified mean and standard deviation through a word embedding layer, a normalization algorithm layer and a data whitening layer in sequence.
[0021] In the application, the Transformer encoder comprises a multi-head self-attention layer, a full connection layer and a position encoding layer; the time sequence information of the vehicle is sequentially input into the multi-head self-attention layer, the full connection layer and the position encoding layer of the encoder, and the context vector of each vehicle is obtained
[0022] In the application, the whole feature context vector I is obtained by using a graph neural network, and the specific process is as follows:
[0023] Building a graph structure: defining the vehicle as a node of the graph; building an edge according to the distance and relative speed between the vehicles; for vehicles r and s, the distance is calculated as: The relative speed is calculated as: v rs = |v r -v s |; the distance threshold is set as d threshold , the relative speed threshold is set as v threshold , all vehicle pairs (r, s) are traversed, and it is judged whether an edge is established according to the threshold; if d rs <d threshold and v rs <v threshold , an edge is established between the vehicle r and the vehicle s; according to the above judgment, the adjacency matrix A is constructed:
[0024]
[0025] It is considered that each vehicle establishes an edge with itself, that is, A rr = 1, so that the node can retain its own information;
[0026] Graph convolution: a graph convolution operation is applied to each node; assuming that there are k layers of graph convolution, in each layer, the new feature representation of the node v is calculated by weighted summation of the feature representations of its neighbor nodes:
[0027]
[0028] Wherein, is the feature representation of the node v in the kth layer; N(v) is the neighbor node set of the node v; d u is the degree of the neighbor node u, that is, the number of edges directly connected to the node u; is the feature representation of the neighbor node u in the k-1th layer; W (k)is the weight matrix used in the k-th layer; b (k) is the bias vector in the k-th layer; σ is the activation function ReLU;
[0029] Feature aggregation: After k-layer graph convolution, the new feature representation of each node is obtained; the node features are aggregated by global average pooling in the last layer to obtain the overall representation of the graph;
[0030] Context vector update: the final feature representation of each node is combined with the features of other nodes to obtain the updated context vector: n is the total number of vehicles;
[0031] The updated context vector is taken as the final context representation of each vehicle at time step t; the context vectors C of all vehicles are combined into an overall feature context vector I: I=[C1,C2,...,C n ].
[0032] In the present application, the updated context vector is taken as the input feature, and the label of the driving intention is taken as the target to train a random forest classifier using the scikit-learn library of Python;
[0033] The trained random forest classifier is used to predict the probability of driving intention; the output driving intention probability matrix is M={ω1,ω2,ω3,ω4,ω5,ω6},wherein ω1→[1,0,0,0,0,0],ω2→[0,1,0,0,0,0],ω3→[0,0,1,0,0,0],ω4→[0,0,0,1,0,0],ω5→[0,0,0,0,1,0],ω6→[0,0,0,0,0,1],ω1~ω6 respectively represent left lane change, straight, right lane change, deceleration left lane change, deceleration straight, deceleration right lane change; only one value of 1 in M with the highest probability of 6 intentions, and the rest are 0, indicating the predicted intention of the current driving behavior.
[0034] In the present application, the decoder includes 3 sub-layers, which are encoding-decoding attention layer, multi-head self-attention layer and fully connected layer;
[0035] The encoding-decoding attention layer calculates the attention weight wherein Q is the query matrix, from the decoder; K is the key matrix, from the encoder (context vector); V is the value matrix, from the encoder (context vector); dk is the dimension of the key; specific parameter learning is obtained through back propagation in the training process;
[0036] The multi-head self-attention layer allows the model to focus on different parts of the input simultaneously, increasing the expressive power of the model:
[0037] MultiHead(Q, K, V) = Concat(head1, head2, …, head h )W O , h = 8
[0038] head i = Attention(QW i Q , KW i K , VW i V )
[0039] Where W i Q , W i K , W i V are weight matrices for each head; W O is the output weight matrix after concatenation; the specific weight matrix is obtained by continuously updating through backpropagation;
[0040] The fully connected layer maps the attention output to the target space to generate the final trajectory distribution probability P(O|I, M):
[0041] P(O|I, M) = softmax(W fc (output) + b fc ),
[0042] Where output is the output of the multi-head self-attention layer; W fc is the weight of the fully connected layer; b fc is the bias of the fully connected layer; the weight and bias are continuously updated through backpropagation during the training process until the model converges.
[0043] Compared with the prior art, the present application has the following advantages:
[0044] 1. The Transformer model used in the present application has a self-attention mechanism, which allows the model to process sequence data without being limited by sequence length, effectively capturing long-range dependencies. This is crucial for trajectory prediction models, as vehicle trajectories often involve long sequences of data. Traditional LSTM and GRU models may encounter problems of gradient vanishing or explosion when processing excessively long sequences, while Transformer effectively solves this problem through self-attention mechanism, improving the accuracy and efficiency of long sequence feature extraction.
[0045] 2、In the present application, the Transformer is used to extract the context information of the vehicle, while the GNN is used to construct the interaction relationship between vehicles and update the context vector of the vehicle. This combination enables the model not only to understand the dynamics of individual vehicles, but also to understand the relative position and motion relationship between vehicles, so as to more accurately predict the vehicle trajectory. The introduction of GNN also helps the model to perform better when dealing with non-Euclidean structure data, such as the relative position and motion state of vehicles on the road, which is difficult for traditional sequence models to capture.
[0046] 3、In the present application, the random forest is used to predict the driving intention of the vehicle, which helps the model to understand the possible direction of action of the vehicle in the future, such as lane changing, straight driving or deceleration, etc. After the driving intention probability matrix M is concatenated with the overall feature context vector I and input into the decoder, the model can consider the driving intention of the vehicle when predicting the trajectory, so as to improve the accuracy and reliability of the trajectory prediction.
[0047] 4、The multi-model fusion method in the present application makes full use of the advantages of each model. The ability of the Transformer to process sequence data, the ability of the GNN to process spatial relationships, and the ability of the random forest to process classification tasks are combined, which enhances the trajectory prediction model in multiple dimensions. This combination also improves the generalization ability of the model, because it can capture and learn the features of the data from different perspectives, making the model more robust in the face of complex and variable traffic environments. In addition, this fusion method also helps to improve the interpretability of the model, because each model can provide specific information about its prediction, which helps to understand the decision-making process of the model, so as to better adjust and optimize the model in practical applications. BRIEF DESCRIPTION OF DRAWINGS
[0048] Figure 1 The flowchart of the method of the present application. DETAILED DESCRIPTION
[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all.
[0050] As Figure 1 shown, the present application discloses a trajectory prediction method for networked autonomous vehicles based on Transformer, which includes the following steps:
[0051] Step 1: Obtain the vehicle driving data, which specifically includes vehicle position information, motion information and working condition information.
[0052] Step two: preprocess the obtained vehicle driving data to obtain the time series information H of each vehicle.
[0053] Convert the obtained vehicle position information, motion information and working condition information into a representation form that can be input into the neural network:
[0054]
[0055]
[0056]
[0057] wherein, is the feature information of the rth vehicle at time t; is the motion information of the rth vehicle at time t; is the working condition information of the rth vehicle at time t. is the lateral and longitudinal position coordinates of the vehicle, is the absolute speed of the rth vehicle at time t; is the acceleration of the rth vehicle at time t; is the steering angle of the rth vehicle at time t; is the throttle position of the rth vehicle at time t; is the clutch position of the rth vehicle at time t; is the right turn signal flag at time t, which is 1 if the vehicle is turning right, otherwise 0; is the left turn signal flag at time t, which is 1 if the vehicle is turning left, otherwise 0;
[0058] Then, the data is uniformly processed to a specified mean and standard deviation through a word embedding layer, a normalization algorithm layer and a data whitening layer in turn.
[0059] Equation (1) defines the input of the trajectory prediction model First, the input is converted to a floating-point number vector s' using the embedding layer, and data whitening processing is performed using the zero-phase component analysis data whitening layer to obtain s" (t) .
[0060] The process of solving ZCA whitening is to zero-mean each dimension of s', and the result is denoted as The covariance matrix of the data is wherein, m is the number of samples.
[0061] The covariance matrix C is subjected to eigenvalue decomposition, that is, C = UΛU T , U is the eigenvector matrix of C, and Λ is the eigenvalue matrix of C.
[0062] The definition of principal component analysis (PCA) whitening is wherein, For the data after PCA whitening processing, ZCA whitening is to transform the data after PCA whitening back to the original space by multiplying the feature vector matrix U, that is,
[0063] In summary, the feature information After the embedding layer and the whitening layer processing, s" is obtained (t) .
[0064] Step three, input the time series information H of each vehicle into the Transformer encoder to obtain the context vector of each vehicle.
[0065] The Transformer encoder includes a multi-head self-attention layer, a fully connected layer, and a position encoding layer; the time series information of the vehicle is input into the multi-head self-attention layer, the fully connected layer, and the position encoding layer of the encoder in turn, and the context vector of each vehicle is obtained
[0066] Step four, input the context vector of each vehicle into the graph neural network to update the context vector of the vehicle, and combine the context vectors of all vehicles into an overall feature context vector I.
[0067] Construct the graph structure: define the vehicle as the node of the graph; construct the edge according to the distance and the relationship of the relative speed between the vehicles; for vehicles r and s, calculate the distance: Calculate the relative speed: v rs = |v r -v s |; set the distance threshold as d threshold , the threshold of the relative speed as v threshold , traverse all vehicle pairs (r, s), and judge whether to establish an edge according to the threshold, if d rs <d threshold and v rs <v threshold , then an edge is established between the vehicle r and the vehicle s; according to the above judgment, the adjacency matrix A is constructed:
[0068]
[0069] Consider that each vehicle establishes an edge with itself, i.e. A rr = 1, to ensure that the node can retain its own information;
[0070] Graph convolution: apply the graph convolution operation to each node, assuming that there are k layers of graph convolution, in each layer, the new feature representation of the node v is calculated by weighted summation of the feature representations of its neighbor nodes:
[0071]
[0072] where, is the feature representation of node v in the k-th layer; N(v) is the set of neighbor nodes of node v; d u is the degree of neighbor node u, which refers to the number of edges directly connected to node u; is the feature representation of neighbor node u in the k-1-th layer; W (k) is the weight matrix used in the k-th layer; b (k) is the bias vector in the k-th layer; σ is the activation function ReLU;
[0073] Feature aggregation: After k-layer graph convolution, the new feature representation of each node is obtained; the node features are aggregated by global average pooling in the last layer to obtain the overall representation of the graph;
[0074] Context vector update: the final feature representation of each node is combined with the features of other nodes to obtain the updated context vector: n is the total number of vehicles;
[0075] The updated context vector is taken as the final context representation of each vehicle at time step t; the context vectors C of all vehicles are combined into an overall feature context vector I: I = [C1, C2, …, C n ].
[0076] Step five, using the random forest algorithm, according to the updated context vector of the vehicle, the driving intention probability matrix M of the current vehicle is obtained.
[0077] The updated context vector is taken as the input feature, and the label of driving intention is taken as the target, a random forest classifier is trained using the scikit-learn library of Python;
[0078] The probability of driving intention is predicted by the trained random forest classifier; the output driving intention probability matrix is M = {ω1, ω2, ω3, ω4, ω5, ω6}, where ω1→[1, 0, 0, 0, 0, 0], ω2→[0, 1, 0, 0, 0, 0], ω3→[0, 0, 1, 0, 0, 0], ω4→[0, 0, 0, 1, 0, 0], ω5→[0, 0, 0, 0, 1, 0], ω6→[0, 0, 0, 0, 0, 1], ω1~ω6 respectively represent left lane change, straight, right lane change, left lane change under deceleration, straight deceleration, and right lane change under deceleration; only one value of 1 in M represents the highest probability of the current driving behavior, and the remaining values are 0, indicating the predicted intention of the current driving behavior.
[0079] Step six, the driving intention probability matrix of the current vehicle is spliced with the overall feature context vector I and input into the decoder, and finally the trajectory distribution probability of the vehicle is output.
[0080] The decoder includes three sub-layers, namely, an encoding-decoding attention layer, a multi-head self-attention layer and a fully connected layer.
[0081] The encoding-decoding attention layer calculates the attention weight through the context vector I and the driving intention probability matrix M, wherein Q is a query matrix from the decoder, K is a key matrix from the encoder (context vector), V is a value matrix from the encoder (context vector), and dk is the dimension of the key; the specific parameter learning is obtained in the training process through back propagation;
[0082] The multi-head self-attention layer allows the model to focus on different parts of the input at the same time, increasing the expression ability of the model:
[0083] MultiHead(Q, K, V) = Concat(head1, head2, …, head h )W O , h = 8
[0084] head i = Attention(QW i Q , KW i K , VW i V )
[0085] wherein W i Q , W i K , W i V are weight matrices of each head; W O is the output weight matrix after splicing; the specific weight matrix is obtained by continuously updating through back propagation;
[0086] The fully connected layer maps the attention output to the target space to generate the final trajectory distribution probability P(O|I, M):
[0087] P(O|I, M) = softmax(W fc (output) + b fc ),
[0088] wherein output is the output of the multi-head self-attention layer; W fc is the weight of the fully connected layer; b fcBias for the fully connected layer; weights and biases are constantly updated during the training process through backpropagation until the model converges.
[0089] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes to the technical solutions and the inventive concept of the present application within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.
Claims
1.A trajectory prediction method for a connected and automated vehicle based on a Transformer, characterized in that, The method comprises the following steps: (1) obtaining vehicle driving data, the vehicle driving data comprising vehicle position information, motion information and working condition information; (2) preprocessing the obtained vehicle driving data to obtain time series information H of each vehicle; (3) inputting the time series information H of each vehicle into a Transformer encoder to obtain a context vector of each vehicle; (4) inputting the context vector of each vehicle into a graph neural network to update the context vector of the vehicle, and combining the context vectors of all vehicles into an overall feature context vector I; (5) using a random forest algorithm to obtain a driving intention probability matrix of the current vehicle according to the updated context vector of the vehicle; (6) inputting the driving intention probability matrix of the current vehicle and the overall feature context vector I after splicing into a decoder to finally output a trajectory distribution probability of the vehicle; The overall feature context vector is obtained by using the graph neural network, and the specific process is as follows: Build graph structure: define vehicles as nodes of the graph; build edges according to the distance between vehicles and the relationship of relative speed, for vehicles r and s, calculate the distance: Calculate relative speed: v rs = |v r -v s |; set the distance threshold as d threshold , the threshold of relative speed as v threshold , traverse all vehicle pairs (r, s), and judge whether to establish an edge according to the threshold, if d rs <d threshold and v rs <v threshold , then an edge is established between vehicle r and vehicle s; according to the above judgment, build the adjacency matrix A: Consider each vehicle to establish an edge with itself, i.e. A rr = 1 to ensure that the node is able to retain its own information; Graph convolution: apply a graph convolution operation to each node, assuming that there are k layers of graph convolution, in each layer, the new feature representation of node v is calculated by weighted summation of the feature representations of its neighbor nodes: where, is the feature representation of node v in the kth layer; N(v) is the set of neighbor nodes of node v; d u is the degree of neighbor node u, which refers to the number of edges directly connected to node u; is the feature representation of neighbor node u in the k-1th layer; W (k) is the weight matrix used in the kth layer; b (k) is the bias vector in the kth layer; σ is the activation function ReLU; Feature aggregation: after k layers of graph convolution, the new feature representation of each node is obtained; the node features are aggregated by global average pooling in the last layer to obtain the overall representation of the graph; Context vector update: Update the context vector for each node with the final feature representation of the node Combine the features of other nodes to get the updated context vector for the node: n is the total number of vehicles; The updated context vector as the final context representation for each vehicle at time step t; the context vectors C of all vehicles are combined into an overall feature context vector I: I = [C1, C2, …, C n ] The updated context vector As input features, the labels of driving intention as target, using Python's scikit-learn library to train a random forest classifier; Through the trained random forest classifier, the probability of the driving intention is predicted, and the output driving intention probability matrix is M={ω1,ω2,ω3,ω4,ω5,ω6}, wherein ω1→[1,0,0,0,0,0],ω2→[0,1,0,0,0,0],ω3→[0,0,1,0,0,0],ω4→[0,0,0,1,0,0],ω5→[0,0,0,0,1,0],ω6→[0,0,0,0,0,1],ω1~ω6 respectively represent left lane changing, straight driving, right lane changing, left lane changing under deceleration, straight driving under deceleration, and right lane changing under deceleration; only one value of 1 in M has the highest probability of 6 kinds of intentions, and the remaining values are 0, indicating the predicted intention of the current driving behavior; The decoder comprises three sub-layers, namely an encoding-decoding attention layer, a multi-head self-attention layer and a fully connected layer; The encoding-decoding attention layer calculates the attention weight by a context vector I and a driving intention probability matrix M, where Q is a query matrix from the decoder; K is a key matrix; V is a value matrix; d k is the dimension of the key; the specific parameter learning is obtained in the training process by back propagation; The multi-head self-attention layer allows the model to simultaneously focus on different parts of the input, increasing the expression ability of the model: MultiHead(Q, K, V) = Concat(head1, head2,..., head h )W O , h = 8 wherein, is the weight matrix for each head; W O is the output weight matrix after final splicing; the specific weight matrix is obtained by continuously updating through back propagation; The fully connected layer maps the attention output to the target space to generate the final trajectory distribution probability P(O|I,M): P(O|I, M) = softmax(W fc (output) + b fc ), wherein output is the output of the multi-head self-attention layer; W fc is the weight of the fully connected layer; b fc is the bias of the fully connected layer; the weight and the bias are constantly updated through backpropagation during the training process until the model converges. 2.The Transformer-based trajectory prediction method for networked autonomous vehicles according to claim 1, wherein, The preprocessing in step (2) converts the vehicle position information, motion information and working condition information in step (1) into the following representation: wherein, is the feature information of the rth vehicle at time t; is the motion information of the rth vehicle at time t; is the working condition information of the rth vehicle at time t; is the lateral and longitudinal position coordinates of the vehicle, is the absolute speed of the rth vehicle at time t; is the acceleration of the rth vehicle at time t; is the steering angle of the rth vehicle at time t; is the throttle position of the rth vehicle at time t; is the clutch position of the rth vehicle at time t; is the right turn signal flag at time t, which is 1 if the right turn signal is on, otherwise 0; is the left turn signal flag at time t, which is 1 if the left turn signal is on, otherwise 0; Then, the data is uniformly processed to a specified mean and standard deviation through a word embedding layer, a normalization algorithm layer and a data whitening layer in sequence. 3.The Transformer-based networked autonomous vehicle trajectory prediction method of claim 1, wherein, The Transformer encoder comprises a multi-head self-attention layer, a full connection layer, and a position encoding layer; time series information of the vehicles is sequentially input to the multi-head self-attention layer, the full connection layer, and the position encoding layer of the encoder, and a context vector of each vehicle is obtained