A vehicle trajectory prediction method based on transformer and LSTM

Abstract

The present application belongs to the technical field of automatic auxiliary driving, and particularly relates to a self-adaptive virtual lane line drawing method in a road environment without lane lines.The present application constructs a spatial undirected graph by taking the obtained target information and the target itself as nodes, converts time sequence information into a graph matrix form, and saves the time sequence thereof. The reconstructed LSTM model is used to process the graph matrix data, and the time sequence of the data is processed by using the cyclic nature of the LSTM. The spatial attention mechanism of the Transformer is used to extract effective features from the spatial graph matrix data and train the model. The inception_mixer module is used to continue processing high-frequency information and low-frequency information, and the effective features in the information are extracted again, thereby guaranteeing the accuracy and efficiency of the present method. The present method can efficiently and long-time predict the trajectories of surrounding vehicles, effectively guarantee the driving safety and road safety, and can be applied to an intelligent traffic system to improve the traffic safety and efficiency, assist route optimization and intersection prediction and the like.

Description

Technical Field

[0001] This invention belongs to the field of vehicle trajectory prediction technology, specifically relating to a vehicle trajectory prediction method based on Transformer and LSTM. Background Technology

[0002] With increasing demand for transportation, intelligent transportation systems (ITS) have become a focal point, and vehicle trajectory prediction is a crucial component of ITS. The purpose of vehicle trajectory prediction is to forecast vehicle routes over a given period to better support the decision-making of ITS systems. However, due to factors such as time variables, environmental variables, and individual variables, vehicle trajectory prediction is highly complex and variable.

[0003] In existing technologies, vehicle trajectory prediction mainly relies on single-model-based algorithms, such as Bayesian networks and Hidden Markov Models. These methods can effectively predict short-term vehicle trajectories, but due to the large number of variables, their performance in predicting long-term trajectories is poor, and they are easily affected by external environmental interference, impacting prediction accuracy. Current deep learning methods can predict vehicle trajectories over long periods, but their accuracy can still be improved. Summary of the Invention

[0004] To address the technical problems existing in the prior art, this invention proposes a vehicle trajectory prediction method based on Transformer and LSTM. This method combines the multi-head attention mechanism of the Transformer model and the long-term memory capability of the LSTM model, and incorporates the Inception-Mixer mechanism to accurately capture high-frequency information in the undirected spatial graph, thereby accurately predicting the vehicle's trajectory.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0006] A vehicle trajectory prediction method based on Transformer and LSTM includes the following steps:

[0007] S1: The vehicle obtains the driving data of surrounding vehicles by interacting with them, including the lateral and longitudinal coordinates of the front center, speed and acceleration, and integrates the vehicle identification number into the data; the data is preprocessed into a 3D tensor using Python's built-in tools.

[0008] S2: Construct a graph data structure from the preprocessed data; first, use matrix multiplication to calculate the number of hops between every two nodes, then classify the relationships between nodes according to the maximum number of hops, and finally obtain an adjacency matrix;

[0009] S3: Create a custom LSTM model; process the graph matrix using Graph_Conv_Block and extract the available edge weights using a custom edge_importance_weighting function, then reconstruct the graph matrix data; finally, use a custom reshape function to synthesize the output and the original input.

[0010] S4: Create a custom Transformer model that includes a spatial attention mechanism; initialize the data of the upper layer of the encoder using Xavier, and then train it using a custom forward function to obtain the output;

[0011] S5: Use the Inception_mixer mechanism to process the data, extract high-frequency information from the data, and concatenate the obtained high-frequency information with low-frequency information to obtain a new output;

[0012] S6: Input the data back into the Transformer with spatial attention mechanism and use the GRU decoder to predict the location.

[0013] As a further preferred technical solution of the present invention, in S1, the driving data of surrounding vehicles is obtained by information interaction with surrounding vehicles. Assuming there are n objects in the past time period t, the original data is processed into a 3D tensor, and a quantization of 5 is set to represent the coordinates (x, y) and velocity (u, v) of an object at a time step, and all coordinates and velocities are standardized to the range (-1, 1).

[0014] Furthermore, as a preferred embodiment of the present invention, the specific steps of S2 include:

[0015] S2.1. Construct a spatial graph from the preprocessed data, setting the number of nodes to 100 and the maximum hop count to 1. Represent the graph as G. t ={V t E t}, the node is represented as Edges are represented as

[0016] S2.2 First, calculate the transit matrix of all nodes using matrix multiplication. Then, determine whether each node is connected by comparing the distance between each node to see if it is less than or equal to the maximum hop count, and obtain the adjacency matrix A. t The formula is expressed as:

[0017]

[0018] S2.3 Regularizing the adjacency matrix: First, calculate the degree matrix based on the sum of each row. Then, regularize the adjacency matrix using matrix multiplication. The formula is:

[0019] A t =G t ×A t .

[0020] As a further preferred technical solution of the present invention, in step S3, a custom LSTM network is created as the first layer network for processing data. Three Graph_Conv_Blocks and one BatchNorm2d are added to this model. This normalization layer is used to normalize the input. The spatial kernel size of each graph convolution layer is determined based on the maximum hop count and number of nodes in the spatial graph at different time intervals. The temporal kernel size is fixed, and the activation function is ReLU. The convolutional layer is used to extract important information from the graph matrix. In the forward function, the data obtained from the graph convolution blocks is computed forward, and the custom function edge_importance_weighting is used to extract node information that meets the edge weight requirements to obtain the output. The output is then concatenated with the original input to form a new matrix, which is then passed to the next layer network.

[0021] As a further preferred technical solution of the present invention, in step S4, a custom Transformer model is created by first spatially encoding the data and allowing data to be transmitted only between neighboring nodes in the attention mechanism; all parameters are iterated through in a loop, and parameters with a dimension greater than 1 are initialized using the Xavier initialization method; in the forward propagation, the output and hidden state are generated by using the GRU decoder through loop iteration, and the results are stored in the corresponding array positions; if the requirements of the teacher_force method are met, it is used as the next input; during the spatial interaction process, the message_passing function is used to process the hidden state and position mask to obtain the output.

[0022] Furthermore, as a preferred embodiment of the present invention, the specific steps of S4 include:

[0023] S4.1. Represent the output obtained in S3 as H, and use the encoding properties of Transformer itself to map the object at each time step t to...

[0024] S4.2. Spatial self-attention mechanism can be viewed as information being transferred on an undirected graph; at time step t, h t i (i = 1, 2, ..., n), representing the query vector, key vector, and value vector for each object, respectively:

[0025]

[0026]

[0027]

[0028] Define the message passed from vehicle i to vehicle j in the spatial graph as follows: The effect of time step t on adjacency matrix A t The attention mechanism is as follows:

[0029]

[0030] Let the set of neighboring vehicles of vehicle i be S={j|A t [i,j]=1}, then the spatial self-attention mechanism is used to obtain the multi-head attention data of vehicle i at time step t, and the data is forward-computed to obtain the output;

[0031] S4.3. Using the teacher_forcing mechanism, the output of the prior time step is used as input to repeat the process of the spatial self-attention mechanism.

[0032] As a further preferred technical solution of the present invention, step S5 inputs the output obtained in step S4, divides the input into several channels, and assigns them to the high-frequency branch and the low-frequency branch respectively; in the high-frequency branch, convolution and max pooling operations are used to extract high-frequency information; in the low-frequency branch, average pooling operation is first used, then spatial attention mechanism is used to process the data, and finally upsampling is performed; finally, the high-frequency result and the low-frequency result are directly concatenated together to obtain the final output.

[0033] Furthermore, as a preferred embodiment of the present invention, the specific steps of S5 include:

[0034] S5.1. Input the result obtained from S4 into another Transformer layer with the inception_mixer mechanism. The first branch X input undergoes a depthwise convolution with a kernel size of 1×R, followed by a depthwise convolution with a kernel size of R×1, where R is the number of graph nodes at time step t. The second branch X input first undergoes a MaxPooling operation to increase data features, followed by a depthwise convolution with a kernel size of R×R. These two branches are for high-frequency information extraction, and the formula is as follows:

[0035] C1=dwC R×1 (dwC 1×R (X))

[0036] C2=dwC R×R (MaxPool(X));

[0037] S5.2, the third branch X first performs an AveagePooling operation to reduce the data features, then uses an R×R depthwise convolution, and finally performs an upsample operation, calling an interpolation function to complete the features. The formula is:

[0038] P3=dwC R×R (AvgPool(X))

[0039] The process of connecting the feature matrices is as follows:

[0040] O=LayerNorm(Concat(C1,C2,P3))

[0041] Finally, spatial self-attention mechanism is used to train the obtained sequences O and X, and the two sequences are multiplied by the query vector matrix W respectively. q , key vector matrix W k Sum value vector matrix W v The formula is as follows:

[0042]

[0043] The final output is obtained.

[0044] As a further preferred embodiment of the present invention, step S6 performs the final Transformer training based on the data obtained in step S5, and uses the GRU decoder to obtain the final predicted position.

[0045] Furthermore, as a preferred embodiment of the present invention, the specific steps of S6 include:

[0046] S6.1. Input the output obtained in S5 into the Transformer layer as described in S4 for training, and finally use the GRU decoder to decode the predicted position sequence.

[0047] S6.2 Model Construction: Total Loss Function, using the predicted location sequence Y pred and the true location sequence Y true The formula for determining the accuracy of predictions is:

[0048]

[0049] The vehicle trajectory prediction method based on Transformer and LSTM described in this invention has the following technical advantages compared with existing technologies:

[0050] This invention constructs a spatial undirected graph using the obtained target information and the target itself as nodes, transforming temporal information into a graph matrix form and preserving its temporal sequence. The reconstructed LSTM model processes the graph matrix data, leveraging the cyclic nature of LSTM to handle the temporal sequence. The Transformer's spatial attention mechanism extracts effective features from the spatial graph matrix data and trains the model. The inception_mixer module further processes high-frequency and low-frequency information, extracting effective features again, ensuring the accuracy and efficiency of this method. This method can efficiently predict the trajectories of surrounding vehicles over long periods, effectively ensuring driving and road safety. It can be applied to intelligent transportation systems to improve traffic safety and efficiency, and assist in route optimization and intersection prediction. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the actual trajectory and predicted trajectory of a vehicle according to an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of a preset traffic scenario in an embodiment of the present invention. Detailed Implementation

[0054] The present invention will be further explained in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand and implement the present invention. However, the following examples are only used to explain the present invention and are not intended to limit the present invention.

[0055] This invention is a vehicle trajectory prediction method based on Transformer and LSTM, and the flowchart of the method is attached. Figure 1 As shown. The technical solution of the present invention will be further described below with reference to specific embodiments:

[0056] S1: Preset traffic scenarios are shown in the appendix. Figure 3 As shown, surrounding vehicle data is acquired through vehicle interaction, and in this example, data from the NGSIM dataset is used instead. All objects in the past time period t are represented by n, with initial data including vehicle ID, vehicle lateral coordinate, vehicle longitudinal coordinate, vehicle speed, and acceleration, respectively. The raw data is processed into a 3D tensor, with a quantization of 5, to represent the coordinates (x, y) and velocity (u, v) of an object at one time step. All coordinates and velocities are then standardized to the range (-1, 1).

[0057] S2: Construct the preprocessed data into a graph matrix structure.

[0058] S2.1: Set num_node to 100, max_hop to 1, initialize the matrix using the np.zeros() method, and then use a for loop to assign a value of 1 to all edges.

[0059] S2.2: Use the np.linalg.matix_power() method to obtain the transition matrix (max_hop+1,100,100); use a for loop to cover the matrix from far to near, excluding infinitely large values.

[0060] S2.3: First, use the np.zeros() function to create an empty matrix. Then, use a for loop to assign the reciprocal of the number of nodes connected to the current node to the empty matrix. Finally, use the np.dot() method to regularize the adjacency matrix.

[0061] S3: Create a custom LSTM network as the first layer for processing the data. The model incorporates a graph convolution module to process graph matrix data; the first layer of this module is BatchNorm2d(in_channels).

[0062] The second layer is Graph_Conv_Block(in_channels,64,kernel_size,1).

[0063] The third layer is Graph_Conv_Block(64,64,kernel_size,1).

[0064] The fourth layer is Graph_Conv_Block(64,64,kernel_size,1).

[0065] The reconstruction module outputs (N*T,T,C), which is input into the LSTM. The activation function used during the encoding process is Tanh, and the decoding process uses ReLU and Tanh functions.

[0066] S4: Create a Transformer model with spatial multi-head self-attention mechanism. Set d_model to 128, d_inner to 512, n_layers to 2, n_head to 4, d_k to 32, d_v to 32, dropout to 0.1, and n_position to 16.

[0067] S4.1: Use Transformer_encoder to add a spatial mask to each data point. The encoder parameters are those mentioned above, the nn.LayerNorm() parameter is given as 60, and the MultiHeadAttention() parameter is also mentioned above.

[0068] S4.2: The key parameter teacher_forcing_ratio of the forward function is set to 0.0. An if statement is used to check whether teacher_forcing_ratio meets the condition. If it does, torch.einsum() is used to sum the spatial mask of the data to obtain hidden, resulting in a tuple of length 4. Finally, the decoder function is used to calculate the output, with parameters set to the last observation position of the upper-level spatial attention mechanism and hidden.

[0069] S5: Use the Inception_mixer mechanism to further extract data.

[0070] S5.1: The first branch first uses a depthwise convolution with a kernel size of (1,1) and the activation function ReLU; then it uses a depthwise convolution with a kernel size of (3,3) and the activation function ReLU. The second branch uses a MaxPooling layer with a kernel size of (3,3) and a stride of (1,1); then it uses a depthwise convolution with a kernel size of (1,1) and the activation function ReLU.

[0071] S5.2: The third branch first uses an AvgPooling layer with a kernel size of (3,3) and a stride of (1,1); then it uses a depthwise convolution with a kernel size of (1,1) and an activation function of ReLU. The data is concatenated using the concatenate function. The data is trained using a spatial self-attention network with the network activation function Softmax to obtain the mask for the input of the next layer.

[0072] S6: Based on the data obtained in S5, train the last layer of Transformer and use the GRU decoder to obtain the final predicted position.

[0073] S6.1: Input the output obtained in S5 into the Transformer layer as described in S4 for training. Set the model parameters in_size to 4, out_size to 2, and the number of masks to 32. Reconstruct the data using torch.randn((32,4,16,22)). The GRU decoder embedding parameters are output_size and hidden_size, and the activation function is SoftMax to obtain the final predicted position sequence. Use Python's built-in matplotlib plotting tool to draw the true and predicted trajectories, as shown in the attached figure. Figure 2 As shown.

[0074] S6.2: Use the compute_RMSE function to calculate the loss, with parameters pred having a size of (N, 2, 25, 120) and true having a size of (N, 2, 25, 120). Calculate the loss using torch.sum(torch.abs(pred-true)**2).

[0075] This invention constructs data as a spatial undirected graph, laying the groundwork for a spatial self-attention mechanism; it uses an Lstm model containing a graph convolution module to process graph matrix data; it updates the self-attention mechanism of the traditional Transformer model to a spatial self-attention mechanism; and it adds an inception_mixer mechanism after the main module and pairs it with a self-attention mechanism.

[0076] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A vehicle trajectory prediction method based on Transformer and LSTM, characterized in that, Includes the following steps: S1: The vehicle obtains the driving data of surrounding vehicles through information interaction with surrounding vehicles, including the lateral coordinates, longitudinal coordinates, speed and acceleration of the front center, and integrates the identification numbers of surrounding vehicles into the driving data; the data is preprocessed into a 3D tensor using built-in Python tools; S2: Construct a graph data structure from the preprocessed data; first, use matrix multiplication to calculate the number of hops between every two nodes, then classify the relationships between nodes according to the maximum number of hops, and finally obtain an adjacency matrix; S3: Create a custom LSTM model, using the adjacency matrix obtained in S2 as input; this LSTM model uses Graph_Conv_Block to process the graph matrix, and also uses a custom edge_importance_weighting function to extract available edge weights, and then reconstructs the graph matrix data; finally, a custom reshape function is used to synthesize the output and the original input. The custom LSTM model in S3 serves as the first layer for data processing. This LSTM model contains three Graph_Conv_Blocks and one BatchNorm2d for input standardization. The spatial kernel size of each graph convolution is determined based on the maximum hop count and number of nodes in the spatial graph at different time intervals, while the temporal kernel size is fixed. The activation function is ReLU. The convolutional layer's role is to extract important information from the graph matrix. In the forward function, the data obtained from the graph convolution blocks is computed forward, and the custom function edge_importance_weighting is used to extract node information that meets the edge weight requirements, resulting in the output. This output is concatenated with the original input to the LSTM model to form a new matrix, which is then passed to the next layer network. S4: Create a custom Transformer model that includes a spatial attention mechanism, using the output obtained in S3 as the input to this Transformer model; this Transformer model uses Xavier to initialize the data of the upper layer of the encoder, and then uses a custom forward function to train and obtain the output; In S4, a custom Transformer model is created. First, the data output from S3 is spatially encoded, and the attention mechanism only allows data to be transmitted between neighboring nodes. All parameters are iterated through in a loop, and parameters with a dimension greater than 1 are initialized using the Xavier initialization method. In the forward propagation, the output and hidden states are generated using the GRU decoder through loop iteration, and the results are stored in the corresponding array positions. If the requirements of the teacher_force method are met, it is used as the next input. During spatial interaction, the message_passing function is used to process the hidden state and position mask to obtain the output; The specific steps of S4 include: S4.1, Express the output obtained in S3 as Using the encoding properties of the Transformer itself, each time step is... Object mapping as ; S4.

2. Spatial self-attention mechanism can be viewed as information being transferred on an undirected graph; at time steps... time , representing the query vector, key vector, and value vector of each object, respectively: ； ； ； Define the spatial diagram from the vehicle Passed to vehicle The message is , The effect of time step on the adjacency matrix The attention mechanism is as follows: ； Vehicle The set of neighboring cars is represented as The time step is obtained using a spatial self-attention mechanism. Vehicle The multi-head attention data is used to perform forward computation to obtain the output. S4.3 Utilize the teacher_forcing mechanism, using the output of the prior time step as input, to repeat the process of the spatial self-attention mechanism; S5: Take the output obtained from S4 as input data, use the Inception_mixer mechanism to process the data, extract the high-frequency information in the data, and concatenate the obtained high-frequency information and low-frequency information together to obtain a new output; S5 takes the output obtained in S4 as input, divides the input into several channels, and assigns them to the high-frequency branch and the low-frequency branch respectively; convolution and max pooling operations are used in the high-frequency branch to extract high-frequency information; in the low-frequency branch, average pooling operation is first used, then spatial attention mechanism is used to process the data, and finally upsampling is performed; finally, the high-frequency result and the low-frequency result are directly concatenated together to obtain the final output. The specific steps of S5 include: S5.

1. Input the output result X obtained in S4 into another Transformer with the inception_mixer mechanism. The first branch X1 first passes through a kernel of size X1. Depthwise convolution, then perform a kernel size of Depth convolution, For time step The number of graph nodes at that time; the second branch X2 first performs a MaxPooling operation to increase data features, and then performs a kernel size of The depthwise convolution; these two branches are for extracting high-frequency information, and the formula is: ； ； S5.2, the third branch X3 first performs the AveagePooling operation to reduce the data features, and then uses... The depthwise convolution is followed by an upsampling operation, which uses an interpolation function to complete the features. The formula is as follows: ； Then, the above results are concatenated, and the formula is: ； Finally, spatial self-attention mechanism is used to process the obtained sequence. and Perform training, and multiply each sequence by the query vector matrix. Key vector matrix Sum value vector matrix The formula is as follows: ； The final output is obtained; S6: Input the output data obtained from S5 into a Transformer with spatial attention mechanism, and use the GRU decoder to predict the position; S6 performs the final Transformer training based on the data obtained in S5, and uses the GRU decoder to obtain the final predicted position. The specific steps of S6 include: S6.

1. Input the output data obtained in S5 back into the same Transformer layer as in S4 for training. Finally, use the GRU decoder to decode the predicted position sequence. S6.2 Model Construction: Total Loss Function, using the predicted location sequence and the true location sequence The formula for determining the accuracy of predictions is: 。 2. The vehicle trajectory prediction method based on Transformer and LSTM according to claim 1, characterized in that, In S1, driving data of surrounding vehicles is obtained through information exchange with surrounding vehicles. (Assuming past...) During the period Each object is processed from its original data into a 3D tensor, and a value of 5 is set to represent the coordinates of an object at one time step. and speed and normalize all coordinates and velocities to between.

3. The vehicle trajectory prediction method based on Transformer and LSTM according to claim 2, characterized in that, The specific steps of S2 include: S2.

1. Construct a spatial graph from the preprocessed data, setting the number of nodes to 100 and the maximum hop count to 1. Represent the graph as follows: The node is represented as The edge is represented as ; S2.2 First, calculate the transit matrix of all nodes using matrix multiplication. Then, determine whether each node is connected by comparing the distance between each node to see if it is less than or equal to the maximum hop count, and obtain the adjacency matrix. The formula is expressed as: ； S2.3 Regularizing the adjacency matrix: First, calculate the degree matrix based on the sum of each row. Then, regularize the adjacency matrix using matrix multiplication. The formula is: 。

Images