Surrounding vehicle trajectory prediction method and system based on MATT-GNN

Abstract

The present invention provides a method and system for predicting surrounding vehicle trajectories based on MATT-GNN. The prediction method first constructs a multi-attention spectral spatiotemporal graph neural network, then inputs the trained multi-attention spectral spatiotemporal graph neural network with real-time collected vehicle information of surrounding traffic scenes, and outputs the predicted future trajectory of the vehicle. The multi-attention spectral spatiotemporal graph neural network includes an encoder, a GRU layer, an MT-stem layer, and a decoder. The MT-stem layer is composed of n MT-stem modules. The MT-stem modules fully extract the spatiotemporal features between surrounding vehicles. The present invention can accurately predict the trajectories of surrounding vehicles in complex traffic scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle intelligent driving technology, and specifically relates to a method and system for predicting the trajectories of vehicles around an autonomous driving vehicle based on MATT-GNN (Mutli-Attention Spectral Temporal-Spatial Graph Neural Network). Background Art

[0002] In recent years, with the continuous advancement of science and technology, scholars both domestically and internationally have deepened their research on autonomous vehicles. How to ensure safe and appropriate interaction between autonomous vehicles and other vehicles in complex traffic scenarios has become a hot topic. Scholars both domestically and internationally generally believe that autonomous vehicles can only become more intelligent and safer if they can predict the trajectories of surrounding vehicles. By predicting the trajectories of surrounding vehicles, autonomous vehicles can react in advance, effectively avoiding traffic accidents.

[0003] Currently, most vehicle trajectory prediction methods are based on the Long Short-Term Memory (LSTM) neural network model. This approach uses the LSTM model to fit the spatiotemporal characteristics of surrounding vehicles, then uses backpropagation to update the gradients of the model weights, ultimately fitting a trajectory prediction model. However, because vehicle interactions often exhibit a non-Euclidean topological structure, recurrent neural networks (RNNs) and convolutional neural networks (CNNs) alone cannot fully represent these interactions and cannot fully express implicit relationships. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a method and system for predicting surrounding vehicle trajectories based on MATT-GNN, which can achieve accurate prediction of surrounding vehicle trajectories in complex traffic scenarios.

[0005] The present invention achieves the above technical objectives through the following technical means.

[0006] The surrounding vehicle trajectory prediction method based on MATT-GNN is as follows:

[0007] Construct a multi-attention spectral spatiotemporal graph neural network: Combine vehicle historical trajectory data with road features, encode them based on the LSTM encoder to obtain features, and construct an adjacency matrix for each vehicle; the adjacency matrix is ​​input into the GRU layer to construct the graph G and its associated adjacency weight matrix W; use the graph G as the input of the MT-stem layer composed of n MT-stem modules, connect the outputs of the n MT-stem modules and send them to the GLU and fully connected layers, output the training loss and trajectory prediction value, and combine them with the graph convolution kernel θ j Then sum it up to get the output value of the jth channel, and connect each output channel to get the result Z; Finally, decode it through the LSTM decoder;

[0008] The real-time collected vehicle information of the surrounding traffic scene is input into the trained multi-attention spectral spatiotemporal graph neural network, and the predicted future trajectory of the vehicle is output; where n≥1.

[0009] Furthermore, the MT-stem module fully extracts the spatiotemporal features between surrounding vehicles. Specifically:

[0010] The graph G is transformed into a spectral matrix representation, and the spectral matrix is ​​placed in the multi-head attention Seq2seq module. The feature tensor component of each node in the spectral matrix is ​​transformed into the frequency domain, and then sent to the one-dimensional convolution and GLU sub-layer to capture the characteristic pattern; then, the tensor with the obtained characteristic pattern is placed in a linear transformation and then input into the scaled dot product attention. The m times of scaled dot product attention results are spliced, and then a linear transformation is performed and converted back to the time domain; then supervised learning is performed on the spectral matrix, and a graph convolutional neural network is applied to the spectral matrix representation, and the inverse operation is performed. Finally, the training loss and trajectory prediction value are output through the output layer.

[0011] Furthermore, the graph G = (X1, W), where X1 is the historical trajectory data of the vehicle, W is the adjacency weight matrix, and Q and K represent query and key value, Q=RW Q 、K=RW K , R is the hidden state vector, W Q and W K is a learnable parameter and d is the time complexity.

[0012] Furthermore, the output value of the j-th channel is:

[0013]

[0014] Among them, GF represents the Fourier transform operator, GF -1 represents the inverse Fourier transform operator, S represents the head attention Seq2seq module, θ ijrefers to the graph convolution kernel consisting of the i-th input channel and the j-th output channel, Λ i is the characteristic matrix of the normalized Laplacian operator, X i is the output of each MT-stem module.

[0015] Furthermore, the decoder output sequence Y is fitted with time and weather features. Specifically, the weather information is represented as a categorical variable, and a linear layer is used to fit the decoder output sequence Y with time and weather features.

[0016] Furthermore, the supervised learning of the spectral matrix is ​​implemented based on a trajectory regularization module, the feature input of the graph neural network is globally pooled, and a multi-label multi-classification algorithm is used for classification.

[0017] Furthermore, the surrounding traffic scene vehicle information includes: vehicle transverse and longitudinal coordinate information, transverse and longitudinal acceleration, transverse and longitudinal speed, vehicle length, vehicle width and the transverse distance between the vehicle center point and the lane boundary line.

[0018] A surrounding vehicle trajectory prediction system based on MATT-GNN, including:

[0019] Trajectory data acquisition module, used to obtain vehicle information in surrounding traffic scenes;

[0020] The data preprocessing module removes noise and performs format conversion on the data collected by the trajectory data acquisition device;

[0021] The trajectory prediction module predicts the trajectories of surrounding vehicles based on preprocessed data and obtains the future trajectory of the predicted vehicle.

[0022] The beneficial effects of the present invention are as follows: the present invention is used to obtain a multi-attention spectral spatiotemporal graph neural network for predicting the future trajectory of a vehicle, including an encoder, a GRU layer, an MT-stem layer, and a decoder. The MT-stem layer is composed of n MT-stem modules. The MT-stem module fully extracts the spatiotemporal features between surrounding vehicles, ensuring more accurate prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the multi-attention spectral spatiotemporal graph neural network structure of the present invention;

[0024] Figure 2 Schematic diagram of the MT-stem layer structure of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0026] The present invention provides a vehicle trajectory prediction system for an autonomous driving vehicle based on a multi-attention spectral spatiotemporal graph neural network, comprising a trajectory data acquisition module, a data preprocessing module and a trajectory prediction module.

[0027] The trajectory data acquisition module, including lidar and road test equipment, identifies vehicle information in the surrounding traffic scene and collects the vehicle's horizontal and vertical coordinate information, horizontal and vertical acceleration, horizontal and vertical speed, vehicle length, vehicle width, and the lateral distance between the vehicle center point and the lane boundary line at a frequency of 5Hz. The surrounding traffic scene includes three parts: autonomous driving vehicles, surrounding vehicles, and non-surrounding vehicles. Autonomous driving vehicles are data collection vehicles, and surrounding vehicles are defined as vehicles within a range of less than 30 meters from the center point of the autonomous driving vehicle; non-surrounding vehicles are vehicles within a range of more than 30 meters from the center point of the autonomous driving vehicle.

[0028] The data preprocessing module performs data preprocessing on the data collected by the trajectory data acquisition device. The preprocessing is specifically as follows: first, the collected data is denoised, and then the data format required by the multi-attention spectral spatiotemporal graph neural network is constructed based on the denoised data.

[0029] The trajectory prediction module predicts the trajectories of surrounding vehicles based on preprocessed data and outputs the predicted future trajectory of the vehicle.

[0030] The present invention provides a method for predicting the trajectory of vehicles around an autonomous driving vehicle based on a multi-attention spectral spatiotemporal graph neural network, which specifically includes the following steps:

[0031] Step (1): Construction of multi-attention spatiotemporal graph neural network

[0032] Step (1.1), input of multi-attention spatiotemporal graph neural network

[0033] The input of the multi-attention spectral spatiotemporal graph neural network includes the historical trajectory of the autonomous driving vehicle and the historical trajectory of surrounding vehicles.

[0034] (1) Historical trajectory of autonomous vehicles

[0035] The historical trajectory sequence of the autonomous vehicle can be expressed as:

[0036] X AV ={p (t) ,p (t-1) ,p (t-2) ,···,p (t-k)}

[0037] Among them, k represents the length of the time series, t represents the current frame, and p (t) represents the historical trajectory of the autonomous vehicle at time t, and:

[0038] p (t) ={local AV_x (t) ,local AV_y (t) ,v AV_x (t) ,v AV_y (t) ,a AV_x (t) ,a AV_y (t) ,L AV ,W AV ,Δd AV_left (t) ,Δd AV_right (t)}

[0039] Among them, local AV_x (t) and local AV_y (t) They represent the horizontal and vertical coordinate positions of the autonomous vehicle at time t, v AV_x (t) and v AV_y (t) represents the horizontal and vertical speed values ​​of the autonomous driving vehicle at time t, a AV_x (t) and a AV_y (t) represents the lateral and longitudinal acceleration values ​​of the autonomous vehicle at time t, L AV 、W AV denote the length and width of the autonomous vehicle, Δd AV_left (t) , Δd AV_right (t) They represent the distance difference between the center point of the autonomous driving vehicle and the left and right lane boundaries of the driving direction at time t.

[0040] (2) Historical trajectories of surrounding vehicles

[0041] The historical trajectory sequence of the surrounding vehicles of the autonomous driving vehicle can be expressed as:

[0042] X surround ={f (t) ,f (t-1) ,f (t-2) ,···,f (t-k)}

[0043] Among them, f (t) represents the historical trajectory of surrounding vehicles at time t, and:

[0044] f (t)={s1 (t) ,s2 (t) ,s3 (t) ,···,s n (t)}

[0045] Among them, s n (t) represents the trajectory information of the nth vehicle around the autonomous vehicle, and:

[0046] s n (t) ={local surround_x (t) ,local surround_y (t) ,v surround_x (t) ,v surround_y (t) ,a surround_x (t) ,a surround_y (t) ,Δlocal x (t) ,Δlocal y (t) ,Δv x (t) ,Δv y (t) ,Δa x (t) ,Δa y (t) ,Δd surround_left (t) ,Δd surround_right (t)}

[0047] Among them, local surround_x (t) and local surround_y (t) They represent the horizontal and vertical coordinate positions of the surrounding vehicles at time t, respectively, surround_x (t) and v surround_y (t) They represent the horizontal and longitudinal speeds of the surrounding vehicles at time t, respectively. surround_x (t) and a surround_y (t) Respectively represent the lateral and longitudinal acceleration values ​​of the surrounding vehicles at time t, Δlocal x (t) and Δlocal y (t) They represent the horizontal and vertical coordinate differences between the center point of the surrounding vehicles and the autonomous driving vehicle at time t, Δvx (t) and Δv y (t) They represent the horizontal and longitudinal speed differences between the center point of the surrounding vehicles and the autonomous driving vehicle at time t, Δa x (t) and Δa y (t) They represent the lateral and longitudinal acceleration differences between the center points of the surrounding vehicles and the autonomous driving vehicle at time t, Δd surround_left (t) , Δd surround_right (t) They represent the distance difference between the center point of the surrounding vehicles and the left and right lane boundaries in the driving direction at time t.

[0048] Step (1.2), multi-attention spatiotemporal graph neural network structure

[0049] The overall architecture of the multi-attention spatiotemporal graph neural network is as follows Figure 1 As shown in the figure, the vehicle historical trajectory data X1 is first combined with the road features, and then encoded based on the LSTM encoder to obtain features, and the adjacency matrix (i.e., input vector and hidden state vector) of each vehicle is constructed; then, the graph G and its associated adjacency weight matrix W are constructed based on the GRU layer; the graph G is used as the input of the MT-stem layer composed of two MT-stem modules, and the outputs of the two MT-stem modules are connected and fed into the GLU and fully connected layers; finally, it is decoded by the LSTM decoder, and the trajectory prediction value is output based on time and weather features.

[0050] (1) Encoder

[0051] This application designs an encoder-decoder architecture for the surrounding vehicle trajectory prediction system. The architecture is built based on multiple LSTMs, respectively called encoder and decoder.

[0052] The encoder is used to process the input sequence X1. At each time step, the encoder recursively takes the input vector and hidden state vector from the previous LSTM layer to generate the input vector and hidden state vector of the current time step, and generates the final vector and hidden state vector after completing the encoding of all time steps.

[0053] (2) GRU layer

[0054] First, the final vector and the hidden state vector are fed into the Gated Recurrent Unit (GRU) layer, which sequentially calculates the spatiotemporal correlation corresponding to each timestamp t. The final hidden state vector R is used as the representation of the entire sequence to obtain the weight matrix W:

[0055] Q=RW Q

[0056] K=RW K

[0057]

[0058] Among them, Q and K represent query and key, which can be calculated by linear projection; W Q and W K is a learnable parameter. By learning the spatiotemporal correlation between surrounding vehicles, the adjacency weight matrix of graph G is obtained; d is the time complexity.

[0059] Construct a graph G = (X1, W). Any two nodes in the graph G may be related, that is, the adjacency relationship between the nodes can be arbitrary.

[0060] (3)MT-stem module

[0061] In detail, the MT-stem module aims to model the internal structure and spatiotemporal correlation of the sequence in the spectral domain, so that the spatiotemporal features between surrounding vehicles can be fully extracted.

[0062] First, the graph G is transformed into a spectral matrix representation based on the Fourier transform operator GFT, and the feature tensor of each node in the spectral matrix becomes linearly independent.

[0063] The spectral matrix is ​​then fed into the multi-head attention Seq2seq module. First, each feature tensor component is transformed to the frequency domain using the discrete Fourier transform (DFT) operator. This is then fed into a one-dimensional convolution (IDConv) and GLU sublayer to capture characteristic patterns. Next, the resulting tensor is subjected to a linear transformation (Linear) and fed into the scaled dot product attention layer. This process is repeated m times, effectively multi-headed, with each calculation being a different head and the linear transformation parameter W. The results of these m scaled dot product attention layers are then concatenated and subjected to another linear transformation (Linear) to obtain the final value. The advantage of the multi-head attention mechanism lies in performing m calculations rather than just one, allowing the model to learn relevant information from different spectral representations. Finally, the inverse discrete Fourier transform (IDFT) operator is used to convert the data back to the time domain.

[0064] Then, the spectral matrix is ​​supervisedly learned based on the trajectory regularization module TCR. Its main purpose is to improve the graph neural network's ability to extract spatiotemporal features of surrounding vehicles, thereby globally pooling the feature input of the graph neural network and using a multi-label multi-classification algorithm for classification, guiding the graph neural network in a supervised manner to only activate class-related features.

[0065] Finally, a graph convolutional neural network Gconv is applied on the spectral matrix representation and the inverse Fourier transform operator IGFT is performed.

[0066] At the end of the MT-stem module, an output layer consisting of a GLU and a fully connected layer FC is added. The model outputs the training loss and trajectory prediction value; the loss function can be expressed as:

[0067]

[0068] in, represents the feedforward loss, X represents the feedback loss, Δ θ represents the training loss; for each timestamp t, {X t-k ,…,X t-1} is the input value, is the predicted value at time stamp t, {B t-k (X),…,B t-1 (X)} is the reconstructed value obtained through feedback.

[0069] (4) MT-stem layer

[0070] The MT-stem layer is constructed by stacking n MT-stem modules with residual connections, and an MT-stem layer is constructed by embedding the multi-head attention Seq2seq and TCR units into the spectral convolution module; n ≥ 1, and in this embodiment, n is preferably two.

[0071] In detail, this paper applies a multi-head attention Seq2seq module to the output of Fourier transform (GFT) to learn the autocorrelation features in trajectory data. GFT, multi-head attention Seq2seq and TCR units are applied to the output X of each MT-stem module. i , followed by the graph convolution kernel θ j Then perform the summation operation.

[0072] Finally, the inverse graphic Fourier transform (IGFT) is applied to the sum to obtain the output value Z of the jth channel. j , which can be written as follows:

[0073]

[0074] Among them, GF, GF -1 and S represent GFT, IGFT and multi-head attention Seq2seq modules respectively, θ ij refers to the graph convolution kernel consisting of the i-th input channel and the j-th output channel, Λ iis the eigenvector matrix of the normalized Laplace operator. The number of eigenvectors used in the GFT is equal to the number of multivariate dimensions without dimensionality reduction.

[0075] Then, by connecting each output channel Z j , and get the result Z.

[0076] (4) Decoder

[0077] The decoder uses the hidden state vector passed by the encoder and the MT-stem layer output result Z, and recursively performs decoding operations by using the LSTM layer; in each decoding step, the decoder inputs the output obtained in the previous step and generates an output sequence Y.

[0078] (5) Time and weather characteristics

[0079] This application also incorporates background features that may affect the driver's trajectory decision-making into trajectory prediction, including time and weather conditions. Different time periods and weather conditions of the day have a significant impact on the vehicle's trajectory. This application divides the day into 12 intervals, with each interval being 2 hours. In addition, this application represents weather information as a categorical variable (sunny / cloudy / light rain / heavy rain) and uses a linear layer to fit the output sequence Y with the time and weather features.

[0080] Step (2), training of multi-attention spatiotemporal graph neural network

[0081] (1) Training data

[0082] Specifically, the data at any sampling moment includes the autonomous vehicle's coordinates, speed, acceleration, vehicle length and width, and distance from the left and right lane boundaries.

[0083] In addition, in each piece of data, the coordinates of the autonomous driving vehicle are set to (0, 0), and the coordinates of the surrounding vehicles are relative coordinates centered on the autonomous driving vehicle, which is used to enhance the generalization ability of the model.

[0084] (2) Model training environment

[0085] This paper uses PyTorch 1.8.1 as the training framework and combines it with the root mean square prop (RMSProp) optimizer to accelerate gradient descent, thereby speeding up the model training process. The learning rate of the RMSProp optimizer is set to 0.0001, which makes the training more accurate in finding the optimal solution while avoiding overfitting.

[0086] By collecting historical vehicle trajectory data in real traffic scenarios as the total data set, the training set, validation set, and test set used in training are all derived from the total data set and divided in a ratio of 7:1:2. The model training rounds are 200 rounds. After each training, the model parameter file is saved and the error is output.

[0087] Step (3), surrounding vehicle trajectory prediction method

[0088] The real-time collected vehicle information of the surrounding traffic scene is input into the trained multi-attention spectral spatiotemporal graph neural network, and the predicted future trajectory of the vehicle is output to provide guidance for the autonomous driving vehicle to plan the driving trajectory.

[0089] The embodiments described are preferred implementations of the present invention, but the present invention is not limited to the above implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention are within the scope of protection of the present invention.

Claims

1. A surrounding vehicle trajectory prediction method based on MATT-GNN, characterized by: Constructing a multi-attention spectral spatiotemporal graph neural network: combining historical vehicle trajectory data with road features, encoding them using an LSTM encoder to obtain features, and constructing an adjacency matrix for each vehicle. This adjacency matrix is ​​then fed into the GRU layer to construct the graph G and its associated adjacency weight matrix W. The graph G is used as the input of the MT-stem layer composed of n MT-stem modules. The outputs of the n MT-stem modules are connected and sent to the GLU and fully connected layers. The training loss and trajectory prediction value are output and combined with the graph convolution kernel θ j Then sum it up to get the output value of the jth channel, connect each output channel to get the result Z; finally, decode it through the LSTM decoder; Input the real-time collected vehicle information of the surrounding traffic scene into the trained multi-attention spectral spatiotemporal graph neural network and output the predicted future trajectory of the vehicle; where n≥1; The MT-stem module enables the spatiotemporal features between surrounding vehicles to be fully extracted. Specifically: the graph G is transformed into a spectral matrix representation, the spectral matrix is ​​placed in the multi-head attention Seq2seq module, the feature tensor component of each node in the spectral matrix is ​​transformed into the frequency domain, and then sent to the one-dimensional convolution and GLU sub-layer to capture the characteristic pattern; then, the tensor of the obtained characteristic pattern is placed in a linear transformation, and then input into the scaled dot product attention, the m times of scaled dot product attention results are spliced, and then a linear transformation is performed and converted back to the time domain; then supervised learning is performed on the spectral matrix, a graph convolutional neural network is applied to the spectral matrix representation, and an inverse operation is performed, and finally the training loss and trajectory prediction value are output through the output layer.

2. The method for predicting surrounding vehicle trajectories based on MATT-GNN according to claim 1, characterized in that: The graph G = (X1, W), where X1 is the vehicle's historical trajectory data, W is the adjacency weight matrix, and Q and K represent query and key value, Q=RW Q 、K=RW K , R is the hidden state vector, W Q and W K is a learnable parameter and d is the time complexity.

3. The method for predicting surrounding vehicle trajectories based on MATT-GNN according to claim 1, characterized in that: The output value of the j-th channel is: Among them, GF represents the Fourier transform operator, GF -1 represents the inverse Fourier transform operator, S represents the head attention Seq2seq module, θ ij refers to the graph convolution kernel consisting of the i-th input channel and the j-th output channel, Λ i is the characteristic matrix of the normalized Laplacian operator, X i is the output of each MT-stem module.

4. The method for predicting surrounding vehicle trajectories based on MATT-GNN according to claim 1, characterized in that: The decoder output sequence Y is fitted with time and weather features. Specifically, weather information is represented as a categorical variable, and a linear layer is used to fit the decoder output sequence Y with time and weather features.

5. The method for predicting surrounding vehicle trajectories based on MATT-GNN according to claim 1, characterized in that: The supervised learning of the spectral matrix is ​​implemented based on the trajectory regularization module, the feature input of the graph neural network is globally pooled, and a multi-label multi-classification algorithm is used for classification.

6. The method for predicting surrounding vehicle trajectories based on MATT-GNN according to claim 1, characterized in that: The vehicle information of the surrounding traffic scene includes: the vehicle's horizontal and vertical coordinate information, horizontal and vertical acceleration, horizontal and vertical speed, vehicle length, vehicle width and the lateral distance between the vehicle center point and the lane boundary line.

7. A prediction system for implementing the surrounding vehicle trajectory prediction method based on MATT-GNN according to any one of claims 1 to 6, characterized in that: include: Trajectory data acquisition module, used to obtain vehicle information in surrounding traffic scenes; The data preprocessing module removes noise and performs format conversion on the data collected by the trajectory data acquisition device; The trajectory prediction module predicts the trajectories of surrounding vehicles based on preprocessed data and obtains the future trajectory of the predicted vehicle.

Images