Vehicle trajectory prediction method and system fusing large kernel convolution and attention mechanism

Abstract

The invention relates to a vehicle trajectory prediction method and system fusing large kernel convolution and an attention mechanism. The system comprises an encoder module, a multi-head space attention module, a large kernel convolution pooling module, a convolution modulation module, an intention recognition module and a decoder module. Spatial interaction between a target vehicle and surrounding vehicles is modeled through a multi-head spatial attention mechanism. Meanwhile, an LKCP module, an intention recognition module and a CM module are creatively introduced. Wherein the LKCP module can additionally consider space interaction among surrounding vehicles, and the space interaction modeling capability among the vehicles is further enhanced; the intention recognition module deduces the maneuvering category of the vehicle based on the historical state information of the vehicle, and provides a basis for multi-modal trajectory prediction; and the CM module further captures the time dependence between moments in the historical trajectory of the target vehicle on the basis of the LSTM, so that the capture of the hidden state of the time dimension is enhanced. And multi-modal trajectory prediction of the target vehicle is realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent vehicle trajectory prediction, and specifically to a vehicle trajectory prediction method and system integrating large kernel convolution and attention mechanism. Background Art

[0002] In the field of trajectory prediction, researchers at home and abroad have proposed a variety of trajectory prediction methods, which can be divided into three categories: methods based on physical models, methods based on traditional machine learning, and methods based on deep learning.

[0003] Methods based on physical models usually rely on the kinematic equations and dynamic equations of the vehicle and assume that the vehicle moves at a constant speed or constant yaw rate. The advantage of this method is that it is simple to calculate, but it is obviously unrealistic to regard the vehicle movement as a constant speed or constant acceleration and deceleration, especially in complex traffic environments. Trajectory prediction methods based on traditional machine learning mainly include Bayesian networks, hidden Markov models, support vector machines, and Gaussian processes. In vehicle trajectory prediction, the Gaussian process method is usually used to quantify the uncertainty of future trajectory prediction and obtain the probability distribution of multiple future trajectories to achieve multimodal prediction. Bayesian networks, hidden Markov models, and support vector machines are often used to decompose the vehicle's driving behavior into lateral and longitudinal directions, and then recombine the decomposed vehicle behavior to finally achieve multimodal prediction. Although these methods can quantify the uncertainty in trajectory prediction and achieve multimodal trajectory prediction, they all ignore the impact of interactions between vehicles on future trajectories.

[0004] Deep learning-based methods can not only consider physical and road-related factors, but also capture the implicit interactions between vehicles in space, making them more suitable for more complex scenarios. Since the vehicle's driving trajectory data has time series characteristics, researchers first used recurrent neural networks ( RNN ) and its variants, such as long short-term memory neural networks ( LSTM ) and gated recurrent units ( GRU ) to capture the temporal correlation of the vehicle’s historical trajectory data. RNN Its variants are used in the short time domain (0-3 s ) trajectory prediction, but performs well in the long time domain (3-5 s ) have limited effects in trajectory prediction, and these methods still have room for improvement in capturing implicit interactions between vehicles. With the continuous deepening of research, convolutional neural networks ( CNN ) and graph neural networks ( GNN ) is also used to extract hidden states from trajectory datasets. GNNThe method constructs a traffic graph, represents the vehicles in the scene as nodes, and the relationship between vehicles as lines between nodes. It is suitable for trajectory prediction scenarios with multiple lanes and traffic congestion, and can achieve high prediction accuracy, but this type of method is highly dependent on map information.

[0005] In actual traffic scenarios, accurately predicting the future trajectories of surrounding vehicles still faces many challenges, mainly including: low prediction accuracy, low real-time performance, implicit interactions between vehicles, and uncertainty in the future trajectories of vehicles, etc. Summary of the invention

[0006] The present invention currently provides a vehicle trajectory prediction method and system that integrates large kernel convolution and attention mechanism to solve the technical problems mentioned in the background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions: Terminology explanation: Multimodal trajectory prediction: Vehicle multimodal trajectory prediction ( Trajectory Prediction ) refers to predicting multiple possible future positions of a vehicle based on its historical motion data (such as position, speed, acceleration, etc.). Its main goal is to make reasonable predictions about the vehicle's future motion trajectory so that the vehicle can make better decisions, planning, and control; Attention Mechanism: Attention Mechanism ( Attention Mechanism ) is a technique that selectively focuses on key parts of input data by dynamically assigning different weights. It simulates the characteristics of human attention and is often used in fields such as natural language processing and computer vision. In deep learning, it allows the model to focus on more important parts when processing information, thereby improving prediction accuracy; Large kernel convolution pooling: Large kernel convolution pooling ( Large Kernel Convolution Pooling , LKP ) is a method proposed in the present invention that combines multi-scale convolution kernels and pooling operations, which can effectively obtain information in the local neighborhood of the vehicle and the larger receptive field, thereby modeling the spatial relationship between surrounding vehicles; Convolution modulation: Convolution modulation ( Convolution Modulation ) is a method that simulates and replaces the self-attention mechanism. It adjusts and transforms the input feature information through convolution operations, thereby weighting, enhancing and changing the representation of the features.

[0008] A vehicle trajectory prediction system integrating large kernel convolution and attention mechanism, the system comprising: an encoder module, a multi-head spatial attention module ( Spatial attention / S ), large kernel convolution pooling module ( Large - kernel Convolution pooling / LKP ), convolution modulation module ( Convolutional modulation / CM ), intent recognition module and decoder module; The data acquisition module is used to collect and track the trajectory data and lane information of the vehicle and other surrounding vehicles within a preset range in real time through the vehicle-mounted camera; The encoder module is used to collect the vehicle historical trajectory data (historical trajectory x and y coordinates, lane information, speed, acceleration) into an embeddable multi-head spatial attention module ( S module) and LKP The shape format of the module; The multi-head spatial attention module is used to calculate the spatial correlation between the target vehicle and each vehicle within the preset range according to the calculation formula of the attention mechanism; Large kernel convolution pooling module, used to capture the interaction between surrounding vehicles other than the target vehicle within a preset range through convolution layers and pooling layers with multiple different convolution kernels; Convolution modulation module is used for feature extraction. LSTM The encoder then captures the temporal correlation of the vehicle’s historical trajectory information again; The intention recognition module and the decoder module output the predicted future trajectory of the target vehicle.

[0009] As a further technical solution of the present invention, the encoder module includes a multi-layer perceptron MLP and long short-term memory neural networks LSTM ; The multi-layer perceptron preprocesses the data. First, the historical status of the vehicle Through linear transformation, as follows: Formula (1): ; in: For Time Vehicle Status information, is the weight matrix of the linear layer, is the bias vector of the linear layer, and During the training process, changes will be continuously updated. is the result after linear transformation; The output of the linear layer in formula (1) is passed through Activation function processing to get the final output : Formula (2): ; in: To be embeddable The encoder vector, It is an adjustable hyperparameter, usually defaulting to 1; Since the vehicle historical trajectory data has time series characteristics, the present invention adopts a long short-term memory neural network LSTM As an encoder, it preliminarily captures the hidden state in the historical trajectory and performs feature encoding. The target vehicle and surrounding vehicles are respectively LSTM The encoder encodes, and each LSTM The encoders share weights, thereby reducing the number of network parameters, model complexity, and the risk of overfitting; The multi-layer perceptron ( MLP ) Output Moment embedding vector enter LSTM Encoder, get The process of hiding the state at all times is as follows: Formula (3): ; Formula (4): ; Formula (5): ; Formula (6): ; Formula (7): ; Formula (8): ; in: , , They are the output of the input gate, the output of the forget gate, and the output of the output gate. , , , are weight matrices, , , , are all weight matrices related to the hidden state at time t-1, , , , is the bias vector, is the activation function, For vehicles exist The state of the candidate cell at time for The candidate cell states at time t, For passing The encoder output is labeled Vehicles in history The hidden state of the moment.

[0010] As a further technical solution of the present invention, the multi-head spatial attention module adopts a multi-head spatial attention mechanism to capture the interaction between the target vehicle and the surrounding vehicles. , the correlation between the target vehicle and each surrounding vehicle is calculated through the calculation formula of the attention mechanism; From the known The target vehicle output by the encoder is The hidden state vector at time , the hidden state vector of the surrounding vehicles is { }, the surrounding vehicles are numbered 1-N, They represent the hidden states of the surrounding vehicles numbered 1-N output by the LSTM encoder at time t, and are input into the multi-head spatial attention module respectively. The calculation formula is as follows: Formula (9): ; Formula (10): ; Formula (11): ; Formula (12): ; in: , and are the query matrix of the target vehicle, the key matrix and the value matrix of the surrounding vehicles respectively obtained by linear layer transformation, , and is the weight matrix of the linear transformation, , and is the corresponding bias term, is the number of channels, output The first head in the multi-head attention The correlation between the target vehicle and other surrounding vehicles calculated at all times; The multi-head spatial attention module in this invention adopts 4 head attention layers, and finally the outputs of the four head attention layers are spliced ​​according to the last dimension to obtain ; Then go through GLU After activation function and residual connection, layer normalization is performed to obtain the hidden state after considering the interaction between the target vehicle and surrounding vehicles: Formula (13): ; in, To consider the hidden state after the interaction between the target vehicle and the surrounding vehicles, LayerNorm is the layer normalization function and GLU is the activation function. The attention obtained by concatenating the outputs of the four head attention layers according to the last dimension, is the bias matrix, for The target vehicle output by the encoder is The hidden state vector at time t.

[0011] As a further technical solution of the present invention, the large kernel convolution pooling module consists of three convolution layers and one adaptive average pooling layer, and the input is ( LSTM The hidden state vector of the surrounding vehicles output by the encoder module), the three convolutional layers are 5 5. Depthwise Convolution , 7 7 dilated convolution DW-D-Conv, 1 1Convolution1 1Conv; First, through 5 5 depth convolution ( ) Capture the detailed features of each surrounding vehicle in the local space; Then, using 7 7 dilated convolutional layers ( DW - D - Conv ) Expand the receptive field so that the network can obtain vehicle interaction information in a larger range while maintaining resolution; Next, use a layer 1 1 convolution (1 1 Conv ) Fuse the features of different channels, integrate the information extracted by multi-layer convolution, and enhance the feature expression ability; Finally, through the adaptive average pooling layer ( Adaptive Average Pooling ) The extracted information is further compressed and mapped into a fixed-size feature representation so that it can be effectively spliced ​​with the hidden features output by the multi-head spatial attention module, thereby improving the ability to model vehicle interaction relationships; The complete feature extraction process is shown in the following formula: Formula (14): ; Formula (15): ; Formula (16): ; Formula (17): ; Formula (18): ; Among them: The input of the large kernel convolution pooling module is the hidden state vector of the surrounding vehicles output by the LSTM encoder module { }, the surrounding vehicles are numbered 1-N, They represent the hidden states of the surrounding vehicles numbered 1-N output by the LSTM encoder at time t, , , and is the hidden state during the convolution process, , and are convolution kernels, representing 5 5, 7 7 and 1 1 small local matrix, , and is the network bias, It is the hidden state output after the features are extracted by the LKCP module.

[0012] As a further technical solution of the present invention, the convolution modulation module is composed of layer normalization, 1 1 convolution, GELU Activation function and 7 7 group convolutions, the input vector of the convolution modulation module is (Multi-head spatial attention module( S ) and large kernel convolution pooling module ( LKP ) output hidden state vector concatenation); First, Processing is done through layer normalization functions. Layer normalization helps standardize input features, ensures that the model stably learns the feature distribution of different time steps, and reduces model learning bias; Then, the results of layer normalization are used for feature extraction through two routes. The first route is 1 1. Convolution is used to simulate the linear transformation in the self-attention mechanism, reconstruct the feature expression, enhance the model's ability to capture deep correlations between time steps in a short period of time, and output the feature vector v , the second route is through 1 1 convolution and 7 7 kernels are grouped and convolved to output feature vectors ; pass Operations,modulate the extracted features to dynamically identify and capture key moments in the prediction process, allowing the model to more accurately predict future trajectories; Finally, through 1 1 Projection convolution corrects the feature dimension of the modulated hidden features to obtain the final output.

[0013] The output of the multi-head spatial attention mechanism and The output of the modules is concatenated and input into the convolution modulation module for time feature extraction. The complete process is shown in the following formula: Formula (19): ; Formula (20): ; Formula (21): ; Formula (22): ; Final Output It is the hidden state of the target vehicle's historical trajectory data generated based on the consideration of vehicle spatial interaction and trajectory data temporal dependency. The input vector of the convolutional modulation module is , To consider the hidden state after the interaction between the target vehicle and surrounding vehicles, is the hidden state output after the LKCP module extracts features, is the hidden state of the input vector of the convolutional modulation module after layer normalization. For the general Pass 1 1 convolution and 7 The 7 kernels are grouped and convolved to produce the feature vector v. Pass 1 1 convolution is performed to output the feature vector.

[0014] As a further technical solution of the present invention, the structure of the intention recognition module is mainly composed of a feedforward layer, a normalized activation layer and two linear classifiers; The information of the target vehicle output from the convolution modulation module Encode and extract all hidden state information of each sample historical data at each time step; The extracted hidden state information is then passed through the feedforward layer and the normalized activation layer for feature extraction. The feedforward layer consists of a fully connected layer ( FC ), the fully connected layer uses ELUActivation function, the normalized activation layer consists of a fully connected layer ( FC ), LayerNorm and ELU Activation function composition; Finally, the output of the normalized activation layer is fed to the vertical classifier and the horizontal classifier respectively, and then the full connection layer ( FC )and The classification activation function calculates the weighted probabilities of each longitudinal and lateral maneuver category at each future prediction time step. The longitudinal maneuver categories include vehicle acceleration, deceleration, and constant speed. The lateral maneuver categories include vehicle left lane change, right lane change, and no lane change (lane keeping).

[0015] As a further technical solution of the present invention, the decoder module consists of two fully connected layers and one The last fully connected layer is used to connect the first fully connected layer and LSTM The information after layer encoding is mapped from feature space to trajectory space. To consider the uncertainty of prediction, it is assumed that the decoder prediction result is considered to obey a binary Gaussian distribution, so the decoder can output the prediction of each future prediction time step. , , , and ,in is the future prediction time step, , the multimodal trajectory follows the total probability theorem, as shown in the following formula: Formula (23): ; in: represents the coordinates at the future prediction time step, For historical time period All vehicle states at all times, in the historical time period At a certain moment The vehicle state is defined as ,in Represents the state of the vehicle, Represents the vehicle number, ,in and Represents the target vehicle in Moment and coordinate, and Represents the target vehicle in The speed and acceleration of the moment, represent The lane where the target vehicle and all vehicles within the preset range around it are located at all times , express The vehicle type (coded by numbers) of the target vehicle and all vehicles within a 90-meter radius around it at the moment; Represents horizontal strategy, Represents vertical strategy, and They represent the probability of the vehicle performing lateral and longitudinal strategic maneuvers based on state X, respectively. is the Gaussian distribution probability of the vehicle's future trajectory predicted based on the vehicle's historical state and lateral and longitudinal maneuvers, ,in , represents the bivariate Gaussian distribution parameters at each time step in the prediction time domain, and The predicted trajectory points are and Expectations in direction, and The predicted trajectory points are and The variance in direction, is the correlation coefficient, and Represents the prediction time step The x-coordinate at the predicted time step The y coordinate of and Represents the prediction time step The Gaussian distribution parameter set at the prediction time step The set of Gaussian distribution parameters at .

[0016] Another object of the present invention is to provide a vehicle trajectory prediction method integrating large kernel convolution and attention mechanism, the method comprising: S1. Collect and track the trajectory data and lane information of the vehicle and other surrounding vehicles within a preset range in real time through the vehicle-mounted camera; S2. Collect the vehicle historical trajectory data (historical trajectory x and y coordinates, lane information, speed, acceleration) are encoded into embeddable S Modules and LKP The shape format of the module; S3, calculating the spatial correlation between the target vehicle and each vehicle within the surrounding preset range according to the calculation formula of the attention mechanism; S4, through multiple convolution layers and pooling layers with different convolution kernels, capturing the interaction between other surrounding vehicles except the target vehicle within the preset range; S5, perform feature extraction, through convolution, and then LSTM The encoder then captures the temporal correlation of the vehicle’s historical trajectory information again; S6. Output the predicted future trajectory of the target vehicle.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The present invention fully utilizes LSTM , CNN Based on the advantages of the convolutional neural network and the attention mechanism, a vehicle trajectory prediction model is proposed. First, the model uses a multi-layer perceptron and LSTM The encoder encodes the historical state information of the vehicle into an embeddable vector and initially captures its hidden state. Subsequently, the spatial interaction between the target vehicle and the surrounding vehicles is modeled through a multi-head spatial attention mechanism. At the same time, the model creatively introduces LKP module, intent recognition module and CM Module. Among them, LKP The module can additionally consider the spatial interactions between surrounding vehicles, further enhancing the spatial interaction modeling capabilities between vehicles; the intention recognition module infers the vehicle's maneuver category based on the vehicle's historical state information, providing a basis for multimodal trajectory prediction; and CM Module in LSTM On the basis of the target vehicle historical trajectory, the temporal dependency between each moment is further captured, and the capture of the hidden state in the time dimension is enhanced. Finally, the decoder outputs and realizes the multimodal trajectory prediction of the target vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 1 is an overall framework diagram of a vehicle trajectory prediction system in an embodiment of the present invention.

[0019] Figure 2 It is a framework diagram of the large-kernel convolution pooling module in an embodiment of the present invention.

[0020] Figure 3 It is a framework diagram of the convolution modulation module in an embodiment of the present invention.

[0021] Figure 4 It is a framework diagram of the intention recognition module in an embodiment of the present invention.

[0022] Figure 5 In the embodiment of the present invention, based on NGSIM Models of the dataset RMSE Comparison results chart.

[0023] Figure 6 In the embodiment of the present invention, based on NGSIM Models of the dataset NLL Comparison results chart.

[0024] Figure 7 This is a visualization diagram of the first trajectory prediction result based on the NGSIM data set in an embodiment of the present invention.

[0025] Figure 8 This is a visualization diagram of the second trajectory prediction result based on the NGSIM data set in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] The purpose of the present invention is to predict the multimodal trajectory of a target vehicle under the condition of fully considering the interaction between vehicles in a traffic scene and the time series characteristics of the vehicle's historical trajectory.

[0028] The current vehicle trajectory prediction problem is basically to predict the future state information of the vehicle based on the historical state information of the vehicle. Assume that the length of the vehicle historical trajectory time period selected by the present invention is , the length of the predicted future trajectory time period is In the historical period At a certain moment The vehicle state is defined as ,in Represents the state of the vehicle, The input of the prediction model in the present invention is the historical time period The states of all vehicles at all times can be expressed as Assume that the target vehicle number is , the number range of other vehicles around is . For a moment, For any vehicle, take the target vehicle as an example, At this moment, its state quantity is defined as ,in and Represents the target vehicle in Moment and coordinate, and Represents the target vehicle in The speed and acceleration of the moment, represent The lane where the target vehicle and all vehicles within a certain range around it are located at the moment , express The vehicle type (coded by numbers) of the target vehicle and all vehicles within a certain range around it at the moment. The output after prediction is the vehicle The trajectory within the time period, assuming For any moment in the future prediction period, . Then in The state of the target vehicle at time instant is defined as The predicted target vehicle in the future time period The state inside is represented as .

[0029] The present invention provides a vehicle trajectory prediction method integrating large kernel convolution and attention mechanism, the method comprising: S1. Collect and track the trajectory data and lane information of the vehicle and other surrounding vehicles within a preset range in real time through the vehicle-mounted camera; S2, through the encoder (by MLP (Multilayer Perceptron)+ LSTM (Long Short-Term Memory Neural Network) will collect the vehicle historical trajectory related data (historical trajectory x and y coordinates, lane information, speed, acceleration) are encoded into embeddable S Modules and LKP The shape format of the module; S3, then S The module will calculate the spatial correlation between the target vehicle and each vehicle within the preset range according to the calculation formula of the attention mechanism; S4, LKP Modules and S The modules are parallel, and through multiple convolution layers and pooling layers with different convolution kernels, the interactions between surrounding vehicles other than the target vehicle within the preset range are captured; S5, then S Modules and LKP The output of the module is spliced ​​and input CM module, further feature extraction, through convolution, LSTM The encoder then captures the temporal correlation of the vehicle’s historical trajectory information again; S6, finally through the intention recognition module and the decoder module (composed of FC (Fully connected layer)+ LSTM (Long Short-Term Memory Neural Network)+ FC (Fully connected layer) outputs the predicted future trajectory of the target vehicle.

[0030] like Figure 1As shown, an embodiment of the present invention provides a vehicle trajectory prediction system integrating large kernel convolution and attention mechanism, the system comprising: a data acquisition module, an encoder module, a multi-head spatial attention module ( Spatial attention / S ), large kernel convolution pooling module ( Large - kernel Convolution pooling / LKP ), convolution modulation module ( Convolutional modulation / CM ), intent recognition module and decoder module; The data acquisition module is used to collect and track the trajectory data and lane information of the vehicle and other surrounding vehicles within a preset range in real time through the vehicle-mounted camera; Specifically, it means: using the on-board camera to collect and track the driving trajectory data of the vehicle and other vehicles within a certain range in real time, and then storing and cleaning these vehicle driving trajectory data to form a usable vehicle driving trajectory data set. Finally, the data set is divided into a training set, a test set, and a validation set. The training set is used to train the prediction model system, and the test set and validation set are used to optimize the model system parameters and evaluate the prediction effect of the model system.

[0031] Encoder module, used to pass the encoder (by MLP (Multilayer Perceptron)+ LSTM (Long Short-Term Memory Neural Network) will collect the vehicle historical trajectory related data (historical trajectory x and y coordinates, lane information, speed, acceleration) are encoded into embeddable S Modules and LKP The shape format of the module; The encoder module includes a multi-layer perceptron MLP and long short-term memory neural networks LSTM ; To extract the preliminary features of historical trajectory data and encode them into suitable LSTM The format processed by the encoder, the present invention LSTM Before the encoder, the data is preprocessed by a multi-layer perceptron. First, the vehicle history status Through linear transformation, as follows: Formula (1): ; in: For Time Vehicle Status information, is the weight matrix of the linear layer, is the bias vector of the linear layer, and During the training process, changes will be continuously updated. is the result after linear transformation; Considering ELU The activation function is compared to ReLU The activation function can maintain a smoother curve in the entire definition domain. Therefore, the output of the linear layer in formula (1) is converted into Activation function processing to get the final output : Formula (2): ; in: To be embeddable The encoder vector, It is an adjustable hyperparameter, usually defaulting to 1; Since the vehicle historical trajectory data has time series characteristics, the present invention adopts a long short-term memory neural network LSTM As an encoder, it preliminarily captures the hidden state in the historical trajectory and performs feature encoding. The target vehicle and surrounding vehicles are respectively LSTM The encoder encodes, and each LSTM The encoders share weights, thereby reducing the number of network parameters, model complexity, and the risk of overfitting; The multi-layer perceptron ( MLP ) Output Moment embedding vector enter LSTM Encoder, get The process of hiding the state at all times is as follows: Formula (3): ; Formula (4): ; Formula (5): ; Formula (6): ; Formula (7): ; Formula (8): ; in: , , They are the output of the input gate, the output of the forget gate, and the output of the output gate. , , , are weight matrices, , , , are all weight matrices related to the hidden state at time t-1, , , , is the bias vector, is the activation function, For vehicles exist The state of the candidate cell at time for The candidate cell states at time t, For passing The encoder output is labeled Vehicles in history The hidden state of the moment.

[0032] The multi-head spatial attention module is used to calculate the spatial correlation between the target vehicle and each vehicle within a preset range (with the target vehicle as the center and a radius of 90 meters around it) according to the calculation formula of the attention mechanism. The core of the attention mechanism is to query ( Q ),key( K ) and value ( V ) The relationship between the three dynamically adjusts the degree of attention to the input information. The present invention adopts a multi-head spatial attention mechanism to capture the interaction between the target vehicle and the surrounding vehicles. , the correlation between the target vehicle and each surrounding vehicle is calculated through the calculation formula of the attention mechanism; From the known The target vehicle output by the encoder is The hidden state vector at time , the hidden state vector of the surrounding vehicles is { }, the surrounding vehicles are numbered 1-N, They represent the hidden states of the surrounding vehicles numbered 1-N output by the LSTM encoder at time t, and are input into the multi-head spatial attention module respectively. The calculation formula is as follows: Formula (9): ; Formula (10): ; Formula (11): ; Formula (12): ; in: , and are the query matrix of the target vehicle, the key matrix and the value matrix of the surrounding vehicles respectively obtained by linear layer transformation, , and is the weight matrix of the linear transformation, , and is the corresponding bias term, is the number of channels, output The first head in the multi-head attention The correlation between the target vehicle and other surrounding vehicles calculated at all times; The multi-head spatial attention module in this invention adopts 4 head attention layers, and finally the outputs of the four head attention layers are spliced ​​according to the last dimension to obtain ; Then go through GLU After activation function and residual connection, layer normalization is performed to obtain the hidden state after considering the interaction between the target vehicle and surrounding vehicles: Formula (13): ; in, To consider the hidden state after the interaction between the target vehicle and the surrounding vehicles, LayerNorm is the layer normalization function and GLU is the activation function. The attention obtained by concatenating the outputs of the four head attention layers according to the last dimension, is the bias matrix, for The target vehicle output by the encoder is The hidden state vector at time t.

[0033] Large kernel convolution pooling module, used to capture the interaction between surrounding vehicles other than the target vehicle within a preset range through convolution layers and pooling layers with multiple different convolution kernels; In order to more accurately predict the future position of the vehicle, the spatial interaction of other surrounding vehicles cannot be ignored. The present invention proposes a large-kernel convolution pooling module, which consists of multi-scale convolution and adaptive average pooling layers. It can effectively obtain information in the local neighborhood of the vehicle and a larger receptive field, thereby modeling the spatial relationship between vehicles. Figure 2 The simplified structural diagram of the module is shown; The large kernel convolutional pooling module consists of three convolutional layers and an adaptive average pooling layer. The input is ( LSTM The hidden state vector of the surrounding vehicles output by the encoder module), the three convolutional layers are , DW - D - Conv , 1 1 Conv ; First, through 5 5 depth convolution ( ) Capture the detailed features of each surrounding vehicle in the local space; Then, using 7 7 dilated convolutional layers ( DW - D - Conv ) Expand the receptive field so that the network can obtain vehicle interaction information in a larger range while maintaining resolution; Next, use a layer 1 1 convolution (1 1 Conv ) Fuse the features of different channels, integrate the information extracted by multi-layer convolution, and enhance the feature expression ability; Finally, through the adaptive average pooling layer ( Adaptive Average Pooling ) The extracted information is further compressed and mapped into a fixed-size feature representation so that it can be effectively spliced ​​with the hidden features output by the multi-head spatial attention module, thereby improving the ability to model vehicle interaction relationships; The complete feature extraction process is shown in the following formula: Formula (14): ; Formula (15): ; Formula (16): ; Formula (17): ; Formula (18): ; Among them: The input of the large kernel convolution pooling module is the hidden state vector of the surrounding vehicles output by the LSTM encoder module { }, the surrounding vehicles are numbered 1-N, They represent the hidden states of the surrounding vehicles numbered 1-N output by the LSTM encoder at time t, , , and is the hidden state during the convolution process, , and are convolution kernels, representing 5 5, 7 7 and 1 1 small local matrix, , and is the network bias, It is the hidden state output after the features are extracted by the LKCP module.

[0034] Convolution modulation module is used for feature extraction. LSTM The encoder then captures the temporal correlation of the vehicle’s historical trajectory information again; Since the future driving path of a vehicle is closely related to its past historical trajectory, the present invention uses a convolutional modulation module to simulate and replace the self-attention mechanism to capture the hidden state of the target vehicle's historical trajectory in the time dimension. Convolutional modulation calculates the large kernel convolution output and the value ( V ) Hadamard product, simulating the calculation process of the self-attention mechanism, that is, using the convolution feature pair v Modulation is performed, this method maintains Transformer While taking advantage of the framework, it can also reduce computational complexity and thus reduce reasoning time. Figure 3 Shown is the architecture of the convolutional modulation module; The convolutional modulation module consists of layer normalization, 1 1 convolution, GELU Activation function and 7 7 group convolutions, the input vector of the convolution modulation module is (Multi-head spatial attention module( S ) and large kernel convolution pooling module ( LKP ) output hidden state vector concatenation); First, Processing is done through layer normalization functions. Layer normalization helps standardize input features, ensures that the model stably learns the feature distribution of different time steps, and reduces model learning bias; Then, the results of layer normalization are used for feature extraction through two routes. The first route is 1 1. Convolution is used to simulate the linear transformation in the self-attention mechanism, reconstruct the feature expression, enhance the model's ability to capture deep correlations between time steps in a short period of time, and output the feature vector v , the second route is through 1 1 convolution and 7 7 kernels are grouped and convolved to output feature vectors ; pass Operations,modulate the extracted features to dynamically identify and capture key moments in the prediction process, allowing the model to more accurately predict future trajectories; Finally, through 1 1 Projection convolution corrects the feature dimension of the modulated hidden features to obtain the final output.

[0035] according to Figure 1 As shown in the architecture, the output of the multi-head spatial attention mechanism and The output of the modules is concatenated and input into the convolution modulation module for time feature extraction. The complete process is shown in the following formula: Formula (19): ; Formula (20): ; Formula (21): ; Formula (22): ; Final Output It is the hidden state of the target vehicle's historical trajectory data generated based on the consideration of vehicle spatial interaction and trajectory data temporal dependency. The input vector of the convolutional modulation module is , To consider the hidden state after the interaction between the target vehicle and surrounding vehicles, is the hidden state output after the LKCP module extracts features, is the hidden state of the input vector of the convolutional modulation module after layer normalization. For the general Pass 1 1 convolution and 7 The 7 kernels are grouped and convolved to produce the feature vector v. Pass 1 1 convolution is performed to output the feature vector.

[0036] Intent recognition module and decoder module (composed of FC (Fully connected layer)+ LSTM (Long Short-Term Memory Neural Network)+ FC (Fully connected layer) outputs the predicted future trajectory of the target vehicle.

[0037] The intention recognition module in the present invention can capture the lateral (left lane change, right lane change, lane keeping) and longitudinal intentions (acceleration, deceleration, constant speed) of the vehicle, thereby helping the model to better cope with the prediction requirements of multimodal trajectories and improve the accuracy and stability of prediction. The intention recognition module structure is mainly composed of a feedforward layer, a normalized activation layer and two linear classifiers. Figure 4 As shown; The information of the target vehicle output from the convolution modulation module Encode and extract all hidden state information of each sample historical data at each time step; The extracted hidden state information is then passed through the feedforward layer and the normalized activation layer for feature extraction. The feedforward layer consists of a fully connected layer ( FC), the fully connected layer uses ELU Activation function, the normalized activation layer consists of a fully connected layer ( FC ), LayerNorm and ELU Activation function composition; Finally, the output of the normalized activation layer is fed to the vertical classifier and the horizontal classifier respectively, and then the full connection layer ( FC )and The classification activation function calculates the weighted probabilities of each longitudinal and lateral maneuver category at each future prediction time step. The longitudinal maneuver categories include vehicle acceleration, deceleration, and constant speed. The lateral maneuver categories include vehicle left lane change, right lane change, and no lane change (lane keeping).

[0038] Since the future trajectory of a vehicle is uncertain in reality, the present invention considers that the future predicted trajectory presents a multimodal distribution. Therefore, the present invention uses a binary Gaussian distribution to quantify the multimodal trajectory, and thus defines and The predicted trajectory points are and Expectations in direction, and The predicted trajectory points are and Variance in direction. is the correlation coefficient.

[0039] Depend on Figure 1 As we know, the decoder module consists of two fully connected layers and one The last fully connected layer is used to connect the first fully connected layer and LSTM The information after layer encoding is mapped from feature space to trajectory space. To consider the uncertainty of prediction, it is assumed that the decoder prediction result is considered to obey a binary Gaussian distribution, so the decoder can output the prediction of each future prediction time step. , , , and ,in is the future prediction time step, , the multimodal trajectory follows the total probability theorem, as shown in the following formula: Formula (23): ; in: represents the coordinates at the future prediction time step, For historical time period All vehicle states at all times, in the historical time period At a certain moment The vehicle state is defined as ,in Represents the state of the vehicle, Represents the vehicle number, ,in and Represents the target vehicle in Moment and coordinate, and Represents the target vehicle in The speed and acceleration of the moment, represent The lane where the target vehicle and all vehicles within the preset range around it are located at all times , express The vehicle type (coded by numbers) of the target vehicle and all vehicles within a 90-meter radius around it at the moment; Represents horizontal strategy, Represents vertical strategy, and They represent the probability of the vehicle performing lateral and longitudinal strategic maneuvers based on state X, respectively. is the Gaussian distribution probability of the vehicle's future trajectory predicted based on the vehicle's historical state and lateral and longitudinal maneuvers, ,in , represents the bivariate Gaussian distribution parameters at each time step in the prediction time domain, and The predicted trajectory points are and Expectations in direction, and The predicted trajectory points are and The variance in direction, is the correlation coefficient, and Represents the prediction time step The x-coordinate at the predicted time step The y coordinate of and Represents the prediction time step The Gaussian distribution parameter set at the prediction time step The set of Gaussian distribution parameters at .

[0040] Evaluation indicators: In the evaluation phase, the root mean square error (RMSE) within the next 5-second prediction period is calculated. RMSE ) and the negative log-likelihood ( NLL ) to evaluate the accuracy of the predicted trajectory. The following are the MSE, NLL and RMSE formula: Formula (24): ; Formula (25): ; Formula (26): ; in: Represents the error between the predicted future coordinate point and the future actual coordinate point. For other related parameters, see formula (23) for detailed explanation. RMSE is used to evaluate the error between the predicted trajectory and the actual trajectory. NLL is used to evaluate the error of driving maneuver behavior.

[0041] Compare models: (1) CV : Early model-based methods used a constant-speed Kalman filter and assumed that the vehicle moves at a constant speed.

[0042] (2) S - LSTM :This model treats each traffic participant separately through LSTM The social pooling layer is used to model the interaction between traffic participants, and the future coordinate positions are predicted based on the historical coordinate position information only.

[0043] (3) PIP :This method is composed of H . Song et al. proposed that, unlike the traditional prediction method that only relies on historical position data, it couples prediction with planning and assists prediction by introducing the planning information of the vehicle itself.

[0044] (4) Comparative Model 4: This method is based on Guo , Hongyan et al. proposed that this method predicts the future trajectory of the target vehicle while considering the motion trend of the ego vehicle by introducing an attention mechanism and a convolutional pooling layer.

[0045] (5) STDAN :based on LSTM The encoder-decoder, spatial attention mechanism, and temporal attention mechanism predict the multimodal trajectory of the target vehicle.

[0046] (6) Att - GCN - LSTM :based on LSTM The encoder-decoder, graph convolutional neural network, and attention mechanism predict the multimodal trajectory of the target vehicle and use the predicted information in the vehicle path planning task.

[0047] Simulation experiment results and visualization: as shown in Table 1-2, Figure 5-Figure 8shown.

[0048] Table 1 is based on NGSIM Models of the dataset RMSE value

[0049] Table 2 is based on NGSIM Models of the dataset NLL value

[0050] Table 1 and Table 2 show the prediction accuracy of the prediction system proposed by the present invention. Figure 5 and Figure 6 The corresponding RMSE and NLL prediction accuracy comparison chart shows that the vehicle trajectory prediction system proposed in the present invention has a higher prediction accuracy. Figure 7 and Figure 8 In the figure, the historical trajectories of the surrounding vehicles and the historical trajectory of the target vehicle are represented by cyan and red, respectively, and the predicted trajectory of the target vehicle and its future true trajectory are represented by green and purple with 50% transparency, respectively. Figure 7 (a), (b), and (c) represent lane keeping in sparse, medium, and dense traffic conditions, respectively. Figure 8 (a), (b), and (c) represent left lane changes in sparse, medium, and dense traffic conditions, respectively. Figure 8 (d), (e) and (f) represent right lane changes in sparse, medium and dense traffic conditions, respectively. The horizontal and vertical coordinates of the trajectory prediction in the pictures are in meters (m).

[0051] It should be noted that, in the present invention, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0052] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A vehicle trajectory prediction system integrating large kernel convolution and attention mechanism, characterized in that: The system includes: a data acquisition module, an encoder module, a multi-head spatial attention module, a large kernel convolution pooling module, a convolution modulation module, an intention recognition module and a decoder module; The data acquisition module is used to collect and track the trajectory data and lane information of the vehicle and other surrounding vehicles within a preset range in real time through the vehicle-mounted camera; The encoder module is used to encode the collected vehicle historical trajectory related data into a model that can be embedded in the spatial attention module and LKP The shape format of the module; The multi-head spatial attention module is used to calculate the spatial correlation between the target vehicle and each vehicle within the preset range according to the calculation formula of the attention mechanism; Large kernel convolution pooling module, used to capture the interaction between surrounding vehicles other than the target vehicle within a preset range through convolution layers and pooling layers with multiple different convolution kernels; Convolution modulation module is used for feature extraction. LSTM The encoder then captures the temporal correlation of the vehicle’s historical trajectory information again; The intention recognition module and the decoder module output the predicted future trajectory of the target vehicle.

2. According to claim 1, a vehicle trajectory prediction system integrating large kernel convolution and attention mechanism is characterized in that: The encoder module includes a multi-layer perceptron MLP and long short-term memory neural networks LSTM ; The multi-layer perceptron preprocesses the data. First, the historical status of the vehicle Through linear transformation, as follows: Formula (1): ; in: For Time Vehicle Status information, is the weight matrix of the linear layer, is the bias vector of the linear layer, and During the training process, changes will be continuously updated. is the result after linear transformation; The output of the linear layer in formula (1) is passed through Activation function processing to get the final output : Formula (2): ; in: To be embeddable The encoder vector, is an adjustable hyperparameter; Long Short-Term Memory Neural Network LSTM As an encoder, it preliminarily captures the hidden state in the historical trajectory and performs feature encoding. The target vehicle and surrounding vehicles are respectively LSTM The encoder encodes, and each LSTM Encoders share weights, thus reducing the number of network parameters; The output of the multi-layer perceptron Moment embedding vector enter LSTM Encoder, get The process of hiding the state at all times is as follows: Formula (3): ; Formula (4): ; Formula (5): ; Formula (6): ; Formula (7): ; Formula (8): ; in: , , They are the output of the input gate, the output of the forget gate, and the output of the output gate. , , , are weight matrices, , , , are all weight matrices related to the hidden state at time t-1, , , , is the bias vector, is the activation function, For vehicles exist The state of the candidate cell at time for The candidate cell states at time t, For passing The encoder output is labeled Vehicles in history The hidden state of the moment.

3. The vehicle trajectory prediction system integrating large kernel convolution and attention mechanism according to claim 1, characterized in that: The multi-head spatial attention module adopts a multi-head spatial attention mechanism to capture the interaction between the target vehicle and the surrounding vehicles at each moment. , the correlation between the target vehicle and each surrounding vehicle is calculated through the calculation formula of the attention mechanism; From the known The target vehicle output by the encoder is The hidden state vector at time , the hidden state vector of the surrounding vehicles is { }, the surrounding vehicles are numbered 1-N, They represent the hidden states of the surrounding vehicles numbered 1-N output by the LSTM encoder at time t, and are input into the multi-head spatial attention module respectively. The calculation formula is as follows: Formula (9): ; Formula (10): ; Formula (11): ; Formula (12): ; in: , and are the query matrix of the target vehicle, the key matrix and the value matrix of the surrounding vehicles respectively obtained by linear layer transformation, , and is the weight matrix of the linear transformation, , and is the corresponding bias term, is the number of channels, output The first head in the multi-head attention The correlation between the target vehicle and other surrounding vehicles calculated at all times; The multi-head spatial attention module uses 4 head attention layers, and finally concatenates the outputs of the four head attention layers according to the last dimension to obtain ; Again will pass GLU After activation function and residual connection, layer normalization is performed to obtain the hidden state after considering the interaction between the target vehicle and surrounding vehicles: Formula (13): ; in, To consider the hidden state after the interaction between the target vehicle and the surrounding vehicles, LayerNorm is the layer normalization function and GLU is the activation function. The attention obtained by concatenating the outputs of the four head attention layers according to the last dimension, is the bias matrix, for The target vehicle output by the encoder is The hidden state vector at time t.

4. The vehicle trajectory prediction system integrating large kernel convolution and attention mechanism according to claim 3 is characterized in that: The large kernel convolutional pooling module consists of three convolutional layers and an adaptive average pooling layer, and the input is , the three convolutional layers are 5 5. Depthwise Convolution , 7 7 dilated convolution DW-D-Conv, 1 1Convolution1 1Conv; First, through 5 5. Depthwise Convolution Capture the detailed features of each surrounding vehicle in the local space; Then, using 7 7. Dilated convolution DW-D-Conv expands the receptive field, allowing the network to obtain vehicle interaction information in a larger range while maintaining resolution; Next, use a layer 1 1Convolution1 1Conv fuses the features of different channels, integrates the information extracted by multi-layer convolution, and enhances the feature expression capability; Finally, the extracted information is further compressed through the adaptive average pooling layer and mapped into a fixed-size feature representation so that it can be effectively spliced ​​with the hidden features output by the multi-head spatial attention module, thereby improving the ability to model vehicle interaction relationships; The feature extraction process is shown as follows: Formula (14): ; Formula(15): ; Formula (16): ; Formula(17): ; Formula(18): ; Among them: The input of the large kernel convolution pooling module is the hidden state vector of the surrounding vehicles output by the LSTM encoder module { }, the surrounding vehicles are numbered 1-N, They represent the hidden states of the surrounding vehicles numbered 1-N output by the LSTM encoder at time t, , , and is the hidden state during the convolution process, , and are convolution kernels, representing 5 5, 7 7 and 1 1 small local matrix, , and is the network bias, It is the hidden state output after the features are extracted by the LKCP module.

5. The vehicle trajectory prediction system integrating large kernel convolution and attention mechanism according to claim 3, characterized in that: The convolutional modulation module consists of layer normalization, 1 1 convolution, GELU Activation function and 7 7 group convolutions, the input vector of the convolution modulation module is ; First, Processed by layer normalization function, layer normalization helps to standardize input features; Then, the results of layer normalization are used for feature extraction through two routes. The first route is 1 1. Convolution is used to simulate the linear transformation in the self-attention mechanism, reconstruct the feature expression, enhance the model's ability to capture deep correlations between time steps in a short period of time, and output the feature vector v , the second route is through 1 1 convolution and 7 7 kernels are grouped and convolved to output feature vectors ; pass Operation, modulate the extracted features to dynamically identify and capture key moments in the prediction process, allowing the model to more accurately predict future trajectories; Finally, through 1 1 Projection convolution corrects the feature dimension of the modulated hidden features to obtain the final output; The output of the multi-head spatial attention mechanism and The output of the modules is concatenated and input into the convolution modulation module for time feature extraction, as shown in the following formula: Formula(19): ; Formula(20): ; Formula(21): ; Formula(22): ; Final Output It is the hidden state of the target vehicle's historical trajectory data generated based on the consideration of vehicle spatial interaction and trajectory data temporal dependency. The input vector of the convolutional modulation module is , To consider the hidden state after the interaction between the target vehicle and surrounding vehicles, is the hidden state output after the LKCP module extracts features, is the hidden state of the input vector of the convolutional modulation module after layer normalization. For the general Pass 1 1 convolution and 7 The 7 kernels are grouped and convolved to produce the feature vector v. Pass 1 1 convolution is performed to output the feature vector.

6. The vehicle trajectory prediction system integrating large kernel convolution and attention mechanism according to claim 3, characterized in that: The structure of the intention recognition module is mainly composed of a feedforward layer, a normalized activation layer and two linear classifiers; The information of the target vehicle output from the convolution modulation module Encode and extract all hidden state information of each sample historical data at each time step; The extracted hidden state information is then passed through the feedforward layer and the normalized activation layer for feature extraction. The feedforward layer consists of a fully connected layer. The fully connected layer uses ELU Activation function, the normalized activation layer consists of a fully connected layer, LayerNorm and ELU Activation function composition; Finally, the output of the normalized activation layer is fed to the vertical classifier and the horizontal classifier respectively, and then the full connection layer and The classification activation function calculates the weighted probabilities of each longitudinal and lateral maneuver category at each future prediction time step. The longitudinal maneuver categories include vehicle acceleration, deceleration, and constant speed. The lateral maneuver categories include vehicle left lane change, right lane change, and no lane change.

7. The vehicle trajectory prediction system integrating large kernel convolution and attention mechanism according to claim 3, characterized in that: The decoder module consists of two fully connected layers and a The last fully connected layer is used to connect the first fully connected layer and LSTM The information after layer encoding is mapped from feature space to trajectory space. To consider the uncertainty of prediction, it is assumed that the decoder prediction result is considered to obey a binary Gaussian distribution, so the decoder can output the prediction of each future prediction time step. , , , and ,in is the future prediction time step, , the multimodal trajectory follows the total probability theorem, as shown in the following formula: Formula(23): ; in: represents the coordinates at the future prediction time step, For historical time period All vehicle states at all times, in the historical time period At a certain moment The vehicle state is defined as ,in Represents the state of the vehicle, Represents the vehicle number, ,in and Represents the target vehicle in Moment and coordinate, and Represents the target vehicle in The speed and acceleration of the moment, represent The lane where the target vehicle and all vehicles within the preset range around it are located at all times , express The vehicle type of the target vehicle and all vehicles within a 90-meter radius around it at all times; Represents horizontal strategy, Represents vertical strategy, and They represent the probability of the vehicle performing lateral and longitudinal strategic maneuvers based on state X, respectively. is the Gaussian distribution probability of the vehicle's future trajectory predicted based on the vehicle's historical state and lateral and longitudinal maneuvers, ,in , represents the bivariate Gaussian distribution parameters at each time step in the prediction time domain, and The predicted trajectory points are and Expectations in direction, and The predicted trajectory points are and The variance in direction, is the correlation coefficient, and Represents the prediction time step The x-coordinate at the predicted time step The y coordinate of and Represents the prediction time step The Gaussian distribution parameter set at the prediction time step The set of Gaussian distribution parameters at .

8. A vehicle trajectory prediction method integrating large kernel convolution and attention mechanism, characterized in that: The method comprises: S1. Collect and track the trajectory data and lane information of the vehicle and other surrounding vehicles within a preset range in real time through the vehicle-mounted camera; S2, encode the collected vehicle historical trajectory related data into a data structure that can be embedded in the spatial attention module and LKP The shape format of the module; S3, calculating the spatial correlation between the target vehicle and each vehicle within the surrounding preset range according to the calculation formula of the attention mechanism; S4, through multiple convolution layers and pooling layers with different convolution kernels, capture the interaction between other surrounding vehicles except the target vehicle within the preset range; S5, perform feature extraction, through convolution, and then LSTM The encoder then captures the temporal correlation of the vehicle’s historical trajectory information again; S6. Output the predicted future trajectory of the target vehicle.

Images