Attention mechanism-based stacked LSTM trajectory prediction method in multi-modal environment

Abstract

The invention discloses a stacked LSTM trajectory prediction method based on an attention mechanism in a multi-modal environment. The method comprises the following steps: aligning modal data through a sliding window mechanism; designing a three-layer stacked LSTM (Long Short Term Memory) network structure, namely an AS-LSTM network structure; the first layer LSTM inputs inertial measurement unit IMU feature splicing PointNet + + extracted point cloud feature, the second layer LSTM inputs first layer linear processed hidden state splicing CNN extracted image feature, and the third layer LSTM inputs second layer linear processed hidden state splicing GPS feature; and an attention mechanism is added to the hidden state output by the third layer of LSTM to calculate a dynamic weight, and the position prediction at the next moment is output through a Dense layer. And by taking the mean square error and the decision coefficient as evaluation indexes, constructing an objective function for minimizing the prediction error. And an objective function value lower than that of other LSTM prediction methods can be obtained after NCLT data set training.

Description

Technical Field

[0001] The present invention belongs to the fields of multimodal fusion technology, vehicle trajectory prediction technology, and deep learning, and specifically relates to a trajectory prediction method of stacked LSTM based on an attention mechanism in a multimodal environment, which is used to improve the accuracy and robustness of LSTM trajectory prediction in a vehicle network environment. Background Art

[0002] Accurately predicting vehicle trajectories is crucial for improving the performance of autonomous driving systems. Existing trajectory prediction methods primarily rely on single sensors or models, failing to effectively handle the fusion of multimodal data in complex environments, resulting in insufficient prediction accuracy. Traditional trajectory prediction methods based on kinematic principles, such as the dynamic window method and temporal elastic band, have achieved some success but perform poorly when dealing with complex environments and uncertainties. In recent years, deep learning-based models, particularly LSTMs, have been widely used for predicting time series data. However, effectively fusing data from different sensors when processing multimodal inputs remains a challenge. The introduction of the attention mechanism offers a new approach to addressing this issue. Existing trajectory prediction models fail to fully exploit the temporal relationships and dynamic characteristics between sensors, resulting in insufficient performance in changing and complex environments.

[0003] Technical comparison with patent CN111027461A "Vehicle trajectory prediction method based on multi-dimensional single-step LSTM network"

[0004] Model elements

[0005] Patent CN111027461A uses data from a monocular camera to acquire coordinate information frame by frame. This information is then separated into horizontal and vertical coordinates to construct a sequence. A single-layer LSTM is used based on a sliding window to predict the horizontal and vertical coordinates of the next time step based on the previous two time steps.

[0006] This invention not only utilizes data from a single camera but also extends it to three monocular cameras. It also incorporates data from other sensor modalities, such as LiDAR, GPS, and IMU data, to comprehensively predict single-step GPS data. It explicitly aligns the timestamps of each sensor data using a sliding window. It expands from a single-layer LSTM to a multi-layer LSTM, and incorporates an attention mechanism in the final layer to maintain focus on the time series of nearby prediction points. This ensures accurate prediction of the next position even when all modalities are missing.

[0007] Feature processing and decision-making mechanism

[0008] Patent CN111027461A requires manual acquisition of camera distortion coefficients to correct video data, and the position information of the next time step is only predicted based on the information of the previous two time steps.

[0009] This method uses a CNN neural network to extract camera data features and automatically adjusts them. Feature extraction reduces data dimensionality and reduces interference from irrelevant data. The horizontal, vertical, and vertical coordinates of the next time step, i.e., the GPS coordinates, are predicted from the previous 4-20 time steps. This allows for better extraction of long-term motion patterns in the surrounding environment and timely correction of predicted positions.

[0010] Technical comparison with patent CN117114051A "Ship trajectory prediction method based on LSTM and self-attention mechanism"

[0011] Model elements

[0012] Patent CN117114051A uses sliding windows and normalization to construct a model input sequence for training. It also predicts the trajectory of the next time step based on a single-layer LSTM and self-attention mechanism.

[0013] Building on this model, the present invention expands the input of unimodal data to multimodal data, expands the single-layer LSTM to three or more layers, and adds a temporal attention mechanism to the last layer to predict the trajectory of the next time step. The integration of multimodal information helps maintain robustness when modalities are missing.

[0014] Feature processing and decision-making mechanism

[0015] Patent CN117114051A collects ship AIS data and pre-processes it, then uses cubic spline interpolation for resampling and uses data differencing to eliminate long-term trajectory trends.

[0016] The present invention is based on the CNN neural network and the PointNet++ neural network, which respectively perform feature processing on image data and 3D point cloud data, reducing the steps of manual parameter optimization and having generalization capabilities for sensors with different electrical and physical properties of the same modality.

[0017] This paper proposes an LSTM model that combines multimodal data features and attention mechanism, which can more accurately capture trajectory changes and dynamic environment changes. Summary of the Invention

[0018] The main purpose of this paper is to improve the trajectory prediction accuracy of LSTM in a multimodal environment. This paper provides a trajectory prediction method for multimodal environments using a stacked LSTM layer based on an attention mechanism. This method stacks multiple LSTM layers to layer-by-layer fuse data from IMU, GPS, LiDAR, and image sensors. It also introduces an attention mechanism to dynamically focus on data at key time steps, thereby improving prediction accuracy and robustness.

[0019] In order to achieve the above-mentioned object of the invention, the trajectory prediction method of the stacked LSTM based on the attention mechanism in a multimodal environment of the present invention includes the following steps:

[0020] S1 performs sliding window alignment on multimodal data based on timestamps to ensure synchronization of sensor data.

[0021] S2 extracts sensor data features and concatenates them with the processed outputs of each layer as inputs to define the model;

[0022] S3 processes the output of the last layer through the attention mechanism and linear layer to obtain the trajectory prediction value at the next moment;

[0023] S4 training and optimization;

[0024] The model is trained using the NCLT dataset or other multimodal sensor datasets. The multimodal sensor dataset must contain at least inertial measurement unit data and global positioning system data, and the mean square error is used as the loss function.

[0025]

[0026] Among them, t1, t2 are the time ranges of the prediction set, P t is the actual trajectory point of the prediction set, is the predicted trajectory point of the model on the prediction set;

[0027] The Adam optimizer is used for optimization. The parameter to be adjusted in the model is set to θ. In the kth round of training, Adam is used to optimize the MSE loss given above. L(θ) is the MSE loss when the parameter is θ.

[0028]

[0029] S4.1 Calculate the gradient g of the kth round k ;

[0030]

[0031] in That is, L(θ) is proportional to the parameters θ i where i = 1, 2, ..., d. d is the number of parameters to be updated. Indicates that the current parameter θ is the parameter θ at the k-1th round k-1 .

[0032] S4.2 First-order and second-order moment estimation;

[0033] m k =β1m k-1 +(1-β1)g k

[0034]

[0035] where mk g k The sliding average, v k for β1 is the first-order moment exponential decay rate, and β2 is the second-order moment exponential decay rate. Generally, β1 = 0.9 and β2 = 0.999.

[0036] S4.3 Bias correction;

[0037]

[0038] in is m after deviation correction in the kth round k , is v after the k-th round of deviation correction k . is the kth power of β1, is β2 to the power of k.

[0039] S4.4 parameter update;

[0040]

[0041] Where α is the learning rate, which can be set to 0.001, and ε is a small constant to prevent the denominator from being 0, which is set to 10 -8 ;

[0042] Each time forward and backward propagation is performed, the gradient g is calculated k After that, the above S4.2-S4.4 will be performed automatically until the model converges or the upper limit of training rounds is reached. Through multiple rounds of training, the model can be adaptively optimized on different sensor data, and finally achieve high-precision trajectory prediction.

[0043] As a further improvement of the present invention, step S1 is specifically as follows: taking the timestamp t0 of the IMU data as a reference, setting the maximum matching time Δt, in milliseconds, and matching the data corresponding to the timestamps of other sensors in the timestamp window of [t0-Δt, t0+Δt] including lidar data, image data and global positioning system data.

[0044] As a further improvement of the present invention, the specific steps of step S2 are as follows:

[0045] S2.1 sensor data feature extraction;

[0046] The LiDAR 3D point cloud data is extracted using the PointNet++ neural network. The image data is reduced in dimension using the CNN neural network to extract the IMG′ feature. The inertial measurement unit and global positioning system data are directly used as the IMU and GPS features. All features are divided and concatenated by timestamp to obtain [Points, IMG′, IMU, GPS] and input into the AS-LSTM model.

[0047] S2.2 AS-LSTM is defined as a multi-layer stacked LSTM with attention;

[0048] The standard LSTM unit is used as follows

[0049] f t =σ(W fh h t-1 +W fx x t +b f )

[0050] i t =σ(W ih h t-1 +W ix x t +b i )

[0051]

[0052] o t =σ(W oh h t-1 +W ox x t +b o )

[0053] h t =o t ·tanh(c t )

[0054] where x t is the input vector at time step t, h t-1 is the hidden state of the previous time step t-1, c t-1 is the cell state at the previous time step t-1, f t It is the forget gate, i t is the input gate, is the candidate cell state, c t is the cell state at the current moment, h t is the hidden state at the current moment, o tis the output gate. W in each formula is the weight matrix and b is the bias, which are all parameters to be updated. Let N be the unit sequence length of a layer of LSTM, D be the dimension of the input feature x, and the state space corresponding to each weight matrix and bias;

[0055] W fh ∈R N×N ,W fx ∈R N×D ,b f ∈R N

[0056] W ih ∈R N×N ,W ix ∈R N×D ,b i ∈R N

[0057] W ch ∈R N×N ,W cx ∈R N×D ,b c ∈R N

[0058] W oh ∈R N×N ,W ox ∈R N×D , b o ∈R N

[0059] All weight matrices W and bias b are learnable parameters. ⊙ is used for element-by-element multiplication of the gate vector and the state vector. σ(·) and tanh(·) are gating functions. The corresponding formula is

[0060]

[0061] Where z is the input number, e -z It is an exponential function with base e. When a vector is input, each number in the vector is calculated separately. For a certain LSTM cell unit, the input vector x at time t is t , simplified to the following formula

[0062] (h t ,c t )=LSTM(x t ,h t-1 ,c t-1 )

[0063] The first layer inputs IMU and Points, and the second layer inputs the output of the first layer. The results after processing And image data IMG', the third layer inputs the second layer output The results after processing The input and output of the AS-LSTM layer are as follows:

[0064]

[0065] The upper right corner subscript (i) represents the i-th layer, for example is the hidden output of the first layer at time t, IMU t is the inertial measurement unit data at time t, IMG′ t Point is the feature of the image data at time t after being processed by the CNN neural network. t The t-time feature of the 3D point cloud data of the multi-threaded lidar extracted by PointNet++, GPS t The GPS data at time t, including longitude, latitude, and altitude.

[0066] As a further improvement of the present invention, step S3 is specifically as follows: the model's attention to different time steps and different modal data is dynamically adjusted according to the current input weight, the correlation of each time step is calculated, and the weight is obtained by normalization using the Softmax function. The final weighted sum is output, and the output is processed by the Dense layer to obtain the final trajectory prediction result, that is, the formula corresponding to the vehicle's position information at the future time is:

[0067]

[0068] Among them, Q is the query matrix, K is the key matrix, and V is the value matrix. is the hidden unit vector of the third layer at time tN, t-N+1, ..., t. Q ,W K ,W V is the weight matrix, which is updated during training. is the hidden output of the third layer LSTM, e i is the scaled dot product The calculated value, d k is the dimension of Q, K. i , K i ) is a scoring function, which is set to the scaled dot product in the present invention. i for e i The value after Softmax normalization, Score for attention, It is a simplified representation of several computational steps of the attention mechanism, input After a series of calculations, the final sum is obtained to get the attention score. t+1It is the global system positioning value at the next moment, which is transformed by the Dense layer on the attention score and corresponds to the predicted trajectory.

[0069] Beneficial effects:

[0070] 1. The prediction accuracy is significantly improved;

[0071] Experiments on the NCLT dataset show that compared with traditional LSTM, Attention LSTM and Seq2Seq LSTM models, AS-LSTM has lower mean square error (MSE) and lower coefficient of determination (R 2 ) is higher, verifying its accuracy advantage.

[0072] 2. Strong robustness in complex environments;

[0073] The complementarity of multimodal data (such as relying on IMU / lidar when GPS fails) enhances the stability of the model in scenarios such as occlusion and signal loss, and is suitable for urban roads and mixed indoor and outdoor environments.

[0074] 3. Efficient feature fusion capability;

[0075] Combining PointNet++ (for processing 3D point clouds) and CNN (for extracting image semantics) effectively captures spatial structure information and avoids the limitations of manual feature engineering.

[0076] 4. Scalability and generalization;

[0077] The model can be flexibly expanded to include sensors of more modalities (such as millimeter-wave radars and depth cameras), and the training framework is compatible with public datasets (such as KITTI and NuScenes), making it widely applicable. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 Shows the structural framework of the AS-LSTM model. DETAILED DESCRIPTION

[0079] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0080] The present invention relates to a trajectory prediction method based on a stacked long short-term memory network with an attention mechanism in a multimodal environment. The method comprises: wherein the structural framework is as follows Figure 1The figure shows how different sensor data is fed into the LSTM network layer by layer and fused to achieve high-precision trajectory prediction. The input and processing flow of sensor data (such as IMU, GPS, point cloud, and image) in the model enhances the focus on key time steps and modal data through stacked LSTM layers and the attention mechanism.

[0081] Modal data is aligned using a sliding window mechanism. A three-layer stacked LSTM network structure, known as the AS-LSTM network, is designed. The first LSTM layer takes as input the IMU features combined with point cloud features extracted by PointNet++. The second LSTM layer takes as input the hidden state of the first layer after linear processing combined with image features extracted by CNN. The third LSTM layer takes as input the hidden state of the second layer after linear processing combined with GPS features. An attention mechanism is applied to the hidden state output of the third LSTM layer to calculate dynamic weights, and a dense layer is used to output the next moment's position prediction. Using mean squared error and coefficient of determination as evaluation metrics, an objective function is constructed to minimize the prediction error. After training on the NCLT dataset, the proposed method achieves lower objective function values than other LSTM prediction methods.

[0082] This paper aims to improve the trajectory prediction accuracy of LSTM under multimodal conditions. It proposes an improved LSTM trajectory prediction method, AS-LSTM, based on multimodal data fusion and an attention mechanism. This method stacks multiple LSTM layers to layer-by-layer fuse data from IMU, GPS, lidar, and image sensors, and introduces an attention mechanism to dynamically focus on data at key time steps, thereby improving prediction accuracy and robustness.

[0083] The AS-LSTM trajectory prediction method proposed in the present invention includes the following steps:

[0084] S1 performs sliding window alignment on multimodal data according to timestamps to ensure synchronization of sensor data.

[0085] Based on the timestamp t0 of the IMU data, set the maximum matching time Δt, which is generally in milliseconds. Match the data corresponding to the timestamps of other sensors (lidar data, image data, GPS data) within the timestamp window [t0-Δt, t0+Δt].

[0086] S2 extracts the sensor data features and concatenates them with the processed outputs of each layer as the input of each layer to define the model.

[0087] S2.1 sensor data feature extraction;

[0088] The LiDAR 3D point cloud data is extracted using the PointNet++ neural network. Image data is reduced in dimension using a CNN neural network, and the Image′ feature is extracted. Inertial measurement unit and global positioning system data are directly used as IMU and GPS features. All features are concatenated by timestamp to generate [Points, Image′, IMU, GPS], which is then fed into the AS-LSTM model.

[0089] S2.2 AS-LSTM definition (multi-layer stacked LSTM with attention);

[0090] The standard LSTM unit is used as follows;

[0091] f t =σ(W fh h t-1 +W fx x t +b f )

[0092] i t =σ(W ih h t-1 +W ix x t +b i )

[0093]

[0094] o t =σ(W oh h t-1 +W ox x t +b o )

[0095] h t =o t ·tanh(c t )

[0096] where x t is the input vector at time step t, h t-1 is the hidden state of the previous time step t-1, c t-1 is the cell state at the previous time step t-1, f t It is the forget gate, i t is the input gate, is the candidate cell state, c t is the cell state at the current moment, h t is the hidden state at the current moment, o tIs the output gate. In each formula, W is the weight matrix and b is the bias, both of which are parameters to be updated. Let N be the length of the unit sequence of a layer of LSTM, D be the dimension of the input feature x, and the state space corresponding to each weight matrix and bias is ⊙ is used for element-by-element multiplication of the gate vector and the state vector. σ(·), tanh(·) are the gating functions, and the corresponding formula is

[0097]

[0098] Where z is the input number. When inputting a vector, each number in the vector is calculated separately. For a certain LSTM cell unit at time t, the input vector x t , simplified to the following formula

[0099] (h t ,c t )=LSTM(x t ,h t-1 ,c t-1 )

[0100] The first layer inputs IMU and Points, and the second layer inputs the output of the first layer. The results after processing And image data Image′, the third layer inputs the second layer output The results after processing The input and output of the AS-LSTM layer are as follows:

[0101]

[0102] The upper right corner subscript (i) represents the i-th layer, for example is the hidden output of the first layer at time t. t is the inertial measurement unit data at time t, IMG′ t Point is the feature of the image data at time t after being processed by the CNN neural network. t The t-time feature of the 3D point cloud data of the multi-threaded lidar extracted by PointNet++, GPS t The GPS data at time t, including longitude, latitude, and altitude.

[0103] S3 processes the output of the last layer through the attention mechanism and linear layer to obtain the trajectory prediction value at the next moment. The model's attention to different time steps and different modal data is dynamically adjusted according to the current input weight. The correlation of each time step is calculated and normalized using the Softmax function to obtain the weight. The final weighted sum is output and processed by the Dense layer to obtain the final trajectory prediction result, that is, the vehicle's position information at the future moment corresponds to the formula:

[0104]

[0105] Among them, Q is the query matrix, K is the key matrix, and V is the value matrix. is the hidden unit vector of the third layer at time tN, t-N+1, ..., t. Q , W K , W V is the weight matrix, which is updated during training. is the hidden output of the third layer LSTM, e i is the scaled dot product The calculated value, d k is the dimension of Q, K. i , K i ) is a scoring function, which is set to the scaled dot product in the present invention. i for e i The value after Soft max normalization, Score for attention, It is a simplified representation of several computational steps of the attention mechanism, input After a series of calculations, the final sum is obtained to get the attention score. t+1 It is the global system positioning value at the next moment, which is transformed by the Dense layer on the attention score and corresponds to the predicted trajectory.

[0106] S4 training and optimization;

[0107] The model is trained using the NCLT dataset. It can also be trained using other multimodal sensor datasets that contain at least inertial measurement unit data and global positioning system data. The mean square error is used as the loss function.

[0108]

[0109] Among them, t1, t2 are the time ranges of the prediction set, P t is the actual trajectory point of the prediction set, is the predicted trajectory point of the model on the prediction set.

[0110] The Adam optimizer is used for optimization. Set the model parameter to be adjusted to θ. In the kth round of training, use Adam to optimize the MSE loss given above. L(θ) is the MSE loss when the parameter is θ.

[0111]

[0112] S4.1 Calculate the gradient g of the kth round k ;

[0113]

[0114] in That is, L(θ) is proportional to the parameters θ i where i = 1, 2, ..., d. d is the number of parameters to be updated. Indicates that the current parameter θ is the parameter θ at the k-1th round k-1 .

[0115] S4.2 First-order and second-order moment estimation;

[0116] m k =β1m k-1 +(1-β1)g k

[0117]

[0118] where m k g k The sliding average, v k for β1 is the first-order moment exponential decay rate, and β2 is the second-order moment exponential decay rate. Generally, β1 = 0.9, β2 = 0.999;

[0119] S4.3 Bias correction;

[0120]

[0121] in is m after deviation correction in the kth round k , is v after the k-th round of deviation correction k . is the kth power of β1, is β2 to the power of k.

[0122] S4.4 parameter update;

[0123]

[0124] Where α is the learning rate, which can be set to 0.001, and ε is a small constant to prevent the denominator from being 0, which is set to 10 -8 .

[0125] Each time forward and backward propagation is performed, the gradient g is calculated k After that, you can repeat S4.2-S4.4 above until the model converges or reaches the upper limit of training rounds. Through multiple rounds of training, the model can be adaptively optimized on different sensor data, ultimately achieving high-precision trajectory prediction.

[0126] S5 method performance

[0127] In the experiment, we used the public NCLT dataset for verification and compared it with traditional LSTM, AttentionLSTM, Seq2Seq LSTM and other models. The experimental results are shown in the following table:

[0128] Table 1 Determination coefficient of each model prediction set at different learning rates under NCLT (without image dimension)

[0129]

[0130] Table 1 shows the changes in the determination coefficients of four LSTM models, including AS-LSTM, traditional LSTM, LSTM with attention mechanism, and decoder-encoder LSTM, on the NCLT dataset prediction set under the condition that the image dimension is not included and the learning rate settings are different.

[0131] Table 2 Mean square error of each model prediction set at different learning rates under NCLT (without image dimension)

[0132]

[0133] Table 2 shows the comparison of mean squared errors of the four models on the NCLT dataset prediction set under different learning rates when the box does not include the image dimension.

[0134] Table 3 Determination coefficient of each model prediction set (including image dimension) at different NCLT learning rates

[0135]

[0136] Table 3 shows the determination coefficient results of the four models on the NCLT dataset prediction set under different learning rate settings, including the image dimension.

[0137] Table 4 Mean square error (including image dimension) of each model prediction set at different learning rates of NCLT

[0138]

[0139]

[0140] Table 4 shows the mean squared error of the four models on the NCLT dataset prediction set under different learning rates when the image dimension is included.

[0141] Experimental results show that the AS-LSTM model exhibits excellent performance in all evaluation indicators, especially in processing high-dimensional data and complex dynamic environments, showing stronger robustness and prediction accuracy.

Claims

1. A stacked LSTM trajectory prediction method based on an attention mechanism in a multimodal environment, characterized by: The following steps are involved: S1 performs sliding window alignment on multimodal data based on timestamps to ensure synchronization of sensor data. S2 extracts sensor data features and concatenates them with the processed outputs of each layer as inputs to define the model; S3 processes the output of the last layer through the attention mechanism and linear layer to obtain the trajectory prediction value at the next moment; S4 training and optimization; The model is trained using the NCLT dataset or other multimodal sensor datasets. The multimodal sensor dataset must contain at least inertial measurement unit data and global positioning system data, and the mean square error is used as the loss function. Among them, t1, t2 are the time ranges of the prediction set, P t is the actual trajectory point of the prediction set, is the predicted trajectory point of the model on the prediction set; The Adam optimizer is used for optimization. The parameter to be adjusted in the model is set to θ. In the kth round of training, Adam is used to optimize the MSE loss given above. L(θ) is the MSE loss when the parameter is θ. S4.1 Calculate the gradient g of the kth round k ; in That is, L(θ) is proportional to the parameters θ i Partial derivative, i = 1, 2, ... d. d is the number of parameters to be updated, Indicates that the current parameter θ is the parameter θ at the k-1th round k-1 ; S4.2 First-order and second-order moment estimation; m k =β1m k-1 +(1-β1)g k where m k g k The sliding average, v k for The sliding average, β1 is the first-order moment exponential decay rate, β2 refers to the second-order moment exponential decay rate, take β1 = 0.9, β2 = 0.999; S4.3 Bias correction; in is m after deviation correction in the kth round k , is v after the k-th round of deviation correction k , is the kth power of β1, is the kth power of β2; S4.4 parameter update; Where α is the learning rate, which can be set to 0.001, and ε is a small constant to prevent the denominator from being 0, which is set to 10 -8 ; Each time forward and backward propagation is performed, the gradient g is calculated k After that, the above S4.2-S4.4 will be performed automatically until the model converges or the upper limit of training rounds is reached. Through multiple rounds of training, the model can be adaptively optimized on different sensor data, and finally achieve high-precision trajectory prediction.

2. The trajectory prediction method based on stacked LSTM with attention mechanism in a multimodal environment according to claim 1, characterized in that: The specific steps of step S1 are as follows: based on the timestamp t0 of the IMU data, a maximum matching time Δt is set in milliseconds, and the data corresponding to the timestamps of other sensors in the timestamp window [t0-Δt, t0+Δt] are matched, including lidar data, image data, and global positioning system data.

3. The trajectory prediction method based on stacked LSTM with attention mechanism in a multimodal environment according to claim 1, characterized in that: The specific steps of step S2 are as follows: S2.1 sensor data feature extraction; The LiDAR 3D point cloud data is extracted using the PointNet++ neural network. The image data is reduced in dimension and the IMG′ feature is extracted using the CNN neural network. The inertial measurement unit and global positioning system data are directly used as the IMU and GPS features. All features are divided and concatenated by timestamp to obtain [Points, IMG′, IMU, GPS] and input into the AS-LSTM model. S2.2 AS-LSTM is defined as a multi-layer stacked LSTM with attention; Use the standard LSTM unit as follows f t =σ(W fh h t-1 +W fx x t +b f ) i t =σ(W ih h t-1 +W ix x t +b i ) o t =σ(W oh h t-1 +W ox x t +b o ) h t =o t ·tanh(c t ) where x t is the input vector at time step t, h t-1 is the hidden state of the previous time step t-1, c t-1 is the cell state at the previous time step t-1, f t It is the forget gate, i t is the input gate, is the candidate cell state, c t is the cell state at the current moment, h t is the hidden state at the current moment, o t is the output gate. W in each formula is the weight matrix and b is the bias, which are all parameters to be updated. Let N be the unit sequence length of a layer of LSTM, D be the dimension of the input feature x, and the state space corresponding to each weight matrix and bias; W fh ∈R N×N ,W fx ∈R N×D ,b f ∈R N W ih ∈R N×N ,W ix ∈R N×D ,b i ∈R N W ch ∈R N×N ,W cx ∈R N×D ,b c ∈R N W oh ∈R N×N ,W ox ∈R N×D ,b o ∈R N All weight matrices W and bias b are learnable parameters. ⊙ is used for element-by-element multiplication of the gate vector and the state vector. σ(·) and tanh(·) are gating functions. The corresponding formula is Where z is the input number, e -z It is an exponential function with base e. When a vector is input, each number in the vector is calculated separately. For a certain LSTM cell unit, the input vector x at time t is t , simplified to the following formula (h t ,c t )=LSTM(x t ,h t-1 ,c t-1 ) The first layer inputs IMU and Points, and the second layer inputs the output of the first layer. The results after processing And image data IMG', the third layer inputs the second layer output Processed results The input and output of the AS-LSTM layer are as follows: z (2) t =W lin (2) h t (2) + blin (2) The upper right corner subscript (i) represents the i-th layer, for example is the hidden output of the first layer at time t, IMU t is the inertial measurement unit data at time t, IMG′ t Point is the feature of the image data at time t after being processed by the CNN neural network. t The t-time feature of the 3D point cloud data of the multi-threaded lidar extracted by PointNet++, GPS t The GPS data at time t, including longitude, latitude, and altitude.

4. The trajectory prediction method based on stacked LSTM with attention mechanism in a multimodal environment according to claim 1, characterized in that: The specific steps of step S3 are as follows: the model's attention to different time steps and different modal data is dynamically adjusted according to the currently input weights. The correlation of each time step is calculated and normalized using the Softmax function to obtain the weights. The weighted sum is finally output and processed by the Dense layer to obtain the final trajectory prediction result, that is, the formula corresponding to the vehicle's position information at the future time is: Among them, Q is the query matrix, K is the key matrix, and V is the value matrix. is the hidden unit vector of the third layer at time tN, t-N+1, ..., t, W Q , W K , W V is the weight matrix, which is updated during training. is the hidden output of the third layer LSTM, e i is the scaled dot product The calculated value, d k is the dimension of Q, K, F(Q i , K i ) is the scoring function, and the annotation is set as the scaled dot product, a i for e i The value after Softmax normalization, Score for attention, It is a simplified representation of several computational steps of the intention mechanism. Input After a series of calculations, the final sum is obtained to get the attention score, GPS t+1 It is the global system positioning value at the next moment, which is transformed by the Dense layer on the attention score and corresponds to the predicted trajectory.

Images