Vehicle trajectory multimodal prediction method considering road information based on LSTM-GNN

Abstract

The present invention provides a multimodal vehicle trajectory prediction method based on LSTM-GNN that takes road information into consideration. The vehicle trajectory prediction model includes an input module, an encoder, an interaction feature extraction module, a road information feature extraction module, and a decoder. The input module contains the historical trajectories of the target vehicle and surrounding traffic vehicles. The encoder encodes the input historical trajectories. The interaction feature extraction module is used to extract the interaction effects between vehicles. The road information feature extraction module extracts road structure information. The prediction method proposed in the present invention takes into account the influence of road structure information on the predicted vehicle trajectory, adds Gaussian noise based on the fused feature vector, and introduces a diversity loss function. The present invention effectively improves the prediction accuracy of future trajectories and enhances the social acceptability and rationality of the predicted trajectories.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and in particular relates to a vehicle trajectory multimodal prediction method based on LSTM-GNN considering road information. Background Art

[0002] Smart driving cars are the strategic direction of the development of the global automotive industry, and automotive intelligence has become the trend and development trend of the automotive industry.

[0003] The prediction module plays a connecting role in the intelligent driving system. The upstream perception module inputs the perception fusion results into the prediction module, which in turn provides input to the downstream decision-making and planning module. Trajectory prediction of surrounding vehicles can improve driving safety and enhance the rationality of the decision-making and planning results of the main vehicle.

[0004] Due to differences in the inputs and intermediate processing steps of prediction models, current vehicle trajectory prediction methods in the field of intelligent driving can be roughly divided into three categories: physical constraint-based prediction methods, behavior-based prediction methods, and learning-based prediction methods. Existing prediction methods rarely consider the impact of road information on the vehicle's future trajectory and do not fully utilize prior knowledge. Vehicle motion is significantly influenced by road structure information. Failure to account for the constraints imposed by road structure on vehicle trajectory can easily lead to low prediction accuracy for future trajectories. Furthermore, due to the uncertainty inherent in vehicle motion, its trajectory is also multimodal, but existing research has rarely considered this. Without considering the multimodality of the predicted trajectory, the future trajectory generated by the prediction model may not conform to driving logic and rules. Summary of the Invention

[0005] The purpose of the embodiment of the present invention is to provide a multimodal prediction method for vehicle trajectories based on LSTM-GNN considering road information, so as to fully utilize the prior knowledge of road information and multimodal output of trajectories, and further improve the accuracy of prediction.

[0006] To solve the above technical problems, the technical solution adopted by the present invention is a vehicle trajectory multimodal prediction method based on LSTM-GNN considering road information, comprising the following steps:

[0007] S1: Obtain vehicle trajectory information. Extract vehicle information from the vehicle trajectory dataset in time series order. The target vehicle's trajectory must have a 3s historical trajectory and a 5s future trajectory. The target vehicle's trajectory length must be at least 1000 meters. In addition, the surrounding traffic vehicles must have a 3s historical trajectory.

[0008] S2: Preprocess the data obtained in S1:

[0009] S2.1 Data cleaning: Filter the acquired data to remove abnormal data, then use a smoothing filter to clean the acquired vehicle trajectory, remove incomplete data, and supplement missing data;

[0010] S2.2: Divide the dataset. Use a sliding window to sample the cleaned data. Each sample should contain 80 frames of vehicle trajectory information, with the first 30 frames as historical trajectory information and the last 50 frames as future trajectory information. Then, divide the data into training, validation, and test sets in proportion.

[0011] S2.3 Map matching of the trajectory: if there is a trajectory that does not conform to the normal form of the vehicle, correct the abnormal trajectory points.

[0012] S3: The encoder encodes the historical trajectory information. The encoder consists of n+1 LSTM networks. Its input is the historical trajectory of the target vehicle and the historical trajectories of surrounding vehicles, and its output is the encoded vehicle dynamics feature vector. The specific steps for constructing the encoder are as follows:

[0013] S3.1: Predicted vehicle v p The historical trajectory of in is the predicted vehicle v p At the time t1, the coordinates are Indicates that t1∈{1,2,3}; surrounding traffic vehicles v i The historical trajectory of in For surrounding traffic vehicles v i At the time t1, the coordinates are Indicates; surrounding traffic vehicles v i ∈{v1,...,v n}, where n is the predicted vehicle v i The number of surrounding traffic vehicles; the predicted vehicle v p The future prediction trajectory of in is the predicted vehicle v p The predicted coordinates at time t2 are Indicates that t2∈{4, 5, 6, 7, 8}; the predicted vehicle v p The future true trajectory of in is the predicted vehicle v p The real coordinates at time t2 are express.

[0014] S3.2: Use Multi-layer Perceptron (MLP) to embed vehicle v j The historical trajectory position information is used to obtain a fixed-length vector where v j ∈{v p , v1, ..., v n};

[0015]

[0016] in, is the embedding function with ReLU nonlinear activation function, W ee is the embedding weight;

[0017] S3.3: Change the vehicle v j Historical trajectory information and fixed-length vectors Input into the encoder LSTM to obtain the dynamic feature vector The encoding process is shown below:

[0018]

[0019] Among them, W encoder is the weight of LSTM.

[0020] S4: Extract the road information feature vector LF based on CNN-LSTM and use the 1D-CNN and LSTM models to encode the road structure information. The process is as follows:

[0021] S4.1: Define candidate lanes based on the current position of the target vehicle. First, search for lane segments within a search radius of 10 meters from the target vehicle's center of mass. Then, expand the lane segments forward and backward until the lane length reaches the required length.

[0022] S4.2: Determine the lanes where the surrounding vehicles are located, and The observation information is sequentially input into 1D-CNN and LSTM for encoding, as shown in the following formula:

[0023]

[0024] in, Lanes encoded using 1D-CNN and LSTM Information feature vector, Indicates the surrounding traffic vehicles v i The lane you are in;

[0025] S4.3: Generate lane information feature vector LF, using attention weight ω i The coded information feature vector of the lane where the surrounding traffic vehicles are located Perform fusion processing as shown below:

[0026]

[0027] S5: The encoded vehicle dynamics feature vector It is then combined with the vehicle-to-vehicle interaction feature vector IF and then fused with the road information feature vector LF. Random noise z is added to the IF and LF fusion feature vectors. Here, Gaussian distribution mixed noise is used. The calculation method of the interaction feature vector IF between vehicles is as follows:

[0028] S5.1: The graph structure can be represented by G = (V, E), and the node is defined as V = {v p , v1, ..., v i ,...,v n}, the edge is defined as E∈V×V; since the graph is a directed graph, the node v p With node v i The edge between nodes v i With node v p The edges between them are different, and the edge E can be expressed as:

[0029]

[0030] in, Represents node v p To node v i Directed edge, node v i Adjacent node v p , and node v i The behavior of node v p behavior; Represents node v i To node v p ;

[0031] S5.2: Use graph neural network (GNN) to model the interaction between vehicles, which can be expressed as follows:

[0032]

[0033] Among them, IF represents the interaction feature vector between vehicles, GNN inter It is an interactive feature encoder composed of two layers of GNN network. is the dynamic characteristic vector of the vehicle, Represents the edges of the graph structure at time t1.

[0034] S6: Use the LSTM network to decode and output the multimodal future trajectory. Input the fused features after adding random noise z into the decoder of the LSTM network to generate the multimodal future trajectory.

[0035] S7: Use the diversity loss function to train a multimodal vehicle trajectory prediction model based on LSTM-GNN that considers road structure information. By arbitrarily sampling a random noise in the distribution to generate k possible prediction trajectories, the "optimal" prediction trajectory is selected based on the L2 Euclidean distance.

[0036] The present invention first inputs the historical trajectory information of the target vehicle and surrounding traffic vehicles into the encoder for encoding, the interaction feature extraction module extracts the interaction influence feature vector between vehicles, and the road information feature extraction module extracts the road structure information feature vector from the high-precision map; then, the historical trajectory feature vector of the target vehicle, the vehicle interaction feature vector and the road information feature vector are spliced, and random noise is added to the spliced ​​feature vector; finally, the final fused feature vector is input into the decoder for decoding to generate a multimodal output trajectory.

[0037] The encoder is composed of a long short-term memory (LSTM) network, which is used to encode the vehicle's dynamic characteristics. After the vehicle's historical trajectory is input into the encoder, the LSTM network is used to encode the input trajectory. The decoder is composed of a long short-term memory (LSTM) network, which is also used to decode and fuse feature vectors to generate a multimodal predicted trajectory. The interaction feature extraction module is composed of a GNN network, which is used to consider the interactive impact of other vehicles on the target vehicle and then extract the interaction features between vehicles. The road information feature extraction module is composed of a CNN-LSTM network, which is used to extract road structure information features.

[0038] The present invention proposes a multimodal vehicle trajectory prediction method based on LSTM-GNN that considers road information. The CNN-LSTM-based road information feature extraction module extracts structural information about the vehicle's lane, further improving trajectory prediction accuracy. Random Gaussian noise is added to the fused feature vector, and a diversity loss function is introduced during model training to generate multimodal sample trajectories, improving the social acceptability and rationality of the trajectories. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is a flow chart of a vehicle multimodal trajectory prediction method based on LSTM-GNN considering road information;

[0041] Figure 2 This is a model structure diagram of the vehicle multimodal trajectory prediction method based on LSTM-GNN considering road information. DETAILED DESCRIPTION

[0042] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0043] This paper proposes a vehicle trajectory multimodal prediction method based on LSTM-GNN considering road information. The model flow chart is as follows: Figure 1 As shown, the specific steps include:

[0044] S1: Obtain vehicle trajectory information. This is obtained from a public dataset and extracted in chronological order. The target vehicle's trajectory must include a 3-second historical trajectory and a 5-second future trajectory. The target vehicle's trajectory must be at least 1000 meters long. Additionally, the surrounding vehicles must have a 3-second historical trajectory.

[0045] S2: Data preprocessing, the specific steps are as follows:

[0046] S2.1: Data cleaning: Filter the acquired data to remove abnormal data. Then, use a smoothing filter to clean the acquired vehicle trajectories, remove incomplete data, and supplement missing data.

[0047] S2.2: Divide the data set; use a sliding window to sample the cleaned data. Each sample should contain 80 frames of vehicle trajectory information, with the first 30 frames as historical trajectory information and the last 50 frames as future trajectory information. Then divide the data into training set, validation set, and test set in proportion.

[0048] S2.3: Map matching of the trajectory; if there is a trajectory that does not conform to the normal form of the vehicle, further correct the abnormal trajectory points.

[0049] S3: The encoder encodes the historical trajectory information; the encoder consists of n+1 LSTM networks, whose input is the historical trajectory of the target vehicle and the historical trajectory of surrounding vehicles, and the output is the encoded vehicle dynamics feature vector The specific implementation process is as follows:

[0050] S3.1: Predicted vehicle v p The historical trajectory of in is the predicted vehicle v pAt the time t1, the coordinates are Represents t1∈{1,2,3}. Surrounding traffic vehicles v i The historical trajectory of in For surrounding traffic vehicles v i At the time t1, the coordinates are Indicates; surrounding traffic vehicles v i ∈{v1,...,v n}, where n is the predicted vehicle v i The number of surrounding vehicles. Predicted vehicle v p The future prediction trajectory of in is the predicted vehicle v p The predicted coordinates at time t2 are Indicates that t2∈{4, 5, 6, 7, 8}. The predicted vehicle v p The future true trajectory of in is the predicted vehicle v p The real coordinates at time t2 are express.

[0051] S3.2: Use Multilayer Perceptron to embed vehicle v j The historical trajectory position information is used to obtain a fixed-length vector where v j ∈{v p , v1,…,v n}.

[0052]

[0053] in, is the embedding function with ReLU nonlinear activation function, W ee is the embedding weight;

[0054] S3.3: Change the vehicle v j Historical trajectory information and fixed-length vectors Input into the encoder LSTM to get the encoding vector The encoding process is shown below:

[0055]

[0056] Among them, W encoder is the weight of LSTM.

[0057] S3: Modeling the interaction between vehicles based on GNN. The interaction is modeled using a directed graph, with nodes representing dynamic encoding features. vehicles.

[0058] S3.1: The graph structure can be represented by G = (V, E), and the node is defined as V = {v p , v1, ..., v i ,...,v n}, the edge is defined as E∈V×V. Since the graph is a directed graph, the node v p With node v i The edge between nodes v i With node v p The edges between them are different, and the edge E can be expressed as:

[0059]

[0060] in, Represents node v p To node v i Directed edge, node v i Adjacent node v p , and node v i The behavior of node v p behavior.

[0061] S3.2: Use graph neural network (GNN) to model the interaction between vehicles, which can be expressed as follows:

[0062]

[0063] Among them, IF represents the interaction feature vector between vehicles, GNN inter It is an interactive encoder composed of two layers of GNN network. is the dynamic characteristic vector of the vehicle, Represents the edges of the graph structure at time t1.

[0064] S4: Extract road information feature vector LF based on CNN-LSTM. Use 1D-CNN and LSTM models to encode road structure information.

[0065] S4.1: Define candidate lanes based on the current position of the target vehicle. First, search for lane segments within a search radius (10 meters) from the target vehicle's center of mass. Then, expand the lane segments forward and backward until the lane line length reaches the required length.

[0066] S4.2: Determine the lanes where the surrounding vehicles are located, and change the lanes where the surrounding vehicles are located to L iThe observation information is sequentially input into 1D-CNN and LSTM for encoding, as shown in the following formula:

[0067]

[0068] in, Lanes encoded using 1D-CNN and LSTM Information feature vector, Indicates the surrounding traffic vehicles v i The lane you are in.

[0069] S43: Generate lane information feature vector LF. Using attention weight ω i Encoded information features of the lanes where surrounding traffic vehicles are located Perform fusion processing as shown below:

[0070]

[0071] S5: Feature fusion. First, the encoded vehicle dynamics feature vector It is then combined with the vehicle-to-vehicle interaction feature vector IF and then fused with the road information feature vector LF. Random noise z is added to the IF and LF fusion feature vectors, and Gaussian distribution mixed noise is used here.

[0072] S6: Use the LSTM network to decode and output the multimodal future trajectory. The fused features after adding random noise are input into the decoder of the LSTM network to generate the multimodal future trajectory.

[0073] S7: Using diversity loss function To train the vehicle trajectory multimodal prediction model based on LSTM-GNN considering road structure information, A random noise is randomly sampled from the distribution to generate k possible prediction trajectories, and the "optimal" prediction trajectory is selected based on the L2 Euclidean distance.

[0074] S7.1: Train the vehicle multimodal prediction model based on LSTM-GNN considering road information proposed in this invention on the preprocessed training set. The input is the historical trajectory information of the predicted vehicle and its surrounding vehicles and the road structure information contained in the high-precision map. The output is the vehicle's multimodal future trajectory.

[0075] S7.2: During the training process, the accuracy of the vehicle trajectory prediction model is calculated using the validation set, and the changes in the loss function during the training process are combined to prevent the model from overfitting. Specifically, the weight parameters and bias parameters of the model proposed in this invention are trained using the Pytorch deep learning framework on an i7-10700F / NVIDIAGeForce RTX 3070 / PCle / SSE2 server. The processed data is passed into the model using the DataLoader class in Pytorch. The optimizer uses Adam, the learning rate is 0.001, the training round (Epoch) is set to 200 rounds, and the model parameters are saved after each round.

[0076] S7.3: Compare the predicted future trajectory with the actual future trajectory, calculate the loss function, and update the network parameters. After the model training is completed, save the corresponding weight parameters and bias parameters.

[0077] S8: Experimental Verification and Analysis. Test and analyze a multimodal vehicle trajectory prediction model based on LSTM-GNN that considers road structure information. Use a preprocessed test set to test the trained model, predict the target vehicle's possible trajectory, and test the model's prediction accuracy.

[0078] S8.1: Evaluation Metrics. Two common evaluation metrics are used to evaluate the model, namely the average displacement error (ADE) and the final displacement error (FDE).

[0079] The average displacement error ADE represents the average L2 distance between the true value and the predicted value, and is calculated as follows:

[0080]

[0081] in, is the predicted vehicle v p The predicted coordinates at time t are expressed as express; is the predicted vehicle v p The real coordinates at time t are expressed as express.

[0082] The final displacement error FDE represents the distance between the predicted final position and the true final position, and is calculated as follows:

[0083]

[0084] S8.2: Ablation Study. Ablation experiments are conducted to verify the effectiveness of each module in the prediction model proposed in this paper. The average displacement error (ADE) and final displacement error (FDE) are used to evaluate the performance of each comparison model. Four comparison models, Model A, Model B, Model C, and Model D, are constructed, as shown in Table 1. Model A is used as the baseline model.

[0085] Table 1. Comparison of models in ablation studies

[0086] Model A B C D History track encoder √ √ √ √ Interactive feature extraction module √ √ √ Road feature extraction module √ √ Diversity loss function √

[0087] Model A only uses historical trajectories to predict future trajectories, inputs historical trajectories into the encoder, and then converts the encoded dynamic feature vector The input is sent to the decoder to decode the future trajectory of the vehicle. Model B is based on Model A and adds an interactive feature extraction module to convert the dynamic feature vector After being fused with the interactive feature vector IF, it is input into the decoder to decode the future trajectory. Model C adds the interactive feature module and the road feature extraction module to the model A, and converts the dynamic feature into The interaction feature vector IF and the road information feature vector LF are fused and input into the decoder to decode the future trajectory. Model D is the vehicle trajectory multimodal prediction method model based on LSTM-GNN considering road information proposed in this invention, as shown in Figure 2 As shown, the proposed model adds an interactive feature module and a road feature extraction module to Model A, and uses a diversity loss function to generate multimodal prediction trajectories. Experimental results show that the proposed model performs optimally, demonstrating the necessity of the interactive feature extraction module, the road information feature extraction module, and the multimodal output.

[0088] S8.3: Comparison with Baseline Models. The prediction model proposed in this paper is compared with commonly used prediction models such as LSTM and DESIRE. The models are evaluated using the average displacement error (ADE) and the final displacement error (FDE). The comparison results are shown in Table 2. The results show that the prediction model proposed in this paper outperforms the other two models.

[0089] Table 2 Model comparison experimental results

[0090]

[0091] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are described briefly because they are generally similar to the method embodiments. For related portions, reference can be made to the description of the method embodiments.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the scope of protection of the present invention.

Claims

1. A vehicle trajectory multimodal prediction method based on LSTM-GNN considering road information, characterized by: The following steps are involved: S1: Obtain vehicle trajectory information. Extract vehicle information from the vehicle trajectory dataset in time series order. The target vehicle's trajectory must have a 3s historical trajectory and a 5s future trajectory. The target vehicle's trajectory length must be at least 1000 meters. In addition, the surrounding traffic vehicles must have a 3s historical trajectory. S2: Preprocess the data obtained in S1: S3: The encoder encodes the historical trajectory information. The encoder consists of n+1 LSTM networks. Its input is the historical trajectory of the target vehicle and the historical trajectory of surrounding vehicles. The output is the encoded vehicle dynamics feature vector. The construction process of the LSTM-based encoder is as follows: S3.1: Predicted vehicle v p The historical trajectory of in is the predicted vehicle v p At the time t1, the coordinates are Indicates that t1∈{1,2,3}; surrounding traffic vehicles v i The historical trajectory of in For surrounding traffic vehicles v i At the time t1, the coordinates are Indicates; surrounding traffic vehicles v i ∈{v1,…,v n }, where n is the predicted vehicle v i The number of surrounding traffic vehicles; the predicted vehicle v p The future prediction trajectory of in is the predicted vehicle v p The predicted coordinates at time t2 are Indicates that t2∈{4,5,6,7,8}; the predicted vehicle v p The future true trajectory of in is the predicted vehicle v p The real coordinates at time t2 are express; S3.2: Use Multi-layer Perceptron (MLP) to embed vehicle v j The historical trajectory position information is used to obtain a fixed-length vector where v j ∈{v p ,v1,…,v n }; in, is the embedding function with ReLU nonlinear activation function, W ee is the embedding weight; S3.3: Change the vehicle v j Historical trajectory information and fixed-length vectors Input into the encoder LSTM to obtain the dynamic feature vector The encoding process is shown below: Among them, W encoder is the weight of LSTM; S4: Extract road information feature vector LF based on CNN-LSTM, and use 1D-CNN and LSTM models to encode road structure information; S4.1: Define candidate lanes based on the current position of the target vehicle. First, search for lane segments within a search radius of 10 meters from the target vehicle's center of mass. Then, expand the lane segments forward and backward until the lane length reaches the required length. S4.2: Determine the lanes where the surrounding vehicles are located, and The observation information is sequentially input into 1D-CNN and LSTM for encoding, as shown in the following formula: in, Lanes encoded using 1D-CNN and LSTM Information feature vector, Indicates the surrounding traffic vehicles v i The lane you are in; S4.3: Generate lane information feature vector LF, using attention weight ω i The coded information feature vector of the lane where the surrounding traffic vehicles are located Perform fusion processing as shown below: S5: The encoded vehicle dynamics feature vector It is then combined with the vehicle-to-vehicle interaction feature vector IF and then fused with the road information feature vector LF. Random noise z is added to the IF and LF fusion feature vectors, and Gaussian distribution mixed noise is used here; The calculation steps of the interaction feature vector IF between vehicles are as follows: S5.1: The graph structure is represented by G = (V, E), and the node is defined as V = {v p ,v1,…,v i ,…,v n }, the edge is defined as E∈V×V; since the graph is a directed graph, the node v p With node v i The edge between nodes v i With node v p The edges between them are different, and the edge E can be expressed as: in, Represents node v p To node v i Directed edge, node v i Adjacent node v p , and node v i The behavior of node v p behavior; Represents node v i To node v p ; S5.2: Use graph neural network (GNN) to model the interaction between vehicles, which can be expressed as follows: Among them, IF represents the interaction feature vector between vehicles, GNN inter It is an interactive feature encoder composed of two layers of GNN network. is the dynamic characteristic vector of the vehicle, Represents the edge of the graph structure at time t1; S6: Use the LSTM network to decode and output the multimodal future trajectory. Input the fused features after adding random noise z into the decoder to generate the multimodal future trajectory. S7: Use the diversity loss function to train a multimodal vehicle trajectory prediction model based on LSTM-GNN that considers road structure information. By arbitrarily sampling a random noise in the distribution to generate k possible prediction trajectories, the optimal prediction trajectory is selected based on the L2 Euclidean distance.

2. The vehicle trajectory multimodal prediction method based on LSTM-GNN considering road information according to claim 1 is characterized in that The specific steps of data preprocessing in S2 are as follows: S2.1: Data cleaning: Filter the acquired data to remove abnormal data, then use a smoothing filter to clean the acquired vehicle trajectories, remove incomplete data, and supplement missing data. S2.2: Dataset division: Use a sliding window to sample the cleaned data. Each sample should contain 80 frames of vehicle trajectory information, with the first 30 frames as historical trajectory information and the last 50 frames as future trajectory information. Then divide the data into training, validation, and test sets in proportion. S2.3: Map matching of trajectory: If there is a trajectory that does not conform to the normal form of the vehicle, correct the abnormal trajectory points.

3. The vehicle trajectory multimodal prediction method based on LSTM-GNN considering road information according to claim 1 is characterized in that The decoder in S6 is composed of a long short-term memory network (LSTM), which is used to decode the fused feature vector to generate a multimodal prediction trajectory.

Images