Anchor point-based multi-modal trajectory prediction method

Abstract

The invention discloses a multi-modal trajectory prediction method based on anchor points. The method belongs to the technical field of automatic driving, and comprises the following operation steps: firstly, preprocessing a pre-collected track data set, and sampling on a track line to obtain candidate anchor points; and then, an MGT network architecture is constructed, interaction between track features and fusion between lane features are realized after a graph neural network technology is fused, the MGT adopts an embedded hierarchical design, the hierarchical structure firstly preliminarily optimizes anchor point positions through an internal dependency relationship between tracks, and further, track surrounding environment information obtained by an API is combined, so that the track surrounding environment information is obtained, and the track surrounding environment information is obtained. Environment features and potential influences of surrounding traffic dynamics are fused to anchor point position features; and finally, decoding the optimized anchor point by using a decoder, and designing a fitting function so as to obtain a complete predicted trajectory coordinate. Multiple factors in the MGT network are fully considered, the anchor point coordinates are optimized step by step, and the credibility and accuracy of prediction are remarkably improved.

Description

Technical Field

[0001] The invention belongs to the technical field of autonomous driving and relates to a multimodal trajectory prediction method based on anchor points. Background Art

[0002] At present, driving a car has become the mainstream choice for public transportation, but the surge in the number of vehicles has also brought about a series of severe traffic challenges. In order to effectively deal with these problems and improve travel efficiency, autonomous driving technology has emerged and is increasingly attracting attention. Among the many core technologies of autonomous driving, trajectory prediction plays a vital role. It accurately predicts the future driving path of the vehicle, plans the optimal driving route for the autonomous driving vehicle, and provides strong support for its decision-making, thereby significantly improving road safety and traffic efficiency.

[0003] Currently, vehicle trajectory prediction tasks are increasingly incorporating high-precision map information, using detailed environmental information to assist in building vehicle features. This approach not only improves prediction accuracy, but also significantly enhances the robustness of the model, ensuring a high degree of consistency between the prediction results and real traffic scenarios. Under the deep learning (DL) framework, trajectory prediction models mainly use two data representation forms: rasterization and vectorization.

[0004] The rasterization method simplifies the processing of geospatial data, but sacrifices detailed information and affects the prediction accuracy. With the development of technology, vectorization methods such as VectorNet remove redundant information through GNN (graph neural network), accurately capture complex interactions between vehicles, and improve the accuracy of trajectory prediction. Predicting the interaction between traffic agents is the key to improving accuracy. The current mainstream modeling methods include RNN (recurrent neural network), CNN (convolutional neural network), GNN, Attention (attention mechanism), and Transformer. RNN is good at processing sequence data and capturing time dependencies; CNN is more efficient in processing spatial relationships; GNN and Attention accurately aggregate information by explicitly modeling interactions between agents; Transformer, with its global receptive field and powerful attention mechanism, directly models complex traffic scenarios and flexibly responds to spatial and temporal challenges. With the development of technology, GNN, Attention and Transformer have taken a dominant position in modeling interactions between traffic agents.

[0005] In summary, based on vectorized representation, using Transformer to model the interaction between traffic agents has shown great potential and significant advantages in vehicle trajectory prediction tasks due to its efficient data processing capabilities, accurate semantic capture capabilities, and excellent model robustness. This method can not only effectively integrate spatial and temporal information, but also accurately depict dynamic interactions in complex scenes, opening up a new way to improve prediction accuracy and reliability. Summary of the invention

[0006] In view of the above problems, the present invention provides a multimodal trajectory prediction method based on anchor points (an innovative vehicle trajectory prediction method), which deeply integrates scene understanding, feature optimization and trajectory decoding technology. This vehicle trajectory prediction method can significantly improve the decision-making ability of the autonomous driving system, and can make more accurate and safe judgments when facing complex and changeable traffic scenes, thereby effectively reducing the occurrence of traffic accidents and improving road safety.

[0007] The technical solution of the present invention is: a multimodal trajectory prediction method based on anchor points described in the present invention comprises the following steps:

[0008] (1) The pre-acquired trajectory data set is segmented according to a ratio of 6:2:2, and then preprocessed to divide the trajectory data and lane line data, and the candidate anchor point coordinates are sampled on the lane line;

[0009] (2) Constructing an MGT network, which combines LFIM and HTransformer networks; the interaction between trajectory features and the fusion between lane features are realized through the LFIM network, and a hierarchical HTransformer network is designed based on the Transformer network to encode trajectory features and environmental features into anchor features respectively, and to fuse features step by step for interaction; finally, MDM is used to decode and fit the anchor points into complete trajectories;

[0010] (3) Send the preprocessed training data to the MGT network for training, and use the AdamW optimizer to back-propagate and update the parameters to optimize the MGT network;

[0011] (4) Send the preprocessed test data to the trained MGT network for testing.

[0012] Furthermore, in step (1), the preprocessing is to process the trajectory data and environmental features (lane line position, whether it is traffic control and whether it is a straight lane, etc.) separately and save them uniformly, and fill in the number of trajectories and lane lines in each file, unify the data format, and facilitate subsequent feature interaction;

[0013] Among them, since the vehicle will not deviate from the lane, the anchor points are evenly sampled on the lane line.

[0014] Furthermore, in step (2), the LFIM network first uses MLP to map the feature graph from the low-dimensional space to the high-dimensional space to extract more information; in order to fuse the state features of the surrounding nodes, LFIM adds node connectivity information, utilizes the interactive features of the graph propagation mechanism, and unifies the pooled node features by adding cluster flags;

[0015] In GCN, the feature update of each node is usually achieved by aggregating the features of its neighbors, and its forward propagation can be expressed as:

[0016]

[0017]

[0018] Where A is the original adjacency matrix, I N is the identity matrix, H (n) is the node feature matrix of the nth layer, W (n) is the weight matrix of the nth layer, is the adjacency matrix with self-connection added, yes The degree matrix of , σ is the activation function;

[0019] Use cluster to perform pooling operation, assuming C i represents the set of nodes in the i-th cluster, H v represents the feature vector of node v, then the pooled feature of the i-th cluster It can be expressed as:

[0020]

[0021] In the formula, is cluster C i The jth element of the pooled feature, H v [j] is the j-th element of the feature of node v.

[0022] Furthermore, in step (2), the HTransformer is built based on the Transformer network and candidate anchor point information is added;

[0023] In order to fuse features, the historical trajectory features that pass through LFIM are first encoded using the Encoder module of HTransformer, and the invalid data filled is controlled by mask. Secondly, in the Decoder module of HTransformer, the candidate anchor points are encoded using the multi-head self-attention mechanism with mask, and the result is fused with the output of the Encoder module of HTransformer using the cross-attention mechanism, so that the historical trajectory features can be fused with the features of the anchor points; the lane line data is also encoded using the Encoder of HTransformer, and the influence of the filling data is eliminated by mask. In the Decoder module of HTransformer, the anchor points of the fused trajectory features are first encoded through the multi-head self-attention mechanism, and the result is cross-attention calculated with the encoded lane line features;

[0024] The self-attention mechanism formula is as follows:

[0025]

[0026] Where Q, K, and V represent the query matrix, key matrix, and value matrix, respectively, and d k is the dimension of each column feature in the key matrix, and softmax is the activation function; this formula first calculates the dot product of the query matrix Q and the key matrix K, and divides it by Scale it and then use the softmax function to normalize it to get the attention weights; finally, multiply the attention weights by the value matrix V to get the weighted sum matrix.

[0027] Furthermore, in step (2), the HTransformer utilizes a masked multi-head attention mechanism in both the Encoder module and the Decoder module.

[0028] Furthermore, in step (2), the fitting function first uses the encoding features of HTransformer to decode the probability and bias of the candidate anchor points. In order to reduce the amount of calculation, K candidate points are first selected, and then the trajectory is fitted using MLP. In order to consider the impact of the environment on the trajectory and improve the credibility and robustness of the trajectory score, the trajectory features are fused with the encoding features during prediction to finally obtain the trajectory score.

[0029] The beneficial effects of the present invention are as follows: the present invention samples and analyzes the scenes of the data to estimate the initial distribution of potential candidate anchor points; then, the designed LFIM is introduced to realize the interaction between trajectory features and the fusion between lane features, and the hierarchical network structure of HTransformer is used to fuse the global historical trajectory features and the surrounding environment information, and the features of the candidate targets are gradually optimized, so that the MGT network can effectively capture the dynamic changes of the target and fully fuse the information of the trajectory and the environment; after the feature optimization stage is completed, the MDM is used for decoding, which not only decodes the possible trajectory paths in the future, but also synchronously generates the prediction probability of each trajectory. This design greatly improves the reliability and diversity of the prediction results; because the MGT network makes full use of the complex information of the surrounding environment to model the interaction behavior between vehicles, less errors are introduced in the prediction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is an operation flow chart of the present invention;

[0031] Figure 2 It is a schematic diagram of the LFIM structure of the present invention;

[0032] Figure 3 It is a schematic diagram of the MLP structure of the present invention;

[0033] Figure 4 It is a schematic diagram of the HTransformer structure in the present invention;

[0034] Figure 5 It is a schematic diagram of the MDM structure in the present invention. DETAILED DESCRIPTION

[0035] The specific technical solutions of the present invention are further described in detail below in conjunction with specific examples.

[0036] The anchor-based multimodal trajectory prediction method described in the present invention has the following specific implementation process: firstly, the pre-acquired trajectory data set is pre-processed to divide the trajectory data and lane line data, and a series of candidate anchor points are obtained by scattering points on the track line for sampling; by filling the data, the data format is aligned, and a training set and a test set are prepared; an MGT (Multimodal Trajectory Prediction Based on Graph and Transformer) network model is constructed, the pre-processed training set is sent to the MGT network model for training, and the MGT network is optimized by back-propagation updating parameters of the AdamW optimizer;

[0037] The main purpose of the sampling algorithm is to generate a set of candidate target points, which represent the possible locations of intelligent agents (such as vehicles and pedestrians) in the future. By sampling and screening the candidate target points, the intentions and future movement patterns of the intelligent agents can be captured more effectively. In the HD Map, points are evenly sampled from the center line of the lane as candidate targets. This is based on the assumption that vehicles usually travel within the lane. In order to ensure the coverage and efficiency of the candidate target points, a large number of candidate points (such as N = 1000) are usually generated, and then the most likely M points (such as M = 50) are selected for subsequent processing.

[0038] The overall structure of the MGT network proposed by the present invention is as follows: Figure 1 As shown in the figure; the MGT network focuses on the local to the global, first classifying the data information, including historical trajectory status information, surrounding environment information, and sampled anchor point location information; using LFIM (Local Feature Interaction Module) to encode the historical trajectory status information and surrounding environment information, and updating the features through message passing, which is conducive to the fusion of subsequent features; global feature fusion uses the HTransformer (Hierarchical Transformer) network, which is composed of stacked Transformers to encode global information; using MDM (Multimodal Decoder Module) to decode the predicted trajectory and the corresponding confidence score, specifically, through PBM (Probability Bias Prediction Module) to obtain the probability of the candidate point and the position bias, and select the top k candidate anchor points to reduce the subsequent amount of calculation, and use k coordinates and corresponding biases to regress the complete trajectory coordinates, and obtain the corresponding prediction score of the trajectory through the coordinates.

[0039] A typical feature of GCN is that it updates node information through message passing. It obtains the features of surrounding nodes based on the graph structure to update its own features. Transformer, on the other hand, updates features based on the interaction of global information. Therefore, it differs from GCN in terms of learning methods and feature properties. Therefore, combining GCN with Transformer can integrate local and global node feature information, which helps to improve the overall network's ability to learn and express features.

[0040] The core idea of ​​LFIM is to use GCN to update the local features of the input; the structure of the LFIM network is as follows: Figure 2As shown in the figure, since low-dimensional features represent less information, the historical trajectory information or lane line environment information is first upgraded through MLP (multi-layer perceptron), which increases the network information extraction capability to a certain extent. The node connectivity information formed by the adjacency matrix is ​​used to update and aggregate messages through the GCN network, thereby updating the node features, which is repeated three times. The specific structure of MLP is shown in the figure. Figure 3 As shown in the figure, a linear layer, a normalization layer, and an activation layer are used, and a residual module is added; each node is updated uniformly. In order to obtain the features of each node later, it is necessary to use the graph index number cluster to extract each graph feature, perform the maximum pooling operation, and fuse the features.

[0041] In GCN, the feature update of each node is usually achieved by aggregating the features of its neighbors, and its forward propagation can be expressed as:

[0042]

[0043]

[0044] Where A is the original adjacency matrix, I N is the identity matrix, H (n) is the node feature matrix of the nth layer, W (n) is the weight matrix of the nth layer, is the adjacency matrix with self-connection added, yes The degree matrix of , σ is the activation function;

[0045] Use cluster to perform pooling operation, assuming C i represents the set of nodes in the i-th cluster, H v represents the feature vector of node v, then the pooled feature of the i-th cluster It can be expressed as:

[0046]

[0047] in, is cluster C i The jth element of the pooled feature, H v [j] is the j-th element of the feature of node v.

[0048] The data sent to MGT is processed by LFIM and then enters the HTransformer network. The structure of the HTransformer network is as follows: Figure 4As shown in the figure; in order to ensure that the network can obtain as many features as possible, the HTransformer network adopts a hierarchical structure, and uses the historical trajectory features through LFIM and the environmental features of the lane line to improve the node features step by step; the network first uses MLP to upgrade the historical trajectory features through LFIM, and adds position encoding, and encodes them using the Encoder module of HTransformer, and uses mask to control the invalid data filled, and passes through the residual network, normalization, and the final output of the feedforward network as the intermediate input of the HTransformer Decoder. In the Decoder part of the HTransformer, the sampled anchor node coordinates are used as input, and the number of valid candidate anchor points is used to obtain the mask matrix, and the candidate anchor points are encoded through the masked multi-head self-attention mechanism, and the residual network and normalization operation are passed to obtain the intermediate output of the HTransformer Decoder, and its output is fused with the output of the Encoder module of HTransformer using the cross attention mechanism, so that the historical trajectory features can be fused with the features of the anchor points, and the anchor point coordinate features are updated.

[0049] The lane line data after LFIM processing is also encoded using the HTransformer Encoder, and the mask is used to eliminate the influence of the padding data. In the HTransformer Decoder module, the anchor point of the fused trajectory features is first encoded through the multi-head self-attention mechanism, and the result is cross-attention calculated with the encoded lane line features to further update the anchor point coordinate features;

[0050] The self-attention mechanism formula is as follows:

[0051]

[0052] Among them, Q, K, V, respectively represent the query matrix, key matrix, and value matrix, and d k is the dimension of each column feature in the key matrix, and softmax is the activation function; this formula first calculates the dot product of the query matrix Q and the key matrix K, and divides it by Scale it, then use the softmax function to normalize it to get the attention weight; finally, add the attention weight to the value matrix V.

[0053] MDM structure is as follows Figure 5As shown in the figure, MDM mainly decodes the predicted trajectory coordinate value and probability value by inputting the encoded features and the local feature information of the historical trajectory; the final prediction value is the coordinate of the trajectory, so the historical trajectory features are spliced ​​on the basis of the input encoded features. This can maximize the use of the features of the historical trajectory and better predict the trajectory of the target; among them, MDM first uses the spliced ​​features to predict the probability and corresponding bias of the candidate anchor points, and selects the anchor points by the probability value to reduce the number of candidate anchor points; in the trajectory prediction part, the feature values ​​of the historical trajectory are used again, and at the same time, they are spliced ​​with the initial position and bias of the candidate coordinates through the MLP module to obtain the predicted trajectory coordinates. In order to increase the credibility and robustness of the trajectory score, the feature values ​​of the encoded part are spliced ​​while the trajectory coordinate prediction value is input, so that the influence of the environmental characteristics is considered when predicting the score.

[0054] In this implementation, the Argoverse trajectory dataset is first obtained from the official website, and is divided and feature constructed. Specifically, the position of the agent in the next 30 frames is predicted based on the initial 20 frames of observation data, and data samples and labels are constructed, and divided according to 6:2:2; the data is stored in the Argoverse / datasets folder, which includes three subfolders, namely train, val, and test; the preprocessed vehicle trajectory data is input into MGT for training, the initial learning rate is 0.001, the optimizer uses AdamW, batch_size is 16, epochs is 100, and model training is started. The loss is obtained through the validation set, and the trained MGT model is tested on the test dataset.

Claims

1. A multimodal trajectory prediction method based on anchor points, characterized in that: The operation steps are as follows: Step (1): Segment the pre-acquired trajectory data set, and then perform preprocessing to divide the trajectory data and lane line data, and sample the candidate anchor point coordinates on the lane line; Step (2): construct an MGT network, which combines LFIM and HTransformer networks; realize the interaction between trajectory features and the fusion between lane features through the LFIM network, design a hierarchical HTransformer network based on the Transformer network, encode trajectory features and environmental features into anchor features respectively, fuse features and interact; finally, use MDM to decode and fit the anchor points into complete trajectories; Step (3): Send the preprocessed training data to the MGT network for training, and use the AdamW optimizer to back-propagate and update the parameters to optimize the MGT network; Step (4): Send the preprocessed test data to the trained MGT network for testing.

2. The anchor point-based multimodal trajectory prediction method according to claim 1, characterized in that: In step (1), the pre-acquired trajectory data set is segmented according to a ratio of 6:2:

2.

3. The anchor point-based multimodal trajectory prediction method according to claim 1, characterized in that: In step (1), the preprocessing is to process the trajectory data and environmental features separately, save them uniformly, fill the number of trajectories and lane lines in each file, and unify the data format.

4. The anchor point-based multimodal trajectory prediction method according to claim 3, characterized in that: Since the vehicle will not deviate from the lane, the anchor points are evenly sampled on the lane line.

5. The anchor point-based multimodal trajectory prediction method according to claim 3, characterized in that: The environmental features include the position of the lane line, whether it is traffic control and whether it is a through lane.

6. The anchor point-based multimodal trajectory prediction method according to claim 1, characterized in that: In step (2), the LFIM network first uses MLP to map the feature graph from the low-dimensional space to the high-dimensional space; in order to integrate the state features of the surrounding nodes, LFIM adds node connectivity information, utilizes the interactive features of the graph propagation mechanism, and unifies the pooled node features by adding cluster flags.

7. The anchor point-based multimodal trajectory prediction method according to claim 6, characterized in that: In GCN, the feature update of each node is achieved by aggregating the features of its neighbors, and its forward propagation can be expressed as: Where A is the original adjacency matrix, I N is the identity matrix, H (n) is the node feature matrix of the nth layer, W (n) is the weight matrix of the nth layer, is the adjacency matrix with self-connection added, yes The degree matrix of , σ is the activation function; Use cluster to perform pooling operation, assuming C i represents the set of nodes in the i-th cluster, H v represents the feature vector of node v, then the pooled feature of the i-th cluster It can be expressed as: In the formula, is cluster C i The jth element of the pooled feature, H v [j] is the j-th element of the feature of node v.

8. The anchor point-based multimodal trajectory prediction method according to claim 1, characterized in that: In step (2), the HTransformer is built based on the Transformer network and candidate anchor point information is added; To fuse features, firstly, the historical trajectory features that pass through LFIM are encoded using the Encoder module of HTransformer, and the invalid data filled is controlled by mask; secondly, in the Decoder module of HTransformer, the candidate anchor points are encoded using the multi-head self-attention mechanism with mask, and the result is fused with the output of the Encoder module of HTransformer using the cross-attention mechanism, so that the historical trajectory features can be fused with the features of the anchor points; the lane line data is also encoded using the Encoder of HTransformer, and the influence of the filling data is eliminated by mask. In the Decoder module of HTransformer, the anchor points of the fused trajectory features are first encoded by the multi-head self-attention mechanism, and the result is cross-attention calculated with the encoded lane line features; The self-attention mechanism formula is as follows: Where Q, K, and V represent the query matrix, key matrix, and value matrix, respectively, and d k is the dimension of each column feature in the key matrix, and softmax is the activation function; the above formula first calculates the dot product of the query matrix Q and the key matrix K, and divides it by Scale it and then use the softmax function to normalize it to get the attention weights; finally, multiply the attention weights by the value matrix V to get the weighted sum matrix.

9. The anchor point-based multimodal trajectory prediction method according to claim 1, characterized in that: In step (2), the HTransformer uses a masked multi-head attention mechanism in both the Encoder module and the Decoder module.

10. The anchor point-based multimodal trajectory prediction method according to claim 1, characterized in that: In step (2), for the MDM, the probability and bias of the candidate anchor points are first obtained using the encoding features of HTransformer; To reduce the amount of calculation, K candidate points are first selected, and then the trajectory is fitted using MLP; In order to consider the impact of the environment on the trajectory and improve the credibility and robustness of the trajectory score, the trajectory features are fused with the encoding features during prediction to finally obtain the trajectory score.

Images