Implementation method for vehicle track prediction

Abstract

The invention discloses an implementation method for vehicle trajectory prediction, and belongs to the technical field of automatic driving and trajectory prediction, and the specific implementation method comprises the steps: 101, carrying out self-supervision pre-training, 102, carrying out the motion coding and scene coding of an intelligent agent, and 103, carrying out the interaction of historical trajectory features and scene features obtained in the step 103 through a Mama module, step 104, global interaction is realized, step 105, strategy head path selection and multi-mode decoding are carried out, and step 106, trajectory prediction is carried out. According to the method, the robust encoder and the interaction module are trained by using the path consistency pre-training task, the noise interference of a data set is reduced, the defect of incomplete scene cognition in the prior art is overcome, the long-distance track matching learning ability of the model is improved, and finally the prediction accuracy and diversity are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic driving and trajectory prediction, and specifically relates to a method for realizing vehicle trajectory prediction. Background Art

[0002] The rapid development of autonomous driving technology has put forward higher requirements on the trajectory prediction ability of vehicles. Trajectory prediction refers to predicting the movement path of a vehicle in the future based on the known historical movement trajectory and environmental information of the vehicle. This technology plays a key role between the perception and decision-making modules in the autonomous driving system, and its accuracy is directly related to the safety and reliability of the system. Accurate trajectory prediction can help autonomous vehicles identify potential collision risks in advance, so as to take corresponding avoidance measures to ensure driving safety. In the urban traffic environment, the movement trajectory of the vehicle is affected by many factors, such as the vehicle's own movement state, the behavior of other vehicles and pedestrians around it, the geometric structure of the road, and traffic signals. The interaction of these factors makes the trajectory prediction problem highly complex and uncertain. In addition, the movement trajectory of vehicles in urban traffic scenes is usually diverse and uncertain, that is, under the same initial conditions, the vehicle may have multiple different movement path options. Therefore, trajectory prediction needs to consider not only the accuracy of the prediction results, but also the diversity of the prediction results to adapt to the complex and changing traffic environment.

[0003] Liu et al. (N. Deo, E. Wolff, and O. Beijbom, "Multimodal trajectory prediction conditioned on lane-graph traversals," in Conference on Robot Learning. PMLR, 2022, pp. 203–212.) improve the accuracy and diversity of prediction by combining lane information and multimodal features. The scheme uses the lane graph structure to capture the relationship between vehicles and lanes, and enhances the understanding of the scene through the attention mechanism. Specifically, the historical trajectory data of the vehicle, high-precision map information, and real-time sensor data are used as input, and the lane graph is modeled through the lane perception encoder to capture the topological relationship between lanes and the motion characteristics of the vehicle. Then, the model generates multimodal future trajectory predictions through multimodal fusion and attention mechanisms to ensure that the prediction results are both accurate and diverse to adapt to complex and changing traffic environments. This method is the most influential graph neural network method, and many researchers have made improvements on this basis. Park et al. (D.Park, H.Ryu, Y.Yang, J.Cho, J.Kim, and K.-J.Yoon, "Leveraging future relationship reasoning for vehicle trajectory prediction," arXiv preprint arXiv:2305.14715, 2023.) predict the future interaction relationships between vehicles by analyzing the historical trajectories and current states of the vehicles, and use these relationships to guide trajectory prediction. Specifically, the historical trajectory characteristics and environmental information of the vehicles are first extracted, and then the future interaction relationships between vehicles, such as the relative position, speed, and acceleration between vehicles, are predicted through the relationship modeling module. Finally, these relationship information and the vehicle's own motion characteristics are combined to generate multimodal future trajectory predictions to ensure that the prediction results are both accurate and diverse to adapt to complex traffic scenarios. This method is a well-known spatiotemporal information modeling method.DEO et al. (M.Liu, H.Cheng, L.Chen, H.Broszio, J.Li, R.Zhao, M.Sester, and MYYang, "Laformer: Trajectory prediction for autonomous driving with lane-aware scene constraints," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp.2039–2049.) aim to capture the complex interactions between vehicles through probabilistic graph models. First, the historical trajectory and environmental information of the vehicle are input into the model to construct a probabilistic graph in which nodes represent vehicles and edges represent the interactions between vehicles. The probabilistic graph is modeled through a graph neural network, which can learn the interactions and motion patterns between vehicles. The model uses probability distribution to represent the future trajectory of the vehicle and generates multimodal prediction results, thereby improving the accuracy and diversity of the prediction and adapting to complex and changing traffic environments. This method is a probabilistic graph interaction method.

[0004] In the field of trajectory prediction for autonomous driving, recurrent neural networks (RNN) and long short-term memory networks (LSTM) are commonly used methods that model the trajectory of vehicles by capturing the dynamic characteristics of time series data. RNN and LSTM can handle temporal dependencies in sequence data and are suitable for modeling the changes in the motion state of vehicles. For example, LSTM controls the flow of information through its internal gating mechanism, thereby effectively capturing the long-term dependencies of vehicle motion. However, these methods may face problems of high computational complexity and difficulty in training when processing large-scale traffic scene data. Graph convolutional networks (GCN) and graph attention networks (GAT) use graph structures to represent the relationship between vehicles and roads in traffic scenes, and model the interaction between vehicles and the topological structure of roads by analyzing the interaction between nodes and edges in the graph. GCN updates the representation of nodes by propagating node feature information in the graph, while GAT uses the attention mechanism to weight the interaction between different nodes, thereby better capturing the complex interaction between vehicles. For example, GCN can be used to model the interactive behavior of vehicles at intersections and predict their trajectories by analyzing the relative positions and speeds between different vehicles. However, these methods may face the problem of low computational efficiency due to overly complex graph structures when dealing with large-scale traffic networks. Transformer-based methods can effectively process long sequence data through the self-attention mechanism, which is suitable for modeling the time series characteristics of vehicle motion trajectories. The self-attention mechanism can capture the global dependencies between different time steps in the sequence, thereby better modeling the long-term dynamic characteristics of vehicle motion. For example, Transformer can be used to model the motion trajectory of vehicles on highways, and predict their future motion paths by analyzing information such as the speed and acceleration of the vehicle at different time steps. However, the Transformer model may face the problem of high computing resource consumption when processing large-scale traffic scene data.

[0005] Methods based on self-supervised learning are generally formulated as mask reconstruction tasks, which enhances their trajectory prediction representation. These methods mask out some of the agent's trajectories and lane segments and use the prediction head to reconstruct the masked elements. However, the existing techniques ignore the impact of noisy data. Because the separation of tasks will lead to mismatches in different data distributions, which in turn will lead to a decrease in prediction accuracy; existing RNN and GCN-based methods mainly effectively enhance the modeling of dynamic scenes, thereby ensuring spatiotemporal interactivity. However, the dynamic selectivity of time-varying trajectories is ignored, resulting in unsatisfactory prediction diversity. Summary of the invention

[0006] The present invention provides a method for implementing vehicle trajectory prediction. Through the present invention, a path consistency pre-training task can be used to train a robust encoder and an interaction module, thereby reducing data set noise interference, solving the defect of incomplete scene cognition in the prior art, improving the long-distance trajectory matching learning ability of the model, and ultimately improving the accuracy and diversity of the prediction.

[0007] The uneven distribution of data sets and lack of scene understanding caused by existing pre-training tasks have brought great difficulties to the accuracy and diversity of trajectory prediction. In order to solve the above technical problems, the present invention provides the following technical solutions: a method for implementing vehicle trajectory prediction, the method for implementing vehicle trajectory prediction comprises the following steps:

[0008] Step 101 first performs self-supervised pre-training; obtains the model parameters of the encoder in step 102;

[0009] Step 102 encodes the motion of the agent and the scene;

[0010] Step 103: The historical trajectory features and scene features obtained above interact through the Mamba module to obtain the combined information of the vehicle scene;

[0011] Step 104 implements global interaction, using a transformer-based attention mechanism global interaction module to perform long-distance information matching of vehicle scene combined information;

[0012] Step 105 performs strategy header path selection and multi-modal decoding;

[0013] Step 106 performs trajectory prediction, performs Kmeans clustering on the sampled trajectory probability distribution, and selects cluster centers as the final K prediction modes, representing the diversified prediction results obtained from the trajectory distribution.

[0014] In step 101 of the present invention, knowledge distillation is divided into a student network and a teacher network, and the input information is divided into two parts of input; the teacher network is constructed by the exponential moving average (EMA) of the weights of the student network, and the student network optimizes its own parameters by learning the output of the teacher network, thereby realizing the transfer of knowledge and improving the robustness of the model.

[0015] The step 101 of the present invention first performs self-supervised pre-training; the specific method is that after the vehicle motion trajectory data obtained after the preparation work forms a time series, different observation paths are constructed by randomly skipping the second-level frames therein, simulating the data loss caused by sensor failure, and providing the model with multiple visual observation perspectives; then, the encoder and the interaction module are trained using knowledge distillation and cross entropy loss; finally, the object association probability distribution under different paths is learned and matched by minimizing the cross entropy loss of the student network parameters, so that the model obtains useful knowledge of the road consistency constraints in the pre-training stage, and the model parameters of the encoder in step 102 are obtained.

[0016] The step 102 of the present invention encodes the motion of the intelligent body and the scene, uses the encoder parameters obtained in step 102 to re-extract the historical trajectory characteristics and scene characteristics of the vehicle, and combines the gated recurrent unit (GRU) and the multi-layer perceptron (MLP) to capture the time series characteristics of the vehicle motion;

[0017] In step 103 of the present invention, the historical trajectory features and scene features obtained above interact through the Mamba module, and the selective state space model (Selective SSM) and gating mechanism are used to refine and compress relevant information, filter out unnecessary information, so as to better capture the social and temporal dynamics between vehicles and obtain the combined information of vehicle scenes.

[0018] Step 104 of the present invention implements global interaction, uses a transformer-based attention mechanism global interaction module to perform long-distance information matching of vehicle scene combined information, uses agent tokens to aggregate global information and assigns it to each lane token, updates the node encoding to include information of nearby agents, enables the model to focus on agents related to the possible path of the target vehicle, obtains global interaction information, and improves the accuracy of trajectory prediction.

[0019] Step 105 of the present invention performs strategy head path selection and multimodal decoding; specifically, the strategy head uses a multi-layer perceptron (MLP) to obtain the global information to generate the outbound edge probability of each node in the graph, indicating the possible future path selection of the target vehicle, and uses the motion state of the target vehicle and the local scene and agent context information to calculate the score of each edge, and converts it into a probability distribution through a softmax layer.

[0020] The implementation method of vehicle trajectory prediction described in the present invention also includes preparatory work; the preparatory work is to obtain necessary data from a high-precision map, and the data includes lane lines, traffic signs, signal lights, sidewalks, stop lines, etc.; extract lane center lines from the map data, which will be used to represent the possible travel paths of the vehicle, and divide the lane center lines into segments of fixed length, each segment corresponding to a node (Node) in the graph neural network; create feature vectors for each node, and these feature vectors include: the x and y coordinates of the node in the bird's-eye view (BEV), the orientation angle of the lane at the node, indicating whether the node is located on the stop line or the crosswalk; then use the nodes to construct a directed graph, in which the nodes represent the lane center lines, and the edges (Edge) represent the possible movement of the vehicle between lanes; according to the possibility of the vehicle moving between lanes, the edges are divided into two categories: Successor Edges: edges connecting adjacent nodes on the same lane, indicating the possibility of the vehicle continuing to travel along the current lane; Proximal Edges: edges connecting adjacent nodes on the same lane, indicating the possibility of the vehicle continuing to travel along the current lane; Edges: Edges connecting adjacent lane nodes, indicating possible lane change behaviors of vehicles; assigning indices to each node and edge in the graph for reference in the model; converting graph structures and node features into vector form f n v =[x n v ,y n v ,θ n v ,L n v ], these vectors will be used as one of the inputs of the model; similarly, to initialize the vectorized state of the vehicle, first collect the vehicle's state information, including position, speed, acceleration and heading angle; the state information is converted into vector form, such as position [x, y], speed [v x ,v y ], acceleration [a x ,a y ], and the heading angle [yaw], integrating the vehicle’s historical trajectory into a state sequence s = [x, y, v, a, ω] as another input.

[0021] The index results tested by the method of the present invention are as follows: 1.22≤ADE_5≤1.24; 0.49≤MR_5≤0.51; 0.89≤ADE_10≤0.91, 0.32≤MR_10≤0.35.

[0022] The basic idea of ​​this method is: first, by designing a self-supervised pre-training task based on path consistency, combined with knowledge distillation and cross entropy loss, the model can deeply learn road consistency constraints and object association knowledge in the pre-training stage. This self-supervised pre-training not only improves the model's ability to understand the scene, but also enhances its robustness in complex traffic scenarios, enabling it to better capture the social and temporal dynamics between vehicles, thus having more advantages in prediction accuracy; then, by combining the global interaction module with the Mamba module, a more comprehensive fusion of agent and scene information is achieved. The global interaction module enables the model to focus on agents related to the possible path of the target vehicle, while the Mamba module further refines and compresses relevant information and filters out unnecessary interference. This combination enables this method to more accurately predict the future trajectory of the vehicle when dealing with multi-vehicle interactions and complex road topologies; finally, a multi-task loss function is used to combine the path consistency loss and the regularization loss of trajectory prediction, and a staged training strategy is adopted, so that the model can make full use of different training data and strategies in the pre-training and fine-tuning stages. The present invention can use path consistency pre-training tasks to train robust encoders and interaction modules, reduce data set noise interference, solve the incomplete scene recognition defects in the prior art, improve the model's long-distance trajectory matching learning ability, and ultimately improve the accuracy and diversity of predictions. The index results tested by the present invention are: 1.22≤ADE_5≤1.24; 0.49≤MR_5≤0.51; 0.89≤ADE_10≤0.91, 0.32≤MR_10≤0.35; ADE_K: minimum average displacement error of the first K prediction results; MR_K: miss rate (MissRate_K) The first K prediction results differ from the actual trajectory by more than 2 meters.

[0023] The beneficial effects of adopting the above technical solution are: 1. The present invention designs a novel self-supervised pre-training task based on path consistency, and trains the encoder and interaction module through knowledge distillation and cross entropy loss. This pre-training task enables the model to maintain the consistency of object identity under different path observations, that is, the association probability distribution between objects should remain consistent, thereby providing a reliable self-supervisory signal for the model, so that it can learn useful knowledge of road consistency constraints in the pre-training stage, laying a solid foundation for subsequent trajectory prediction tasks. At the same time, the knowledge distillation technology is introduced, and through the structure of the teacher network and the student network, the student network can learn the output of the teacher network, realize the transfer of knowledge and improve the robustness of the model. 2. The present invention innovatively introduces the Mamba module, which refines and compresses relevant information through the selective state space model (Selective SSM) and the gating mechanism, filters out unnecessary information, thereby enhancing the model's ability to model long sequences and better capture the social and temporal dynamics between vehicles. In addition, a global interaction module based on the attention mechanism is designed, which uses agent tokens to aggregate global information and assigns it to each lane token, updates the node encoding to include the information of nearby agents, enables the model to focus on agents related to the possible path of the target vehicle, improves the accuracy of trajectory prediction, and better captures the interaction and influence between vehicles. 3. The present invention uses a strategy head to generate the outgoing edge probability of each node in the graph through a multi-layer perceptron (MLP), indicating the possible future path selection of the target vehicle, and considers the motion state of the target vehicle and the local scene and agent context information, which can more accurately evaluate the probability of different paths. The multimodal decoder selectively aggregates the relevant context information along the sampled path by sampling the path generated by the strategy head, and uses the multi-head scaled dot product attention mechanism to generate the future trajectory prediction of the target vehicle, so that the model can capture a wider range of possible future trajectories and enhance the diversity and robustness of the prediction. At the same time, Kmeans clustering is used to optimize the sampled trajectory, and the cluster center is selected as the final K prediction mode to further improve the diversity and accuracy of the prediction results. 4. The present invention defines a multi-task loss function, which combines the path consistency loss and the regularization loss of trajectory prediction for end-to-end training of the model, so that the model can simultaneously improve the performance of path consistency and trajectory prediction in the pre-training and fine-tuning stages. A phased training strategy is adopted, which relies on real trajectories for pre-training at the beginning of training, and then uses the paths sampled from the path strategy for fine-tuning, so that the model can make full use of different training data and strategies at different stages, improving training efficiency and model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below in conjunction with specific embodiments.

[0026] Example 1

[0027] Traffic trajectory prediction for a certain city scene

[0028] Data preparation:

[0029] 1. Dataset preparation: Collect information about urban scenes, such as using a car camera to shoot scenes and collect high-precision maps of the city online, including 1,000 driving scenes, with a sampling frequency of 2 Hz and a prediction time of 6 seconds, covering the entire city traffic.

[0030] 2. High-precision map data extraction: Extract lane centerlines, traffic signs, signal lights, sidewalks, stop lines and other information from high-precision maps. Segment the lane centerlines into segments of fixed length, and each segment serves as a node in the graph neural network.

[0031] 3. Feature vector construction: The feature vector of each node includes: the x and y coordinates of the node in the bird's eye view (BEV), the direction of the lane at the node, and whether the node is located at the stop line or the crosswalk. Vehicle state information is vectorized: position [x, y], speed [vx, vy], acceleration [ax, ay], heading angle [yaw], integrated into a state sequence s = [x, y, v, a, ω] as input.

[0032] Model implementation:

[0033] 1. Self-supervised pre-training (step 101): Using the collected dataset, randomly skip second-level frames to construct different observation paths and simulate sensor failures. Using knowledge distillation, the teacher network is constructed through the exponential moving average (EMA) of the student network weights, and the student network optimizes its own parameters through cross entropy loss.

[0034] 2. Agent motion coding and scene coding (step 102): Use GRU and MLP to capture the time series characteristics of vehicle motion, and combine pre-trained encoder parameters to extract historical trajectory features and scene features.

[0035] 3. Interaction module (step 103): Using the Mamba module, the information is refined through the Selective State Space Model (SelectiveSSM) and the gating mechanism to filter out unnecessary data.

[0036] 4. Global interaction (step 104): Use the Transformer-based attention mechanism to perform global information matching, aggregate global information and assign it to lane tokens, and update node encoding.

[0037] 5. Strategy head path selection and multimodal decoding (step 105): The strategy head generates the outgoing edge probability of each node through MLP, calculates the edge score and converts it into a probability distribution through softmax.

[0038] 6. Trajectory prediction (step 106): Perform K-means clustering on the trajectory probability distribution and select the cluster centers as the final K prediction modes.

[0039] Finally, a series of test indicators are obtained by comparing the predicted values ​​with the true values. The test indicators of this embodiment are: ADE_5 (5-second average displacement error): 1.22, MR_5 (5-second omission rate): 0.49, ADE_10 (10-second average displacement error): 0.89, MR_10 (10-second omission rate): 0.32. The indicators meet the required range.

[0040] The above embodiments are only used to illustrate rather than limit the technical solutions of the present invention. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that the present invention can still be modified or replaced by equivalents. Any modification or partial replacement that does not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for implementing vehicle trajectory prediction, characterized in that: The method for implementing the vehicle trajectory prediction comprises the following steps: Step 101 first performs self-supervised pre-training; obtains the model parameters of the encoder in step 102; Step 102 encodes the motion of the agent and the scene; Step 103: The historical trajectory features and scene features obtained above interact through the Mamba module to obtain the combined information of the vehicle scene; Step 104 implements global interaction, using a transformer-based attention mechanism global interaction module to perform long-distance information matching of vehicle scene combined information; Step 105 performs strategy header path selection and multi-modal decoding; Step 106 performs trajectory prediction, performs Kmeans clustering on the sampled trajectory probability distribution, and selects cluster centers as the final K prediction modes, representing the diversified prediction results obtained from the trajectory distribution.

2. The method for implementing vehicle trajectory prediction according to claim 1, characterized in that: In step 101, knowledge distillation is divided into a student network and a teacher network, and the input information is divided into two parts; the teacher network is constructed by the exponential moving average of the weights of the student network, and the student network optimizes its own parameters by learning the output of the teacher network, thereby realizing the transfer of knowledge and improving the robustness of the model.

3. The method for implementing vehicle trajectory prediction according to claim 1, characterized in that: The step 101 first performs self-supervised pre-training; the specific method is that after the vehicle motion trajectory data obtained after the preparation work forms a time series, different observation paths are constructed by randomly skipping the second-level frames therein, simulating the data loss caused by sensor failure, and providing the model with multiple visual observation perspectives; then the encoder and the interaction module are trained using knowledge distillation and cross entropy loss; finally, the object association probability distribution under different paths is learned and matched by minimizing the cross entropy loss of the student network parameters, so that the model obtains useful knowledge of the road consistency constraints in the pre-training stage, and the model parameters of the encoder in step 102 are obtained.

4. A method for implementing vehicle trajectory prediction according to any one of claims 1 to 3, characterized in that: The step 102 encodes the motion of the intelligent body and the scene, uses the encoder parameters obtained in step 102 to re-extract the historical trajectory features and scene features of the vehicle, and combines the gated recurrent unit and the multi-layer perceptron to capture the time series characteristics of the vehicle motion.

5. A method for implementing vehicle trajectory prediction according to any one of claims 1 to 3, characterized in that: In step 103, the historical trajectory features and scene features obtained above interact through the Mamba module, and the selective state space model and gating mechanism are used to refine and compress relevant information and filter out unnecessary information, so as to better capture the social and temporal dynamics between vehicles and obtain the combined information of vehicle scenes.

6. A method for implementing vehicle trajectory prediction according to any one of claims 1 to 3, characterized in that: The step 104 implements global interaction, uses a transformer-based attention mechanism global interaction module to perform long-distance information matching of vehicle scene combined information, uses agent tokens to aggregate global information and assigns it to each lane token, updates node encoding to include information of nearby agents, enables the model to focus on agents related to the possible path of the target vehicle, obtains global interaction information, and improves the accuracy of trajectory prediction.

7. A method for implementing vehicle trajectory prediction according to any one of claims 1 to 3, characterized in that: The step 105 performs strategy head path selection and multimodal decoding; specifically, the strategy head uses the obtained global information through a multi-layer perceptron to generate the outgoing edge probability of each node in the graph, indicating the possible future path selection of the target vehicle, and uses the motion state of the target vehicle and the local scene and agent context information to calculate the score of each edge, and converts it into a probability distribution through a softmax layer.

8. A method for implementing vehicle trajectory prediction according to any one of claims 1 to 3, characterized in that: The method for implementing vehicle trajectory prediction also includes preparatory work; the preparatory work is to obtain necessary data from a high-precision map, the data including lane lines, traffic signs, signal lights, sidewalks, and stop lines; extract lane center lines from the map data, these center lines will be used to represent the possible driving paths of the vehicle, and divide the lane center lines into segments of fixed length, each segment corresponding to a node in the graph neural network; Create feature vectors for each node. These feature vectors include: the x and y coordinates of the node in the bird's-eye view, the angle of the lane at the node, and whether the node is located at a stop line or a crosswalk. Then, a directed graph is constructed using the nodes, where the nodes represent the center lines of the lanes and the edges represent the possible movement of the vehicle between lanes. The edges are divided into two categories according to the possibility of the vehicle moving between lanes: successor edges: edges connecting adjacent nodes on the same lane, indicating the possibility of the vehicle continuing to move along the current lane; adjacent edges: edges connecting nodes in adjacent lanes, indicating the possible lane change behavior of the vehicle. An index is assigned to each node and edge in the graph for reference in the model. The graph structure and node features are converted into vector form f n v =[x n v ,y n v ,θ n v ,L n v ], these vectors will be used as one of the inputs of the model; similarly, to initialize the vectorized state of the vehicle, first collect the vehicle's state information, including position, speed, acceleration and heading angle; the state information is converted into vector form, such as position [x, y], speed [v x ,v y ], acceleration [a x ,a y ], and the heading angle [yaw], integrating the vehicle’s historical trajectory into a state sequence s = [x, y, v, a, ω] as another input.

9. A method for implementing vehicle trajectory prediction according to any one of claims 1 to 3, characterized in that: The index results tested by the method are: 1.22≤ADE_5≤1.24; 0.49≤MR_5≤0.51; 0.89≤ADE_10≤0.91, 0.32≤MR_10≤0.35.

Images