A deep learning trajectory prediction method introducing vehicle kinematic constraints

Abstract

A deep learning trajectory prediction method introducing vehicle kinematic constraints is proposed to solve the problem of lack of physical consistency in trajectory prediction. The invention relates to the field of autonomous driving. The invention includes a deep learning prediction module, a physical model prediction module and a model optimization training module. The deep learning prediction module receives vehicle and map information, extracts features and interacts using a gated recurrent unit (GRU) and a graph convolution subgraph network, and outputs a first predicted trajectory. The physical model prediction module uses a PID controller to calculate a control quantity based on a reference position state, and drives a kinematic model to close loop and deduce a second predicted trajectory that conforms to the kinematic constraints. The model optimization training module constructs a joint loss based on the prediction error and the physical consistency error between the two trajectories, and optimizes the network parameters through back propagation, finally loads the optimal model to output the trajectory.

Description

Technical Field

[0001] This invention belongs to the field of autonomous vehicle technology, specifically a deep learning trajectory prediction method that incorporates vehicle kinematic constraints. Background Technology

[0002] As autonomous driving technology advances towards advanced Level 3 and even Level 4 applications, the safety and stability of vehicles driving on complex urban roads have become a core bottleneck restricting their widespread adoption. Trajectory prediction, as the core connecting the perception system and the decision-making and planning system, directly determines whether autonomous vehicles can implement safe avoidance strategies based on the accuracy and reasonableness of their predictions. Currently, research on trajectory prediction mainly falls into two major technical routes: traditional methods based on physical models and data-driven methods based on deep learning.

[0003] Traditional physical models based on kinematics or dynamics, following Newton's laws and vehicle geometric constraints, possess interpretability and physical reliability. However, when faced with complex interaction scenarios such as congestion games and mixed pedestrian-vehicle traffic, these methods struggle to capture complex traffic coupling relationships because they cannot perceive the potential interaction intentions between traffic participants and complex environmental topology information. This results in severe error divergence in long-term time-domain tasks, making it difficult to support safe decision-making for autonomous driving in complex road conditions.

[0004] In contrast, currently widely adopted prediction methods based on deep learning, such as Long Short-Term Memory (LSTM), Graph Convolutional Networks (GCN), and Transformers, excel in capturing environmental semantics and interactive intentions due to their powerful high-dimensional feature extraction capabilities, achieving prediction accuracy far exceeding traditional physical models on public datasets. However, deep learning is essentially a data-driven "black box" system. The training process relies solely on probabilistic fitting of sample data, lacking explicit modeling of the inherent physical constraints of vehicle dynamics and kinematics. This makes the model prone to generating physically infeasible abnormal predictions in uncovered edge cases, outputting abnormal trajectories that violate kinematic common sense, such as lateral translation and sudden changes in turning angle.

[0005] The lack of environmental semantic perception in physical models and the blind pursuit of deep learning ("only looking at data, not understanding physics") have led to a significant gap between interpretability and high-dimensional perception capabilities. Therefore, how to retain the advantages of high-dimensional perception in deep learning while deeply internalizing explicit physical and dynamic constraints into the network parameters has become a key technical challenge to be solved in the field of autonomous driving trajectory prediction.

[0006] To address the aforementioned issues, some existing technologies have attempted to combine physical models with deep learning. Specifically, regarding patents CN 106950956 A and CN 119037470 A, these two patents primarily focus on shallow fusion at the input end, where environmental cognitive information and kinematic state are only used as features input to the network. The physical model does not substantially participate in the network's training and update process, and the network parameters still rely solely on positional errors for optimization, failing to fundamentally solve the problem of "black box" models generating unreasonable trajectories.

[0007] Regarding patents CN 120606865 A and CN 118885967 B, these two patents generate initial trajectories using deep learning networks, and then use kinematic equations to smooth and constrain these trajectories. This approach struggles to correct severe physical deviations in the initial trajectory, significantly increasing the computational burden during the inference phase. This is fundamentally different from the efficient strategy of this invention, which internalizes physical laws into the network parameters, thereby achieving physical supervision during the training phase and independent prediction during the application phase.

[0008] Regarding patent CN 11576182 B, this patent only uses vehicle kinematics formulas to construct mathematical penalty terms, applying discrete numerical constraints to the predicted trajectory points. While this mathematical constraint forces the trajectory points to satisfy geometric relationships, it lacks consideration for control continuity, leading to potentially severe oscillations in the control quantity over time in the predicted trajectory. In contrast, the trajectory generated by this invention utilizes PID control, which implicitly incorporates smooth and asymptotic control logic, ensuring the dynamic stability of the prediction results from the outset.

[0009] Regarding patent CN 116495007 A, this patent only uses thresholds such as maximum speed and turning angle to set static physical boundaries, and implements constraints by penalizing exceeding limits. While this passive threshold truncation method defines the edge of the physically feasible region, it cannot provide the dynamic evolution logic of the vehicle over time. In contrast, this invention utilizes a PID closed-loop system to generate an ideal reference trajectory with a clear control intent, directly guiding the neural network to learn the optimal control process, ensuring that the predicted trajectory does not exceed numerical limits, and simultaneously meeting the requirements of smoothness and asymmetry in vehicle motion in terms of control logic.

[0010] Regarding patent CN 116424365 B, this patent ignores the core role of the backpropagation mechanism in physical fusion, while this invention uses the joint loss function to explicitly update the network weights, so that the network internalizes physical constraints during training, and achieves a deep unification of physical model and data-driven approach. Summary of the Invention

[0011] To address the limitations of traditional physical models in perceiving complex environmental interactions and exhibiting large prediction errors over long time domains, as well as the problems of deep learning prediction models lacking physical constraints and easily generating abnormal trajectories that violate kinematic laws, this invention provides a deep learning trajectory prediction method that incorporates vehicle kinematic constraints. This invention aims to deeply internalize vehicle kinematic constraints and control logic into the neural network parameters by constructing a closed-loop joint training mechanism between the physical model and deep learning, thereby significantly improving the physical consistency and executability of the predicted trajectory while maintaining high-dimensional environmental perception accuracy.

[0012] The technical solution adopted by this invention to solve the technical problem is as follows:

[0013] A deep learning trajectory prediction method incorporating vehicle kinematic constraints addresses the lack of physical consistency in trajectory prediction. This invention relates to the field of autonomous driving. It includes a deep learning prediction module, a physical model prediction module, and a model optimization and training module. The deep learning prediction module receives historical vehicle information and a high-resolution vector map, extracts temporal features using a gated recurrent unit (GRU), extracts map topological features through a subgraph network of a graph convolutional network, and outputs a first predicted trajectory after interaction. The physical model prediction module uses a PID controller to calculate control inputs based on reference position states, drives the vehicle kinematic model to perform closed-loop deduction, and generates a second predicted trajectory that conforms to kinematic constraints. The model optimization and training module constructs a joint loss function, calculates the prediction error between the first predicted trajectory and the actual trajectory, and the physical consistency error between the first and second predicted trajectories. Then, it uses backpropagation of the joint loss function to optimize the parameters of the deep learning prediction network, and finally loads the trained optimal deep learning prediction network model to output the vehicle trajectory.

[0014] The method includes the following steps:

[0015] Step 1: Design the deep learning prediction module, which includes the following sub-steps:

[0016] Step 1.1: Collect historical state data of the target vehicle and surrounding traffic participants as vehicle historical information, specifically including lateral and longitudinal position coordinates, speed, acceleration, and heading angle. Encode this data as tensors and input them into the GRU to extract the temporal features of vehicle motion. Simultaneously, construct a lane map based on a high-definition vector map, discretize the lane centerline into map nodes, and use the subgraph network of the graph convolutional network to preliminarily encode the geometry and connection relationships of the lane nodes, extracting map topological features containing geometric information.

[0017] Step 1.2: Input the vehicle motion temporal features and map topology features obtained in Step 1.1 into the interactive network, including the following sub-steps:

[0018] Step 1.2.1: Utilize attention mechanism 1 to perform vehicle-to-node interaction. Inject vehicle motion temporal features into map node features, enabling map features to reflect the current traffic congestion and occupancy status.

[0019] Step 1.2.2: Perform node-to-node interactions using a graph convolutional network. Apply multi-scale graph convolution to the updated map nodes to propagate node information across the lane topology map, capturing the global topology and long-distance dependencies of the road network, and generating fused deep map node features.

[0020] Step 1.2.3: Node-to-vehicle interaction is performed through attention mechanism 2, transmitting the fused deep map node features to vehicle features to extract and fuse road geometry information. Simultaneously, attention mechanism 2 performs vehicle-to-vehicle interaction to extract social interaction information between the target vehicle and other traffic participants, ultimately outputting the final vehicle features that fuse road geometry information and social interaction information.

[0021] Step 1.3: Input the final vehicle features output from Step 1.2 into the trajectory decoder composed of a fully connected layer (MLP) to calculate the displacement coordinates at future time steps and generate the vehicle's first predicted trajectory. .

[0022] Step 2: Design the physical model prediction module, which includes the following sub-steps:

[0023] Step 2.1: Obtain the vehicle's initial time. The reference position state is determined and initialized to the current system state of the vehicle's physical model.

[0024] Step 2.2: Calculate the lateral position deviation between the current system state and the reference position state. Heading deviation The front wheel angle is calculated using a PID controller at the current moment. With longitudinal acceleration The calculation formula is as follows:

[0025] (1)

[0026] (2)

[0027] in, These are the proportional, integral, and derivative gain coefficients for lateral control, respectively. This is the heading deviation feedback gain coefficient; These are the proportional and integral gain coefficients for longitudinal control, respectively. For the desired vehicle speed; This represents the lateral deviation difference value.

[0028] Step 2.3: Calculate the control quantity, front wheel steering angle. With longitudinal acceleration The data is input into the vehicle's kinematics bicycle model, and the next moment is deduced based on the current system state. The vehicle position status is updated using the following equation:

[0029] (3)

[0030] (4)

[0031] (5)

[0032] (6)

[0033] (7)

[0034] in, These are the distances from the vehicle's center of gravity to the front and rear axles, respectively. It is the centroid sideslip angle; For time step.

[0035] Step 2.4, based on the calculations in Step 2.3 The vehicle's position status is updated to the current system status at each moment, and the reference position status for the next moment is obtained. Steps 2.2 to 2.3 are repeated until the future is generated. The second predicted trajectory sequence at time 1 .

[0036] Step 3: Design the model optimization and training module, which includes the following sub-steps:

[0037] Step 3.1: Calculate the first predicted trajectory output by the deep learning prediction module. With respect to the vehicle's actual driving trajectory The positional deviation between them is used to construct the prediction error loss term. The calculation formula is as follows:

[0038] (8)

[0039] in, To predict the length of the time domain, This represents the Euclidean norm.

[0040] Step 3.2: Calculate the first predicted trajectory output by the deep learning prediction module. The second predicted trajectory output by the physical model prediction module The trajectory error between them is used to construct a physical consistency loss term. The calculation formula is as follows:

[0041] (9)

[0042] Step 3.3: Introduce the balance coefficient The prediction error loss term Physical consistency loss term By performing weighted summation, we obtain the joint loss function ultimately used for network optimization. :

[0043] (10)

[0044] Step 3.4: Based on the calculated joint loss function The weight parameters of the deep learning prediction module are updated through backpropagation until the network converges to obtain the optimal prediction model. Attached Figure Description

[0045] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0046] The present invention will now be described in detail with reference to the accompanying drawings.

[0047] This invention proposes a deep learning trajectory prediction method incorporating vehicle kinematics model constraints. The method includes a deep learning prediction module, a physical model prediction module, and a model optimization and training module. The deep learning prediction module receives historical vehicle information and a high-resolution vector map, extracts temporal features using a GRU, extracts map topological features through a lane map convolutional subgraph network, and outputs a first predicted trajectory after interaction. The physical model prediction module uses a PID controller to calculate control inputs based on reference position states, drives the vehicle kinematics model to perform closed-loop deduction, and generates a second predicted trajectory that conforms to kinematic constraints. The model optimization and training module constructs a joint loss function, calculates the prediction error between the first predicted trajectory and the actual trajectory, and the physical consistency error between the first and second predicted trajectories. Then, it uses backpropagation of the joint loss function to optimize the deep learning prediction network parameters, and finally loads the trained optimal deep learning prediction network model to output the vehicle trajectory. (Refer to...) Figure 1 The illustration specifically includes the following steps:

[0048] Step 1: Design the deep learning prediction module, which includes the following sub-steps:

[0049] Step 1.1: Collect historical state data of the target vehicle and surrounding traffic participants over the past 2 seconds (20 frames total) at a sampling rate of 10Hz. The state data includes horizontal and vertical position coordinates. ,speed acceleration and heading angle The aforementioned historical state sequence is input into the GRU network. The GRU network contains a single-layer GRU structure with 128 hidden units, used to extract the temporal feature vector of the vehicle's motion. The GRU update formula is as follows:

[0050] (11)

[0051] (12)

[0052] (13)

[0053] (14)

[0054] in, This represents the current input state. This is the hidden state from the previous moment. To update the door, To reset the door, The output is the vehicle motion timing feature. These are the weight matrices for the corresponding gates. These are the bias terms for the corresponding gates. This is the Sigmoid activation function.

[0055] Lane maps are constructed based on high-resolution vector maps, discretizing the complex lane centerlines into a sequence of map nodes containing position coordinates and direction attributes. Encoding is performed using a subgraph network of a lane map convolutional network. This subgraph network consists of three stacked 1D convolutional layers with a kernel size of 3 and channel numbers of 64, 64, and 128 respectively. Each layer is followed by a normalization layer and a ReLU activation function. The final output map node feature dimension is set to 128, consistent with the dimension of the vehicle motion temporal feature vector extracted by GRU, facilitating subsequent feature interaction and fusion. Through these operations, the geometry and connectivity of lane nodes are initially encoded, thereby extracting initial map topological features containing local geometric information.

[0056] Step 1.2: Input the vehicle motion temporal features and map topology features obtained in Step 1.1 into the interactive network, including the following sub-steps:

[0057] Step 1.2.1: Perform vehicle-to-node interaction using attention mechanism 1. Using map topology features as the query term Q, and vehicle motion temporal features as the key term K and value term V, calculate using formula (15):

[0058] (15)

[0059] Where Q is the query matrix, K is the key matrix, and V is the value matrix; The dimension of the feature vector is set to 128 in this embodiment; This is the normalization factor. The formula is used to calculate the attention weight of each map node to surrounding vehicles, and the dynamic information of vehicles is weighted and aggregated into the map node features, enabling the updated map nodes to reflect the current traffic congestion and occupancy status of the lane.

[0060] Step 1.2.2: Perform node-to-node interactions using a graph convolutional network. Apply multi-scale graph convolution to the updated map nodes to propagate node information across the lane topology map, capturing the global topology and long-distance dependencies of the road network, and generating fused deep map node features.

[0061] Step 1.2.3: Perform node-to-vehicle interaction through attention mechanism 2. Using the attention mechanism described in formula (15), the fused deep map node features are passed back to the vehicle features. The vehicle motion temporal features are used as the query term Q, and the enhanced deep map node features are used as the key term K and value term V. By calculating the correlation between the vehicle and the map nodes, the road geometry is extracted and fused and passed to the vehicle features to improve the vehicle's perception of the surrounding environment. At the same time, vehicle-to-vehicle interaction is performed through attention mechanism 2. Using the attention mechanism described in formula (15), social interaction information between vehicles is extracted. The vehicle features are used as the query term Q, key term K, and value term V. The mutual influence weights between the target vehicle and other traffic participants are calculated, and the final vehicle features are obtained by fusing road geometry information and social interaction information.

[0062] Step 1.3: Input the final vehicle features output from Step 1.2 into the trajectory decoder. This decoder consists of two MLP layers. The first layer maps the feature dimension from 128 to 64 and connects it to the ReLU activation function; the second layer maps the dimension to 60 and is used to output the predicted value. To prevent overfitting, a dropout layer with a dropout rate of 0.1 is added between the layers.

[0063] The decoder regresses and predicts coordinate displacement increments over 3 seconds, with each time step being 0.1 seconds. Then, using the vehicle's current coordinates as a reference, these coordinate displacement increments are accumulated over time to obtain the absolute coordinates for each future time step, thereby generating the vehicle's first predicted trajectory. .

[0064] Step 2: Design the physical model prediction module, which includes the following sub-steps:

[0065] Step 2.1: Obtain the vehicle's initial time. The reference position status, specifically including coordinates Longitudinal velocity Heading angle The reference position state is the vehicle's actual driving trajectory data, which is initialized to the current system state of the vehicle's physical model.

[0066] Step 2.2: Calculate the lateral position deviation between the current system state and the reference position state. Heading deviation The front wheel angle is calculated using a PID controller at the current moment. With longitudinal acceleration The calculation formula is as follows:

[0067] (16)

[0068] (17)

[0069] in, These are the proportional, integral, and derivative gain coefficients for lateral control, respectively. This is the heading deviation feedback gain coefficient; These are the proportional and integral gain coefficients for longitudinal control, respectively. The desired vehicle speed is the longitudinal speed in the reference position state. This represents the lateral deviation difference value.

[0070] Step 2.3: Calculate the control quantity, front wheel steering angle. With longitudinal acceleration Input into the vehicle kinematics bicycle model, based on the current system state Predicting the next moment In this embodiment, the vehicle location status is determined by setting a time step. Distance from center of mass to front axle Distance from center of mass to rear axle The updated equation is:

[0071] (18)

[0072] (19)

[0073] (20)

[0074] (twenty one)

[0075] (twenty two)

[0076] in, These are the distances from the vehicle's center of gravity to the front and rear axles, respectively. The sideslip angle is the angle of the centroid. For time step.

[0077] Step 2.4, based on the calculations in Step 2.3 Real-time vehicle location status Update to the current system state, obtain the reference position state for the next moment, and repeat steps 2.2 to 2.3 until the future is generated. The second predicted trajectory sequence at time 1 .

[0078] Step 3: Design the model optimization and training module, which includes the following sub-steps:

[0079] Step 3.1: Calculate the first predicted trajectory output by the deep learning prediction module. With respect to the vehicle's actual driving trajectory The prediction error loss term between Both trajectories are absolute coordinate sequences in the global coordinate system, with a data dimension of T×2, corresponding to the next 3 seconds. Coordinates. The calculation formula is as follows:

[0080] (twenty three)

[0081] in, To predict the time domain length (T=30), Denotes the Euclidean norm. Indicates time A two-dimensional coordinate vector.

[0082] Step 3.2: Calculate the first predicted trajectory output by the deep learning prediction module. The second predicted trajectory output by the physical model prediction module The trajectory error between them is used to construct a physical consistency loss term. The calculation formula is as follows:

[0083] (twenty four)

[0084] Step 3.3: Introduce the balance coefficient The prediction error loss term Physical consistency loss term By performing weighted summation, we obtain the joint loss function ultimately used for network optimization. :

[0085] (25)

[0086] Step 3.4: Based on the calculated joint loss function The Adam optimizer was used to train the deep learning prediction module via backpropagation, with an initial learning rate of 0.001, a batch size of 32, and 50 training epochs. During training, gradient descent was used to calculate gradients and update only the weight parameters of the deep learning prediction module, while strictly keeping the parameters of the physical model prediction module fixed, until the network converged and the optimal deep learning prediction model was obtained.

[0087] Step 3.5: After training, only the saved optimal deep learning prediction model is loaded for inference. Using the historical state data and map information of the target vehicle as input, the same data preprocessing and feature extraction operations as in steps 1.1 and 1.2 are performed. The extracted features are input into the optimal deep learning prediction model, and the network is used for forward propagation to directly output the vehicle prediction trajectory sequence that conforms to kinematic constraints.

Claims

1. A deep learning trajectory prediction method incorporating vehicle kinematic constraints, characterized in that: The deep learning prediction module in this method is used to receive vehicle historical information and high-definition vector map, extract temporal features using gated recurrent unit (GRU), extract map topology features through graph convolutional subgraph network, and output the first predicted trajectory after interaction; the physical model prediction module calculates the control quantity and the deviation of the current position state based on the reference position state, uses PID controller to drive the vehicle kinematic model to perform closed-loop deduction, and generates the second predicted trajectory that conforms to kinematic constraints. The model optimization training module constructs a joint loss based on the prediction error and the physical consistency error between the first and second prediction trajectories, and optimizes the network parameters through backpropagation. The specific execution process of the deep learning prediction module includes the following sub-steps: Step 1.1: Collect historical vehicle status data and extract vehicle motion temporal features through a gated recurrent unit (GRU). Simultaneously, construct a lane map based on a high-definition vector map, discretize the lane centerline into map nodes, and use a subgraph network of a graph convolutional network to preliminarily encode the geometry and connection relationships of the lane nodes, extracting initial map topological features containing local geometric information. Step 1.2: Input the vehicle motion temporal features obtained in Step 1.1 and the initial map topology features into the interactive network, including the following sub-steps: Step 1.2.1: Use the attention mechanism to perform vehicle-to-node interaction, inject the vehicle motion time sequence features into the map node features, so that the map features can reflect the current traffic congestion and occupancy status. Step 1.2.2: Utilize graph convolutional networks to perform node-to-node interactions, apply multi-scale graph convolution to the updated map nodes, propagate node information on the lane topology map, capture the global topology and long-distance dependencies of the road network, and generate fused deep map node features. Step 1.2.3: Perform node-to-vehicle interaction through the attention mechanism, pass the fused deep map node features to vehicle features, and extract and fuse road geometric information; At the same time, the attention mechanism performs vehicle-to-vehicle interaction, extracts social interaction information of the target vehicle and other traffic participants, and finally outputs the final vehicle features that integrate road geometry information and social interaction information. Step 1.3: Input the final vehicle features output in Step 1.2 into the trajectory decoder to calculate the displacement coordinates at future times and generate the first predicted trajectory of the vehicle. The specific execution process of the physical model prediction module includes the following sub-steps: Step 2.1: Obtain the vehicle's initial time. The reference position state is determined and initialized to the current system state of the vehicle's physical model; Step 2.2: Calculate the lateral position deviation between the current system state and the reference position state. Heading deviation ; The front wheel angle is calculated using a PID controller at the current moment. With longitudinal acceleration The calculation formula is as follows: in, These are the proportional, integral, and derivative gain coefficients for lateral control, respectively. This is the heading deviation feedback gain coefficient; These are the proportional and integral gain coefficients for longitudinal control, respectively. For the desired vehicle speed; This represents the difference in lateral deviation. Step 2.3: Set the current time Control of front wheel steering angle With longitudinal acceleration The data is input into the kinematic model, and the next moment is deduced based on the current system state. The vehicle position status is updated using the following equation: in, These are the distances from the vehicle's center of gravity to the front and rear axles, respectively. The sideslip angle is the angle of the centroid. For time step; Step 2.4, based on the calculations in Step 2.3 The vehicle's position status is updated to the current system status at each moment, and the reference position status for the next moment is obtained. Steps 2.2 to 2.3 are repeated until the future is generated. The second predicted trajectory sequence at each time point; The specific execution process of the model optimization and training module includes the following sub-steps: Step 3.1: Calculate the first predicted trajectory output by the deep learning prediction module. With respect to the vehicle's actual driving trajectory The positional deviation between them is used to construct the prediction error loss term. The calculation formula is as follows: in, To predict the length of the time domain, Denotes the Euclidean norm; Step 3.2: Calculate the first predicted trajectory output by the deep learning prediction module. The second predicted trajectory output by the physical model prediction module The trajectory error between them is used to construct a physical consistency loss term. The calculation formula is as follows: Step 3.3: Introduce the balance coefficient The prediction error loss term Physical consistency loss term By performing weighted summation, we obtain the joint loss function ultimately used for network optimization. : Step 3.4: Based on the calculated joint loss function The weight parameters of the deep learning prediction module are updated through backpropagation until the network converges to obtain the optimal prediction model; the optimal deep learning prediction network model is loaded, and the final vehicle prediction trajectory is output.

Images