Vehicle multi-mode motion trail prediction method considering dynamic executable constraint

Abstract

The invention discloses a vehicle multi-modal motion trajectory prediction method considering dynamic executable constraints, aims to consider the influence of a vehicle motion model on prediction performance, and specifically comprises the following steps: (1) coding a vehicle trajectory and a road map by using an encoder through an LSTM network; (2) encoding the interaction relationship between the predicted vehicle and the surrounding vehicles and the interaction relationship between the predicted vehicle and the road map through an encoder; and (3) modeling the predicted vehicle into a two-degree-of-freedom kinematics model, incorporating the model into a decoder, and outputting a dynamically executable multi-modal prediction trajectory. According to the method, the influence of the motion of the vehicle in the interaction scene and the two-degree-of-freedom kinematics model are brought into the motion trail prediction of the vehicle, and finally the purpose of predicting the multi-modal motion trail of the vehicle is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of automobile electronic control and intelligence, and is a vehicle multi-modal motion trajectory prediction method considering dynamic executable constraints. Background Art

[0002] Existing motion trajectory prediction models generally ignore the impact of the autonomous driving system's own behavior on the environment, as well as the interactive motion characteristics of the vehicle and surrounding vehicles in traffic scenarios. They only focus on prediction accuracy and lack consideration of the real needs and explainability in practical applications.

[0003] Due to the uncertainty of traffic environment and driving behavior, the interaction between vehicles is nonlinear and strongly coupled, and is easily affected by multiple uncertain factors. Therefore, vehicle trajectories usually show a certain degree of uncertainty. In order to objectively reflect the real situation, the motion trajectory prediction model needs to consider the influence of objective conditions such as surrounding vehicles and roads, and the output results should have multiple possibilities, that is, to generate an acceptable and easy-to-interpret probability distribution. This type of model generally mines the intrinsic characteristics of the driver from real environment data, directly trains and learns the probability distribution of the predicted trajectory, and outputs multimodal trajectory prediction results for the target vehicle, such as CVAE, GAN, LSTM, Transformer, etc.

[0004] Most of the existing generative models directly output predicted driving trajectories, ignoring the motion dynamics of traffic participants. The essence of vehicle motion is to generate a driving trajectory through driving action input and motion dynamics conversion. They lack consideration of the impact of the vehicle motion model on the prediction performance and find it difficult to capture the dynamic information of the longitudinal and lateral coupling of the vehicle, which can easily lead to positional discontinuity in the trajectory prediction results and may even result in situations that do not conform to physical facts.

[0005] Some existing patents, such as the invention patent with patent number CN113657433B, discloses a vehicle trajectory multimodal prediction method that can improve the stability and accuracy of intelligent vehicles' prediction of the trajectory of traffic vehicles in the traffic environment, and the invention patent with patent number CN114997484A discloses a multi-stage multimodal vehicle trajectory prediction method that can constrain the predicted trajectory of the vehicle within a reasonable range with the help of lane lines. The former establishes a multimodal trajectory prior prediction model through the full probability formula, and combines the road constraints to filter the trajectory, and then uses the Bayesian Nash equilibrium theory to correct the trajectory probability to achieve accurate prediction of the vehicle's future trajectory. The latter generates candidate trajectories by establishing and combining the first model and the second model prediction, and outputs the predicted trajectory with the candidate trajectory and the top-k algorithm. Both pursue prediction accuracy, and neither considers the impact of the movement / dynamics of traffic participants in the interactive scene. Summary of the invention

[0006] The present invention aims to consider the impact of vehicle motion in interactive scenarios and proposes a vehicle multimodal motion trajectory considering dynamically executable constraints. The method first encodes the vehicle trajectory and the road map through an encoder, then encodes the interaction relationship between the predicted vehicle and the surrounding vehicles, as well as the interaction relationship between the predicted vehicle and the road map through the encoder, and finally models the predicted vehicle as a two-degree-of-freedom kinematic model, and incorporates this model into a decoder to output a multimodal predicted trajectory with dynamic executableness.

[0007] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0008] The present invention provides a vehicle multimodal motion trajectory prediction method considering dynamic executable constraints, comprising the following steps: S1: encoding the vehicle trajectory and the road map by an encoder, the specific steps are as follows:

[0009] S11: An LSTM network with 32 hidden dimensions is used to encode the trajectory of the vehicle, and a trajectory encoding vector including the current state characteristics and historical state characteristics of the predicted vehicle and its surrounding vehicles is obtained, as shown in equation (1):

[0010]

[0011] where (x, y), v, a and are the position, velocity, acceleration and heading of the vehicle in the local coordinate system. The local coordinate system is centered on the current position of the predicted vehicle, and its heading is aligned on the x-axis. h is the historical time step, N is the number of vehicles, including up to 10 surrounding vehicles within a 30-meter radius of the predicted vehicle, and S represents the vehicle trajectory encoding vector;

[0012] S12: Obtain the vector information of the road sampling point through road information encoding; each road sampling point consists of a set of feature vectors Indicates that (x r ,y r ) is the position relative to the predicted vehicle, is the direction of each road sampling point; the road sampling points on the same lane share the same lane features, and use 0-1 encoding to represent whether the lane has the features of turning, intersection and traffic light control; the road sampling points and lane features are processed by the fully connected layer respectively and then spliced ​​to obtain the road map encoding vector;

[0013] S2: Encode the interaction relationship between the predicted vehicle and surrounding vehicles, and the interaction relationship between the predicted vehicle and the road map through an encoder. The specific steps are as follows:

[0014] S21: Taking the vehicle trajectory encoding vector in step S11 as input, encode the interaction relationship between the predicted vehicle and the surrounding vehicles; first, the predicted vehicle is given different attention weights for the surrounding vehicles through the self-attention mechanism; then, the multi-head attention mechanism shown in equation (2) is used to provide different self-attention results for each surrounding vehicle; finally, through two position fully connected feedforward layers and two summation and normalization layers, the vehicle-to-vehicle interaction encoding vector is obtained;

[0015]

[0016] Where h is the total number of attention heads; Q is the query vector for predicting vehicle features; KV is the key-value pair vector; W i Q , W i K , W i V , W O The attention function calculates the attention weights of the predicted vehicle to the surrounding vehicles in a mapping manner, and normalizes the weight sum by introducing the softmax function, as shown below:

[0017]

[0018] Where d is the dimension of the key vector;

[0019] S22: Taking the vehicle trajectory encoding vector in step S11 and the road map encoding vector in step S12 as input, encode and predict the interaction relationship between the vehicle and the road map; adopt the multimodal attention mechanism shown in equation (4) to ensure the diversity of the attention of the predicted vehicle to different parts of the map. It does not connect and project each head into a low-dimensional vector, but directly outputs the result of each head; then, through two positional fully connected feedforward layers and two summation and normalization layers, obtain the vehicle and road map interaction encoding vector;

[0020] MultiHead(Q m ,K m ,V m )=(head1,…,head h ) (4)

[0021] Where Q m is the query vector for predicting vehicle features; K m and V m are the key vector and value vector of the road map feature respectively;

[0022] S3: Model the predicted vehicle as a two-degree-of-freedom kinematic model and incorporate this model into the decoder to output a multimodal predicted trajectory of dynamic executableness;

[0023] S31: A discretized latent variable z∈Z is introduced to express the uncertainty of predicting the potential driving behavior of the vehicle. Assuming that the posterior distribution of the driving behavior follows a normal distribution, the mean and variance of the distribution are obtained by neural network fitting to meet the multimodal output requirements; the model output and the current state probability distribution are converted into the distribution of the latent variable, as shown in the following formula:

[0024]

[0025] In the formula, With p θ (z|x) represents the neural network model; and θ are the weight parameters of the two neural networks respectively; Z is the discrete feasible domain of the latent variable z, and its range represents the diversity of behaviors;

[0026] S32: The vehicle trajectory encoding vector, the road map encoding vector, the vehicle-to-vehicle interaction encoding vector, the vehicle-to-road map interaction encoding vector, and the hidden variable z in step S31 are used as the input of the decoder and sent to a 128-dimensional gated recurrent unit (GRU), and each GRU unit outputs a control operation that follows the parameters of the binary Gaussian distribution;

[0027] S33: A two-degree-of-freedom kinematic model is used to model the motion state of the predicted vehicle. Assuming that the control input in each sampling step remains unchanged, the discrete system state equation under zero-order hold shown in equation (6) is obtained;

[0028]

[0029] At each time step, the control input satisfies the following Gaussian distribution N(μ u ,Σ u ):

[0030]

[0031] where μ ω is the average yaw rate of the vehicle; μ a is the average acceleration of the vehicle; σ ω is the standard deviation of the heading angle change rate; σ a is the standard deviation of acceleration; ρ ωa is the correlation coefficient between ω and a;

[0032] The state space equation of equation (6) is linearly discretized at the current state and control through the Jacobian matrix to obtain the state covariance equation of the predicted vehicle, as shown in equation (8);

[0033]

[0034]

[0035] S34: Substitute the control input average value in equation (7) into the system state equation in equation (6) to obtain the average state of the predicted vehicle; substitute the control input variance in equation (7) into the state covariance equation in equation (8) to obtain the state covariance of the predicted vehicle; add the average state and state covariance of the predicted vehicle at the corresponding step in the prediction time domain to obtain the multimodal prediction trajectory of the predicted vehicle.

[0036] Compared with the prior art, the advantages of the present invention are:

[0037] 1. Aiming at the problems that the existing trajectory prediction models lack consideration of system dynamic constraints, lack of interpretability, and are difficult to apply to behavior planning algorithms, a vehicle multimodal motion trajectory prediction model considering dynamic interaction characteristics is proposed. First, the vehicle trajectory and road map are encoded by the encoder. Then, the encoder encodes the relationship between the predicted vehicle and the surrounding vehicles, as well as the relationship between the predicted vehicle and the road map. Finally, the predicted vehicle is modeled as a two-degree-of-freedom kinematic model, and the model is integrated into the decoder to output a multimodal predicted trajectory with dynamic executableness. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0039] Figure 1 A flowchart of a vehicle multimodal motion trajectory prediction method considering dynamic executable constraints according to the present invention;

[0040] Figure 2 A vehicle interaction trajectory encoding vector flow chart of the vehicle multimodal motion trajectory prediction method considering dynamic executable constraints according to the present invention;

[0041] Figure 3 A vehicle trajectory and map encoding vector flow chart of the vehicle multimodal motion trajectory prediction method considering dynamic executable constraints described in the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0043] The present invention will be further described below in conjunction with the accompanying drawings.

[0044] See also Figure 1 The vehicle multimodal motion trajectory prediction method considering dynamic executable constraints of the present invention specifically comprises the following steps:

[0045] S1: Encode the vehicle trajectory and road map through the encoder. The specific steps are as follows:

[0046] S11: An LSTM network with 32 hidden dimensions is used to encode the trajectory of the vehicle, and a trajectory encoding vector including the current state characteristics and historical state characteristics of the predicted vehicle and its surrounding vehicles is obtained, as shown in equation (1):

[0047]

[0048] where (x, y), v, a and are the position, velocity, acceleration and heading of the vehicle in the local coordinate system. The local coordinate system is centered on the current position of the predicted vehicle, and its heading is aligned on the x-axis. h is the historical time step, N is the number of vehicles, including up to 10 surrounding vehicles within a 30-meter radius of the predicted vehicle, and S represents the vehicle trajectory encoding vector;

[0049] S12: Obtain the vector information of the road sampling point through road information encoding; each road sampling point consists of a set of feature vectors Indicates that (x r ,y r ) is the position relative to the predicted vehicle, is the direction of each road sampling point; the road sampling points on the same lane share the same lane features, and use 0-1 encoding to represent whether the lane has the features of turning, intersection and traffic light control; the road sampling points and lane features are processed by the fully connected layer respectively and then spliced ​​to obtain the road map encoding vector;

[0050] S2: Encode the interaction relationship between the predicted vehicle and surrounding vehicles, and the interaction relationship between the predicted vehicle and the road map through an encoder. The specific steps are as follows:

[0051] S21: See Figure 2 , the vehicle trajectory encoding vector flow chart, takes the vehicle trajectory encoding vector in step S11 as input, encodes the interaction relationship between the predicted vehicle and the surrounding vehicles; first, the predicted vehicle is given different attention weights for the surrounding vehicles through the self-attention mechanism; then, the multi-head attention mechanism shown in equation (2) is used to provide different self-attention results for each surrounding vehicle; finally, through two position fully connected feedforward layers, and two summation and normalization layers, the vehicle-to-vehicle interaction encoding vector is obtained;

[0052]

[0053] Where h is the total number of attention heads; Q is the query vector for predicting vehicle features; KV is the key-value pair vector; W i Q , W i K , W i V , W O The attention function calculates the attention weights of the predicted vehicle to the surrounding vehicles in a mapping manner, and normalizes the weight sum by introducing the softmax function, as shown below:

[0054]

[0055] Where d is the dimension of the key vector;

[0056] S22: See Figure 3 , a flow chart of vehicle trajectory and map encoding vector, taking the vehicle trajectory encoding vector in step S11 and the road map encoding vector in step S12 as input, encoding and predicting the interaction relationship between the vehicle and the road map; adopting the multimodal attention mechanism shown in equation (4) to ensure the diversity of the attention of the predicted vehicle to different parts of the map, it does not connect and project each head into a low-dimensional vector, but directly outputs the result of each head; then, through two positional fully connected feedforward layers and two summation and normalization layers, the vehicle and road map interaction encoding vector is obtained;

[0057] MultiHead(Q m ,K m ,V m )=(head1,…,head h ) (4)

[0058] Where Q m is the query vector for predicting vehicle features; K m and V m are the key vector and value vector of the road map feature respectively;

[0059] S3: Two-degree-of-freedom vehicle model, which models the predicted vehicle as a two-degree-of-freedom kinematic model and incorporates this model into the decoder to output a multimodal predicted trajectory of dynamic executableness;

[0060] S31: A discretized latent variable z∈Z is introduced to express the uncertainty of predicting the potential driving behavior of the vehicle. Assuming that the posterior distribution of the driving behavior follows a normal distribution, the mean and variance of the distribution are obtained by neural network fitting to meet the multimodal output requirements; the model output and the current state probability distribution are converted into the distribution of the latent variable, as shown in the following formula:

[0061]

[0062] In the formula, With p θ (z|x) represents the neural network model; and θ are the weight parameters of the two neural networks respectively; Z is the discrete feasible domain of the latent variable z, and its range represents the diversity of behaviors;

[0063] S32: The vehicle trajectory encoding vector, the road map encoding vector, the vehicle-to-vehicle interaction encoding vector, the vehicle-to-road map interaction encoding vector, and the hidden variable z in step S31 are used as the input of the decoder and sent to a 128-dimensional gated recurrent unit (GRU), and each GRU unit outputs a control operation that follows the parameters of the binary Gaussian distribution;

[0064] S33: A two-degree-of-freedom kinematic model is used to model the motion state of the predicted vehicle. Assuming that the control input in each sampling step remains unchanged, the discrete system state equation under zero-order hold shown in equation (6) is obtained;

[0065]

[0066] At each time step, the control input satisfies the following Gaussian distribution N(μ u ,Σ u ):

[0067]

[0068] where μ ω is the average yaw rate of the vehicle; μ a is the average acceleration of the vehicle; σ ω is the standard deviation of the heading angle change rate; σ a is the standard deviation of acceleration; ρ ωa is the correlation coefficient between ω and a;

[0069] The state space equation of equation (6) is linearly discretized at the current state and control through the Jacobian matrix to obtain the state covariance equation of the predicted vehicle, as shown in equation (8);

[0070]

[0071]

[0072] S34: Substitute the control input average value in equation (7) into the system state equation in equation (6) to obtain the average state of the predicted vehicle; substitute the control input variance in equation (7) into the state covariance equation in equation (8) to obtain the state covariance of the predicted vehicle; add the average state and state covariance of the predicted vehicle at the corresponding step in the prediction time domain to obtain the multimodal prediction trajectory of the predicted vehicle.

[0073] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A vehicle multimodal motion trajectory prediction method considering dynamic executable constraints, characterized in that: The following steps are involved: S1: Encode the vehicle trajectory and road map through the encoder. The specific steps are as follows: S11: An LSTM network with 32 hidden dimensions is used to encode the trajectory of the vehicle, and a trajectory encoding vector including the current state characteristics and historical state characteristics of the predicted vehicle and its surrounding vehicles is obtained, as shown in equation (1): where (x, y), v, a and are the position, velocity, acceleration and heading of the vehicle in the local coordinate system. The local coordinate system is centered on the current position of the predicted vehicle, and its heading is aligned on the x-axis. h is the historical time step, N is the number of vehicles, including up to 10 surrounding vehicles within a 30-meter radius of the predicted vehicle, and S represents the vehicle trajectory encoding vector; S12: Obtain the vector information of the road sampling points through road information encoding; each road sampling point consists of a set of feature vectors Indicates that (x r ,y r ) is the position relative to the predicted vehicle, is the direction of each road sampling point; the road sampling points on the same lane share the same lane features, and use 0-1 encoding to represent whether the lane has the features of turning, intersection and traffic light control; the road sampling points and lane features are processed by the fully connected layer respectively and then spliced ​​to obtain the road map encoding vector; S2: Encode the interaction relationship between the predicted vehicle and surrounding vehicles, and the interaction relationship between the predicted vehicle and the road map through an encoder. The specific steps are as follows: S21: Taking the vehicle trajectory encoding vector in step S11 as input, encode the interaction relationship between the predicted vehicle and the surrounding vehicles; first, the predicted vehicle is given different attention weights for the surrounding vehicles through the self-attention mechanism; then, the multi-head attention mechanism shown in equation (2) is used to provide different self-attention results for each surrounding vehicle; finally, through two position fully connected feedforward layers and two summation and normalization layers, the vehicle-to-vehicle interaction encoding vector is obtained; Where h is the total number of attention heads; Q is the query vector for predicting vehicle features; KV is the key-value pair vector; W i Q , W i K , W i V , W O The attention function calculates the attention weights of the predicted vehicle to the surrounding vehicles in a mapping manner, and normalizes the weight sum by introducing the softmax function, as shown below: Where d is the dimension of the key vector; S22: Taking the vehicle trajectory encoding vector in step S11 and the road map encoding vector in step S12 as input, encode and predict the interaction relationship between the vehicle and the road map; adopt the multimodal attention mechanism shown in equation (4) to ensure the diversity of the attention of the predicted vehicle to different parts of the map. It does not connect and project each head into a low-dimensional vector, but directly outputs the result of each head; then, through two positional fully connected feedforward layers and two summation and normalization layers, obtain the vehicle and road map interaction encoding vector; MultiHead(Q m ,K m ,V m )=(head1,…,head h ) (4) Where Q m is the query vector for predicting vehicle features; K m and V m are the key vector and value vector of the road map feature respectively; S3: Model the predicted vehicle as a two-degree-of-freedom kinematic model and incorporate this model into the decoder to output a multimodal predicted trajectory of dynamic executableness; S31: A discretized latent variable z∈Z is introduced to express the uncertainty of predicting the potential driving behavior of the vehicle. Assuming that the posterior distribution of the driving behavior follows a normal distribution, the mean and variance of the distribution are obtained by neural network fitting to meet the multimodal output requirements; the model output and the current state probability distribution are converted into the distribution of the latent variable, as shown in the following formula: In the formula, With p θ (z|x) represents the neural network model; and θ are the weight parameters of the two neural networks respectively; Z is the discrete feasible domain of the latent variable z, and its range represents the diversity of behaviors; S32: The vehicle trajectory encoding vector, the road map encoding vector, the vehicle-to-vehicle interaction encoding vector, the vehicle-to-road map interaction encoding vector, and the hidden variable z in step S31 are used as the input of the decoder and sent to a 128-dimensional gated recurrent unit (GRU), and each GRU unit outputs a control operation that follows the parameters of the binary Gaussian distribution; S33: A two-degree-of-freedom kinematic model is used to model the motion state of the predicted vehicle. Assuming that the control input in each sampling step remains unchanged, the discrete system state equation under zero-order hold shown in equation (6) is obtained; in, Control variable u = [ω, a]; At each time step, the control input satisfies the following Gaussian distribution N(μ u ,Σ u ): where μ ω is the average yaw rate of the vehicle; μ a is the average acceleration of the vehicle; σ ω is the standard deviation of the heading angle change rate; σ a is the standard deviation of acceleration; ρ ωa is the correlation coefficient between ω and a; The state space equation of equation (6) is linearly discretized at the current state and control through the Jacobian matrix to obtain the state covariance equation of the predicted vehicle, as shown in equation (8); S34: Substitute the control input average value in equation (7) into the system state equation in equation (6) to obtain the average state of the predicted vehicle; substitute the control input variance in equation (7) into the state covariance equation in equation (8) to obtain the state covariance of the predicted vehicle; add the average state and state covariance of the predicted vehicle at the corresponding step in the prediction time domain to obtain the multimodal prediction trajectory of the predicted vehicle.

Images