Motion prediction method based on long and short term memory network and multivariate Gaussian distribution

Abstract

The invention relates to a motion prediction method based on a long and short term memory network and multivariate Gaussian distribution. The method comprises the following steps: acquiring data, and constructing model input information; extracting data in the coordinates to calculate a mean value and a variance of a motion prediction data set; performing normalization processing, and inputting normalized data into the recurrent neural network model; solving a probability density function by using parameters output and decomposed by the neural network; calculating the gradient of the probability density function to the trainable parameters of the model; a cycle with the total cycle index of PREDLEN is constructed, model input is prevdata, and the init idle state of an LSTM unit trained by observation part data is constructed; according to the method, the future trajectory of the traffic participant on the road can be estimated in advance, the coping capacity of the system for unexpected situations in the driving process is enhanced, the driving smoothness of the system is improved, and fuel consumption reduction and power consumption reduction of the whole vehicle can be further achieved based on the technology.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving motion behavior prediction, and in particular to a motion prediction method based on long short-term memory network and multivariate Gaussian distribution. Background Art

[0002] Autonomous driving motion prediction involves predicting the future trajectory of an autonomous vehicle to better plan its route. Currently, the main technical solutions for autonomous driving motion prediction include physics-based methods, machine learning-based methods, and hybrid model-based methods. Physics-based methods predict the vehicle's future trajectory by modeling its kinematics and dynamics; machine learning-based methods learn the vehicle's future motion patterns through training on large amounts of data; and hybrid model-based methods combine these two approaches to achieve better prediction results.

[0003] Among physics-based models, commonly used models include linear models, nonlinear models, and hybrid models. The simplest model is the linear model, which assumes that the car will move in a straight line over a period of time. The nonlinear model models the car's motion as a nonlinear function. Hybrid models combine multiple models of different types to achieve better prediction results.

[0004] Commonly used machine learning algorithms include decision trees, support vector machines, neural networks, and deep learning. Decision trees are simple yet effective algorithms that classify data based on input features; support vector machines are powerful and flexible algorithms that can handle high-dimensional data and have excellent generalization capabilities; neural networks, similar to the neural network structure of the human brain, can handle complex nonlinear problems; and deep learning, based on neural networks, can handle large amounts of data and has strong expressive power.

[0005] Among the methods based on hybrid models, commonly used algorithms include particle filtering, Kalman filtering, and extended Kalman filtering. Among them, particle filtering is a filtering algorithm based on the Monte Carlo method, which can handle nonlinear problems and has good robustness; Kalman filtering is an optimal linear unbiased estimation algorithm, which can handle linear problems and has good real-time performance; and extended Kalman filtering is an algorithm that generalizes Kalman filtering to nonlinear problems.

[0006] Physics-based approaches predict a vehicle's future trajectory by modeling its kinematics and dynamics. Machine learning-based approaches learn the vehicle's future motion patterns through training on large amounts of data. Hybrid model-based approaches combine multiple models of different types to achieve better predictions.

[0007] However, existing machine learning-based methods are mainly used for motion prediction of pedestrians, passenger cars and non-motor vehicles. There is currently no relevant research demonstrating the experimental effect of this method on commercial vehicles. At the same time, the lack of high-quality motion prediction data in the commercial vehicle field limits the performance of data-driven methods.

[0008] Physics-based approaches require modeling of vehicle kinematics and dynamics, requiring extensive expertise and experience. Machine learning-based approaches require extensive data training, consuming significant time and computing resources, and suffer from poor system interpretability.

[0009] Hybrid model-based methods need to combine multiple models of different types, thus requiring more computing resources and time. Summary of the Invention

[0010] The present invention proposes a motion prediction method based on long short-term memory network and multivariate Gaussian distribution, which solves the technical problems of prediction systems such as insufficient application research of commercial vehicle motion prediction technology, lack of commercial vehicle motion prediction data set, and poor interpretability of machine learning methods.

[0011] In order to solve the problems in the above background technology, the present invention is achieved through the following technical solutions:

[0012] A motion prediction method based on a long short-term memory network and a multivariate Gaussian distribution comprises the following steps:

[0013] S1, based on the world coordinate system, collects historical trajectories of surrounding traffic participants detected by the vehicle's perception system during normal driving, and constructs a motion prediction dataset. The motion prediction dataset includes the following elements: time-related frame labels, object types, object marker labels, coordinates, sizes, and target orientation angles;

[0014] S2, feature screening: selecting one or more elements in the motion prediction dataset to form the model input information;

[0015] S3, data cleaning: obtaining the normalization factor of the motion prediction dataset: extracting all x and y data in the coordinates to calculate the mean and variance of the motion prediction dataset;

[0016] S4, data set establishment: according to the minimum observation sequence length of the model, the motion prediction data set is processed to generate sequence data;

[0017] S5, normalization processing: normalize the x and y data, and then input the normalized data into the recurrent neural network model composed of LSTM neurons;

[0018] S6, decomposes the neural network output into five parameters of a two-dimensional Gaussian distribution, namely μ x , μ y , σ x , σ y ,ρ xy , construct a binary Gaussian distribution model about the target x and y, and bring the target position x and y into the binary Gaussian distribution model to obtain the probability density function;

[0019] S7, define the optimizer, calculate the gradient of the probability density function with respect to the model's trainable parameters θ, and clip the calculated gradient to prevent gradient explosion;

[0020] S8, given the observation sequence length OBS_LEN and the prediction sequence length PRED_LEN, implements the model reasoning;

[0021] S9, constructs a loop with a total number of loops PRED_LEN, the model input is prev_data, and the initial_state state of the LSTM unit trained with the observed partial data.

[0022] Preferably, the one or more elements in step S2 are time-related frame labels, object types and coordinates.

[0023] Preferably, in step S3, the formula for normalizing the motion prediction data set is:

[0024]

[0025] Where μ is the data mean and σ is the variance.

[0026] Preferably, step S4 is: extracting data with the same object tag that meets the sequence length, and naming it as an array traj; and making x a segment of the object's historical trajectory that meets the length be recorded as traj[idx:idx+sequence_length,:], and y a segment of the trajectory that is one position behind the x trajectory, and recorded as traj[idx+1:idx+sequence_length+1,:].

[0027] Preferably, in step S6, the formula for obtaining the probability density function is:

[0028]

[0029]

[0030] At the same time, to ensure numerical stability, the result gradient is clipped, where ε = e -20 , ρ is the parameter predicted by the neural network, and the formula is:

[0031]

[0032] Then average l and perform L2 regularization on l, where θ is the trainable parameter of the LSTM model. Substitute l into l to get L:

[0033]

[0034] Preferably, in step S7, during the gradient descent process, where η is the learning rate, a threshold T is preset to prevent gradient explosion, and f(θ) is clipped:

[0035]

[0036] Where f(θ) is the function representing the parameters of the neural network. To derive f(θ);

[0037] calculate The norm of , scaling the items exceeding the threshold to within the range of T:

[0038]

[0039] Repeat steps S4, S5, S6, and S7 to complete the model training.

[0040] Preferably, step S8 is: initializing the LSTM unit state, extracting the historical trajectory data of the traffic participant whose historical trajectory data length is greater than or equal to OBS_LEN, normalizing the input observation data based on the normalization factor obtained in step S3, and then inputting the current observed trajectory of the target into the trained LSTM network, and returning the LSTM unit state as initial_state, assigning the end position observation data to prev_data, and realizing model reasoning.

[0041] Preferably, the step S9 is: the LSTM model predicts the statistical parameters of the multivariate Gaussian distribution model according to the input and output, which is recorded as

[0042]

[0043]

[0044]

[0045] Where Z is the current multivariate Gaussian distribution random sampling value, that is, the predicted target position of the next position sample under the output historical observation, that is, X(x,y); is the neural network prediction parameter. Based on formula (8), when predicting K trajectories, for the next candidate position at the same time, the predicted multivariate Gaussian distribution can be sampled K times.

[0046] Compared with the prior art, the present invention has the following beneficial technical effects:

[0047] 1. For post-fusion autonomous driving systems, this method can reduce system algorithm processing time, improve system response speed, and reduce the algorithm processing hardware load. For each traffic participant's K trajectory prediction, the prediction time is 1 / K, compared to a method based entirely on the LSTM model.

[0048] 2. It enables trajectories of traffic participants three seconds into the future, facilitating autonomous driving decision-making and enabling control systems to handle emergencies in advance. This improves the autonomous driving system's ability to respond to emergencies and reduces the risk of accidents involving autonomous vehicles.

[0049] 3. In dynamic interaction scenarios, obtaining the trajectory distribution of traffic participants over the next three seconds can improve the ability of autonomous vehicles to navigate complex interaction scenarios such as intersections.

[0050] 4. The autonomous driving system can achieve predictive control based on the future trajectory distribution of traffic participants, improving the control compliance of the autonomous vehicle and enhancing the driving comfort and satisfaction of the safety officer;

[0051] 5. Based on the future trajectory distribution of traffic participants, the autonomous driving system can improve the vehicle's fuel and power saving levels and reduce vehicle energy consumption levels according to predictive control strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the process of the present invention;

[0053] Figure 2 Schematic diagram of the training model of the present invention;

[0054] Figure 3 Schematic diagram of the reasoning model of the present invention. DETAILED DESCRIPTION

[0055] like Figure 1 As shown, a motion prediction method based on long short-term memory network and multivariate Gaussian distribution includes the following steps:

[0056] S1, based on the world coordinate system, collects historical trajectories of surrounding traffic participants detected by the vehicle's perception system during normal driving, and constructs a motion prediction dataset. The motion prediction dataset includes the following elements: time-related frame labels, object types, object marker labels, coordinates, sizes, and target orientation angles;

[0057] S2, feature screening: selecting one or more elements in the motion prediction dataset to form the model input information;

[0058] S3, data cleaning: obtaining the normalization factor of the motion prediction dataset: extracting all x and y data in the coordinates to calculate the mean and variance of the motion prediction dataset;

[0059] S4, data set establishment: according to the minimum observation sequence length of the model, the motion prediction data set is processed to generate sequence data;

[0060] S5, normalization processing: normalize the x and y data, and then input the normalized data into the recurrent neural network model composed of LSTM neurons;

[0061] S6, decomposes the neural network output into five parameters of a two-dimensional Gaussian distribution, namely μ x , μ y , σ x , σ y ,ρ xy , construct a binary Gaussian distribution model about the target x and y, and bring the target position x and y into the binary Gaussian distribution model to obtain the probability density function;

[0062] S7, define the optimizer, calculate the gradient of the probability density function with respect to the model's trainable parameters θ, and clip the calculated gradient to prevent gradient explosion;

[0063] S8, given the observation sequence length OBS_LEN and the prediction sequence length PRED_LEN, implements the model reasoning;

[0064] S9, constructs a loop with a total number of loops PRED_LEN, the model input is prev_data, and the initial_state state of the LSTM unit trained with the observed partial data.

[0065] The one or more elements in step S2 are time-related frame labels, object types, and coordinates.

[0066] In step S3, the formula for normalizing the motion prediction data set is:

[0067]

[0068] Where μ is the data mean and σ is the variance.

[0069] Step S4 is: extract the data with the same object label that meets the sequence length and name it as an array traj; and let x be a trajectory segment that meets the length on the object's historical trajectory, recorded as traj[idx:idx+sequence_length,:], and y be the trajectory segment of x that is shifted one position, recorded as traj[idx+1:idx+sequence_length+1,:].

[0070] In step S6, the formula for obtaining the probability density function is:

[0071]

[0072]

[0073] At the same time, to ensure numerical stability, the result gradient is clipped, where ε = e -20 , ρ is the parameter predicted by the neural network, and the formula is:

[0074]

[0075] Then average l and perform L2 regularization on l, where θ is the trainable parameter of the LSTM model. Substitute l into l to get L:

[0076]

[0077] In step S7, during the gradient descent process, where η is the learning rate, a threshold T is preset to prevent gradient explosion and f(θ) is clipped:

[0078]

[0079] Where f(θ) is the function representing the parameters of the neural network. To derive f(θ);

[0080] calculate The norm of , scaling the items exceeding the threshold to within the range of T:

[0081]

[0082] Repeat steps S4, S5, S6, and S7 to complete the model training.

[0083] Step S8 is: initialize the LSTM unit state, extract the historical trajectory data of traffic participants whose historical trajectory data length is greater than or equal to OBS_LEN, normalize the input observation data based on the normalization factor obtained in step S3, then input the current observed trajectory of the target into the trained LSTM network, and return the LSTM unit state as initial_state, assign the last position observation data to prev_data, and realize the reasoning of the model.

[0084] Step S9 is: the LSTM model predicts the statistical parameters of the multivariate Gaussian distribution model based on the input and output, which is recorded as

[0085]

[0086]

[0087]

[0088] Where Z is the current multivariate Gaussian distribution random sampling value, that is, the predicted target position of the next position sample under the output historical observation, that is, X(x,y); is the neural network prediction parameter. Based on formula (8), when predicting K trajectories, for the next candidate position at the same time, the predicted multivariate Gaussian distribution can be sampled K times.

Claims

1. A motion prediction method based on long short-term memory network and multivariate Gaussian distribution, characterized by: The following steps are involved: S1, based on the world coordinate system, collects historical trajectories of surrounding traffic participants detected by the vehicle's perception system during normal driving, and constructs a motion prediction dataset. The motion prediction dataset includes the following elements: time-related frame labels, object types, object marker labels, coordinates, sizes, and target orientation angles; S2, feature screening: selecting one or more elements in the motion prediction dataset to form the model input information; S3, data cleaning: obtaining the normalization factor of the motion prediction dataset: extracting all x and y data in the coordinates to calculate the mean and variance of the motion prediction dataset; S4, data set establishment: according to the minimum observation sequence length of the model, the motion prediction data set is processed to generate sequence data; S5, normalization processing: normalize the x and y data, and then input the normalized data into the recurrent neural network model composed of LSTM neurons; S6, decomposes the neural network output into five parameters of a two-dimensional Gaussian distribution, namely μ x , μ y , σ x , σ y ,ρ xy , construct a binary Gaussian distribution model about the target x and y, and bring the target position x and y into the binary Gaussian distribution model to obtain the probability density function; S7, define the optimizer, calculate the gradient of the probability density function with respect to the model's trainable parameters θ, and clip the calculated gradient to prevent gradient explosion; S8, given the observation sequence length OBS_LEN and the prediction sequence length PRED_LEN, implements the model reasoning; S9, constructs a loop with a total number of loops PRED_LEN, the model input is prev_data, and the initial_state state of the LSTM unit trained with the observed partial data.

2. The motion prediction method based on long short-term memory network and multivariate Gaussian distribution according to claim 1, characterized in that: The one or more elements in step S2 are time-related frame labels, object types, and coordinates.

3. The motion prediction method based on long short-term memory network and multivariate Gaussian distribution according to claim 1, characterized in that: In step S3, the formula for normalizing the motion prediction data set is: Where μ is the data mean and σ is the variance.

4. The motion prediction method based on long short-term memory network and multivariate Gaussian distribution according to claim 3, characterized in that: The step S4 is as follows: extracting data with the same object tag that meets the sequence length and naming it as an array of traj; and making x a segment of the object's historical trajectory that meets the length be recorded as traj[idx:idx+sequence_length,:], and y a segment of the trajectory that is one position behind the x trajectory be recorded as traj[idx+1:idx+sequence_length+1,:].

5. The motion prediction method based on long short-term memory network and multivariate Gaussian distribution according to claim 1, characterized in that: In step S6, the formula for obtaining the probability density function is: At the same time, to ensure numerical stability, the result gradient is clipped, where ε = e -20 , ρ is the parameter predicted by the neural network, and the formula is: Then average l and perform L2 regularization on l, where θ is the trainable parameter of the LSTM model. Substitute l into l to get L:

6. The motion prediction method based on long short-term memory network and multivariate Gaussian distribution according to claim 1, characterized in that: In S7, during the gradient descent process, η is the learning rate. To prevent gradient explosion, a threshold T is preset and f(θ) is clipped: Where f(θ) is the function representing the parameters of the neural network. To derive f(θ); calculate The norm of , scaling the items exceeding the threshold to within the range of T: Repeat steps S4, S5, S6, and S7 to complete the model training.

7. The motion prediction method based on long short-term memory network and multivariate Gaussian distribution according to claim 2, characterized in that: Step S8 is as follows: initializing the LSTM unit state, extracting the historical trajectory data of traffic participants whose historical trajectory data length is greater than or equal to OBS_LEN, normalizing the input observation data based on the normalization factor obtained in step S3, and then inputting the current observed trajectory of the target into the trained LSTM network, returning the LSTM unit state as initial_state, and assigning the end position observation data to prev_data to implement model reasoning.

8. The motion prediction method based on long short-term memory network and multivariate Gaussian distribution according to claim 3, characterized in that: The step S9 is: the LSTM model predicts the statistical parameters of the multivariate Gaussian distribution model according to the input and output, which is recorded as Where Z is the current multivariate Gaussian distribution random sampling value, that is, the predicted target position of the next position sample under the output historical observation, that is, X(x,y); is the neural network prediction parameter. Based on formula (8), when predicting K trajectories, for the next candidate position at the same time, the predicted multivariate Gaussian distribution can be sampled K times.

Images