A multi-stage and multi-modal vehicle trajectory prediction method

Abstract

The present invention discloses a multi-stage multi-modal vehicle trajectory prediction method, comprising the following steps: S1: collecting vehicle data, constructing a data set based on data samples; S2: dividing the data set into a training set, a validation set and a test set; S3: establishing a first model to predict an initial endpoint, correcting the initial endpoint based on a lane line, generating a candidate endpoint based on the corrected initial endpoint, and establishing a second model based on the candidate endpoint to generate a multi-modal trajectory; S4: training the first model and the second model; S5: inputting the data in the test set into the first model to generate an initial endpoint, generating a number of candidate endpoints based on the initial endpoint and the lane line, generating a number of candidate trajectories based on the candidate endpoint and the second model, and outputting a predicted trajectory based on the candidate trajectory and the top‑k algorithm. The present invention has the beneficial effect of constraining the predicted trajectory of the vehicle within a reasonable range with the help of the lane line, so that the predicted trajectory is accurate and safe.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle trajectory prediction, and in particular to a multi-stage and multi-modal vehicle trajectory prediction method. Background Art

[0002] For autonomous vehicles to drive safely and efficiently in complex traffic composed of human drivers, they need to be able to make active decisions, such as deciding when to change lanes, overtake, or slow down to allow other vehicles to merge. This requires autonomous vehicles to know the future trajectories of surrounding vehicles so that they can plan their driving status in advance and avoid danger. It can be seen that vehicle trajectories play a very important guiding role in analyzing vehicle driving information, driver driving habits and road load conditions.

[0003] In the existing technology, images are generally used as the input of the algorithm, which limits the scope of application of the algorithm. In real applications, vehicle trajectory prediction algorithms are deployed on vehicle chips, which have limited hardware resources, and constructing an image requires a large amount of resources. At the same time, many existing technologies do not consider the physical constraints of lane lines, and it is easy for the predicted trajectory to appear outside the lane line or even outside the curb. At this time, the predicted trajectory is basically unusable for subsequent tasks.

[0004] For example, a "vehicle trajectory analysis method and device" disclosed in Chinese patent literature, with publication number: CN105513405B, and application date: January 12, 2016, obtains multiple vehicle positions reported by the vehicle to be predicted at different times, merges the multiple vehicle positions, merges two vehicle positions whose distance is less than a first preset value into one vehicle position, generates a vehicle trajectory according to each vehicle position obtained after the merging process, and displays the generated vehicle trajectory, which can avoid repeated drawing points and enable users to understand the vehicle trajectory of the vehicle to be predicted more clearly. However, there is a problem that the physical constraints of the lane are not taken into account, and it is easy for the predicted trajectory to appear outside the lane line or even outside the curb. Summary of the invention

[0005] In view of the deficiency that the prior art does not take the physical constraints of the lane into consideration and the predicted trajectory may easily appear outside the lane line or even outside the curb, the present invention proposes a multi-stage and multi-modal vehicle trajectory prediction method, which can constrain the predicted vehicle trajectory within a reasonable range with the help of lane lines, so that the predicted trajectory is accurate and safe.

[0006] The following is a technical solution of the present invention, a multi-stage and multi-modal vehicle trajectory prediction method, comprising the following steps:

[0007] S1: Collect vehicle data, construct historical information and future information based on the vehicle data, construct data samples based on the historical information, future information and lane lines, and construct a data set based on the data samples;

[0008] S2: The data set is divided into a training set, a validation set and a test set;

[0009] S3: Establishing a first model to predict an initial endpoint, correcting the initial endpoint based on the lane line, generating a candidate endpoint based on the corrected initial endpoint, and establishing a second model based on the candidate endpoint to generate a multimodal trajectory;

[0010] S4: training the first model and the second model;

[0011] S5: The data in the test set is input into the first model to generate an initial endpoint, several candidate endpoints are generated based on the initial endpoint and the lane line, several candidate trajectories are generated based on the candidate endpoints and the second model, and the predicted trajectory is output based on the candidate trajectories and the top-k algorithm.

[0012] In this solution, vehicle data is collected, data samples are constructed based on the historical information, future information and lane lines, and a data set is constructed based on the data samples. The constructed data set is used for model training to ensure the validity of the data. The preprocessed data set is divided into a training set for training the model, a verification set for verifying the training results of the model, and a test set for testing the trained model. The training set ensures the validity of the training model and the accuracy of the output results. A first model is established to predict the initial endpoint, the initial endpoint is corrected based on the lane lines, and a candidate endpoint is generated based on the corrected initial endpoint. A second model is established based on the candidate endpoint to generate a multimodal trajectory, which can constrain the predicted trajectory of the vehicle within a reasonable range with the help of lane lines, so that the predicted trajectory is accurate and safe.

[0013] Preferably, the vehicle data comprises geographic coordinates, speed and acceleration, and the geographic coordinates are expressed as transverse coordinates and longitudinal coordinates of a two-dimensional coordinate system.

[0014] In this solution, the vehicle data includes geographic coordinates, speed and acceleration. The geographic coordinates are expressed as horizontal and vertical coordinates of a two-dimensional coordinate system. The vehicle's position and speed information is recorded to verify past trajectories and predicted trajectories. The horizontal and vertical coordinates facilitate two-dimensional calculations, improve the model training speed and ensure the accuracy of the model output.

[0015] Preferably, the lane lines are left and right lane lines of the target vehicle.

[0016] In this scheme, the physical constraints of the lane are not considered, and it is easy for the predicted trajectory to appear outside the lane line or even outside the curb. At this time, the predicted trajectory is basically unusable for subsequent tasks, ensuring the practicality and accuracy of the experimental output.

[0017] Preferably, the data volume ratio of the training set, validation set and test set is 7:1:2.

[0018] In this solution, the preprocessed data set is divided into a training set for training the model, a validation set for verifying the training results of the model, and a test set for testing the trained model. The data volume ratio is 7:1:2 for training set: validation set: test set. Make full use of existing data to ensure the effectiveness of model training and the accuracy of model output.

[0019] Preferably, the prediction method of the initial endpoint is: sending the features of the vehicle at each moment to the encoder of the fully connected layer and the long short-term memory network to obtain the hidden state, sending the hidden state to the decoder and the fully connected layer of the long short-term memory network, and outputting the trajectory of the last moment.

[0020] In this scheme, for the target vehicle as an example, the historical information X in the past T moments, X = X t-T+1 ,X t-T+2 ,...,X t , t is the current time, T is the past time. The feature X at the i-th time i , X i =x 1 ,x 2 ,...,x n , n represents the number of features. The feature X of each vehicle at each moment i The hidden state H is obtained by sending it to the encoder of the fully connected layer (FC) and the long-short term memory network (LSTM), respectively. H = h t-T+1 ,h t-T+2 ,...,h t , the hidden state H is sent to the decoder and FC layer of LSTM, the future trajectory of the last moment is output, and the initial end point is obtained, so as to facilitate the modeling based on the initial end point to obtain the predicted trajectory.

[0021] Preferably, the method for generating the candidate endpoints is to generate the candidate endpoints by using a square grid with a certain interval around the corrected initial endpoint.

[0022] In this scheme, assuming that the expressions of the left and right lanes of the target vehicle at the current moment are LANE1 and LANE2, then LANE1_y=f(x p ) and LANE2_y=f(x p ), the coordinate of the position after lane line correction is Y fixed =(x,(LANE1_y+LANE1_y+y p ) / 3). With the corrected position of the lane line as the center, 16 candidate endpoints are selected using a 4x4 grid with a spacing of 2 meters, denoted as Y n , where n∈{1,2,...,16}, the candidate endpoints.

[0023] Preferably, the candidate endpoints are classified in a method of: if the distance between the candidate endpoint and the true endpoint is less than a set threshold, it is a positive sample, otherwise, it is a negative sample.

[0024] In this solution, if the distance between the candidate endpoint and the true endpoint is less than the set threshold, it is a positive sample, otherwise, it is a negative sample. The candidate endpoints are classified based on the distance between the candidate endpoint and the true endpoint, which facilitates the judgment of the validity of the candidate endpoint and improves the accuracy of the predicted trajectory.

[0025] Preferably, the positive samples and negative samples are concatenated with the hidden state and sent to the decoder of the long short-term memory network, and classified and regressed in the fully connected layer, the true value label of the classification is the positive sample or the negative sample, and the true value label of the regression is the future position of the sample.

[0026] In this scheme, the positive samples and negative samples are concatenated with the hidden state and sent to the decoder of the long short-term memory network, and classification and regression are performed in the fully connected layer. The classification branch determines whether the sample is a positive sample or a negative sample, and the regression branch predicts the future M positions of the target vehicle, thereby accelerating the response speed of the fully connected layer and improving the accuracy of the predicted trajectory.

[0027] Preferably, the function used for predicting the initial endpoint is the L2 loss function, the function used for classification is the cross entropy loss function, and the function used for regression is the L2 loss function.

[0028] In this scheme, the function used to predict the initial endpoint is the L2 loss function, the function used for classification is the cross entropy loss function, and the function used for regression is the L2 loss function. The error of the L2 loss function is the sum of all square differences between the true value and the predicted value. The cross entropy describes the distance between the probability distribution of the correct answer and the predicted value. The smaller the cross entropy, the closer the distribution of the two probabilities. Setting positive and negative samples, using the cross entropy loss function to calculate faster, the cross entropy loss function and the L2 loss function are used to improve the accuracy of the predicted trajectory.

[0029] Preferably, several trajectories with the highest classification scores are used as multimodal trajectories.

[0030] The beneficial effect of the present invention is that the predicted trajectory of the vehicle can be constrained within a reasonable range by means of lane lines, so that the predicted trajectory is accurate and safe. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flow chart of a multi-stage and multi-modal vehicle trajectory prediction method of the present invention.

[0032] Figure 2 The present invention discloses a candidate trajectory sampling diagram of a multi-stage and multi-modal vehicle trajectory prediction method. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0034] Example: Figure 1 As shown, a multi-stage and multi-modal vehicle trajectory prediction method includes the following steps:

[0035] S1: Data collection and data preprocessing. Collect vehicle data, including geographic coordinates, speed, and acceleration, build historical information and future information based on vehicle data, build data samples based on historical information, future information, and lane lines, and build data sets based on data samples.

[0036] S2: Dataset division. The dataset is divided into training set, validation set and test set.

[0037] S3: Construct a trajectory prediction model. Predict the initial endpoint, correct the initial endpoint based on the lane line, generate a candidate endpoint based on the corrected initial endpoint, and generate a multi-modal trajectory based on the candidate endpoint.

[0038] S4: Model training. Train the prediction model in stages.

[0039] S5: The data in the test set is input into the first model to generate an initial endpoint, several candidate endpoints are generated based on the initial endpoint and the lane line, several candidate trajectories are generated based on the candidate endpoints and the second model, and the predicted trajectory is output based on the candidate trajectories and the top-k algorithm.

[0040] S1: Data collection and data preprocessing. Collect vehicle data, including geographic coordinates, speed, and acceleration, build historical information and future information based on vehicle data, build data samples based on historical information, future information, and lane lines, and build data sets based on data samples.

[0041] Specifically, the geographic coordinates, speed, acceleration and other data of the vehicle are continuously collected, and the geographic coordinates are represented as horizontal coordinates and vertical coordinates in a two-dimensional coordinate system.

[0042] In specific applications, a vehicle at a certain moment is selected as the target vehicle, and the historical information of the target vehicle in the past T moments and the future information in M ​​moments are obtained. The historical information and future information specifically include the horizontal coordinate, the vertical coordinate, the speed, and the acceleration. The left and right lanes of the target vehicle at the current moment are expressed as LANE1 and LANE2 in the two-dimensional coordinate system with the target vehicle as the origin. A data sample containing historical information, future information, LANE1, and LANE2 is constructed. The motion trajectories of all vehicles are sampled at intervals of K seconds, and a data set is constructed based on the data samples.

[0043] S2: Dataset division.

[0044] The preprocessed data set is divided into a training set for training the model, a validation set for verifying the training results of the model, and a test set for testing the trained model. The data volume ratio is 7:1:2 for training set: validation set: test set. Make full use of existing data to ensure the effectiveness of model training and the accuracy of model output.

[0045] S3: Trajectory prediction model. Predict the initial endpoint, correct the initial endpoint based on the lane line, generate candidate endpoints based on the corrected initial endpoint, and generate multi-modal trajectories based on the candidate endpoints.

[0046] S31: Build a model to estimate the initial endpoint.

[0047] For the target vehicle, for example, the historical information X at the past T time points, X = X t-T+1 ,X t-T+2 ,...,X t , t is the current time, T is the past time. The feature X at the i-th time i , X i =x 1 ,x 2 ,...,x n ,n represents the number of features.

[0048] The feature X of each vehicle at each moment i The hidden state H is obtained by sending it to the encoder of the fully connected layer (FC) and the long-short term memory network (LSTM), respectively. H = h t-T+1 ,h t-T+2 ,...,h t, the hidden state H is sent to the decoder and FC layer of LSTM, and the future trajectory of the last moment is output.

[0049] S32: Generate candidate endpoints.

[0050] like Figure 2 As shown, line 1 is the historical trajectory of the target vehicle, line 2 is the real future trajectory of the target vehicle, and dots A, B, C, D, and E are the last moment position, current moment position, predicted initial position, position after lane line correction, and sampled candidate position of the target vehicle, respectively.

[0051] Given the predicted initial end point position Y = (x p ,y p ). Based on the observation that the driving trajectory of a general vehicle is along the lane line, the lane line can be used to correct the predicted initial end point. The lane lines on the left and right sides of the current target vehicle are used to constrain the initial end point position so that the predicted trajectory is within a reasonable range.

[0052] Assuming that the left and right lanes of the target vehicle at the current moment are LANE1 and LANE2, then LANE1_y=f(x p ) and LANE2_y=f(x p ).

[0053] The coordinate of the position after lane line correction is Y fixed =(x,(LANE1_y+LANE1_y+y p ) / 3).

[0054] Taking the corrected position of the lane line as the center, 16 candidate endpoints are selected using a 4x4 grid with a spacing of 2 meters, denoted as Y n , where n∈{1,2,...,16}.

[0055] For these candidate endpoints Y n If the distance between the candidate endpoint and the real endpoint is less than the set threshold, it is classified as a positive sample, otherwise it is classified as a negative sample. The positive and negative samples include the candidate endpoint Y n And two elements of positive and negative sample labels.

[0056] In order to generate a reasonable multi-modal trajectory, the corrected position at the last moment and the projection points of the position on the left and right lane lines are sampled as multiple points as candidate endpoints, and then a multi-modal trajectory is generated based on the candidate endpoints.

[0057] S33: Generate multimodal trajectories based on candidate endpoints.

[0058] The positive samples and negative samples are concatenated with the hidden state H output by the LSTM encoder in step S31, and recorded as H_cat.

[0059] Copy H_cat to M copies and send them to lstm decoder for classification FC and regression FC. The classification branch determines whether the sample is a positive sample or a negative sample, and the regression branch predicts the M future positions of the target vehicle. Among them, the true value label of the classification branch is the positive sample or negative sample determined in step S32, and the true value label of the regression branch is the M future positions of each sample. Based on the candidate endpoints, a multimodal trajectory is generated.

[0060] S4: Model training.

[0061] The models of step S31 and step S33 are trained in stages. The loss functions used in step S31 are all L2 loss functions. The classification branch in step S33 uses the cross entropy loss function, and the regression branch uses the L2 loss function.

[0062] S5: The data in the test set is input into the first model to generate an initial endpoint, several candidate endpoints are generated based on the initial endpoint and the lane line, several candidate trajectories are generated based on the candidate endpoints and the second model, and the predicted trajectory is output based on the candidate trajectories and the top-k algorithm.

[0063] Save the trained model, input the data in the test set into the trained network model, and predict the possible trajectory coordinates of the vehicle in the future. Use the network in step S31 to predict the initial destination. Select multiple candidate destinations according to the strategy in step S32. Use the network in step S33 to predict multiple trajectories, and select the L trajectories with the highest classification scores as the multimodal output of the model.

[0064] By using lane lines to constrain the output predicted endpoint, the predicted trajectory is safer and more reasonable. The algorithm uses lightweight network LSTM and FC layers, which can be easily ported to vehicle chips.

[0065] This application aims at vehicle trajectory prediction, but this method is not only applicable to predicting vehicle trajectory, but also to scenarios such as autonomous driving and unmanned delivery. The vehicle is equipped with a camera to capture surrounding vehicles, and the positions of vehicles within a certain range of lateral and longitudinal distances are mapped to the reference system of this vehicle, generating the trajectory history of this vehicle and surrounding vehicles. According to the method of this application, the vehicle trajectory of this vehicle and surrounding vehicles is predicted. In order to utilize the interaction information between different surrounding vehicles and the target vehicle, a social tensor is introduced to encode several most important neighbor features. Based on safe driving routes and distance perception, the trajectory model parameters are optimized, and the predicted trajectory is corrected to ensure the coordination between this vehicle and surrounding vehicles.

Claims

1. A multi-stage and multi-modal vehicle trajectory prediction method, characterized in that: The following steps are involved: S1: Collect vehicle data, construct historical information and future information based on the vehicle data, construct data samples based on the historical information, future information and lane lines, and construct a data set based on the data samples; S2: Divide the dataset into training set, validation set and test set; S3: Establish a first model to predict an initial endpoint, correct the initial endpoint based on the lane line, generate a candidate endpoint based on the corrected initial endpoint, and establish a second model based on the candidate endpoint to generate a multi-modal trajectory; The predicted initial end point position (x p ,y p ), the expressions of the left and right lane lines of the target vehicle at the current moment are LANE1 and LANE2 respectively, and the position coordinates after lane line correction are (x p ,(LANE1_y+LANE2_y+y p ) / 3); Centered on the corrected position using the lane line, 16 candidate endpoints Y are selected using a 4x4 grid with a spacing of 2 meters. n , for Y n Classification is performed. If the distance between the sample and the true endpoint is less than the set threshold, it is classified as a positive sample, otherwise it is classified as a negative sample. S4: training the first model and the second model; S5: The data in the test set is input into the first model to generate an initial endpoint, several candidate endpoints are generated based on the initial endpoint and the lane line, several candidate trajectories are generated based on the candidate endpoints and the second model, and the predicted trajectory is output based on the candidate trajectories and the top-k algorithm.

2. A multi-stage and multi-modal vehicle trajectory prediction method according to claim 1, characterized in that: The vehicle data includes geographic coordinates, speed and acceleration, and the geographic coordinates are expressed as lateral coordinates and longitudinal coordinates of a two-dimensional coordinate system.

3. The multi-stage and multi-modal vehicle trajectory prediction method according to claim 1, characterized in that: The lane lines are the left and right lane lines of the target vehicle.

4. The multi-stage and multi-modal vehicle trajectory prediction method according to claim 1, characterized in that: The data volume ratio of the training set, validation set and test set is 7:1:

2.

5. The multi-stage and multi-modal vehicle trajectory prediction method according to claim 1, characterized in that: The prediction method of the initial endpoint is: sending the features of the vehicle at each moment to the encoder of the full connection layer and the long short-term memory network to obtain the hidden state, sending the hidden state to the decoder and the full connection layer of the long short-term memory network, and outputting the trajectory of the last moment.

6. The multi-stage and multi-modal vehicle trajectory prediction method according to claim 1, characterized in that: The candidate endpoints are classified, and the classification method is: if the distance between the candidate endpoint and the true endpoint is less than a set threshold, it is a positive sample, otherwise, it is a negative sample.

7. A multi-stage and multi-modal vehicle trajectory prediction method according to claim 6, characterized in that: The positive samples and negative samples are concatenated with the hidden state and sent to the decoder of the long short-term memory network, and classified and regressed in the fully connected layer. The true value label of the classification is the positive sample or the negative sample, and the true value label of the regression is the future position of the sample.

8. The multi-stage and multi-modal vehicle trajectory prediction method according to claim 1, characterized in that: The function used to predict the initial endpoint is the L2 loss function, the function used for classification is the cross entropy loss function, and the function used for regression is the L2 loss function.

9. The multi-stage and multi-modal vehicle trajectory prediction method according to claim 7, characterized in that: The trajectories with the highest scores of the classification are used as multimodal trajectories.

Images