A method and system for predicting object trajectory

Abstract

The present invention proposes a method and system for predicting the trajectory of an object, the method comprising: obtaining historical trajectory information and surrounding environment information of a target object to be predicted; encoding and fusing the historical trajectory information and surrounding environment information of the target object to obtain a first fusion feature; inputting the first fusion feature into a trained trajectory prediction model for trajectory prediction: setting a prediction time window, the prediction time window includes a first period in the future that is relatively short term and a second period in the future that is relatively long term, first predicting the coordinates of the target point predicted in the first period based on the first fusion feature, then fusing the coordinates of the target point predicted in the first period with the first fusion feature to obtain a second fusion feature, and then predicting the coordinates of the target point predicted in the second period based on the second fusion feature with the coordinates of the target point predicted in the first period as a condition. The present invention can ensure the prediction accuracy of each stage in the project in stages by predicting step by step.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method and system for predicting the trajectory of an object. Background Art

[0002] In the field of autonomous driving, how to understand the behavior of the targets around the autonomous driving has become an important part of the autonomous driving system. The trajectory prediction task aims to predict the future driving trajectory of the target (such as pedestrians, vehicles and other traffic participants) based on the current or historical trajectory and environmental information of the target. The trajectory prediction result is one of the important information for the autonomous driving system to make advance decisions. In the trajectory prediction of autonomous driving, the future trajectory of traffic participants is inherently uncertain, and is affected by multiple factors such as sensor measurement noise, environmental changes, and unknown intentions of other traffic participants, which makes the prediction of the intended trajectory of the target object prone to deviation. Short-term trajectory prediction (such as within a few seconds) usually has higher accuracy, because in a short period of time, the vehicle state and trajectory will not change drastically, and the trajectory prediction error obtained by the kinematic model is small and the accuracy is high. This is because in the short term, the behavior and intention of the vehicle are easier to predict and less affected by external interference.

[0003] However, the accuracy of medium- and long-term predictions usually decreases as the prediction time increases. This is because small errors in the initial prediction may be amplified over time, resulting in significant deviations from the true trajectory. The challenges faced by long-term predictions include dealing with longer dynamic changes, more uncertainties, and more complex interaction patterns. But for autonomous vehicles, predictions that are too short often cannot meet the requirements of the project, especially considering that autonomous vehicles must be forward-looking and intelligent. This requires the vehicle to be able to not only do a good job of short-term intention prediction, but also long-term trajectory prediction.

[0004] In summary, the existing trajectory prediction has the following difficulties: 1) The target intention is difficult to predict, such as sudden acceleration and deceleration without warning. 2) Since the future behavior pattern of the target object has a certain degree of uncertainty, the trajectory prediction accuracy of the target object is different for different time windows. For example, the prediction of a short time window of 1 to 3 seconds is often more accurate than the prediction of a long time window of 5 to 7 seconds. 3) For autonomous driving, planning decisions are often based on the speed of the vehicle, the speed of surrounding targets, and the surrounding traffic conditions. The target objects are analyzed according to different time windows to make planning decisions. 4) The current mainstream trajectory prediction algorithms often predict the target object based on a fixed time window, which will result in the predicted trajectory not having a certain degree of adaptability. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for predicting the trajectory of an object to solve the above-mentioned problem. According to the short-term prediction results, a conditional probability model based on the results is established. Through a step-by-step prediction method, the loss function is established in sections, and the model is trained through different combination coefficients, so that the prediction accuracy of each stage can be guaranteed in sections in the project.

[0006] The present invention proposes a method for predicting the trajectory of an object, comprising the following steps:

[0007] Obtain the historical trajectory information and surrounding environment information of the target object to be predicted;

[0008] Encode and fuse the historical trajectory information and surrounding environment information of the target object to obtain the first fusion feature;

[0009] The first fusion feature is input into the trained trajectory prediction model for trajectory prediction: a prediction time window is set, and the prediction time window includes a first period in the short term and a second period in the long term. The coordinates of the target point predicted in the first period are first predicted based on the first fusion feature, and then the coordinates of the target point predicted in the first period are fused with the first fusion feature to obtain the second fusion feature. Then, the coordinates of the target point predicted in the second period are predicted based on the second fusion feature, which is conditioned by the coordinates of the target point predicted in the first period.

[0010] In one embodiment, the predicting of the target point coordinates for the first period based on the first fusion feature specifically includes:

[0011] Generate a first-period candidate trajectory target point list and a first-period trajectory target point coordinate list of the target object based on the surrounding environment information of the target object;

[0012] A probability list of candidate trajectory target points of the target object in the first period is predicted based on the first fusion feature and the candidate trajectory target point list in the first period;

[0013] According to the probability list of the target object's first-period candidate trajectory target points, k first-period candidate trajectory target points with the highest probability are selected and their coordinates are obtained as the first-period target point coordinates;

[0014] Combined with the target point coordinate list of the first period trajectory of the target object, the offset of the target point coordinate of the first period is corrected to obtain the first period predicted target point coordinate of the target object.

[0015] In one embodiment, the step of predicting the target point coordinates predicted in the second period based on the second fusion feature and taking the target point coordinates predicted in the first period as a condition specifically includes:

[0016] Generate a second-period candidate trajectory target point list of the target object based on the surrounding environment information of the target object;

[0017] Generate a second period trajectory target point coordinate list based on the second fusion feature, which is conditioned on the first period candidate trajectory target point list;

[0018] A second-period trajectory target point probability list of the target object is predicted based on the second fusion feature and the second-period candidate trajectory target point list;

[0019] According to the probability list of the target points of the second-period trajectory of the target object, K target points of the second-period trajectory with the highest probability are selected as the predicted target points of the second-period trajectory of the target object;

[0020] The second period predicted target point of the target object is combined with the second period trajectory target point coordinate list to obtain the second period predicted target point coordinates of the target object.

[0021] In one embodiment, the specific process of training the trajectory prediction model includes:

[0022] Setting a first period and a second period of prediction, and using the trajectory prediction model to perform trajectory prediction on the target object;

[0023] Obtain the true value of the trajectory of the target object in the first period and the true value of the trajectory of the target object in the second period;

[0024] Calculate the one-hot vector of the true value of the target point probability in the first period according to the true value of the target's trajectory in the first period and the list of target points of the candidate trajectory in the first period;

[0025] The first loss function Loss1 is constructed by the cross entropy of the one-hot vector of the true value of the target point probability in the first period and the probability list of the candidate trajectory target points in the first period;

[0026] According to the true value of the first period trajectory of the target object, the true value of the coordinates of the target point of the first period trajectory of the target object is obtained;

[0027] The second loss function Loss2 is constructed based on the true value of the trajectory target point coordinates in the first period and the predicted target point coordinates in the first period;

[0028] Calculate the one-hot vector of the true value of the target point probability in the second period according to the true value of the target's second period trajectory and the list of target points of the second period candidate trajectory;

[0029] The third loss function Loss3 is constructed by the cross entropy of the one-hot vector of the true value of the target point probability in the second period and the probability list of the target point of the trajectory in the second period;

[0030] According to the real value of the second period trajectory of the target object, the real value of the coordinates of the target point of the second period trajectory of the target object is obtained;

[0031] The fourth loss function Loss4 is constructed based on the true value of the target point coordinates of the trajectory in the second period and the predicted target point coordinates in the second period;

[0032] Construct a combined loss function Loss based on the first loss function Loss1, the second loss function Loss2, the third loss function Loss3 and the fourth loss function Loss4;

[0033] The parameters of the trajectory prediction model are optimized through back propagation according to the combined loss function Loss.

[0034] In one embodiment, the formula of the combined loss function Loss is: Loss=λ1Loss1+λ2Loss2+λ3Loss3+λ4Loss4;

[0035] Among them, λ1, λ2, λ3, and λ4 are respectively the fine-tuning parameters that need to be set for the trajectory prediction model.

[0036] In one embodiment, the trajectory prediction model is a feed-forward neural network model.

[0037] In one embodiment, the feature fusion of the target point coordinates of the predicted trajectory in the first period with the first fusion feature to obtain the second fusion feature specifically includes: feature concatenation, numerical addition or data subtraction of the target point coordinates of the predicted trajectory in the first period with the first fusion feature.

[0038] In one embodiment, when the target object to be predicted is a moving vehicle, the surrounding environment includes surrounding roads and surrounding objects, and the surrounding objects include surrounding pedestrians and surrounding vehicles.

[0039] The present invention also proposes a system for predicting object trajectories, which is used to implement the method for predicting object trajectories as described above, comprising:

[0040] An information collection module is used to collect historical trajectory information and surrounding environment information of the target object to be predicted;

[0041] An information fusion module is used to encode and fuse the historical trajectory information and surrounding environment information of the target object to obtain a first fusion feature;

[0042] A trajectory prediction module, used to predict the coordinates of a target point in a first period of the target object based on the first fusion feature, and to predict the coordinates of the target point in a second period of the target object based on the coordinates of the target point in the first period of the target object;

[0043] The output module is used to output the first period predicted target point coordinates and / or the second period predicted target point coordinates of the target object.

[0044] In one embodiment, the trajectory prediction module includes:

[0045] A first-period target point probability prediction unit, used to predict a first-period candidate trajectory target point probability list of the target object according to the first fusion feature and the first-period candidate trajectory target point list generated based on the target object's surrounding environment information;

[0046] A first-period target point position prediction unit, configured to predict the first-period predicted target point coordinates of the target object according to the first-period candidate trajectory target point probability list of the target object and the first-period trajectory target point coordinate list generated based on the surrounding environment information of the target object;

[0047] A feature fusion unit, used for fusing the first period predicted target point coordinates of the target object with the first fusion feature to obtain a second fusion feature;

[0048] A second-period target point probability prediction unit, used to predict a second-period trajectory target point probability list of the target object according to the second fusion feature and the second-period candidate trajectory target point list generated based on the target object's surrounding environment information;

[0049] A second-period target point coordinate prediction unit, used to generate a second-period trajectory target point coordinate list based on the first-period candidate trajectory target point list according to the second fusion feature;

[0050] The second period target point position prediction unit is used to predict the second period predicted target point coordinates of the target object according to the second period trajectory target point probability list and the second period trajectory target point coordinate list of the target object.

[0051] Compared with the prior art, the method and system for predicting object trajectories of the present invention have the following beneficial effects:

[0052] 1) The present invention combines historical trajectories with the model to predict the predicted behavior probability of the trajectory in the short term in the future, and uses this as the conditional input of the next level time period to obtain the trajectory output of a longer time period, that is, based on the conditional probability, the behavior trajectory of the future time period is segmented and step-by-step predicted, and the prediction results of different stages are put into the long-term prediction, so that the model predicts the trajectory more accurately.

[0053] 2) The present invention sets a combined loss function for predictions in different periods and adjusts the weight coefficient of the loss function in each period to fine-tune the overall model, thereby improving the performance of the trajectory prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of a flow chart of a method for predicting an object trajectory according to an embodiment of the present invention;

[0055] Figure 2A schematic diagram of a specific process of performing trajectory prediction using a trajectory prediction model in a method for predicting an object trajectory according to an embodiment of the present invention;

[0056] Figure 3 This is a diagram of the operational steps of a method for predicting an object trajectory according to an embodiment of the present invention for predicting the trajectory of an autonomous driving vehicle. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments of the specification. It should be noted that many specific details are set forth in the following description to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0058] Secondly, "one embodiment" or "an embodiment" in this application refers to a specific feature, structure or characteristic that can be included in at least one implementation of the present invention. The "in one embodiment" and "an embodiment" that appear in different places in this specification do not all refer to the same embodiment, nor are they separate or selective embodiments that are mutually exclusive with other embodiments. The terms "including" and "comprising" indicate the presence of the claimed features, but do not exclude the presence of one or more other features. The term "and / or" used in this application includes any and all combinations of one or more of the relevant listed items.

[0059] The present invention proposes a method for predicting the trajectory of an object, see Figure 1 , including the following steps:

[0060] Step S1, obtaining the historical trajectory information and surrounding environment information of the target object to be predicted. When the target object to be predicted is a moving vehicle, the surrounding environment includes surrounding roads and surrounding objects, and the surrounding objects include surrounding pedestrians and surrounding vehicles. When obtaining the historical trajectory information of the target object, the historical trajectory time window T0 of the target object must be set first. For example, T0 can be 20 frames. Of course, the time window can be adjusted according to actual needs.

[0061] Step S2: Encode and fuse the historical trajectory information and surrounding environment information of the target object to obtain a first fused feature, which is a high-dimensional feature vector that fuses the global environment information and historical trajectory information of the target object. Encoding refers to converting data information into a feature vector that can be input into a trajectory prediction model, and the feature vector contains the information required by the trajectory prediction model for trajectory prediction.

[0062] Step S3: Input the first fusion feature into the trained trajectory prediction model to perform trajectory prediction. For specific steps, see Figure 2 ,as follows:

[0063] Step S31, adjusting and setting the prediction time window according to needs and experiments, wherein the prediction time window includes a first period in the short term in the future (that is, the short term future trajectory time window of the target object, hereinafter referred to as short term T1) and a second period in the long term in the future (that is, the long term future trajectory time window of the target object, hereinafter referred to as long term T2).

[0064] Then make predictions according to the time window segment:

[0065] Step S32: obtaining the coordinates of the predicted target point in the first period based on the first fusion feature prediction.

[0066] Step S33: Fusing the coordinates of the target point predicted in the first period with the first fusion feature to obtain a second fusion feature.

[0067] Step S34: obtaining the predicted target point coordinates for the second period based on the predicted target point coordinates for the first period based on the second fusion feature prediction.

[0068] Both the first period and the second period can be adjusted according to actual needs. When performing trajectory prediction for multiple targets, similar to the trajectory prediction for a single target, the historical trajectory information of each target in the target scene and its surrounding environment information are obtained, the interaction characteristics between the targets are determined, and then short-term (T1) trajectory prediction and long-term (T2) trajectory prediction are performed step by step. In autonomous driving, short-term T1 is generally a time window of 1 to 3 seconds, and the T1 value of one embodiment of the present invention is 3 seconds; long-term T2 is generally a time window of 5 to 7 seconds, and the T2 value of one embodiment of the present invention is 7 seconds.

[0069] The step S32, predicting the target point coordinates for the first period based on the first fusion feature, specifically includes:

[0070] Generate a first-period candidate trajectory target point list and a first-period trajectory target point coordinate list of the target object based on the surrounding environment information of the target object;

[0071] A probability list of candidate trajectory target points of the target object in the first period is predicted based on the first fusion feature and the candidate trajectory target point list in the first period;

[0072] According to the probability list of the target object's first-period candidate trajectory target points, k first-period candidate trajectory target points with the highest probability are selected and their coordinates are obtained as the first-period target point coordinates;

[0073] Combined with the target point coordinate list of the first period trajectory of the target object, the offset of the target point coordinate of the first period is corrected to obtain the first period predicted target point coordinate of the target object.

[0074] The step S33, fusing the coordinates of the target point of the predicted trajectory in the first period with the first fusion feature to obtain the second fusion feature, specifically includes: performing feature concatenation, numerical addition or data subtraction on the coordinates of the target point of the predicted trajectory in the first period and the first fusion feature.

[0075] The step S34, based on the second fusion feature prediction, obtains the predicted target point coordinates for the second period based on the predicted target point coordinates for the first period, specifically including:

[0076] Generate a second-period candidate trajectory target point list of the target object based on the surrounding environment information of the target object;

[0077] Generate a second period trajectory target point coordinate list based on the second fusion feature, which is conditioned on the first period candidate trajectory target point list;

[0078] A second-period trajectory target point probability list of the target object is predicted based on the second fusion feature and the second-period candidate trajectory target point list;

[0079] According to the probability list of the target points of the second-period trajectory of the target object, K target points of the second-period trajectory with the highest probability are selected as the predicted target points of the second-period trajectory of the target object;

[0080] The second period predicted target point of the target object is combined with the second period trajectory target point coordinate list to obtain the second period predicted target point coordinates of the target object.

[0081] The specific process of training the trajectory prediction model in one embodiment of the present invention includes:

[0082] Setting a first period (short-term T1 value) and a second period (long-term T2 value) of prediction, and using the trajectory prediction model to predict the trajectory of the target object;

[0083] Obtain the true value of the trajectory of the target object in the first period and the true value of the trajectory of the target object in the second period;

[0084] Calculate the one-hot vector of the true value of the target point probability in the first period according to the true value of the target's trajectory in the first period and the list of target points of the candidate trajectory in the first period;

[0085] The first loss function Loss1 is constructed by the cross entropy of the one-hot vector of the true value of the target point probability in the first period and the probability list of the candidate trajectory target points in the first period;

[0086] According to the true value of the first period trajectory of the target object, the true value of the coordinates of the target point of the first period trajectory of the target object is obtained;

[0087] The second loss function Loss2 is constructed based on the true value of the trajectory target point coordinates in the first period and the predicted target point coordinates in the first period;

[0088] Calculate the one-hot vector of the true value of the target point probability in the second period according to the true value of the target's second period trajectory and the list of target points of the second period candidate trajectory;

[0089] The third loss function Loss3 is constructed by the cross entropy of the one-hot vector of the true value of the target point probability in the second period and the probability list of the target point of the trajectory in the second period;

[0090] According to the real value of the second period trajectory of the target object, the real value of the coordinates of the target point of the second period trajectory of the target object is obtained;

[0091] The fourth loss function Loss4 is constructed based on the true value of the target point coordinates of the trajectory in the second period and the predicted target point coordinates in the second period;

[0092] Construct a combined loss function Loss based on the first loss function Loss1, the second loss function Loss2, the third loss function Loss3 and the fourth loss function Loss4;

[0093] The parameters of the trajectory prediction model are optimized through back propagation according to the combined loss function Loss.

[0094] This training method can be used to fine-tune the entire prediction model by adjusting the weights of the loss functions of different periods. At the same time, in actual engineering, by analyzing the overall performance of the model, you can set the coefficients in the combined loss function to zero and perform separate training for a certain period (such as short-term or long-term).

[0095] The formula of the combined loss function Loss of an embodiment of the present invention is: Loss=λ1Loss1+λ2Loss2+λ3Loss3+λ4Loss4;

[0096] Among them, λ1, λ2, λ3, and λ4 are respectively the fine-tuning parameters that need to be set for the trajectory prediction model.

[0097] The trajectory prediction model of an embodiment of the present invention is a feedforward neural network model, such as MLP, etc., but the present invention can also apply other types of neural network models, and this application does not impose any limitation on the type of model.

[0098] The present invention also proposes a system for predicting the trajectory of an object, which is used to implement the method for predicting the trajectory of an object as described above, and includes an information acquisition module, an information fusion module Model_0, a trajectory prediction module and an output module. The information acquisition module is used to collect the historical trajectory information and surrounding environment information of the target object to be predicted. The information fusion module Model_0 is used to encode and fuse the historical trajectory information and surrounding environment information of the target object to obtain a first fusion feature. The trajectory prediction module is used to predict the first period predicted target point coordinates of the target object based on the first fusion feature, and predict the second period predicted target point coordinates of the target object based on the first period predicted target point coordinates. The output module is used to output the first period predicted target point coordinates and / or the second period predicted target point coordinates of the target object.

[0099] Specifically, the trajectory prediction module includes a first-period target point probability prediction unit Model_1, a first-period target point position prediction unit Model_2, a feature fusion unit, a second-period target point probability prediction unit Model_3, a second-period target point coordinate prediction unit Model_4, and a second-period target point position prediction unit. The first-period target point probability prediction unit Model_1 is used to predict the first-period candidate trajectory target point probability list of the target object according to the first fusion feature and the first-period candidate trajectory target point list generated based on the target object's surrounding environment information. The first-period target point position prediction unit Model_2 is used to predict the first-period predicted target point coordinates of the target object according to the first-period candidate trajectory target point probability list of the target object, combined with the first-period trajectory target point coordinate list generated based on the target object's surrounding environment information. The feature fusion unit is used to perform feature fusion on the first-period predicted target point coordinates of the target object with the first fusion feature to obtain the second fusion feature. The second-period target point probability prediction unit Model_3 is used to predict the second-period trajectory target point probability list of the target object according to the second fusion feature and the second-period candidate trajectory target point list generated based on the target object's surrounding environment information. The second-period target point coordinate prediction unit Model_4 is used to generate a second-period trajectory target point coordinate list based on the first-period candidate trajectory target point list according to the second fusion feature. The second-period target point position prediction unit is used to predict the second-period predicted target point coordinates of the target object according to the second-period trajectory target point probability list and the second-period trajectory target point coordinate list.

[0100] The following takes the above-mentioned system and method for predicting the trajectory of an object as an example to predict the trajectory of a moving vehicle, and elaborates on the above-mentioned prediction method, model training method, etc. The operation steps are as follows, see Figure 3 :

[0101] Step 1: Extract the state information of the target object (target vehicle) to be predicted.

[0102] Obtain the state information of the target object to be predicted. Including the vehicle's location (x, y, z), speed (v x , v y , v z ), acceleration information (a x , a y , a z ). According to the state information of the target object, the corresponding trajectory prediction model can be selected through appropriate rules, such as selecting the corresponding speed interval to refine the prediction model under different speed interval states.

[0103] Step 2: Extract the historical (T0 takes 20 frames) trajectory information of the target vehicle.

[0104] Receive information from the perception fusion module provided by the autonomous vehicle and obtain the historical trajectory information of the predicted target. Assuming that the timing starts from the occurrence of the trajectory, it is agreed that this time is t19, and the 20 frames before this can be recorded as t0, t1, t2...t19, then write the historical trajectory points from t0 to t19 into the array container, recorded as {p0, p1, p2, p3,...p 19}.

[0105] Step 3: Obtain information about surrounding objects of the target vehicle.

[0106] Based on the location information around the target vehicle and combined with the perception signal, screening is performed within a certain range to obtain information about other objects around the target vehicle, including a list of information about other vehicles and pedestrians.

[0107] Step 4: Sample the target point coordinates of the target vehicle's short-term (T1) trajectory.

[0108] Similar to the process of extracting the historical (T0) trajectory information of the target vehicle in step 2, a list of target point coordinates of the short-term (T1) trajectory of the target vehicle can be obtained through certain rule screening.

[0109] Step 5: Extract the short-term (T1) true value information of the target vehicle.

[0110] Through the recorded data, the historical information (T0) and short-term trajectory (T1) information of the target vehicle can be obtained from the information of each time series of the perceived target vehicle by dividing it according to the time window. The actual value of the short-term (T1) trajectory of the target vehicle is obtained, which is recorded as {Point1, Point2, Point3...Point t1}.

[0111] Step 6: Extract the true value information of the long-term (T2) trajectory of the target vehicle.

[0112] Similar to the process in step 5, the long-term (T2) trajectory information of the target object can be obtained by dividing the information of each time series of the perceived target vehicle according to the time window through the recorded data. The true value of the long-term (T2) trajectory of the target vehicle is obtained, denoted as {Point t1 ,Point t1+1 ,Point t1+2 ……Point t2}.

[0113] Step 7: Extract the surrounding road information of the target vehicle.

[0114] 7.1 According to the coordinates of the positioning point, send it to the high-precision map to obtain the lane line number information around the high-precision map.

[0115] 7.2 Use the surrounding lane line numbers and the high-precision map API to obtain information about points on the lane line.

[0116] 7.3 Obtain the trajectory information of other surrounding vehicles through the perception fusion module.

[0117] Step 8: Encode the historical (T0) trajectory of the target vehicle.

[0118] The historical trajectory of the target vehicle is encoded, and the encoding format of each moment can be in the form of a one-dimensional array. An embodiment of the present invention is in the form of: (x, y, Δx, Δy, step, 0, 0, 0, 0, vector_id), where x and y represent the coordinates of the trajectory, Δx, Δy represent the coordinate vector of the trajectory, which is equivalent to subtracting the coordinate difference of the previous moment from the current moment, and is expressed by the formula Vector_t = (Δx, Δy)t = Point t -Point t-1 , vector_id represents the number of the vehicle trajectory to which the current trajectory encoding vector belongs. Then there are 20 moments from step 0 to step 19, which are represented by a coding matrix of 20 times 10. The above is an encoding example described in the present invention, and the present invention does not impose any limitation on the specific encoding format of the target vehicle historical trajectory.

[0119] Step 9: Encode the surrounding environment of the target vehicle.

[0120] 9.1 According to the lane line number obtained in step 7.1, obtain the information of each small lane line segment on the lane line through the map interface.

[0121] 9.2 Encode the information of the small lane line segment obtained in step 9.1. The encoding format of an embodiment of the present invention is (x, y, Δx, Δy, traffic_ctrl, is_turns, is_intersect, vector_id), wherein x and y are the coordinates of the center point of the line segment, Δx and Δy are the normalized vector directions of the line segment based on the center point, traffic_ctrl is the traffic control signal, such as 0 indicates traffic control (red light), 1 indicates no traffic control (green light), is_turns indicates turning information, such as (1,0) indicates a left turn lane, (0,1) indicates a right turn lane, (0,0) indicates a straight lane, is_intersect indicates intersection information, 0 indicates a non-intersection, 1 indicates an intersection, and vector_id indicates the number of the lane line to which the current lane encoding vector belongs. The above is an example of vehicle surrounding environment encoding described in the present invention, and the present invention does not impose any limitation on the specific encoding format.

[0122] Step 10: Generate a short-term (T1) candidate trajectory target point list for the target vehicle.

[0123] According to the surrounding environment information obtained in step 3, a short-term (T1) candidate trajectory target point list of the target vehicle is generated through a certain rule method. The present invention does not limit the method of the target point.

[0124] Step 11: Generate a list of long-term (T2) candidate trajectory target points for the target vehicle.

[0125] According to the surrounding environment information obtained in step 3, a long-term (T2) candidate trajectory target point list of the target vehicle is generated through a certain rule method. The present invention does not limit the method of the target point.

[0126] Step 12: Generate a feature vector that integrates the global environment information and the historical trajectory information of the target vehicle.

[0127] Through the information fusion module (the information fusion module of one embodiment of the present invention is a neural network model Model_0), the module takes steps 8 and 9 as input, performs feature fusion through the model, and generates a high-dimensional feature vector including global environmental information and target vehicle trajectory information, that is, the first fusion feature. The present invention does not limit the type of the neural model. If the dimension of the high-dimensional feature vector is set to d, the feature vector calculated by Model_0 can be represented by a two-dimensional tensor (b, d), where b is the batch size of the model training, and the value of d depends on the model size and the richness of the feature representation requirements, and can generally be 256, 512 or 1024. To take a sample of a high-dimensional feature vector, γ={γ1,γ2,γ3…γ d} represents the high-dimensional vector.

[0128] Step 13: Calculate the short-term candidate trajectory target point probability list of the target vehicle.

[0129] Through the first period target point probability prediction unit (the first period target point probability prediction unit of one embodiment of the present invention is a neural network model Model_1), the model takes the high-dimensional feature vector obtained in step 12 and the short-term (T1) candidate trajectory target point list of the target vehicle obtained in step 10 as input, through the neural network model, one embodiment of which is MLP, and through the softmax activation function, the short-term candidate trajectory target point probability list of the target vehicle is calculated: The present invention does not limit the model type of Model_1.

[0130] Step 14: Construct a one-hot vector of the true value of the short-term (T1) target point probability.

[0131] The target vehicle short-term (T1) trajectory true value information extracted in step 5 is compared with the target vehicle short-term (T1) candidate trajectory target point list obtained in step 10, and the Euclidean distance between each candidate trajectory target point and the true value of the trajectory in the T1 period is calculated respectively. The shortest distance is selected as the true candidate point, and the one-hot vector P is constructed based on this. t1 (x i ), (0, 0, 0, ... 1, 0, 0, 0), where 1 is the number of the real coordinate point in the candidate table, and the coordinate difference (Δx, Δy) between the real candidate point and the real point of the short-term (T1) future trajectory is calculated.

[0132] Step 15: Construct the first loss function Loss1.

[0133] Cross entropy is used to construct the classification loss function for target candidate point judgment, that is, the first loss function Loss1, as shown in the formula: in, is the probability list obtained in step 13, p(x i ) is the true value one-hot vector obtained in step 14. This loss function can optimize the model parameters in Model_1 through back propagation.

[0134] Step 16: Select the high probability short-term (T1) target point coordinates.

[0135] This step illustrates a method for selecting target candidate points based on segmented likelihood probability. According to the target vehicle short-term candidate trajectory target point probability list {p1, p2, p3, p4...} calculated in step 13, and according to the corresponding short-term trajectory target point coordinate list obtained in step 4, the k target points with the highest probability are selected to obtain the coordinate list of these points. This patent does not limit the specific value of k. In this embodiment, the value of k is 50, and the coordinates of the target point with the highest probability are marked as

[0136] Steps 17 to 23 describe a method for constructing a probability list of candidate points for long-term target trajectory prediction by using short-term predicted candidate target points as conditional probabilities. The specific steps are as follows:

[0137] Step 17: Encode the target point of the short-term trajectory of the target vehicle.

[0138] Encode the short-term target points sampled in step 4 so that the data format matches the model input format. For example, one encoding example is to encode the sampled short-term target points into a three-dimensional matrix tensor of (b, n, 2), where b is the batch size of the model training, n is the number of short-term sampling points, and 2 is the two coordinate dimensions of x and y. The present invention does not specifically limit the specific form of encoding.

[0139] Step 18: Calculate and output the short-term (T1) predicted target point coordinates of the target vehicle.

[0140] 18.1 Send the results of step 16 and step 17 to the first period target point position prediction unit (in this example, the first period target point position prediction unit is the deep learning model Model_2), calculate the offset of the predicted target point, and according to the encoding format given in step 17, the model calculates a three-dimensional matrix tensor of (b, n, 2), in which the last two dimensions (n, 2) are the corrected offsets of the target points predicted by the model. Denoted as

[0141] 18.2 Add the corrected offset in step 18.1 and the original coordinate value of the target point in step 13 to obtain the coordinate value of the short-term predicted target point.

[0142] The above coordinate point list can be concisely written as {p t1 ,p t2 ,p t3 …… tk ,|t1}, the target point prediction of the future trajectory at time t1 calculated by the model is recorded as

[0143] Step 19: Copy and concatenate the feature point coordinates to obtain the second fusion feature.

[0144] This step describes a method for fusing the short-term prediction target conditional probability with the feature points of the high-dimensional environment feature vector. The specific steps are as follows:

[0145] 19.1 First, combine the short-term predicted target point coordinates of step 18 with the short-term target candidate trajectory target point probability list {p1, p2, p3, p4...} obtained in step 13, and select k' points from high to low probability. The present invention does not limit the value of k'. In this example, the value of k' is 1, and the coordinate vector {x t1 ,y t1};

[0146] 19.2 The high-dimensional vector γ in step 12 is converted into {γ1,γ2,γ3...γ d} and the coordinate point list obtained in step 19.1 The vector γ′ is obtained by concatenating them to a dimension of d′=d+2k′. In this example, the value of k′ is 1, and the vector γ′={γ1,γ2,γ3…γ d ,x t1 ,y t1}.

[0147] It should be noted that the method of integrating the short-term prediction target into the subsequent long-term prediction target calculation is not limited to feature concatenation, but can also be numerical addition, subtraction, or any other operation. At the same time, this patent does not limit the value of k'.

[0148] Step 20: Construct a one-hot vector of the true value of the long-term (T2) target point probability.

[0149] The target vehicle long-term (T2) trajectory true value information extracted in step 6 is compared with the target vehicle long-term (T2) candidate trajectory target point list obtained in step 11, and the Euclidean distance between each candidate point and the true value of the trajectory in the T2 period is calculated respectively. The shortest distance is selected as the true candidate point, and the long-term (T2) target point probability true value one-hot vector P is constructed based on this. t2 (x i ), (0, 0, 0, ... 1, 0, 0, 0), where 1 is the number of the real coordinate point in the candidate table, and the coordinate difference (Δx′, Δy′) between the real candidate point and the real point of the long-term (T2) trajectory is calculated.

[0150] Step 21: Obtain the true value of the target point coordinates of the short-term (T1) trajectory.

[0151] From the short-term (T1) true value of the target vehicle in step 5 {Point1, Point2, Point3...Pointt1}Get the actual value of the short-term (T1) trajectory target point coordinate Point t1 ={x t1 ,y t1}.

[0152] Step 22: Construct the second loss function Loss2, that is, the short-term (T1) future trajectory target point displacement L2 loss function. The calculation formula of the second loss function Loss2 is:

[0153]

[0154] In the formula, (x t1 ,y t1 ) is the true value of the target point coordinate of the short-term (T1) trajectory obtained in step 21, The short-term (T1) predicted target point coordinates are calculated as the output of the model in step 18.

[0155] Step 23: Calculate a long-term (T2) trajectory target point probability list based on the short-term (T1) candidate trajectory target points.

[0156] The target vehicle long-term (T2) candidate trajectory target point generated in step 11 and the concatenated second fusion feature vector γ′={γ1,γ2,γ3...γ d ,x t1 ,y t1} is used as the input of the target point probability prediction unit in the second period (in this embodiment, the neural network model Model_3). After model inference calculation, a long-term (T2) trajectory target point probability list based on the short-term (T1) candidate trajectory target point is obtained, which is recorded as

[0157]

[0158] Step 24: Construct the third loss function Loss3.

[0159] Cross entropy is used to construct the classification loss function for target candidate point judgment, that is, the third loss function Loss3, as shown in the formula: in, is the probability list of long-term (T2) trajectory target points conditioned on the short-term (T1) candidate trajectory target points calculated in step 23, P t2 (x i ) is the one-hot vector of the true value of the long-term (T2) target point probability obtained in step 20. This loss function can be used to optimize the model parameters in Model_3 through back-propagation.

[0160] Step 25: Generate a long-term trajectory target point coordinate list based on the short-term (T1) candidate trajectory target point list.

[0161] The vector γ′ obtained in step 19 is converted into {γ1,γ2,γ3...γ d ,x t1 ,y t1} as input, the vector already contains the coordinate information of the short-term predicted target point t1, and after the inference of the second period target point coordinate prediction unit (the conditional probability model Model_4 in this embodiment), the long-term (T2) trajectory target point coordinate list based on the short-term (T1) candidate trajectory target point list is calculated, and the process is similar to step 18, and the specific operations are as follows:

[0162] 25.1 Take the vector γ′ obtained in step 19 as input, and calculate the offset of the predicted target point through Model_4. According to the encoding format given in this paper, the model calculates the three-dimensional matrix tensor (b,n,2) in the same way as step 17. The last two dimensions (n,2) are the corrected offsets of the target points predicted by the model, recorded as

[0163] 25.2 Add the corrected offset in step 25.1 to the original coordinate value of the target point to obtain the coordinate value list of the long-term trajectory target point

[0164] The above coordinate point list can be concisely written as {p t1 ,p t2 ,p t3 …… tk ,|t2}, the target point prediction of the future trajectory at time t2 calculated by the model is recorded as

[0165] Step 26: predict the long-term (T2) predicted target point coordinates of the target vehicle.

[0166] Based on the coordinate points in step 25 and the probability list obtained in step 23 By sorting the probability from high to low, the trajectory target point list obtained in step 25 Select K points from This patent does not limit the value of K. In an example in this patent, the value of K is 1, and the predicted coordinates are obtained. That is the long-term (T2) predicted target point coordinates of the target vehicle.

[0167] Step 27: Obtain the true value of the target point coordinates of the long-term (T2) trajectory.

[0168] The long-term (T2) trajectory true value information of the target vehicle obtained in step 6 is used to obtain the true value coordinates Point of the trajectory target point at time T2 t2 , denoted as {x t2 ,y t2}.

[0169] Step 28: Construct the fourth loss function Loss4.

[0170] The calculation formula of the fourth loss function Loss4 is as follows:

[0171] Where P t1 The actual value of the target point coordinate of the short-term (T1) trajectory obtained in step 21 is Point t1 , The short-term (T1) predicted target point coordinates are calculated as the output of the model in step 18.

[0172] Step 29: Construct the combined loss function Loss.

[0173] The calculation formula of the combined loss function Loss is: Loss = λ1Loss1+λ2Loss2+λ3Loss3+λ4Loss t ;

[0174] Among them, Loss1, Loss2, Loss3, and Loss4 are the step-by-step losses calculated in step 15, step 22, step 24, and step 28, respectively, and λ1, λ2, λ3, and λ4 are fine-tuning parameters set according to model requirements.

[0175] In summary, steps 17 to 23 describe a method for constructing a probability list of candidate points for long-term target trajectory prediction by using short-term predicted candidate target points as conditional probabilities. The specific features are as follows: encoding the short-term trajectory target points of the target vehicle (step 17), calculating and outputting the coordinates of the short-term (T1) trajectory predicted target points through a deep learning model (step 18), fusing the short-term predicted target conditional probability and the first fusion feature (step 19), calculating the true value one-hot vector of the long-term target point probability (step 20), obtaining the true value of the short-term (T1) trajectory target point coordinates (step 21), constructing the short-term (T1) future trajectory target point displacement L2 loss function Loss2 (step 22), and calculating the long-term (T2) trajectory target point probability list conditioned on the short-term (T1) candidate trajectory target points (step 23).

[0176] Use cross entropy to construct a classification loss function for target candidate point judgment (step 24), generate a long-term trajectory target point coordinate list based on the short-term (T1) candidate trajectory target point (step 25), obtain the target vehicle long-term (T2) candidate point coordinate prediction with the highest probability (step 26), obtain the true value coordinates of the trajectory target point at time T2 (step 27), construct a short-term displacement L2 loss function (step 28), and set a combined loss function (step 29). Steps 24 to 29 describe a method for a segmented prediction trajectory target point judgment classification combined loss function. In this combined loss function, the entire prediction model can be fine-tuned by adjusting the weights of the loss functions at different times, realizing the segmented training and step-by-step training functions of the entire model. At the same time, in actual engineering, by analyzing the overall performance of the model, a certain model (such as short-term or long-term) can be trained separately by setting the coefficients in the combined loss function to zero.

[0177] The above steps implement the training method process of the conditional probability target trajectory prediction model described in the present invention. In the actual application deployment process, the deployment can refer to the general model deployment method.

[0178] It should be noted that the term "vehicle" or other similar terms include general motor vehicles, such as passenger cars (including sports utility vehicles, buses, trucks, etc.), various commercial vehicles, and include hybrid electric vehicles, electric vehicles, plug-in hybrid electric vehicles, etc. A hybrid electric vehicle is a vehicle with two or more power sources, such as gasoline-powered and electric-powered vehicles. The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0179] The present invention has the following beneficial effects:

[0180] 1) The present invention combines historical trajectories with the model to predict the predicted behavior probability of the trajectory in the short term in the future, and uses this as the conditional input of the next level time period to obtain the trajectory output of a longer time period, that is, based on the conditional probability, the behavior trajectory of the future time period is segmented and step-by-step predicted, and the prediction results of different stages are put into the long-term prediction, so that the model predicts the trajectory more accurately.

[0181] 2) The present invention sets a combined loss function for predictions in different periods and adjusts the weight coefficient of the loss function in each period to fine-tune the overall model, thereby improving the performance of the trajectory prediction model.

[0182] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0183] The construction and arrangement of the present application shown in a plurality of different exemplary embodiments are only exemplary. Although only a few embodiments are described in detail in this disclosure, it should be readily understood by those who refer to this disclosure that many modifications are possible (e.g., size, structure, shape and proportion, as well as parameter values, mounting arrangements, use of materials, color, directional changes, etc.) without substantially departing from the novel teachings and advantages of the subject matter described in the application. For example, the element shown as integrally formed can be composed of multiple parts or elements, the position of the element can be inverted or otherwise changed, and the nature or number or position of the discrete element can be changed or changed. Therefore, all such modifications are intended to be included in the scope of the present invention. The order or sequence of any process or method steps can be changed or reordered according to alternative embodiments. In the claims, any "device plus function" clause is intended to cover the structure described herein that performs the function, and is not only structurally equivalent but also equivalent structure. Without departing from the spirit and essence of the present invention, a person skilled in the art can make various corresponding modifications and changes according to the present invention, but these corresponding modifications and changes should all fall within the scope of protection of the present invention.

Claims

1. A method for predicting the trajectory of an object, characterized in that: The steps include: Obtain the historical trajectory information and surrounding environment information of the target object to be predicted; Encode and fuse the historical trajectory information and surrounding environment information of the target object to obtain the first fusion feature; The first fusion feature is input into the trained trajectory prediction model for trajectory prediction: a prediction time window is set, and the prediction time window includes a first period in the short term and a second period in the long term. The coordinates of the target point predicted in the first period are first predicted based on the first fusion feature, and then the coordinates of the target point predicted in the first period are fused with the first fusion feature to obtain the second fusion feature. Then, the coordinates of the target point predicted in the second period are predicted based on the second fusion feature, which is conditioned by the coordinates of the target point predicted in the first period.

2. The method for predicting an object trajectory according to claim 1, characterized in that: The step of predicting the target point coordinates for the first period based on the first fusion feature prediction specifically includes: Generate a first-period candidate trajectory target point list and a first-period trajectory target point coordinate list of the target object based on the surrounding environment information of the target object; A probability list of candidate trajectory target points of the target object in the first period is predicted based on the first fusion feature and the candidate trajectory target point list in the first period; According to the probability list of the target object's first-period candidate trajectory target points, k first-period candidate trajectory target points with the highest probability are selected and their coordinates are obtained as the first-period target point coordinates; Combined with the target point coordinate list of the first period trajectory of the target object, the offset of the target point coordinate of the first period is corrected to obtain the first period predicted target point coordinate of the target object.

3. The method for predicting the trajectory of an object according to claim 2, characterized in that: The step of predicting the target point coordinates predicted in the second period based on the second fusion feature and taking the target point coordinates predicted in the first period as a condition specifically includes: Generate a second-period candidate trajectory target point list of the target object based on the surrounding environment information of the target object; Generate a second period trajectory target point coordinate list based on the second fusion feature, which is conditioned on the first period candidate trajectory target point list; A second-period trajectory target point probability list of the target object is predicted based on the second fusion feature and the second-period candidate trajectory target point list; According to the probability list of the target points of the second-period trajectory of the target object, K target points of the second-period trajectory with the highest probability are selected as the predicted target points of the second-period trajectory of the target object; The second period predicted target point of the target object is combined with the second period trajectory target point coordinate list to obtain the second period predicted target point coordinates of the target object.

4. The method for predicting the trajectory of an object according to claim 3, characterized in that: The specific process of training the trajectory prediction model includes: Setting a first period and a second period of prediction, and using the trajectory prediction model to perform trajectory prediction on the target object; Obtain the true value of the trajectory of the target object in the first period and the true value of the trajectory of the target object in the second period; Calculate the one-hot vector of the true value of the target point probability in the first period according to the true value of the target's trajectory in the first period and the list of target points of the candidate trajectory in the first period; The first loss function Loss1 is constructed by the cross entropy of the one-hot vector of the true value of the target point probability in the first period and the probability list of the candidate trajectory target points in the first period; According to the true value of the first period trajectory of the target object, the true value of the coordinates of the target point of the first period trajectory of the target object is obtained; The second loss function Loss2 is constructed based on the true value of the trajectory target point coordinates in the first period and the predicted target point coordinates in the first period; Calculate the one-hot vector of the true value of the target point probability in the second period according to the true value of the target's second period trajectory and the list of target points of the second period candidate trajectory; The third loss function Loss3 is constructed by the cross entropy of the one-hot vector of the true value of the target point probability in the second period and the probability list of the target point of the trajectory in the second period; According to the real value of the second period trajectory of the target object, the real value of the coordinates of the target point of the second period trajectory of the target object is obtained; The fourth loss function Loss4 is constructed based on the true value of the target point coordinates of the trajectory in the second period and the predicted target point coordinates in the second period; Construct a combined loss function Loss based on the first loss function Loss1, the second loss function Loss2, the third loss function Loss3 and the fourth loss function Loss4; The parameters of the trajectory prediction model are optimized through back propagation according to the combined loss function Loss.

5. The method for predicting the trajectory of an object according to claim 4, characterized in that: The formula of the combined loss function Loss is: Loss = λ1Loss1+λ2Loss2+λ3Loss3+λ4Loss4; Among them, λ1, λ2, λ3, and λ4 are respectively the fine-tuning parameters that need to be set for the trajectory prediction model.

6. The method for predicting an object trajectory according to claim 1, characterized in that: The trajectory prediction model is a feed-forward neural network model.

7. The method for predicting an object trajectory according to claim 1, characterized in that: The step of fusing the coordinates of the target point of the predicted trajectory in the first period with the first fusion feature to obtain the second fusion feature specifically includes: performing feature concatenation, numerical addition, or data subtraction on the coordinates of the target point of the predicted trajectory in the first period with the first fusion feature.

8. The method for predicting an object trajectory according to claim 1, characterized in that: When the target object to be predicted is a moving vehicle, the surrounding environment includes surrounding roads and surrounding objects, and the surrounding objects include surrounding pedestrians and surrounding vehicles.

9. A system for predicting the trajectory of an object, characterized in that: The method for predicting the trajectory of an object according to any one of claims 1 to 8 comprises: An information collection module is used to collect historical trajectory information and surrounding environment information of the target object to be predicted; An information fusion module is used to encode and fuse the historical trajectory information and surrounding environment information of the target object to obtain a first fusion feature; A trajectory prediction module, used to predict the coordinates of a target point in a first period of the target object based on the first fusion feature, and to predict the coordinates of the target point in a second period of the target object based on the coordinates of the target point in the first period of the target object; The output module is used to output the first period predicted target point coordinates and / or the second period predicted target point coordinates of the target object.

10. The system for predicting object trajectories according to claim 9, characterized in that: The trajectory prediction module comprises: A first-period target point probability prediction unit, used to predict a first-period candidate trajectory target point probability list of the target object according to the first fusion feature and the first-period candidate trajectory target point list generated based on the target object's surrounding environment information; A first-period target point position prediction unit, configured to predict the first-period predicted target point coordinates of the target object according to the first-period candidate trajectory target point probability list of the target object and the first-period trajectory target point coordinate list generated based on the surrounding environment information of the target object; A feature fusion unit, used for fusing the first period predicted target point coordinates of the target object with the first fusion feature to obtain a second fusion feature; A second-period target point probability prediction unit, used to predict a second-period trajectory target point probability list of the target object according to the second fusion feature and the second-period candidate trajectory target point list generated based on the target object's surrounding environment information; A second-period target point coordinate prediction unit, used to generate a second-period trajectory target point coordinate list based on the first-period candidate trajectory target point list according to the second fusion feature; The second period target point position prediction unit is used to predict the second period predicted target point coordinates of the target object according to the second period trajectory target point probability list and the second period trajectory target point coordinate list of the target object.

Images