Multi-modal motion trail prediction method and device

Abstract

The invention provides a multi-modal motion trail prediction method and device. The method comprises the following steps: pre-constructing a multi-modal motion trail prediction model; deploying the multi-modal motion trail prediction model to a target vehicle; acquiring current map information and obstacle track information of the target vehicle; according to the current map information, the obstacle track information and a multi-mode motion track prediction model, obstacle track prediction is carried out, and an obstacle track prediction result is obtained; and determining a current automatic driving decision of the target vehicle according to the obstacle trajectory prediction result. According to the method and the device, the prediction accuracy, feasibility and safety can be improved by constructing the multi-modal motion trail prediction model, and the method and the device are favorable for adapting to a complex traffic mode.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more specifically, to a multimodal motion trajectory prediction method and device. Background Art

[0002] As a core technology in the field of autonomous driving, autonomous driving motion prediction carries the heavy responsibility of analyzing the future behavior of traffic participants around autonomous driving vehicles. This technology provides accurate prediction basis for autonomous driving vehicles by analyzing the historical trajectories, surrounding environment information and future plans of traffic participants (including vehicles, pedestrians, bicycles, etc.), thereby assisting them in making correct behavioral decisions. At present, commonly used autonomous driving trajectory prediction is usually based on physical models, such as motion vehicle dynamics models or kinematic models. The car driving data is first obtained, and then the trajectory prediction calculation is performed based on the physical model and the car driving data. However, it has been found in practice that the existing methods have low accuracy and cannot adapt to more complex traffic patterns. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a multimodal motion trajectory prediction method and device, which can improve the prediction accuracy, feasibility and safety by constructing a multimodal motion trajectory prediction model, and is conducive to adapting to complex traffic patterns.

[0004] The first aspect of the present application provides a multimodal motion trajectory prediction method, comprising:

[0005] Pre-build a multi-modal motion trajectory prediction model;

[0006] Deploying the multimodal motion trajectory prediction model onto a target vehicle;

[0007] Obtaining current map information and obstacle trajectory information of the target vehicle;

[0008] Predicting the trajectory of the obstacle according to the current map information, the obstacle trajectory information and the multimodal motion trajectory prediction model to obtain an obstacle trajectory prediction result;

[0009] The current autonomous driving decision of the target vehicle is determined according to the obstacle trajectory prediction result.

[0010] Furthermore, the pre-constructing of a multi-modal motion trajectory prediction model includes:

[0011] Acquire a sample data set; wherein the sample data set includes at least obstacle trajectory information and high-precision map information;

[0012] Extracting features from the sample data set to obtain historical movement trajectories of obstacles, historical movement trajectories of the vehicle, map lane structured features, and lane reference lines;

[0013] Constructing an original trajectory prediction model; wherein the original trajectory prediction model at least includes a self-attention aggregator, a cross-attention fusion device, a map feature encoder constructed based on a graph-attention graph attention network, a trajectory feature encoder with a two-layer MLP structure, and a trajectory prediction decoder;

[0014] The original trajectory prediction model is trained by using the obstacle historical motion trajectory, the vehicle historical motion trajectory, the lane line structured features and the lane reference line to obtain a trajectory prediction result and a trained trajectory prediction model;

[0015] Performing a model evaluation on the trained trajectory prediction model according to the trajectory prediction result and the sample data set to obtain a model evaluation index;

[0016] When it is determined according to the model evaluation index that the trained trajectory prediction model passes the performance evaluation, the trained trajectory prediction model is determined as a trained multimodal motion trajectory prediction model.

[0017] Furthermore, the obtaining of the sample data set includes:

[0018] Obtaining raw data; wherein the raw data includes at least the Argoverse data set and the record data packet collected by the actual vehicle;

[0019] Parsing the original data to obtain parsed data;

[0020] A sample data set is constructed based on the parsed data, wherein the sample data set includes at least obstacle trajectory information and high-precision map information; the obstacle trajectory information includes at least obstacle coordinate information within a historical time window, obstacle speed information within the historical time window, obstacle angle information within the historical time window, and obstacle size information; the high-precision map information includes at least sampling point coordinates of a lane centerline, lane turning information, traffic control information, and intersection information.

[0021] Furthermore, the feature extraction of the sample data set to obtain the obstacle historical motion trajectory, the vehicle historical motion trajectory, the map lane structured features and the lane reference line includes:

[0022] Extracting features from the obstacle trajectory information to obtain a historical movement trajectory of the obstacle; wherein the historical movement trajectory of the obstacle at least includes obstacle trajectory coordinate points, obstacle speeds corresponding to the obstacle trajectory coordinate points, and obstacle directions corresponding to the obstacle trajectory coordinate points;

[0023] Acquire lane line structured features according to the high-precision map information; wherein the lane line structured features at least include the coordinates of the road center point within a preset range, road turning information, intersection information, the successor relationship between lanes, and the left and right neighbor relationship of the lanes;

[0024] Acquire lane line information and the historical movement trajectory of the vehicle according to the high-precision map information;

[0025] A lane reference line of a vehicle obstacle is generated according to the lane line information and the historical motion trajectory of the vehicle.

[0026] Furthermore, the original trajectory prediction model is trained by using the obstacle historical motion trajectory, the vehicle historical motion trajectory, the lane line structured features, and the lane reference line to obtain a trajectory prediction result and a trained trajectory prediction model, including:

[0027] Performing feature encoding processing through the trajectory feature encoder, the obstacle historical motion trajectory, the vehicle historical motion trajectory and the lane line structured features to obtain a trajectory feature vector and a map feature vector;

[0028] Performing feature fusion processing on the trajectory feature vector and the map feature vector to obtain a fused spatiotemporal feature vector;

[0029] Decoding the spatiotemporal feature vector by the trajectory prediction decoder to obtain a candidate prediction trajectory;

[0030] Decoding the candidate predicted trajectory by the trajectory prediction decoder to obtain a predicted refined trajectory;

[0031] Generating a trajectory prediction result according to the candidate predicted trajectory and the predicted refined trajectory;

[0032] Calculating a training loss based on the candidate predicted trajectory, the trajectory prediction result, and the lane reference line;

[0033] The model parameters of the original trajectory prediction model are optimized according to the training loss to obtain a trained trajectory prediction model.

[0034] Furthermore, the feature encoding process is performed on the trajectory feature encoder, the obstacle historical motion trajectory, the vehicle historical motion trajectory and the lane line structured feature to obtain a trajectory feature vector and a map feature vector, including:

[0035] The obstacle historical motion trajectory and the vehicle historical motion trajectory are spliced ​​to obtain the historical motion trajectory of the entire prediction scene;

[0036] The historical motion trajectory is encoded by the trajectory feature encoder to obtain a trajectory feature vector;

[0037] The lane line structured features are encoded by the map feature encoder to obtain a map feature vector.

[0038] Furthermore, the performing feature fusion processing on the trajectory feature vector and the map feature vector to obtain a fused spatiotemporal feature vector includes:

[0039] Aggregating the trajectory feature vector through the self-attention aggregator to obtain a vehicle history time series feature vector;

[0040] The vehicle historical time series feature vector and the map feature vector are cross-fused by the cross-attention fuser to obtain a feature vector of the interaction relationship between vehicle modeling and the road;

[0041] Adding the interaction relationship feature vector to the vehicle history time series feature vector to obtain an added feature vector;

[0042] The added feature vectors are aggregated through a self-attention aggregator to obtain a fused spatiotemporal feature vector.

[0043] Furthermore, the calculating of the training loss according to the candidate predicted trajectory, the trajectory prediction result and the lane reference line includes:

[0044] Obtaining the real trajectory of the obstacle according to the sample data set;

[0045] Performing a smooth L1 loss calculation based on the candidate predicted trajectory and the real trajectory of the obstacle to obtain a first loss value;

[0046] Performing a smooth L1 loss calculation based on the trajectory prediction result and the real trajectory of the obstacle to obtain a second loss value;

[0047] Calculating regression loss according to the first loss value and the second loss value;

[0048] Determining a predicted trajectory to be calculated having the smallest distance from the lane reference line from the trajectory prediction results;

[0049] Calculating a lane loss according to the predicted trajectory to be calculated and the lane reference line;

[0050] The regression loss and the lane loss are weighted and summed according to a preset weight value to obtain the training loss.

[0051] Furthermore, the trained trajectory prediction model is evaluated according to the trajectory prediction result and the sample data set to obtain a model evaluation index, including:

[0052] Determining an optimal obstacle prediction trajectory and a trajectory probability of the optimal obstacle prediction trajectory according to the trajectory prediction result;

[0053] Performing average Euclidean distance calculation based on the optimal obstacle prediction trajectory and the actual obstacle trajectory to obtain a minimum average displacement error;

[0054] Calculating the Euclidean distance between the end point of the optimal obstacle prediction trajectory and the end point of the obstacle real trajectory to obtain the minimum final displacement error;

[0055] Calculating an endpoint error according to the trajectory probability, the optimal obstacle predicted trajectory, and the obstacle true trajectory;

[0056] Determining a corresponding candidate lane according to the optimal obstacle prediction trajectory;

[0057] Calculating a lateral displacement error between an end point of the optimal obstacle prediction trajectory and a center line of the candidate lane;

[0058] The minimum lane final displacement error is determined by taking the minimum value of the lateral displacement error;

[0059] Calculating the MR value according to the endpoint error, a preset distance threshold, the optimal obstacle prediction trajectory and the obstacle real trajectory;

[0060] The minimum average displacement error, the minimum final displacement error, the endpoint error, the minimum lane final displacement error and the MR value are summarized to obtain a model evaluation index.

[0061] A second aspect of the present application provides a multimodal motion trajectory prediction device, the multimodal motion trajectory prediction device comprising:

[0062] A model building unit, used to pre-build a multi-modal motion trajectory prediction model;

[0063] A model deployment unit, used to deploy the multimodal motion trajectory prediction model to a target vehicle;

[0064] An acquisition unit, used to acquire current map information and obstacle trajectory information of the target vehicle;

[0065] A trajectory prediction unit, configured to predict the trajectory of an obstacle according to the current map information, the obstacle trajectory information and the multimodal motion trajectory prediction model, and obtain an obstacle trajectory prediction result;

[0066] A decision determination unit is used to determine the current automatic driving decision of the target vehicle according to the obstacle trajectory prediction result.

[0067] Furthermore, the model building unit includes:

[0068] An acquisition subunit, used to acquire a sample data set; wherein the sample data set at least includes obstacle trajectory information and high-precision map information;

[0069] An extraction subunit, used to extract features from the sample data set to obtain the obstacle historical motion trajectory, the vehicle historical motion trajectory, the map lane structured features and the lane reference line;

[0070] A construction subunit is used to construct an original trajectory prediction model; wherein the original trajectory prediction model at least includes a self-attention aggregator, a cross-attention fusion device, a map feature encoder constructed based on a graph-attention graph attention network, a trajectory feature encoder with a two-layer MLP structure, and a trajectory prediction decoder;

[0071] A training subunit, used to train the original trajectory prediction model through the obstacle historical motion trajectory, the vehicle historical motion trajectory, the lane line structured features and the lane reference line to obtain a trajectory prediction result and a trained trajectory prediction model;

[0072] An evaluation subunit, configured to perform a model evaluation on the trained trajectory prediction model according to the trajectory prediction result and the sample data set to obtain a model evaluation index;

[0073] The determination subunit is used to determine the trained trajectory prediction model as a trained multimodal motion trajectory prediction model when it is determined according to the model evaluation index that the trained trajectory prediction model passes the performance evaluation.

[0074] Furthermore, the acquisition subunit includes:

[0075] A first acquisition module is used to acquire raw data; wherein the raw data at least includes an Argoverse data set and a record data packet collected by a real vehicle;

[0076] A parsing module, used for parsing the original data to obtain parsed data;

[0077] A construction module is used to construct a sample data set based on the parsed data, wherein the sample data set includes at least obstacle trajectory information and high-precision map information; the obstacle trajectory information includes at least obstacle coordinate information within a historical time window, obstacle speed information within the historical time window, obstacle angle information within the historical time window, and obstacle size information; the high-precision map information includes at least sampling point coordinates of the lane centerline, lane turning information, traffic control information, and intersection information.

[0078] Furthermore, the extraction subunit includes:

[0079] An extraction module, configured to perform feature extraction on the obstacle trajectory information to obtain a historical movement trajectory of the obstacle; wherein the historical movement trajectory of the obstacle at least includes an obstacle trajectory coordinate point, an obstacle speed corresponding to the obstacle trajectory coordinate point, and an obstacle direction corresponding to the obstacle trajectory coordinate point;

[0080] A second acquisition module is used to acquire lane line structured features according to the high-precision map information; wherein the lane line structured features at least include the coordinates of the road center point within a preset range, road turning information, intersection information, the relationship between the front and rear lanes, and the relationship between the left and right neighbors of the lanes;

[0081] The second acquisition module is further used to acquire lane line information and a historical motion trajectory of the vehicle according to the high-precision map information;

[0082] The first generating module is used to generate a lane reference line of a vehicle obstacle according to the lane line information and the historical motion trajectory of the vehicle.

[0083] Furthermore, the training subunit includes:

[0084] An encoding module, used for performing feature encoding processing through the trajectory feature encoder, the obstacle historical motion trajectory, the vehicle historical motion trajectory and the lane line structured features to obtain a trajectory feature vector and a map feature vector;

[0085] A fusion module, used for performing feature fusion processing on the trajectory feature vector and the map feature vector to obtain a fused spatiotemporal feature vector;

[0086] A decoding module, configured to decode the spatiotemporal feature vector through the trajectory prediction decoder to obtain a candidate prediction trajectory;

[0087] The decoding module is further used to decode the candidate predicted trajectory through the trajectory prediction decoder to obtain a predicted refined trajectory;

[0088] A second generating module, configured to generate a trajectory prediction result according to the candidate predicted trajectory and the predicted refined trajectory;

[0089] A first calculation module, used to calculate the training loss according to the candidate predicted trajectory, the trajectory prediction result and the lane reference line;

[0090] The optimization module is used to optimize the model parameters of the original trajectory prediction model according to the training loss to obtain a trained trajectory prediction model.

[0091] Furthermore, the encoding module is specifically used to splice the obstacle historical motion trajectory and the vehicle historical motion trajectory to obtain the historical motion trajectory of the entire prediction scene;

[0092] The encoding module is further configured to encode the historical motion trajectory through the trajectory feature encoder to obtain a trajectory feature vector;

[0093] The encoding module is specifically used to encode the lane line structured features through the map feature encoder to obtain a map feature vector.

[0094] Furthermore, the fusion module is specifically used to aggregate the trajectory feature vector through the self-attention aggregator to obtain the vehicle history time series feature vector;

[0095] The fusion module is further configured to perform cross-fusion processing on the vehicle historical time series feature vector and the map feature vector through the cross-attention fuser to obtain a feature vector of the interaction relationship between vehicle modeling and the road;

[0096] The fusion module is further configured to add the interaction relationship feature vector to the vehicle history time series feature vector to obtain an added feature vector;

[0097] The fusion module is specifically used to aggregate the added feature vectors through a self-attention aggregator to obtain a fused spatiotemporal feature vector.

[0098] Furthermore, the first calculation module is specifically used to obtain the real trajectory of the obstacle according to the sample data set;

[0099] The first calculation module is further configured to perform a smooth L1 loss calculation based on the candidate predicted trajectory and the real trajectory of the obstacle to obtain a first loss value;

[0100] The first calculation module is further configured to perform a smooth L1 loss calculation based on the trajectory prediction result and the real trajectory of the obstacle to obtain a second loss value;

[0101] The first calculation module is further configured to calculate a regression loss according to the first loss value and the second loss value;

[0102] The first calculation module is further configured to determine, from the trajectory prediction results, a predicted trajectory to be calculated having a minimum distance from the lane reference line;

[0103] The first calculation module is further configured to calculate a lane loss according to the predicted trajectory to be calculated and the lane reference line;

[0104] The first calculation module is further specifically used to perform weighted sum calculation on the regression loss and the lane loss according to a preset weight value to obtain a training loss.

[0105] Furthermore, the evaluation subunit includes:

[0106] A determination module, configured to determine an optimal obstacle prediction trajectory and a trajectory probability of the optimal obstacle prediction trajectory according to the trajectory prediction result;

[0107] A second calculation module is used to calculate the average Euclidean distance according to the optimal obstacle prediction trajectory and the actual obstacle trajectory to obtain a minimum average displacement error;

[0108] The second calculation module is further used to calculate the Euclidean distance between the end point of the optimal obstacle prediction trajectory and the end point of the obstacle real trajectory to obtain the minimum final displacement error;

[0109] The second calculation module is further used to calculate an endpoint error according to the trajectory probability, the optimal obstacle predicted trajectory and the obstacle true trajectory;

[0110] The determination module is further used to determine the corresponding candidate lane according to the optimal obstacle prediction trajectory;

[0111] The second calculation module is further used to calculate the lateral displacement error between the end point of the optimal obstacle prediction trajectory and the center line of the candidate lane;

[0112] The determination module is further used to determine the minimum lane final displacement error by taking the minimum value of the lateral displacement error;

[0113] The second calculation module is further used to calculate the MR value according to the endpoint error, the preset distance threshold, the optimal obstacle prediction trajectory and the obstacle real trajectory;

[0114] The summarizing module is used to summarize the minimum average displacement error, the minimum final displacement error, the endpoint error, the minimum lane final displacement error and the MR value to obtain a model evaluation index.

[0115] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the multimodal motion trajectory prediction method described in any one of the first aspects of the present application.

[0116] A fourth aspect of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the multimodal motion trajectory prediction method described in any one of the first aspects of the present application is executed.

[0117] The beneficial effects of the present application are as follows: the method and device can effectively improve the accuracy, feasibility and safety of model prediction without reducing the time performance of the model. At the same time, the model that integrates map information and historical trajectory information can enable the obstacle trajectory characteristics to obtain map environment feature information, thereby enhancing the model's understanding and prediction capabilities of the obstacle's motion behavior. In addition, by adding lane loss training, multiple trajectories can be encouraged to show different distributions, covering feasible maneuvers in a map-aware manner, thereby helping autonomous driving make safer decisions. Finally, adding the minimum lane final error during the model evaluation process can capture the diverse quantity and quality of multimodal outputs based on map data, thereby improving the scientificity and diversity of model evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0119] Figure 1 A schematic diagram of a flow chart of a multimodal motion trajectory prediction method provided in an embodiment of the present application;

[0120] Figure 2 A schematic diagram of a flow chart of another multimodal motion trajectory prediction method provided in an embodiment of the present application;

[0121] Figure 3 A schematic diagram of an example flow of feature coding provided in an embodiment of the present application;

[0122] Figure 4 A schematic diagram of an example flow of feature fusion provided in an embodiment of the present application;

[0123] Figure 5 A schematic diagram of an example flow of feature decoding provided in an embodiment of the present application;

[0124] Figure 6 A schematic diagram of the structure of a multi-modal motion trajectory prediction device provided in an embodiment of the present application;

[0125] Figure 7 A schematic diagram of the structure of another multimodal motion trajectory prediction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0126] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0127] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0128] Example 1

[0129] Please see Figure 1 , Figure 1 A schematic flow chart of a multimodal motion trajectory prediction method provided in this embodiment. The multimodal motion trajectory prediction method includes:

[0130] S101. Pre-build a multi-modal motion trajectory prediction model.

[0131] In this embodiment, the model building process may include three sub-processes: feature extraction, model training, and model evaluation.

[0132] Regarding feature extraction, this method can extract features of interest to the model from the raw data and preprocess the raw data into feature data that can be used by the model. It is mainly divided into three parts: trajectory information extraction, map information extraction, and reference lane generation.

[0133] Regarding model training, this method can use deep learning methods to build a motion prediction model, which can learn the motion laws of objects from historical trajectory data and contextual information (maps, interactions between targets) and predict their future motion trajectories. Model training mainly includes four parts: feature encoding, feature fusion, feature decoding, and loss design (or loss supervision).

[0134] Regarding model evaluation, this method can use the minimum average displacement error minADE, the minimum final displacement error minFDE, and the minimum lane final displacement error minLaneFDE as indicators to evaluate the effectiveness of the training model.

[0135] In this embodiment, the model can provide the autonomous driving vehicle with the ability to predict the movement behavior of surrounding obstacles, thereby helping the autonomous driving system to make more accurate decisions and plans.

[0136] S102: deploying the multimodal motion trajectory prediction model to the target vehicle.

[0137] S103: Obtain current map information and obstacle trajectory information of the target vehicle.

[0138] S104: Predict the trajectory of the obstacle according to the current map information, the obstacle trajectory information and the multimodal motion trajectory prediction model to obtain an obstacle trajectory prediction result.

[0139] S105. Determine the current autonomous driving decision of the target vehicle according to the obstacle trajectory prediction result.

[0140] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and no limitation is made in this embodiment.

[0141] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0142] It can be seen that the implementation of the multimodal motion trajectory prediction method described in this embodiment can improve the prediction accuracy, feasibility and safety by constructing a multimodal motion trajectory prediction model, which is conducive to adapting to complex traffic patterns.

[0143] Example 2

[0144] Please see Figure 2 , Figure 2 A schematic flow chart of a method for constructing a multimodal motion trajectory prediction model provided in this embodiment. The method for constructing a multimodal motion trajectory prediction model includes:

[0145] S201, obtaining original data.

[0146] In this embodiment, the original data includes at least the Argoverse data set (such as an open data set for autonomous driving) and record data packets collected by a real vehicle (such as data automatically collected using sensor equipment on a test section).

[0147] S202: Analyze the original data to obtain analyzed data.

[0148] S203: Construct a sample data set according to the parsed data.

[0149] In this embodiment, the sample data set includes at least obstacle trajectory information and high-precision map information.

[0150] In this embodiment, the obstacle trajectory information includes at least the obstacle coordinate information within the historical time window, the obstacle speed information within the historical time window, the obstacle angle information within the historical time window, and the obstacle size information, that is, the xy coordinates, speed, direction, etc. of the obstacle trajectory coordinate point.

[0151] In this embodiment, the high-precision map information includes at least the sampling point coordinates of the lane centerline, lane turning information, traffic control information, and intersection information.

[0152] S204: Extract features from the obstacle trajectory information to obtain the obstacle's historical motion trajectory.

[0153] In this embodiment, the obstacle historical motion trajectory includes at least obstacle trajectory coordinate points, obstacle speeds corresponding to the obstacle trajectory coordinate points, and obstacle directions corresponding to the obstacle trajectory coordinate points.

[0154] S205. Acquire lane line structured features according to high-precision map information.

[0155] In this embodiment, the lane line structured features include at least the coordinates of the road center point within a preset range, road turning information, intersection information, the successor relationship between lanes, and the left and right neighbor relationship of the lanes.

[0156] S206. Obtain lane line information and the historical movement trajectory of the vehicle based on the high-precision map information.

[0157] S207: Generate a lane reference line for the vehicle obstacle according to the lane line information and the historical motion trajectory of the vehicle.

[0158] In this embodiment, the method can use the lane line information and historical trajectory information of the high-precision map to generate lane reference lines for obstacles.

[0159] S208: Construct an original trajectory prediction model.

[0160] In this embodiment, the original trajectory prediction model includes at least a self-attention aggregator, a cross-attention fusion device, a map feature encoder constructed based on a graph-attention network, a trajectory feature encoder with a two-layer MLP structure, and a trajectory prediction decoder.

[0161] S209 , performing feature encoding processing through a trajectory feature encoder, the obstacle historical motion trajectory, the vehicle historical motion trajectory, and the lane line structured features to obtain a trajectory feature vector and a map feature vector.

[0162] As an optional implementation, feature encoding processing is performed through a trajectory feature encoder, an obstacle historical motion trajectory, a vehicle historical motion trajectory, and a lane line structured feature to obtain a trajectory feature vector and a map feature vector, including:

[0163] The historical motion trajectory of the obstacle and the historical motion trajectory of the vehicle are spliced ​​together to obtain the historical motion trajectory of the entire prediction scene;

[0164] The historical motion trajectory is encoded by a trajectory feature encoder to obtain a trajectory feature vector;

[0165] The lane line structured features are encoded through a map feature encoder to obtain a map feature vector.

[0166] In the embodiment, the method can concatenate the obstacle's historical motion trajectory and the vehicle's historical motion trajectory to obtain the historical motion trajectory of the entire prediction scene; then the obtained historical motion trajectory is sent to a trajectory feature encoder composed of two layers of MLP to obtain a trajectory feature vector; finally, the lane line structured features are sent to a map feature encoder to obtain a map feature vector. The map feature encoder is a deep neural network based on the Graph-attention graph attention network.

[0167] Please see Figure 3 , Figure 3 An example flowchart of feature encoding is shown.

[0168] S210: Perform feature fusion processing on the trajectory feature vector and the map feature vector to obtain a fused spatiotemporal feature vector.

[0169] As an optional implementation, feature fusion processing is performed on the trajectory feature vector and the map feature vector to obtain a fused spatiotemporal feature vector, including:

[0170] The trajectory feature vector is aggregated through the self-attention aggregator to obtain the vehicle history time series feature vector;

[0171] The vehicle historical time series feature vector and the map feature vector are cross-fused through the cross-attention fuser to obtain the interactive relationship feature vector between vehicle modeling and the road.

[0172] Add the interaction relationship feature vector and the vehicle history time series feature vector to obtain the added feature vector;

[0173] The added feature vectors are aggregated through the self-attention aggregator to obtain the fused spatiotemporal feature vector.

[0174] In this embodiment, the method can perform a self-attention operation on the trajectory feature vector to aggregate the historical time series information of the obstacle; then perform a cross-attention operation on the historical time series information and the map feature vector to model the interaction relationship between the obstacle and the road; finally, the interaction relationship between the obstacle and the road is added to the historical time series information, and a self-attention operation is performed to obtain a feature vector that integrates spatiotemporal information.

[0175] Please see Figure 4 , Figure 4 An example flowchart of feature fusion is shown.

[0176] S211 . Decode the spatiotemporal feature vector through a trajectory prediction decoder to obtain a candidate prediction trajectory.

[0177] In this embodiment, the method can send the fused trajectory feature vector to the trajectory prediction decoder to obtain the trajectory prediction decoding result. The trajectory prediction decoder first performs self-attention modeling on different prediction modes, and enhances the model's ability to understand future uncertainties by modeling the mutual influence between different prediction trajectories; then, through an MLP layer, the first decoded trajectory candidate is obtained.

[0178] S212: Decode the candidate prediction trajectory through a trajectory prediction decoder to obtain a predicted refined trajectory.

[0179] In this embodiment, the method may use the trajectory candidate obtained in step S211 as the input of the entire decoding process to generate a refinement of the predicted trajectory again.

[0180] S213: Generate a trajectory prediction result according to the candidate prediction trajectory and the predicted refined trajectory.

[0181] In this embodiment, the final predicted trajectory is obtained by adding the trajectory candidates and the trajectory refinement.

[0182] Please see Figure 5 , Figure 5 An example flowchart of feature decoding is shown.

[0183] S214: Calculate the training loss based on the candidate predicted trajectories, the trajectory prediction results, and the lane reference lines.

[0184] As an optional implementation, the training loss is calculated based on the candidate predicted trajectory, the trajectory prediction result and the lane reference line, including:

[0185] Obtain the true trajectory of the obstacle based on the sample data set;

[0186] A smooth L1 loss is calculated based on the candidate predicted trajectory and the actual trajectory of the obstacle to obtain a first loss value;

[0187] A smooth L1 loss is calculated based on the trajectory prediction result and the actual trajectory of the obstacle to obtain the second loss value;

[0188] Calculate the regression loss based on the first loss value and the second loss value;

[0189] Determine the predicted trajectory to be calculated with the minimum distance from the lane reference line from the trajectory prediction results;

[0190] Calculate lane loss based on the predicted trajectory to be calculated and the lane reference line;

[0191] The regression loss and lane loss are weighted and summed according to the preset weight values ​​to obtain the training loss.

[0192] In this embodiment, regarding the loss design: (1) regression loss

[0193] The Huber Loss of the trajectory candidate generated in step S211 and the final trajectory generated in step S213 and the true value are calculated respectively, and the two are added to obtain the regression loss result.

[0194] (2) Lane loss

[0195] According to the generated lane reference line, the distance between the final trajectory of the six modes and the lane reference line is calculated, and the predicted trajectory with the smallest distance and the lane reference line are selected to calculate the L1 Loss as the result of Lane Loss.

[0196] (3) Training Loss

[0197] It is obtained by the weighted sum of regression loss and lane loss. Specifically, the weights of regression loss and lane loss are 1.0 and 0.1 respectively.

[0198] S215. Optimize the model parameters of the original trajectory prediction model according to the training loss to obtain a trained trajectory prediction model.

[0199] S216: Determine the best obstacle prediction trajectory and the trajectory probability of the best obstacle prediction trajectory according to the trajectory prediction result.

[0200] S217: Calculate the average Euclidean distance based on the best obstacle prediction trajectory and the actual obstacle trajectory to obtain the minimum average displacement error.

[0201] In this embodiment, the minimum average displacement error min-ADE is the average Euclidean distance between the best predicted trajectory and the true trajectory, wherein the best refers to the trajectory with the minimum endpoint error.

[0202] S218. Calculate the Euclidean distance between the end point of the optimal obstacle prediction trajectory and the end point of the obstacle real trajectory to obtain the minimum final displacement error.

[0203] In this embodiment, the minimum final displacement error min-FDE is the Euclidean distance between the best predicted endpoint and the actual trajectory endpoint, wherein the best refers to the trajectory with the minimum endpoint error.

[0204] S219: Calculate the endpoint error according to the trajectory probability, the best obstacle prediction trajectory and the actual obstacle trajectory.

[0205] In this embodiment, the endpoint error Brier-minFDE is similar to minFDE. The only difference is that we add (1.0-p)^2 to the endpoint L2 distance, where p corresponds to the probability of the best predicted trajectory.

[0206] S220: Determine a corresponding candidate lane according to the optimal obstacle prediction trajectory.

[0207] S221. Calculate the lateral displacement error between the end point of the optimal obstacle prediction trajectory and the center line of the candidate lane.

[0208] S222. Determine the minimum lane final displacement error by taking the minimum value of the lateral displacement error.

[0209] In this embodiment, the minimum lane final displacement error Min-LaneFDE is the minimum value of the lateral displacement error between the best predicted end point and the center line of the candidate lane.

[0210] S223, calculating the MR value according to the endpoint error, the preset distance threshold, the best obstacle prediction trajectory and the obstacle real trajectory.

[0211] In this embodiment, Miss Rate (MR) is the proportion of cases where the Euclidean distance between the predicted trajectory and the true value is greater than 2.0 meters based on the endpoint error.

[0212] S224, summarizing the minimum average displacement error, the minimum final displacement error, the endpoint error, the minimum lane final displacement error and the MR value to obtain a model evaluation index.

[0213] S225. When it is determined according to the model evaluation index that the trained trajectory prediction model passes the performance evaluation, the trained trajectory prediction model is determined as a trained multimodal motion trajectory prediction model.

[0214] In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, and no limitation is made in this embodiment.

[0215] In this embodiment, the execution subject of the method may also be a smart device such as a smart phone, a tablet computer, etc., which is not limited in this embodiment.

[0216] It can be seen that the implementation of the multimodal motion trajectory prediction method described in this embodiment can improve the prediction accuracy, feasibility and safety by constructing a multimodal motion trajectory prediction model, which is conducive to adapting to complex traffic patterns.

[0217] Example 3

[0218] Please see Figure 6 , Figure 6 Schematic diagram of the structure of a multi-modal motion trajectory prediction device provided in this embodiment. Figure 6 As shown, the multimodal motion trajectory prediction device includes:

[0219] A model building unit 310, used to pre-build a multi-modal motion trajectory prediction model;

[0220] A model deployment unit 320, for deploying the multimodal motion trajectory prediction model to a target vehicle;

[0221] An acquisition unit 330 is used to acquire current map information and obstacle trajectory information of the target vehicle;

[0222] The trajectory prediction unit 340 is used to predict the trajectory of the obstacle according to the current map information, the obstacle trajectory information and the multi-modal motion trajectory prediction model to obtain the obstacle trajectory prediction result;

[0223] The decision determination unit 350 is used to determine the current autonomous driving decision of the target vehicle according to the obstacle trajectory prediction result.

[0224] In this embodiment, the explanation of the multimodal motion trajectory prediction device can refer to the description in Embodiment 1 or Embodiment 2, and will not be further elaborated in this embodiment.

[0225] It can be seen that the implementation of the multimodal motion trajectory prediction device described in this embodiment can improve the prediction accuracy, feasibility and safety by constructing a multimodal motion trajectory prediction model, which is conducive to adapting to complex traffic patterns.

[0226] Example 4

[0227] Please see Figure 7 , Figure 7 Schematic diagram of the structure of a multi-modal motion trajectory prediction device provided in this embodiment. Figure 7 As shown, the multimodal motion trajectory prediction device includes:

[0228] A model building unit 310, used to pre-build a multi-modal motion trajectory prediction model;

[0229] A model deployment unit 320, for deploying the multimodal motion trajectory prediction model to a target vehicle;

[0230] An acquisition unit 330 is used to acquire current map information and obstacle trajectory information of the target vehicle;

[0231] The trajectory prediction unit 340 is used to predict the trajectory of the obstacle according to the current map information, the obstacle trajectory information and the multi-modal motion trajectory prediction model to obtain the obstacle trajectory prediction result;

[0232] The decision determination unit 350 is used to determine the current autonomous driving decision of the target vehicle according to the obstacle trajectory prediction result.

[0233] As an optional implementation, the model building unit 310 includes:

[0234] The acquisition subunit 311 is used to acquire a sample data set; wherein the sample data set at least includes obstacle trajectory information and high-precision map information;

[0235] The extraction subunit 312 is used to extract features from the sample data set to obtain the historical motion trajectory of the obstacle, the historical motion trajectory of the vehicle, the structured features of the map lane, and the lane reference line;

[0236] A construction subunit 313 is used to construct an original trajectory prediction model; wherein the original trajectory prediction model at least includes a self-attention aggregator, a cross-attention fusion device, a map feature encoder constructed based on a graph-attention graph attention network, a trajectory feature encoder with a two-layer MLP structure, and a trajectory prediction decoder;

[0237] The training subunit 314 is used to train the original trajectory prediction model through the historical motion trajectory of the obstacle, the historical motion trajectory of the vehicle, the structured features of the lane line and the lane reference line to obtain the trajectory prediction result and the trained trajectory prediction model;

[0238] An evaluation subunit 315 is used to perform model evaluation on the trained trajectory prediction model according to the trajectory prediction result and the sample data set to obtain a model evaluation index;

[0239] The determination subunit 316 is configured to determine the trained trajectory prediction model as a trained multimodal motion trajectory prediction model when it is determined according to the model evaluation index that the trained trajectory prediction model passes the performance evaluation.

[0240] As an optional implementation, the acquiring subunit 311 includes:

[0241] The first acquisition module is used to acquire raw data; wherein the raw data at least includes the Argoverse data set and the record data packet collected by the actual vehicle;

[0242] A parsing module is used to parse the original data to obtain parsed data;

[0243] A construction module is used to construct a sample data set based on the parsed data, wherein the sample data set includes at least obstacle trajectory information and high-precision map information; the obstacle trajectory information includes at least obstacle coordinate information within a historical time window, obstacle speed information within a historical time window, obstacle angle information within a historical time window, and obstacle size information; the high-precision map information includes at least sampling point coordinates of the lane centerline, lane turning information, traffic control information, and intersection information.

[0244] As an optional implementation, the extraction subunit 312 includes:

[0245] An extraction module is used to extract features from obstacle trajectory information to obtain a historical movement trajectory of the obstacle; wherein the historical movement trajectory of the obstacle at least includes obstacle trajectory coordinate points, obstacle speeds corresponding to the obstacle trajectory coordinate points, and obstacle directions corresponding to the obstacle trajectory coordinate points;

[0246] The second acquisition module is used to acquire the lane line structured features according to the high-precision map information; wherein the lane line structured features at least include the coordinates of the road center point within a preset range, road turning information, intersection information, the successor relationship between lanes, and the left and right neighbor relationship of the lanes;

[0247] The second acquisition module is also used to obtain lane line information and the historical movement trajectory of the vehicle based on the high-precision map information;

[0248] The first generation module is used to generate a lane reference line of a vehicle obstacle according to the lane line information and the historical motion trajectory of the vehicle.

[0249] As an optional implementation, the training subunit 314 includes:

[0250] The encoding module is used to perform feature encoding processing through the trajectory feature encoder, the obstacle historical motion trajectory, the vehicle historical motion trajectory and the lane line structured features to obtain the trajectory feature vector and the map feature vector;

[0251] A fusion module is used to perform feature fusion processing on the trajectory feature vector and the map feature vector to obtain a fused spatiotemporal feature vector;

[0252] A decoding module is used to decode the spatiotemporal feature vector through a trajectory prediction decoder to obtain a candidate prediction trajectory;

[0253] The decoding module is further used to decode the candidate prediction trajectory through a trajectory prediction decoder to obtain a prediction refined trajectory;

[0254] The second generation module is used to generate a trajectory prediction result according to the candidate prediction trajectory and the prediction refinement trajectory;

[0255] A first calculation module, used for calculating the training loss according to the candidate predicted trajectory, the trajectory prediction result and the lane reference line;

[0256] The optimization module is used to optimize the model parameters of the original trajectory prediction model according to the training loss to obtain the trained trajectory prediction model.

[0257] As an optional implementation, the encoding module is specifically used to splice the historical motion trajectory of the obstacle and the historical motion trajectory of the vehicle to obtain the historical motion trajectory of the entire prediction scene;

[0258] The encoding module is specifically used to encode the historical motion trajectory through a trajectory feature encoder to obtain a trajectory feature vector;

[0259] The encoding module is specifically used to encode the lane line structured features through a map feature encoder to obtain a map feature vector.

[0260] As an optional implementation, the fusion module is specifically used to aggregate the trajectory feature vector through a self-attention aggregator to obtain a vehicle historical time series feature vector;

[0261] The fusion module is specifically used to perform cross-fusion processing on the vehicle historical time series feature vector and the map feature vector through a cross-attention fuser to obtain the interaction relationship feature vector between the vehicle modeling and the road;

[0262] The fusion module is further used to add the interaction relationship feature vector and the vehicle history time series feature vector to obtain the added feature vector;

[0263] The fusion module is specifically used to aggregate the added feature vectors through a self-attention aggregator to obtain a fused spatiotemporal feature vector.

[0264] As an optional implementation, the first calculation module is specifically used to obtain the real trajectory of the obstacle according to the sample data set;

[0265] The first calculation module is further configured to perform a smooth L1 loss calculation based on the candidate predicted trajectory and the real trajectory of the obstacle to obtain a first loss value;

[0266] The first calculation module is further used to perform a smooth L1 loss calculation based on the trajectory prediction result and the actual trajectory of the obstacle to obtain a second loss value;

[0267] A first calculation module is further used to calculate the regression loss according to the first loss value and the second loss value;

[0268] The first calculation module is further used to determine the predicted trajectory to be calculated with the smallest distance from the lane reference line from the trajectory prediction result;

[0269] The first calculation module is further used to calculate the lane loss according to the predicted trajectory to be calculated and the lane reference line;

[0270] The first calculation module is specifically used to perform weighted sum calculation on the regression loss and the lane loss according to a preset weight value to obtain the training loss.

[0271] As an optional implementation, the evaluation subunit 315 includes:

[0272] A determination module, used to determine the best obstacle prediction trajectory and the trajectory probability of the best obstacle prediction trajectory according to the trajectory prediction result;

[0273] The second calculation module is used to calculate the average Euclidean distance according to the best obstacle prediction trajectory and the actual obstacle trajectory to obtain the minimum average displacement error;

[0274] The second calculation module is also used to calculate the Euclidean distance between the end point of the best obstacle prediction trajectory and the end point of the obstacle real trajectory to obtain the minimum final displacement error;

[0275] The second calculation module is further used to calculate the endpoint error according to the trajectory probability, the best obstacle predicted trajectory and the obstacle real trajectory;

[0276] The determination module is further used to determine a corresponding candidate lane according to the optimal obstacle prediction trajectory;

[0277] The second calculation module is also used to calculate the lateral displacement error between the end point of the optimal obstacle prediction trajectory and the center line of the candidate lane;

[0278] The determination module is further used to determine the minimum lane final displacement error by taking the minimum value of the lateral displacement error;

[0279] The second calculation module is further used to calculate the MR value according to the endpoint error, the preset distance threshold, the best obstacle prediction trajectory and the obstacle real trajectory;

[0280] The summary module is used to summarize the minimum average displacement error, the minimum final displacement error, the endpoint error, the minimum lane final displacement error and the MR value to obtain the model evaluation index.

[0281] In this embodiment, the explanation of the multimodal motion trajectory prediction device can refer to the description in Embodiment 1 or Embodiment 2, and will not be further elaborated in this embodiment.

[0282] It can be seen that the implementation of the multimodal motion trajectory prediction device described in this embodiment can improve the prediction accuracy, feasibility and safety by constructing a multimodal motion trajectory prediction model, which is conducive to adapting to complex traffic patterns.

[0283] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the multimodal motion trajectory prediction method in embodiment 1 or embodiment 2 of the present application.

[0284] An embodiment of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the multimodal motion trajectory prediction method in Embodiment 1 or Embodiment 2 of the present application is executed.

[0285] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0286] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0287] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0288] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0289] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0290] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A multimodal motion trajectory prediction method, characterized in that: include: Pre-build a multi-modal motion trajectory prediction model; Deploying the multimodal motion trajectory prediction model onto a target vehicle; Obtaining current map information and obstacle trajectory information of the target vehicle; Predicting the trajectory of the obstacle according to the current map information, the obstacle trajectory information and the multimodal motion trajectory prediction model to obtain an obstacle trajectory prediction result; The current autonomous driving decision of the target vehicle is determined according to the obstacle trajectory prediction result.

2. The multimodal motion trajectory prediction method according to claim 1, characterized in that: The pre-built multi-modal motion trajectory prediction model includes: Acquire a sample data set; wherein the sample data set includes at least obstacle trajectory information and high-precision map information; Extracting features from the sample data set to obtain historical movement trajectories of obstacles, historical movement trajectories of the vehicle, map lane structured features, and lane reference lines; Constructing an original trajectory prediction model; wherein the original trajectory prediction model at least includes a self-attention aggregator, a cross-attention fusion device, a map feature encoder constructed based on a graph-attention graph attention network, a trajectory feature encoder with a two-layer MLP structure, and a trajectory prediction decoder; The original trajectory prediction model is trained by using the obstacle historical motion trajectory, the vehicle historical motion trajectory, the lane line structured features and the lane reference line to obtain a trajectory prediction result and a trained trajectory prediction model; Performing a model evaluation on the trained trajectory prediction model according to the trajectory prediction result and the sample data set to obtain a model evaluation index; When it is determined according to the model evaluation index that the trained trajectory prediction model passes the performance evaluation, the trained trajectory prediction model is determined as a trained multimodal motion trajectory prediction model.

3. The multimodal motion trajectory prediction method according to claim 2, characterized in that: The obtaining of the sample data set comprises: Obtaining raw data; wherein the raw data includes at least the Argoverse data set and the record data packet collected by the actual vehicle; Parsing the original data to obtain parsed data; A sample data set is constructed based on the parsed data, wherein the sample data set includes at least obstacle trajectory information and high-precision map information; the obstacle trajectory information includes at least obstacle coordinate information within a historical time window, obstacle speed information within the historical time window, obstacle angle information within the historical time window, and obstacle size information; the high-precision map information includes at least sampling point coordinates of a lane centerline, lane turning information, traffic control information, and intersection information.

4. The multimodal motion trajectory prediction method according to claim 2, characterized in that: The feature extraction of the sample data set to obtain the obstacle historical motion trajectory, the vehicle historical motion trajectory, the map lane structured features and the lane reference line includes: Extracting features from the obstacle trajectory information to obtain a historical movement trajectory of the obstacle; wherein the historical movement trajectory of the obstacle at least includes obstacle trajectory coordinate points, obstacle speeds corresponding to the obstacle trajectory coordinate points, and obstacle directions corresponding to the obstacle trajectory coordinate points; Acquire lane line structured features according to the high-precision map information; wherein the lane line structured features at least include the coordinates of the road center point within a preset range, road turning information, intersection information, the successor relationship between lanes, and the left and right neighbor relationship of the lanes; Acquire lane line information and the historical movement trajectory of the vehicle according to the high-precision map information; A lane reference line of a vehicle obstacle is generated according to the lane line information and the historical motion trajectory of the vehicle.

5. The multimodal motion trajectory prediction method according to claim 2, characterized in that: The original trajectory prediction model is trained by using the obstacle historical motion trajectory, the vehicle historical motion trajectory, the lane line structured features and the lane reference line to obtain a trajectory prediction result and a trained trajectory prediction model, including: Performing feature encoding processing through the trajectory feature encoder, the obstacle historical motion trajectory, the vehicle historical motion trajectory and the lane line structured features to obtain a trajectory feature vector and a map feature vector; Performing feature fusion processing on the trajectory feature vector and the map feature vector to obtain a fused spatiotemporal feature vector; Decoding the spatiotemporal feature vector by the trajectory prediction decoder to obtain a candidate prediction trajectory; Decoding the candidate predicted trajectory by the trajectory prediction decoder to obtain a predicted refined trajectory; Generating a trajectory prediction result according to the candidate predicted trajectory and the predicted refined trajectory; Calculating a training loss based on the candidate predicted trajectory, the trajectory prediction result, and the lane reference line; The model parameters of the original trajectory prediction model are optimized according to the training loss to obtain a trained trajectory prediction model.

6. The multimodal motion trajectory prediction method according to claim 5, characterized in that: The feature encoding process is performed by the trajectory feature encoder, the obstacle historical motion trajectory, the vehicle historical motion trajectory and the lane line structured feature to obtain a trajectory feature vector and a map feature vector, including: The obstacle historical motion trajectory and the vehicle historical motion trajectory are spliced ​​to obtain the historical motion trajectory of the entire prediction scene; The historical motion trajectory is encoded by the trajectory feature encoder to obtain a trajectory feature vector; The lane line structured features are encoded by the map feature encoder to obtain a map feature vector.

7. The multimodal motion trajectory prediction method according to claim 5, characterized in that: The performing feature fusion processing on the trajectory feature vector and the map feature vector to obtain a fused spatiotemporal feature vector includes: Aggregating the trajectory feature vector through the self-attention aggregator to obtain a vehicle history time series feature vector; The vehicle historical time series feature vector and the map feature vector are cross-fused by the cross-attention fuser to obtain a feature vector of the interaction relationship between vehicle modeling and the road; Adding the interaction relationship feature vector to the vehicle history time series feature vector to obtain an added feature vector; The added feature vectors are aggregated through a self-attention aggregator to obtain a fused spatiotemporal feature vector.

8. The multimodal motion trajectory prediction method according to claim 5, characterized in that: The calculating the training loss according to the candidate predicted trajectory, the trajectory prediction result and the lane reference line includes: Obtaining the real trajectory of the obstacle according to the sample data set; Performing a smooth L1 loss calculation based on the candidate predicted trajectory and the real trajectory of the obstacle to obtain a first loss value; Performing a smooth L1 loss calculation based on the trajectory prediction result and the real trajectory of the obstacle to obtain a second loss value; Calculating regression loss according to the first loss value and the second loss value; Determining a predicted trajectory to be calculated having the smallest distance from the lane reference line from the trajectory prediction results; Calculating a lane loss according to the predicted trajectory to be calculated and the lane reference line; The regression loss and the lane loss are weighted and summed according to a preset weight value to obtain the training loss.

9. The multimodal motion trajectory prediction method according to claim 8, characterized in that: The performing model evaluation on the trained trajectory prediction model according to the trajectory prediction result and the sample data set to obtain a model evaluation index includes: Determining an optimal obstacle prediction trajectory and a trajectory probability of the optimal obstacle prediction trajectory according to the trajectory prediction result; Performing average Euclidean distance calculation based on the optimal obstacle prediction trajectory and the actual obstacle trajectory to obtain a minimum average displacement error; Calculating the Euclidean distance between the end point of the optimal obstacle prediction trajectory and the end point of the obstacle real trajectory to obtain the minimum final displacement error; Calculating an endpoint error according to the trajectory probability, the optimal obstacle predicted trajectory, and the obstacle true trajectory; Determining a corresponding candidate lane according to the optimal obstacle prediction trajectory; Calculating a lateral displacement error between an end point of the optimal obstacle prediction trajectory and a center line of the candidate lane; The minimum lane final displacement error is determined by taking the minimum value of the lateral displacement error; Calculating the MR value according to the endpoint error, a preset distance threshold, the optimal obstacle prediction trajectory and the obstacle real trajectory; The minimum average displacement error, the minimum final displacement error, the endpoint error, the minimum lane final displacement error and the MR value are summarized to obtain a model evaluation index.

10. A multimodal motion trajectory prediction device, characterized in that: The multimodal motion trajectory prediction device comprises: A model building unit, used to pre-build a multi-modal motion trajectory prediction model; A model deployment unit, used to deploy the multimodal motion trajectory prediction model to a target vehicle; An acquisition unit, used to acquire current map information and obstacle trajectory information of the target vehicle; A trajectory prediction unit, configured to predict the trajectory of an obstacle according to the current map information, the obstacle trajectory information and the multimodal motion trajectory prediction model, and obtain an obstacle trajectory prediction result; A decision determination unit is used to determine the current automatic driving decision of the target vehicle according to the obstacle trajectory prediction result.

Images