Multimodal trajectory prediction method based on data-driven modal representation and regional attention

Abstract

The present application relates to a multimodal trajectory prediction method based on data-driven modal representation and regional attention. This method redefines the standard for measuring the intensity of vehicle interaction as the intention of the target vehicle, and constructs an IRA module to allocate attention to surrounding vehicles located in different areas. The attention matrix is ​​used to weightedly aggregate interactions in different areas, which can model non-local interactions in a larger range without introducing noise generated by surrounding vehicles unrelated to the surrounding area; a DIM module is constructed, which comprehensively considers the modal commonality and individuality of trajectories of the same maneuver type, and further learns personalized high-dimensional modal representations of the trajectories in a data-driven manner, thereby guiding the model to output more accurate and reasonable multimodal trajectories.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a multimodal trajectory prediction method based on data-driven modal representation and regional attention. Background Art

[0002] Accurate and reliable vehicle trajectory prediction is one of the core functions of autonomous vehicles and the basis for high-quality local motion planning and decision-making. However, due to the complex interactions between vehicles and the diversity of maneuver categories, accurately predicting future trajectories is a challenging task.

[0003] In dynamic, dense traffic scenarios, a vehicle's motion is influenced not only by its own historical path but also by surrounding vehicles. For example, a target vehicle needs to consider surrounding vehicles to avoid collisions when attempting to change lanes. Therefore, modeling inter-vehicle interactions is crucial for trajectory prediction. However, choosing the appropriate scope for interaction modeling is a thorny issue. Some existing methods only consider surrounding vehicles close to the target vehicle, ignoring non-local interactions. This results in incomplete interaction representation and compromises trajectory prediction accuracy. Other methods model interactions by considering a wider range of surrounding vehicles to capture the full interaction. This causes information from irrelevant surrounding vehicles to become noise in the interaction representation, ultimately interfering with model prediction. Furthermore, influenced by individual driver characteristics, physiological, and psychological factors, a target vehicle may have multiple plausible maneuver options within the same traffic scenario. Even when performing the same maneuver, different vehicles may execute it differently. Predicting a unimodal trajectory results in multimodal averaging, leading to modal collapse. Consequently, researchers have conducted research on multimodal trajectory prediction. Most existing methods tend to use one-hot encoding as a modal prior for different maneuver categories to decode multimodal trajectories. However, one-hot encoding is a low-dimensional sparse vector that is difficult to effectively represent complex trajectory patterns. In addition, all trajectory modes with the same maneuver category are represented by the same one-hot encoding, ignoring the differences within the trajectory. Summary of the Invention

[0004] In order to overcome at least one deficiency in the prior art, the present application provides a multimodal trajectory prediction method based on data-driven modal representation and regional attention.

[0005] First, a multimodal trajectory prediction method based on data-driven modal representation and regional attention is provided, including:

[0006] Obtain the motion information of the target vehicle and its surrounding vehicles over a historical period of time; the motion information includes the position, velocity, and acceleration in the X direction, and the position, velocity, and acceleration in the Y direction;

[0007] The motion information of the target vehicle and its surrounding vehicles in the historical time period is input into the multimodal trajectory prediction model to obtain the multimodal trajectory prediction result of the target vehicle. The multimodal trajectory prediction model includes: the intention-based regional attention mechanism module IRA, the personalized modal representation module DIM, and the long short-term memory artificial neural network.

[0008] The intention-based regional attention mechanism module IRA is used to obtain the interaction between vehicles based on the motion information of the target vehicle and its surrounding vehicles in the historical time period;

[0009] The personalized modal representation module DIM is used to obtain the future modal representation of the target vehicle based on the motion information of the target vehicle and its surrounding vehicles in the historical time period;

[0010] The long short-term memory artificial neural network is used to obtain the multimodal trajectory prediction results of the target vehicle based on the interaction between vehicles, the future modal representation of the target vehicle and the motion information of the target vehicle in the historical time period.

[0011] In one embodiment, the intent-based regional attention mechanism module IRA includes: a two-dimensional multi-head attention mechanism 2D MHA, a convolution module, and a three-dimensional multi-head attention mechanism 3DMHA;

[0012] The target vehicle’s motion information in the historical time period is input into the two-dimensional multi-head attention mechanism 2D MHA to obtain the driver’s intention;

[0013] The motion information of the target vehicle and its surrounding vehicles in the historical time period is represented as a tensor;

[0014] The tensor is input into the convolution module for convolution operation to obtain the scene representation;

[0015] The driver's intention and scene representation are input into the 3D multi-head attention mechanism 3D MHA to obtain the interaction between vehicles.

[0016] In one embodiment, the driver’s intention and scene representation are input into a 3D multi-head attention mechanism (3D MHA) to obtain the interactions between vehicles, including:

[0017] The 3D multi-head attention mechanism (3D MHA) consists of multiple subspaces. For each subspace, the query vector Q is calculated based on the driver's intention; the key vector K and value vector V are calculated based on the scene representation.

[0018] The query vector Q and the key vector K are calculated through the softmax function to obtain the attention score matrix;

[0019] Multiply the attention score matrix and the value vector V to obtain the interaction representation corresponding to the subspace;

[0020] The interaction representations corresponding to all subspaces are weighted and aggregated to obtain the interaction between vehicles.

[0021] In one embodiment, the personalized modality representation module DIM includes: a historical modality representation module E H , historical personality representation calculation module, fitter, classifier, future modality representation calculation module;

[0022] The motion information of the target vehicle and its surrounding vehicles in the historical time period is input into the historical modal representation module E H , and get the historical modal representation R H ;

[0023] Represent the historical mode R H and historical commonality representation M H Input into the historical personality representation calculation module to obtain the historical personality representation σ H ; Historical personality representation σ H historical personality representations encompassing multiple modalities;

[0024] Representing historical personality H Input to the fitter to get the future personality representation

[0025] Representing future personality and future common representation M F Input into the future modal representation calculation module to obtain the future modal representation of the target vehicle Future modal representation of the target vehicle Future modal representations including multiple modalities;

[0026] The classifier is used to characterize R according to historical modalities H , and obtain the probability of the target vehicle executing multiple modes.

[0027] In one embodiment, the future commonality representation M F , obtained in the following way:

[0028] The training samples are divided into three subsets according to the lateral displacement within a set time period; the three subsets represent left turn, straight ahead and right turn respectively; each subset includes multiple samples;

[0029] For each subset, the future trajectory data of the sample is input into the future modality representation module E F , get the future modal representation corresponding to each sample;

[0030] Find the future modal representation R corresponding to all samples F The future common representation corresponding to each subset is obtained by taking the mean ofF .

[0031] In one embodiment, the historical commonality representation M H , obtained in the following way:

[0032] The training samples are divided into three subsets according to the lateral displacement within a set time period; the three subsets represent left turn, straight ahead and right turn respectively; each subset includes multiple samples;

[0033] For each subset, the historical trajectory data of the sample is input into the historical modality representation module E H , get the historical modal representation corresponding to each sample;

[0034] Find the historical modal representation R corresponding to all samples H The historical common representation corresponding to each subset is obtained by taking the mean of H .

[0035] In one embodiment, the long short-term memory artificial neural network includes a first long short-term memory artificial neural network unit and a second long short-term memory artificial neural network unit;

[0036] Inputting the target vehicle's motion information in the historical time period and the interaction between vehicles into the first long short-term memory artificial neural network unit to obtain an output result;

[0037] The output result and the future modal representation of the target vehicle are spliced ​​together, and the spliced ​​result is input into the second long short-term memory artificial neural network unit to output the multimodal trajectory prediction result of the target vehicle.

[0038] In a second aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multimodal trajectory prediction method based on data-driven modal representation and regional attention.

[0039] In a third aspect, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned multimodal trajectory prediction method based on data-driven modal representation and regional attention.

[0040] Compared with the prior art, this application has the following beneficial effects:

[0041] 1. This application redefines the criterion for measuring the intensity of vehicle interactions as the intention of the target vehicle, and constructs an IRA module to allocate attention to surrounding vehicles in different areas. The attention matrix is ​​used to weightedly aggregate interactions in different areas, which can model non-local interactions on a larger scale without introducing noise generated by unrelated surrounding vehicles.

[0042] 2. This application constructs a DIM module, which comprehensively considers the modal commonalities and individualities of trajectories of the same maneuver type, and further learns the personalized high-dimensional modal representation of the trajectory in a data-driven manner, thereby guiding the model to output more accurate and reasonable multimodal trajectories.

[0043] 3. This application conducted comparative experiments and ablation experiments on a real vehicle trajectory dataset to comprehensively evaluate the proposed method. The results show that the model constructed in this application has obvious advantages, especially in long-term prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present application may be better understood by referring to the following description in conjunction with the accompanying drawings, which together with the following detailed description are incorporated into and form a part of this specification. In the drawings:

[0045] Figure 1 A flowchart of a multimodal trajectory prediction method based on data-driven modal representation and regional attention according to an embodiment of the present application is shown;

[0046] Figure 2 shows the structural block diagram of the multimodal trajectory prediction model;

[0047] Figure 3 Shows the structural block diagram of the three-dimensional multi-head attention mechanism 3D MHA;

[0048] Figure 4 shows a structural block diagram of the personalized modality representation module DIM;

[0049] Figure 5 The experimental results of the ablation experiment on two datasets are shown, where (a) is the experimental result of the ablation experiment on the HighD dataset, and (b) is the experimental result of the ablation experiment on the NGSIM dataset;

[0050] Figure 6 The attention distribution diagrams of samples corresponding to the three maneuver categories are shown, where (a) is the attention distribution diagram of samples corresponding to the left-turn category, (b) is the attention distribution diagram of samples corresponding to the straight-ahead category, and (c) is the attention distribution diagram of samples corresponding to the right-turn category.

[0051] Figure 7 The performance comparison charts of different models on different data sets are shown, where (a) is the performance comparison chart of different models on the HighD data set, and (b) is the performance comparison chart of different models on the NGSIM data set;

[0052] Figure 8 A schematic diagram of the target vehicle's multimodal trajectory and the corresponding probabilities is shown. DETAILED DESCRIPTION

[0053] Exemplary embodiments of the present application are described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of actual embodiments are described in this specification. However, it should be understood that in the process of developing any such actual embodiment, many implementation-specific decisions may be made to achieve the developer's specific goals, and these decisions may vary from one implementation to another.

[0054] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show the device structure closely related to the solution according to the present application, while other details that are not closely related to the present application are omitted.

[0055] It should be understood that the present application is not limited to the described embodiments due to the following description with reference to the accompanying drawings. In this document, where feasible, the embodiments may be combined with each other, features between different embodiments may be replaced or borrowed, and one or more features may be omitted in one embodiment.

[0056] The present invention provides a multimodal trajectory prediction method based on data-driven modal representation and regional attention. Figure 1 A flowchart of a multimodal trajectory prediction method based on data-driven modal representation and regional attention according to an embodiment of the present application is shown. Figure 1 , methods include:

[0057] Step S11, obtaining motion information of the target vehicle and its surrounding vehicles in a historical time period; the motion information includes the position, velocity and acceleration in the X direction, and the position, velocity and acceleration in the Y direction.

[0058] Here, the motion information of the target vehicle and its surrounding vehicles at time t in the historical time period is represented by x t express, in, is the motion information of the target vehicle at time t, is the motion information of the surrounding N vehicles at time t.

[0059] Step S12: inputting the motion information of the target vehicle and its surrounding vehicles in the historical time period into the multimodal trajectory prediction model to obtain the multimodal trajectory prediction result of the target vehicle; Figure 2 The structural block diagram of the multimodal trajectory prediction model is shown in Figure 2 ,The multimodal trajectory prediction model includes : the intention-based regional attention mechanism module IRA, the ,individualized modal representation module DIM, and the long and short-term memory ,artificial neural network;

[0060] The intention-based regional attention mechanism module IRA is used to obtain the interaction between vehicles based on the motion information of the target vehicle and its surrounding vehicles in the historical time period;

[0061] The personalized modal representation module DIM is used to obtain the future modal representation of the target vehicle based on the motion information of the target vehicle and its surrounding vehicles in the historical time period;

[0062] The long short-term memory artificial neural network is used to obtain the multimodal trajectory prediction results of the target vehicle based on the interaction between vehicles, the future modal representation of the target vehicle and the motion information of the target vehicle in the historical time period.

[0063] This embodiment redefines the criterion for measuring the intensity of vehicle interactions as the target vehicle's intention and constructs an IRA module to distribute attention to surrounding vehicles in different areas. The attention matrix is ​​used to weight and aggregate interactions in different areas, allowing for modeling of non-local interactions on a larger scale without introducing noise from unrelated surrounding vehicles. This embodiment constructs a DIM module that comprehensively considers the modal commonalities and individuality of trajectories of the same maneuver type. It further learns personalized high-dimensional modal representations of trajectories in a data-driven manner, thereby guiding the model to output more accurate and reasonable multimodal trajectories.

[0064] In one embodiment, to achieve accurate trajectory prediction in dynamic traffic, it is necessary to model the complex interactions between the target vehicle and its surrounding vehicles. Using the proposed intent-based regional attention mechanism (IRA), attention is allocated to different regions of the traffic scene guided by the target vehicle's intent. This mechanism allows for weighted aggregation of inter-vehicle interactions in different regions, enabling the modeling of larger-scale non-local interactions without introducing noise, resulting in inter-vehicle interactions (IVIs). The functions of each IRA module are described below.

[0065] The intention-based regional attention mechanism module IRA includes: two-dimensional multi-head attention mechanism 2D MHA, convolution module, and three-dimensional multi-head attention mechanism 3D MHA.

[0066] Here, the target vehicle’s motion information in the historical time period is input into the two-dimensional multi-head attention mechanism 2DMHA to obtain the driver’s intention INT; the driver’s intention corresponding to the motion information at time t is INT t express.

[0067] The motion information of the target vehicle and its surrounding vehicles in the historical time period is represented as a tensor F, where the motion information at time t is represented as a tensor F t , where the tensor F t The representation is as follows:

[0068]

[0069] Where (m,n) represents the tensor F t The grid position of represents the motion information of vehicle k at time t, represents the position of vehicle k at time t, represents the X-direction position of vehicle k at time t, represents the Y-direction position of vehicle k at time t, represents the speed of vehicle k in the X direction at time t, represents the speed of vehicle k in the Y direction at time t, represents the acceleration of vehicle k in the X direction at time t, represents the acceleration of vehicle k in the Y direction at time t. mn (·) is an indicator function if In the grid (m, n), its value is 1, otherwise it is 0.

[0070] The tensor F is input into the convolution module for convolution operation to obtain the scene representation SR; here, the convolution module includes two full convolution layers.

[0071] The driver intention INT and scene representation SR are input into the three-dimensional multi-head attention mechanism 3DMHA to obtain the vehicle-to-vehicle interaction IVI.

[0072] In one embodiment, Figure 3 The structural diagram of the three-dimensional multi-head attention mechanism 3D MHA is shown in Figure 3 , the vehicle-to-vehicle interaction IVI can be obtained in the following way:

[0073] The three-dimensional multi-head attention mechanism 3D MHA includes multiple subspaces. First, for each subspace, the query vector Q is calculated according to the driver's intention: the key vector K and value vector V are calculated according to the scene representation;

[0074]

[0075]

[0076]

[0077] in, and Both represent linear transformations, Int t represents the driver's intention at time t, W i Q ,W i K and W iV Denotes the linear transformation corresponding to subspace i and The learnable parameter matrix, represents the query vector at time t in subspace i, represents the key vector at time t in subspace i, represents the value vector at time t in subspace i, SR t Represents the scene representation at time t.

[0078] Then, the query vector Q and the key vector K are calculated through the softmax function to obtain the attention score matrix:

[0079]

[0080] in, is the attention score matrix at time t in subspace i, which is used to measure the driver’s attention to different areas under intention guidance. i is the scale factor in subspace i, used to represent dimension.

[0081] Finally, the attention score matrix is ​​multiplied by the value vector V to obtain the interaction representation corresponding to the subspace; the interaction representations corresponding to all subspaces are weighted and aggregated to obtain the interaction between vehicles. It is expressed by the following formula:

[0082]

[0083]

[0084] in, is the interaction representation in subspace i at time t, n is the number of subspaces, which can be 4, IVI t is the interaction between vehicles at time t, W O1 and W O2 are model parameters.

[0085] In one embodiment, a personalized modal representation module (DIM) was constructed to address the modal collapse problem in unimodal trajectory prediction. This module can learn a personalized high-dimensional modal representation for each trajectory in a data-driven manner. Furthermore, for trajectories of the same maneuver category (modality), the modal representation takes into account both commonality and individuality, enabling more accurate multimodal trajectory prediction. Figure 4 The structural diagram of the personalized modality representation module DIM is shown in FIG. Figure 4 , personalized modal representation module DIM, including: historical modal representation module E H , historical personality representation calculation module, fitter, classifier, future modality representation calculation module;

[0086] The motion information of the target vehicle and its surrounding vehicles in the historical time period is input into the historical modal representation module E H , and get the historical modal representation R H ;

[0087] Represent the historical mode R H and historical commonality representation M H Input into the historical personality representation calculation module to obtain the historical personality representation σ H ; Among them, σ H =R H -M H Here, the historical commonality represents M H It includes three modes, including left turn, straight ahead and right turn. Therefore, the historical personality representation σ H It also includes three modes of historical personality representation.

[0088] Representing historical personality H Input to the fitter Fit to obtain the future personality representation

[0089] Representing future personality and future common representation M F Input into the future modal representation calculation module to obtain the future modal representation of the target vehicle Here, the future commonality represents M F Including 3 modalities, future personality representation It also includes 3 modes, including left turn, straight ahead and right turn. Therefore, the future modal representation of the target vehicle Includes future modal representation of 3 modalities.

[0090] The classifier is used to characterize R according to historical modalities H , obtaining the probability of the target vehicle executing multiple modes. The different values ​​of the probability of multiple modes guide the model to output the future trajectory of the target vehicle in different modes (left turn, straight, right turn).

[0091] In the above embodiment, in order to obtain a more predictive deep representation from the vehicle's historical and future motion information, the historical modal representation module E is firstly trained using supervised learning and self-supervised learning. H and future modal representation module E F Train and use supervised learning and self-supervised learning to encode and decode the vehicle's historical and future motion information. H and E F Computing the future common representation M F and historical commonality representation M H, save them as known input data of the personalized modal representation module DIM. F You can obtain it in the following ways:

[0092] First, according to the set time period [T h +1,T h +T f ] lateral displacement within The training samples are divided into three subsets C l ,C k and C r , where T h +1 is the start time of the set time period, T h +T f To set the end time of the time period, is time T h +T f The position in the Y direction, is time T h +1 in the Y direction, C l ,C k and C r They are subsets of training samples that meet the conditions Δ<-1.5m, |Δ|<1.5m, and Δ>1.5m. The three subsets represent left turn, straight ahead, and right turn, respectively. Each subset includes multiple samples, and each sample includes historical trajectory data and future trajectory data. Figure 4 , each subset includes 2 samples.

[0093] Then, for each subset, the future trajectory data of the sample is input into the future modality representation module E F , get the future modal representation corresponding to each sample;

[0094] Then, find the future modal representation R corresponding to all samples F The future common representation corresponding to each subset is obtained by taking the mean of F Here, M F It includes future common representations corresponding to three subsets.

[0095] Among them, the historical commonality represents M H , which can be obtained in the following ways:

[0096] The training samples are divided into three subsets according to the lateral displacement within a set time period; the three subsets represent left turn, straight ahead and right turn respectively; each subset includes multiple samples;

[0097] For each subset, the historical trajectory data of the sample is input into the historical modality representation module E H, get the historical modal representation corresponding to each sample;

[0098] Find the historical modal representation R corresponding to all samples H The historical common representation corresponding to each subset is obtained by taking the mean of H .

[0099] In one embodiment, the long short-term memory artificial neural network includes a first long short-term memory artificial neural network unit LSTM and a second long short-term memory artificial neural network unit LSTM;

[0100] Inputting the target vehicle's motion information in the historical time period and the interaction between vehicles into the first long short-term memory artificial neural network unit to obtain an output result;

[0101] The output is concatenated with the future modal representation of the target vehicle and fed into a second long short-term memory artificial neural network unit to output a multimodal trajectory prediction for the target vehicle. The multimodal trajectory prediction for the target vehicle includes trajectory predictions for the target vehicle in multiple modes, which may include left turns, going straight, and right turns.

[0102] Furthermore, the multimodal trajectory prediction model mentioned above is a multimodal trajectory prediction model obtained after training. During the model training process, two public vehicle trajectory datasets, NGSIM and HighD datasets, are used to train the model. The NGSIM dataset comes from the Next Generation Simulation Research Project initiated by the Federal Highway Administration of the United States. The dataset is derived from image information extracted by digital cameras and contains data on all vehicles passing through the US-101 and I-80 highways within 45 minutes. The HighD dataset contains 11.5 hours of measurements from 6 locations with a sampling frequency of 25Hz, recording data of 110,000 vehicles and 45,000 kilometers of mileage. The dataset is divided into training and test sets in proportion, and the original dataset is downsampled. Specifically, the NGSIM dataset is downsampled by 2 times, while the HighD dataset is downsampled by 5 times. The processed dataset uses 40 frames (i.e., 8 seconds) as a test sample, and the first 15 frames (i.e., 3 seconds) are the historical trajectory sequence T h =15, the next 25 frames (ie 5 seconds) are the future trajectory sequence T f =25.

[0103] An embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, a multimodal trajectory prediction method based on data-driven modal representation and regional attention is implemented.

[0104] An embodiment of the present application provides a computer program product, including a computer program / instruction, which implements a multimodal trajectory prediction method based on data-driven modal representation and regional attention when the computer program / instruction is executed by a processor.

[0105] To verify the effectiveness of the method of this application, experimental verification and evaluation were carried out on two publicly available datasets, HighD and NGSIM. The NGSIM dataset comes from the next generation simulation research project initiated by the Federal Highway Administration of the United States. The dataset is derived from image information extracted by digital cameras and contains data on all vehicles passing through the US-101 and I-80 highways within 45 minutes. The complete HighD dataset was captured at a frequency of 25Hz for 1.65 hours at each of the six locations, including 110,000 vehicles with a mileage of 45,000 kilometers and 5,600 complete lane changes. The processed data is tested in 40 frames (ie 8 seconds), and the first 15 frames (ie 3 seconds) are the historical trajectory sequence T h =15, the next 25 frames (ie 5 seconds) are the future trajectory sequence T f =25.

[0106] To test the model's effectiveness, this application uses the root mean square error (RMSE) to verify the model results. The RMSE formula is shown below.

[0107]

[0108] Where m is the number of samples, and are the true position and predicted position of sample i at time t. h 、T f is the length of the historical trajectory sequence and the length of the future trajectory sequence. RMSE measures the average size of the displacement error between the predicted position and the true position within the prediction time range. A larger RMSE indicates a larger error.

[0109] First, to verify the effectiveness of the component modules, we designed an ablation experiment. In this ablation experiment, the RD-Net model was decomposed into three modules: LSTM, IRA, and DIM, to demonstrate the effective improvement in prediction accuracy achieved by each module. Specifically, we considered the following models and tested them under the same parameter settings.

[0110] +IRA(k2): This model is composed of LSTM+IRA. The effectiveness of IRA is verified by comparing it with LSTM. Different values ​​of k2 in k2∈{1,2,3} constitute different models, which can be used to explore the impact of the size of the region represented by each vector in SR on the model performance. The output of this model is a single-mode trajectory.

[0111] +DIM: This model is a combination of LSTM and DIM, used to verify the effectiveness of DIM compared to LSTM. This model outputs multimodal trajectories based on different lateral maneuver categories. To quantitatively describe the improvement brought by each module, the following formula is used:

[0112]

[0113] Among them, m s and m are the performance evaluations of LSTM with and without module S, respectively. s Indicates the performance improvement brought by module S.

[0114] Figure 5 The experimental results of the ablation experiment on two datasets are shown, where (a) is the experimental result of the ablation experiment on the HighD dataset, and (b) is the experimental result of the ablation experiment on the NGSIM dataset. Figure 5 As can be seen, the overall performance of the DIM model is higher than that of the IRA model, indicating that unimodal trajectory prediction leads to mode collapse, severely limiting prediction accuracy. Therefore, as long as DIM is used for multimodal trajectory prediction, model performance can be greatly improved, even without IRA for inter-vehicle interaction modeling. Second, the prediction performance of the model with IRA is affected by the size of the convolution kernel k2. When k2 = 1, the region-level attention mechanism degenerates into a target-level attention mechanism, assigning attention weights to each surrounding vehicle rather than to each region in the traffic scene. When k2 = 2, the overall model performance is optimal. This is because the future trajectory of the target vehicle is influenced not only by its surrounding vehicles but also by the surrounding drivable space without vehicles. Therefore, when k2 = 2, the scene representation (SR) incorporates information about the surrounding vehicles and the drivable space, which is beneficial for improving the model's prediction accuracy. Finally, the DIM and IRA modules overall bring more significant improvements on the HighD dataset, as noise in the NGSIM dataset interferes with the modal representation and modeling of inter-vehicle interactions. In addition, the performance improvement brought by IRA almost increases with the increase of prediction horizon, which indicates that non-local cross-vehicle interactions have a more significant impact on long-term trajectory prediction and the positive effect of IRA is more obvious in long-term prediction.

[0115] In addition, in order to intuitively show the regional attention distribution guided by the target vehicle intention, the attention distribution of some samples corresponding to the three maneuver categories is visualized. Figure 6 The attention distribution diagrams of samples corresponding to the three maneuver categories are shown, where (a) is the attention distribution diagram of samples corresponding to the left-turn category, (b) is the attention distribution diagram of samples corresponding to the straight-ahead category, and (c) is the attention distribution diagram of samples corresponding to the right-turn category. Figure 6As can be seen, when the target vehicle tends to turn left or right, it pays more attention to surrounding vehicles in the target lane; when the target vehicle tends to maintain a straight line, it pays more attention to itself and nearby vehicles. The visualization results show that vehicle-to-vehicle interactions are determined not only by vehicle distance but also by the target vehicle's intention. The IRA module effectively weights and aggregates interactions guided by the target vehicle's intention.

[0116] To verify the performance of the model, the RMSE index of the proposed model RD-Net and the existing models Convolutional Social-LSTM (CS-LSTM), Social-GAN (S-LSTM) and PIP1 within 5 seconds was visualized. Figure 7 The performance comparison charts of different models on different datasets are shown, where (a) is the performance comparison chart of different models on the HighD dataset, and (b) is the performance comparison chart of different models on the NGSIM dataset. Figure 7 It can be seen that the model RD-Net of the present application outperforms almost all existing models, which shows that the non-local vehicle-to-vehicle interactions modeled by IRA and the personalized future modal representation learned by DIM greatly improve the model performance. On the HighD dataset, when the prediction horizon is 1 second, the performance of RD-Net is slightly lower than that of the optimal model, but RD-Net works better in long-term predictions (3-5 seconds). For the NGSIM dataset, when the prediction horizon is 1 second, the performance of RD-Net is comparable to the best performance of these comparison models, and RD-Net outperforms all other models in other prediction horizons (2-5 seconds). In addition, the prediction performance of all models on HighD is significantly better than that on NGSIM. As the prediction horizon increases, the performance advantage of RD-Net increases. This is because non-local interactions and trajectory modalities have a greater impact on the future trajectory of the target vehicle in long-term predictions. Therefore, the performance improvement brought by RD-Net's modules IRA and DIM is more significant in long-term predictions.

[0117] At the same time, in order to qualitatively describe the prediction performance of the proposed model, the prediction results of the RD-Net model of this application are visualized. Figure 8 The target vehicle multimodal trajectory and the corresponding probability diagram are shown. Figure 8 It can be seen that even in congested traffic scenarios, RD-Net can accurately and smoothly predict the multimodal trajectories and corresponding probabilities of target vehicles, which intuitively illustrates the excellent performance of RD-Net.

[0118] In summary, this application has the following beneficial effects:

[0119] 1. This application redefines the criterion for measuring the intensity of vehicle interactions as the intention of the target vehicle, and constructs an IRA module to allocate attention to surrounding vehicles in different areas. The attention matrix is ​​used to weightedly aggregate interactions in different areas, which can model non-local interactions on a larger scale without introducing noise generated by unrelated surrounding vehicles.

[0120] 2. This application constructs a DIM module, which comprehensively considers the modal commonalities and individualities of trajectories of the same maneuver type, and further learns the personalized high-dimensional modal representation of the trajectory in a data-driven manner, thereby guiding the model to output more accurate and reasonable multimodal trajectories.

[0121] 3. This application conducted comparative experiments and ablation experiments on a real vehicle trajectory dataset to comprehensively evaluate the proposed method. The results show that the model constructed in this application has obvious advantages, especially in long-term prediction.

[0122] The above descriptions are merely examples of various embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multimodal trajectory prediction method based on data-driven modal representation and regional attention, characterized by: include: Obtaining motion information of the target vehicle and its surrounding vehicles over a historical period; the motion information includes position, velocity, and acceleration in the X direction, and position, velocity, and acceleration in the Y direction; Inputting the motion information of the target vehicle and its surrounding vehicles in a historical time period into a multimodal trajectory prediction model to obtain a multimodal trajectory prediction result of the target vehicle; The multimodal trajectory prediction model includes: an intention-based regional attention mechanism module IRA, a personalized modal representation module DIM, and a long short-term memory artificial neural network; The intention-based regional attention mechanism module IRA is used to obtain the interaction representation between vehicles based on the motion information of the target vehicle and its surrounding vehicles in a historical time period; The personalized modal representation module DIM is used to obtain a future modal representation of the target vehicle based on the motion information of the target vehicle and its surrounding vehicles in a historical time period; The long short-term memory artificial neural network is used to obtain a multimodal trajectory prediction result of the target vehicle based on the interaction representation between the vehicles, the future modal representation of the target vehicle, and the motion information of the target vehicle in a historical time period; The intention-based regional attention mechanism module IRA includes: a two-dimensional multi-head attention mechanism 2D MHA, a convolution module, and a three-dimensional multi-head attention mechanism 3D MHA; Input the target vehicle's motion information in the historical time period into the two-dimensional multi-head attention mechanism 2D MHA to obtain the driver's intention; Representing the motion information of the target vehicle and its surrounding vehicles in a historical period as a tensor; Inputting the tensor into the convolution module for convolution operation to obtain scene representation; The driver intention and the scene representation are input into the three-dimensional multi-head attention mechanism 3D MHA to obtain the interaction representation between vehicles.

2. The method according to claim 1, wherein in, The driver intention and the scene representation are input into the 3D multi-head attention mechanism 3D MHA to obtain the interaction representation between vehicles, including: The three-dimensional multi-head attention mechanism 3D MHA includes multiple subspaces. For each subspace, a query vector Q is calculated according to the driver's intention; a key vector K and a value vector V are calculated according to the scene representation; The query vector Q and the key vector K are calculated by the softmax function to obtain an attention score matrix; Multiplying the attention score matrix and the value vector V to obtain the interaction representation corresponding to the subspace; The interaction representations corresponding to all subspaces are weighted and aggregated to obtain the interaction representation between the vehicles.

3. The method according to claim 1, wherein The personalized modality representation module DIM includes: a historical modality representation module , historical personality representation calculation module, fitter, classifier, future modality representation calculation module; The motion information of the target vehicle and its surrounding vehicles in the historical time period is input into the historical modal characterization module , and obtain the historical modal representation ; Representing historical modalities and historical commonalities Input into the historical personality representation calculation module to obtain the historical personality representation ; The historical personality representation historical personality representations encompassing multiple modalities; Representing the historical personality Input into the fitter to obtain future personality representation ; Representing future personality and future common representations Input into the future modal representation calculation module to obtain the future modal representation of the target vehicle ; Future modal representation of the target vehicle Future modal representations including multiple modalities; The classifier is used to represent the historical modality , and obtain the probability of the target vehicle executing multiple modes.

4. The method according to claim 3, wherein The future common representation , obtained in the following way: The training samples are divided into three subsets according to the lateral displacement within a set time period; the three subsets represent left turn, straight ahead and right turn respectively; each subset includes multiple samples; For each subset, the future trajectory data of the sample is input into the future modality representation module , get the future modal representation corresponding to each sample; Find the future modal representation corresponding to all samples The future common representation corresponding to each subset is obtained by taking the mean of .

5. The method according to claim 3, wherein The historical commonality , obtained in the following way: The training samples are divided into three subsets according to the lateral displacement within a set time period; the three subsets represent left turn, straight ahead and right turn respectively; each subset includes multiple samples; For each subset, the historical trajectory data of the sample is input into the historical modal representation module , get the historical modal representation corresponding to each sample; Find the historical modal representation corresponding to all samples The historical commonality representation corresponding to each subset is obtained by taking the mean of .

6. The method according to claim 1, wherein The long short-term memory artificial neural network comprises a first long short-term memory artificial neural network unit and a second long short-term memory artificial neural network unit; Inputting the motion information of the target vehicle in the historical time period and the interaction representation between the vehicles into the first long short-term memory artificial neural network unit to obtain an output result; The output result and the future modal representation of the target vehicle are spliced ​​together, and the spliced ​​result is input into the second long short-term memory artificial neural network unit to output the multimodal trajectory prediction result of the target vehicle.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multimodal trajectory prediction method based on data-driven modal representation and regional attention as described in any one of claims 1 to 6.

8. A computer program product, characterized in that The invention comprises a computer program / instruction, which, when executed by a processor, implements the multimodal trajectory prediction method based on data-driven modal representation and regional attention as described in any one of claims 1 to 6.

Images