Multimodal vehicle trajectory prediction model training method, device and trajectory prediction method

Abstract

The present application relates to a multimodal vehicle trajectory prediction model training method, device and trajectory prediction method. The model training method includes dividing the vehicle motion trajectory training data into multiple categories; performing weight iterative pruning on the training data of each category based on the decoder module to obtain a mask matrix corresponding to each category; training the encoder module and the decoder module based on the training data of each category and the mask matrix corresponding to each category to obtain a trained multimodal vehicle trajectory prediction model that can output multiple trajectory prediction results corresponding to multiple categories. The multimodal vehicle trajectory prediction model training method of the present application adopts a data-driven approach, fully considers different driving operation intentions, combines the vehicle's historical motion trajectory state information and the characteristics of different driving operation intentions, and trains the multimodal vehicle trajectory prediction model. The generated vehicle trajectory prediction model can output multiple reasonable predicted trajectories.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and specifically to a multimodal vehicle trajectory prediction model training method, device, and trajectory prediction method. Background Art

[0002] Accurate trajectory prediction is a prerequisite for autonomous vehicles to achieve autonomous path planning and avoid driving risks in road traffic systems, providing a strong guarantee for their safe operation. Accurately predicting the motion trajectories of surrounding vehicles is a major challenge in the transportation sector. Model-driven trajectory prediction methods focus on vehicles or pedestrians, using kinematic or statistical physics methods, such as Kalman filtering. However, these methods fail to consider the impact of other traffic participants on the subject and fail to fully utilize large amounts of real-world vehicle data. With the advancement of deep learning, data-driven methods, such as recurrent neural networks (RNNs), enable models to automatically learn vehicle motion characteristics and predict future trajectories. Long-short-term memory networks (LSTMs) have limited long-term prediction capabilities, and their prediction error increases dramatically with increasing prediction time. Transformer models, while offering strong long-term prediction capabilities, suffer from a large number of model parameters and computational complexity, making them overly complex and a major drawback. Furthermore, these models lack a comprehensive understanding of driving behavior. Specifically, under the same driving situation, the target vehicle may adopt different driving intentions. Therefore, generating only a single predicted trajectory significantly impacts prediction accuracy. Summary of the Invention

[0003] In order to overcome at least one deficiency in the prior art, embodiments of the present application provide a multimodal vehicle trajectory prediction model training method, device, and trajectory prediction method.

[0004] In a first aspect, a multimodal vehicle trajectory prediction model training method is provided, comprising:

[0005] The vehicle motion trajectory training data is divided into multiple categories, including left deviation, right deviation and straight travel;

[0006] The multimodal vehicle trajectory prediction model includes an encoder module and a decoder module. The decoder module performs iterative weight pruning on the training data of each category to obtain multiple mask matrices corresponding to each category.

[0007] The encoder module and decoder module are trained based on the training data of each category and multiple mask matrices corresponding to each category to obtain a trained multimodal vehicle trajectory prediction model. The trained multimodal vehicle trajectory prediction model can output multiple trajectory prediction results corresponding to multiple categories.

[0008] In one embodiment, weights of the training data of each category are iteratively pruned based on the decoder module to obtain multiple mask matrices corresponding to each category, including:

[0009] Initialize the network parameters of the decoder module and perform multiple rounds of pre-training on the decoder module using vehicle trajectory training data to obtain the base network;

[0010] The training data of each category is input into the base network separately, and multiple rounds of pre-training are performed to obtain the subnet corresponding to each category;

[0011] The weight parameters of multiple fully connected layers of the subnet corresponding to each category are iteratively pruned to determine multiple mask matrices corresponding to each category.

[0012] In one embodiment, the encoder module and the decoder module are trained based on the training data of each category and the multiple mask matrices corresponding to each category to obtain a trained multimodal vehicle trajectory prediction model, including:

[0013] The training process is repeated multiple times. Each training includes: inputting the training data of each category into the encoder module to obtain the high-dimensional space vector corresponding to each category; inputting the training data of each category, the corresponding multiple mask matrices and the corresponding high-dimensional space vector into the decoder module to obtain the trajectory prediction result corresponding to each category;

[0014] After each training session, the multimodal vehicle trajectory prediction model is tested based on the vehicle motion trajectory test data. When the multimodal vehicle trajectory prediction model meets the test requirements, the training ends and the trained multimodal vehicle trajectory prediction model is obtained.

[0015] In one embodiment, the training data of each category is input into the encoder module to obtain the high-dimensional space vector corresponding to each category, including:

[0016] The encoder module consists of a multi-head attention module, a residual connection layer, a normalization layer, and a feedforward neural network connected in sequence;

[0017] For each category, the training data is input into the multi-head attention module to obtain the weight distribution of the Value vector;

[0018] The weight distribution of the Value vector and the training data are input into the residual connection layer for residual connection, and the result of the residual connection is input into the normalization layer for normalization to obtain the normalized vector;

[0019] Input the normalized vector into the feedforward neural network to obtain the dimension-reduced vector;

[0020] The reduced-dimensional vector and the normalized vector are input into the residual connection layer for residual connection, and the result of the residual connection is input into the normalization layer for normalization to obtain the high-dimensional space vector corresponding to the category.

[0021] In one embodiment, the training data of each category, the corresponding multiple mask matrices, and the corresponding high-dimensional space vector are input into the decoder module to obtain the trajectory prediction results corresponding to each category, including:

[0022] The decoder module includes a masked multi-head attention module, a multi-head attention module, a residual connection layer, a normalization layer, a feedforward neural network, and a fully connected layer network;

[0023] For each category, the training data and multiple mask matrices are input into the masked multi-head attention module to obtain the first weight distribution of the Value vector;

[0024] Input the first weight distribution of the Value vector and the training data into the residual connection layer for residual connection, and input the result of the residual connection into the normalization layer for normalization to obtain the first normalized vector;

[0025] Input the first normalized vector, high-dimensional space vector and multiple mask matrices into the multi-head attention module to obtain the second weight distribution of the Value vector;

[0026] The second weight distribution of the Value vector and the first normalized vector are input into the residual connection layer for residual connection, and the result of the residual connection is input into the normalization layer for normalization to obtain the second normalized vector;

[0027] Input the second normalized vector into the feedforward neural network to obtain a vector after dimensionality reduction;

[0028] The reduced-dimensional vector and the second normalized vector are input into the residual connection layer for residual connection, and the result of the residual connection is input into the normalization layer for normalization to obtain the third normalized vector;

[0029] The third normalized vector is input into the fully connected layer network to obtain the trajectory prediction result corresponding to the category.

[0030] In one embodiment, the multimodal vehicle trajectory prediction model is tested based on vehicle motion trajectory test data. When the multimodal vehicle trajectory prediction model meets the test requirements, the training ends, including:

[0031] The vehicle motion trajectory test data includes multiple categories of test data; each category of test data is input into the encoder module to obtain the high-dimensional space vector corresponding to each category;

[0032] Input the test data of each category, the corresponding multiple mask matrices and the corresponding high-dimensional space vector into the decoder module to obtain the trajectory prediction result corresponding to each category;

[0033] Perform linear transformation and Softmax function processing on the high-dimensional space vector corresponding to each category to obtain the distribution probability of each category;

[0034] Calculate the error between the trajectory prediction result and the actual trajectory result corresponding to the category with the largest distribution probability;

[0035] If the error is within the set range, the multimodal vehicle trajectory prediction model meets the test requirements and the training ends.

[0036] In a second aspect, a multimodal vehicle trajectory prediction model training device is provided, comprising:

[0037] A data classification unit is used to classify the vehicle motion trajectory training data into multiple categories, including left deviation, right deviation and straight travel;

[0038] The mask matrix acquisition unit, used in the multimodal vehicle trajectory prediction model, includes an encoder module and a decoder module. The weights of the training data of each category are iteratively pruned based on the decoder module to obtain multiple mask matrices corresponding to each category.

[0039] The training unit is used to train the encoder module and the decoder module based on the training data of each category and the multiple mask matrices corresponding to each category to obtain a trained multimodal vehicle trajectory prediction model. The trained multimodal vehicle trajectory prediction model can output multiple trajectory prediction results corresponding to multiple categories.

[0040] In one embodiment, the mask matrix obtaining unit is further configured to:

[0041] Initialize the network parameters of the decoder module and perform multiple rounds of pre-training on the decoder module using vehicle trajectory training data to obtain the base network;

[0042] The training data of each category is input into the base network separately, and multiple rounds of pre-training are performed to obtain the subnet corresponding to each category;

[0043] The weight parameters of multiple fully connected layers of the subnet corresponding to each category are iteratively pruned to determine multiple mask matrices corresponding to each category.

[0044] In one embodiment, the training unit is further configured to:

[0045] The training process is repeated multiple times. Each training includes: inputting the training data of each category into the encoder module to obtain the high-dimensional space vector corresponding to each category; inputting the training data of each category, the corresponding multiple mask matrices and the corresponding high-dimensional space vector into the decoder module to obtain the trajectory prediction result corresponding to each category;

[0046] After each training session, the multimodal vehicle trajectory prediction model is tested based on the vehicle motion trajectory test data. When the multimodal vehicle trajectory prediction model meets the test requirements, the training ends and the trained multimodal vehicle trajectory prediction model is obtained.

[0047] In a third aspect, a multimodal vehicle trajectory prediction method is provided, comprising:

[0048] Input the trajectory sequence to be predicted into the multimodal vehicle trajectory prediction model and output multiple trajectory prediction results;

[0049] The multimodal vehicle trajectory prediction model is obtained according to the multimodal vehicle trajectory prediction model training method described above.

[0050] Compared with the prior art, this application has the following beneficial effects:

[0051] 1. Using a data-driven approach, we fully consider different driving intentions, combine the vehicle's historical motion trajectory state information and the characteristics of different driving intentions, and perform multimodal trajectory prediction through a sparse weight sharing decoder to generate multiple reasonable predicted trajectories.

[0052] 2. Design an iterative weight pruning unit to perform parameter sparsification on the self-attention mechanism in the Transformer. Through multiple iterative pruning steps, different subnet masks are generated for data with different driving intentions, making their structure adaptable to the high-dimensional feature space of each category's dataset. This allows the extraction of hidden information from different categories of trajectories and enables the accurate prediction of future trajectories to a high degree.

[0053] 3. A parallel training strategy is adopted to feed datasets of different driving operation intentions into corresponding subnets for training. This ensures that some parameters in the network are updated only in specific subnets. This reduces the computational overhead of the model while ensuring model efficiency, improves the accuracy of trajectory prediction, and enables accurate and reliable vehicle trajectory prediction in complex traffic scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present application may be better understood by referring to the following description in conjunction with the accompanying drawings, which together with the following detailed description are incorporated into and form a part of this specification. In the drawings:

[0055] Figure 1A flowchart of a multimodal vehicle trajectory prediction model training method according to an embodiment of the present application is shown;

[0056] Figure 2 A schematic diagram of a multimodal vehicle trajectory prediction model training method according to an embodiment of the present application is shown;

[0057] Figure 3 The following is a structural block diagram of a multimodal vehicle trajectory prediction model training device according to an embodiment of the present application;

[0058] Figure 4 A diagram showing a comparison of the root mean square error between the multimodal vehicle trajectory prediction model of the present application and the existing model;

[0059] Figure 5 The predicted result diagram of the vehicle moving forward and deviating to the left is shown;

[0060] Figure 6 The figure shows the predicted results when the vehicle is traveling in the opposite direction and deviating to the right. DETAILED DESCRIPTION

[0061] Exemplary embodiments of the present application are described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of actual embodiments are described in this specification. However, it should be understood that in the process of developing any such actual embodiment, many implementation-specific decisions may be made to achieve the developer's specific goals, and these decisions may vary from one implementation to another.

[0062] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show the device structure closely related to the solution according to the present application, while other details that are not closely related to the present application are omitted.

[0063] It should be understood that the present application is not limited to the described embodiments due to the following description with reference to the accompanying drawings. In this document, where feasible, the embodiments may be combined with each other, features between different embodiments may be replaced or borrowed, and one or more features may be omitted in one embodiment.

[0064] The embodiments of the present application provide a multimodal vehicle trajectory prediction model training method, device and trajectory prediction method. During the model training process, a data category division unit based on unsupervised clustering is designed. The clustering algorithm is used to screen out data sets of different driving operation categories through the difference in the vehicle's longitudinal coordinate displacement. In order to reduce the computational complexity of the model and improve operational efficiency, a mask matrix acquisition unit is designed to perform parameter sparsification processing on the fully connected layer for calculating Q, K, and V for the self-attention mechanism in the decoder, generate corresponding multiple mask matrices for the multi-category data set, and obtain sub-networks for each category. Based on the Transformer encoder module and the sparse weight sharing decoder module, the generated modal vehicle trajectory prediction model can output multiple reasonable predicted trajectories.

[0065] Figure 1 A flowchart of a multimodal vehicle trajectory prediction model training method according to an embodiment of the present application is shown. In the multimodal vehicle trajectory prediction model training method, the multimodal vehicle trajectory prediction model includes an encoder module and a decoder module, including:

[0066] Step S11, dividing the vehicle motion trajectory training data into multiple categories, including left deviation, right deviation and straight travel;

[0067] In this step, the vehicle trajectory dataset of German highways (HighD dataset) is used. The HighD dataset includes 11.5 hours of measurement values ​​from 6 locations with a sampling frequency of 25 Hz. It records data of 110,000 vehicles and 45,000 kilometers of mileage.

[0068] The difference Δy between the vertical coordinates of the end point and the starting point of the motion trajectory within a period of time in the vehicle trajectory dataset is obtained. The K-Means clustering method in unsupervised learning is used to learn the characteristics of Δy. The vehicle trajectory dataset is divided into multiple categories, and the vehicle trajectory data of each category is divided into vehicle motion trajectory training data and vehicle motion trajectory test data. Figure 2 The schematic diagram of the multimodal vehicle trajectory prediction model training method according to the embodiment of the present application is shown. Step S11 can be performed by Figure 2 The data processing module is implemented in .

[0069] Step S12, weights of each category of training data are iteratively pruned based on the decoder module to obtain multiple mask matrices corresponding to each category; Step S12 can be performed by Figure 2 The weight iterative pruning module in is implemented.

[0070] Step S13, based on the training data of each category and the mask matrix corresponding to each category, the encoder module and the decoder module are trained to obtain a trained multimodal vehicle trajectory prediction model. The trained multimodal vehicle trajectory prediction model can output multiple trajectory prediction results corresponding to multiple categories. Step S12 can be Figure 2 The encoder module and decoder module are implemented in .

[0071] The multimodal vehicle trajectory prediction model training method of the embodiment of the present application adopts a data-driven approach, fully considers different driving operation intentions, combines the vehicle's historical motion trajectory state information and the characteristics of different driving operation intentions, and trains the multimodal vehicle trajectory prediction model. The generated modal vehicle trajectory prediction model can output multiple reasonable predicted trajectories.

[0072] The traditional Transformer self-attention mechanism has a large number of parameters, but some parameters not only contribute little to the final output result, but also contain a lot of redundancy. The embodiment of the present application uses a weight iterative pruning method to perform parameter sparsification on the fully connected layer of the self-attention mechanism in the Transformer decoder to calculate the query, key and value vectors, thereby reducing the model size, improving the running speed, and preventing overfitting to a certain extent. Specifically, in one embodiment, in step S12, the training data of each category is subjected to weight iterative pruning based on the decoder module to obtain multiple mask matrices corresponding to each category, including:

[0073] Step S121: Initialize the network parameters of the decoder module and perform multiple rounds of pre-training on the decoder module using the vehicle motion trajectory training data to obtain a base network. Here, it is assumed that the base network is over-parameterized and contains multiple categories of solutions. Initialize the mask matrix M for each category. c =1, where c = 1, 2, 3, represents the category number. Here, the elements in the mask matrix correspond to multiple weight parameters of the fully connected layer in the decoder.

[0074] Step S122: input the training data of each category into the base network respectively, perform multiple rounds of pre-training, and obtain the subnet corresponding to each category;

[0075] Step S123 , performing iterative weight pruning on the weight parameters of multiple fully connected layers of the subnet corresponding to each category to determine multiple mask matrices corresponding to each category.

[0076] In this step, each fully connected layer has multiple weight parameters. The absolute values ​​of the weight parameters of each fully connected layer are sorted in ascending order, and the first α% of weight parameters with the smallest absolute values ​​are pruned, where α is the pruning rate at each pruning. If a parameter of a fully connected layer is pruned, the corresponding element in the mask matrix corresponding to that weight parameter is set to 0. Through the above operation, multiple mask matrices corresponding to each category in the left offset, right offset, and straight row are obtained.

[0077] In this embodiment, a weight iterative pruning unit is designed to perform parameter sparsification on the self-attention mechanism in the Transformer. Through multiple iterative pruning steps, different subnet masks are generated for data with different driving operation intentions, so that their structure adapts to the high-dimensional feature space of each category of the dataset, thereby extracting the hidden information of trajectories of different categories and accurately predicting future trajectories to a great extent.

[0078] In one embodiment, in step S13, the encoder module and the decoder module are trained based on the training data of each category and the multiple mask matrices corresponding to each category to obtain a trained multimodal vehicle trajectory prediction model, including:

[0079] The training process is repeated multiple times. Each training includes: inputting the training data of each category into the encoder module to obtain the high-dimensional space vector corresponding to each category; inputting the training data of each category, the corresponding multiple mask matrices and the corresponding high-dimensional space vector into the decoder module to obtain the trajectory prediction result corresponding to each category;

[0080] After each training session, the multimodal vehicle trajectory prediction model is tested based on the vehicle motion trajectory test data. When the multimodal vehicle trajectory prediction model meets the test requirements, the training ends and the trained multimodal vehicle trajectory prediction model is obtained.

[0081] In this embodiment, during the training process, for each category of training data, the training data is first divided into multiple training subsets according to a certain data size, and each training is performed using a different training subset.

[0082] Specifically, in the above embodiment, in the specific process of training the encoder module, the training data of each category is input into the encoder module respectively. Here, the encoder module includes a multi-head attention module, a residual connection layer, a normalization layer and a feedforward neural network connected in sequence, and the high-dimensional space vector corresponding to each category is obtained, which may include:

[0083] Step S131: For each category, input the training data into the multi-head attention module to obtain the weight distribution of the Value vector;

[0084] In this step, the multi-head attention module includes multiple attention units, each of which includes three fully connected layers, which are used to process the training data to obtain the Query, Key, and Value vectors respectively. The specific formula used is as follows:

[0085] Q=W q X

[0086] K=W k X

[0087] V=W v X(1)

[0088] Among them, Q, K, V represent Query, Key and Value vectors respectively, and W q ,W k ,W v Represent the weight parameters of the three fully connected layers respectively, and X represents the training data;

[0089] Then, the Softmax function is used to process the Query, Key, and Value vectors to obtain the weight distribution of the Value vector output by each attention unit. The specific formula used is as follows:

[0090]

[0091] Among them, d k represents the dimension of the Key vector, and attention(Q,K,V) represents the weight distribution of the Value vector output by each attention unit;

[0092] Then, the weight distribution of the Value vector output by each attention unit is spliced ​​together to obtain the weight distribution of the Value vector output by the multi-head attention module.

[0093] Step S132: Input the weight distribution of the Value vector and the training data into the residual connection layer for residual connection, and input the result of the residual connection into the normalization layer for normalization to obtain a normalized vector;

[0094] Step S133, inputting the normalized vector into a feedforward neural network to obtain a vector after dimensionality reduction;

[0095] In this step, the structure of the feedforward neural network is simple, consisting of two linear layers and a ReLU activation function in the middle, which increases the dimension of the vector and then reduces it.

[0096] In step S134, the reduced-dimensional vector and the normalized vector are input into the residual connection layer for residual connection, and the result of the residual connection is input into the normalization layer for normalization processing to obtain a high-dimensional space vector corresponding to the category.

[0097] In this embodiment, the network's ability to capture sequential nonlinearity mainly comes from the attention module. Self-attention, as a variant of attention, reduces dependence on external information and is better at capturing the internal correlation of data or features. It filters out a small amount of important information from a large amount of input information and ignores other information. That is, it solves long-term dependency problems by calculating the mutual influence between vectors.

[0098] In one embodiment, in the aforementioned embodiment, the training data of each category, the corresponding multiple mask matrices, and the corresponding high-dimensional space vector are input into the decoder module. Here, the decoder module includes a masked multi-head attention module, a multi-head attention module, a residual connection layer, a normalization layer, a feedforward neural network, and a fully connected layer network; obtaining the trajectory prediction result corresponding to each category may specifically include:

[0099] Step S141: For each category, input the training data and multiple mask matrices into the masked multi-head attention module to obtain the first weight distribution of the Value vector;

[0100] In this step, the masked multi-head attention module also includes multiple attention units. Each attention unit includes three fully connected layers, which are used to process the training data and the mask matrix corresponding to the fully connected layer to obtain the Query, Key and Value vectors. The specific formula used is as follows:

[0101] Q=M q ⊙W q )X

[0102] K=(M k ⊙W k )X

[0103] V=(M v ⊙W v )X (3)

[0104] Among them, Q, K, V represent Query, Key and Value vectors respectively, and W q ,W k ,W v Represents the weight parameters of the three fully connected layers, M q Represents the mask matrix corresponding to the fully connected layer that generates the Query vector, M k Represents the mask matrix corresponding to the fully connected layer that generates the Key vector, M v Represents the mask matrix corresponding to the fully connected layer that generates the Value vector; X represents the training data.

[0105] Then, the Softmax function is used to process the Query, Key, and Value vectors to obtain the weight distribution of the Value vector output by each attention unit. The specific formula used is as follows:

[0106]

[0107] Among them, d k represents the dimension of the Key vector, and attention(Q,K,V) represents the weight distribution of the Value vector output by each attention unit;

[0108] Then, the weight distribution of the Value vector output by each attention unit is spliced ​​to obtain the first weight distribution of the Value vector output by the masked multi-head attention module.

[0109] Step S142: Input the first weight distribution of the Value vector and the training data into the residual connection layer for residual connection, and input the result of the residual connection into the normalization layer for normalization to obtain a first normalized vector;

[0110] Step S143: Input the first normalized vector, the high-dimensional space vector, and the multiple mask matrices into a multi-head attention module to obtain a second weight distribution of the Value vector;

[0111] In this step, the multi-head attention module includes multiple attention units, each of which includes three fully connected layers, which are used to process the first normalized vector, the high-dimensional space vector obtained in step S134, and the mask matrix corresponding to the fully connected layer to obtain the Query, Key, and Value vectors. The specific formula used is as follows:

[0112] Q=M q ⊙W q )X 归一化

[0113] K=M k ⊙W k )enc

[0114] V=(M v ⊙W v )enc (5)

[0115] Among them, Q, K, V represent Query, Key and Value vectors respectively, and W q ,W k ,W v Represents the weight parameters of the three fully connected layers, M q Represents the mask matrix corresponding to the fully connected layer that generates the Query vector, M kRepresents the mask matrix corresponding to the fully connected layer that generates the Key vector, M v Represents the mask matrix corresponding to the fully connected layer that generates the Value vector; X 归一化 Represents the first normalized vector, and enc represents the high-dimensional space vector.

[0116] Then, the Softmax function is used to process the Query, Key, and Value vectors to obtain the weight distribution of the Value vector output by each attention unit. The specific formula used is as follows:

[0117]

[0118] Among them, d k represents the dimension of the Key vector, and attention(Q,K,V) represents the weight distribution of the Value vector output by each attention unit;

[0119] Then, the weight distribution of the Value vector output by each attention unit is spliced ​​together to obtain the second weight distribution of the Value vector output by the multi-head attention module.

[0120] Step S144: Input the second weight distribution of the Value vector and the first normalized vector into the residual connection layer for residual connection, and input the result of the residual connection into the normalization layer for normalization to obtain a second normalized vector;

[0121] Step S145, inputting the second normalized vector into a feedforward neural network to obtain a vector after dimensionality reduction;

[0122] Step S146: Input the dimension-reduced vector and the second normalized vector into a residual connection layer for residual connection, and input the result of the residual connection into a normalization layer for normalization to obtain a third normalized vector;

[0123] Step S147: Input the third normalized vector into the fully connected layer network to obtain the trajectory prediction result corresponding to the category.

[0124] In one embodiment, the multimodal vehicle trajectory prediction model is tested based on vehicle motion trajectory test data. When the multimodal vehicle trajectory prediction model meets the test requirements, the training ends, including:

[0125] In step S151, the vehicle motion trajectory test data includes test data of multiple categories; the test data of each category is input into the encoder module respectively to obtain the high-dimensional space vector corresponding to each category; the specific implementation process of this step is consistent with steps S131 to S134.

[0126] In step S152, the test data of each category, the corresponding multiple mask matrices, and the corresponding high-dimensional space vector are input into the decoder module to obtain the trajectory prediction result corresponding to each category. The specific implementation process of this step is consistent with steps S141 to S147.

[0127] Step S153: Perform linear transformation and Softmax function processing on the high-dimensional space vector corresponding to each category to obtain the distribution probability P(c|x) of each category. Specifically, the following formula can be used:

[0128] P(c|x)=softmax(wx+b)(7)

[0129] Here, x represents a high-dimensional space vector, and w and b are learnable parameters in the linear transformation.

[0130] Step S154 , calculating the error between the trajectory prediction result corresponding to the category with the largest distribution probability and the actual trajectory result; here, the root mean square error (RMSE) can be calculated.

[0131] In step S155 , if the error is within the set range, the multimodal vehicle trajectory prediction model meets the test requirements and the training ends.

[0132] The present application also provides a multimodal vehicle trajectory prediction method, including:

[0133] The trajectory sequence to be predicted is input into the multimodal vehicle trajectory prediction model, and multiple trajectory prediction results are output; the multimodal vehicle trajectory prediction model is obtained according to the multimodal vehicle trajectory prediction model training method of the above embodiment.

[0134] Based on the same inventive concept as the multimodal vehicle trajectory prediction model training method, this embodiment also provides a corresponding multimodal vehicle trajectory prediction model training device. Figure 3 The following is a structural block diagram of a multimodal vehicle trajectory prediction model training device according to an embodiment of the present application, which includes:

[0135] A data classification unit 31 is used to classify the vehicle motion trajectory training data into a plurality of categories, including left deviation, right deviation and straight travel;

[0136] The mask matrix acquisition unit 32 is used for the multimodal vehicle trajectory prediction model, including an encoder module and a decoder module; the weight of the training data of each category is iteratively pruned based on the decoder module to obtain multiple mask matrices corresponding to each category;

[0137] The training unit 33 is used to train the encoder module and the decoder module based on the training data of each category and the mask matrix corresponding to each category to obtain a trained multimodal vehicle trajectory prediction model. The trained multimodal vehicle trajectory prediction model can output multiple trajectory prediction results corresponding to multiple categories.

[0138] The multimodal vehicle trajectory prediction model training device of the embodiment of the present application adopts a data-driven approach, fully considers different driving operation intentions, combines the vehicle's historical motion trajectory state information and the characteristics of different driving operation intentions, and trains the multimodal vehicle trajectory prediction model. The generated modal vehicle trajectory prediction model can output multiple reasonable predicted trajectories.

[0139] In one embodiment, the mask matrix obtaining unit 32 is further configured to:

[0140] Initialize the network parameters of the decoder module and perform multiple rounds of pre-training on the decoder module using vehicle trajectory training data to obtain the base network;

[0141] The training data of each category is input into the base network separately, and multiple rounds of pre-training are performed to obtain the subnet corresponding to each category;

[0142] The weight parameters of multiple fully connected layers of the subnet corresponding to each category are iteratively pruned to determine multiple mask matrices corresponding to each category.

[0143] In this embodiment, a weight iterative pruning unit is designed to perform parameter sparsification on the self-attention mechanism in the Transformer. Through multiple iterative pruning steps, different subnet masks are generated for data with different driving operation intentions, so that their structure adapts to the high-dimensional feature space of each category of the dataset, thereby extracting the hidden information of trajectories of different categories and accurately predicting future trajectories to a great extent.

[0144] In one embodiment, the training unit 33 is further configured to:

[0145] The training process is repeated multiple times. Each training includes: inputting the training data of each category into the encoder module to obtain the high-dimensional space vector corresponding to each category; inputting the training data of each category, the corresponding multiple mask matrices and the corresponding high-dimensional space vector into the decoder module to obtain the trajectory prediction result corresponding to each category;

[0146] After each training session, the multimodal vehicle trajectory prediction model is tested based on the vehicle motion trajectory test data. When the multimodal vehicle trajectory prediction model meets the test requirements, the training ends and the trained multimodal vehicle trajectory prediction model is obtained.

[0147] To verify the effectiveness of this application, a large-scale natural vehicle trajectory dataset HighD from publicly available German highways was used for model training and testing. This dataset contains 11.5 hours of measurements from 6 locations, with a sampling frequency of 25Hz, and records data from 110,000 vehicles and 45,000 kilometers of mileage. The processed dataset is a dataset of three driving operation categories: left deviation, right deviation, and staying straight. 40 frames (i.e., 8 seconds) are used as a test sample, and the first 15 frames (i.e., 3 seconds) are the historical trajectory sequence T h =15, the next 25 frames (ie 5 seconds) are the future trajectory sequence T f =25.

[0148] In order to test the effect of the multimodal vehicle trajectory prediction model, the root mean square error (RMSE) is used to verify the model effect. The larger the RMSE, the greater the error. The root mean square error calculation formula is as follows:

[0149]

[0150] Among them, T h 、T f is the length of the historical trajectory sequence and the length of the future trajectory sequence, x t 、y t is the real coordinate of the target vehicle at time t, x pred t 、y pred t is the predicted coordinate of the target vehicle at time t.

[0151] In order to verify the performance of the multimodal vehicle trajectory prediction model, this application visualizes the RMSE indicators of the existing models Convolutional Social-LSTM (CS-LSTM) and Social-GAN (S-GAN) within 1-5 seconds. Figure 4 The root mean square error comparison results of the multimodal vehicle trajectory prediction model of the present application and the existing model are shown. Figure 4 As can be seen from the figure, although the proposed method lags behind existing models in short-term trajectory prediction (1-2 seconds), it significantly outperforms existing models in long-term trajectory prediction (3-5 seconds). Furthermore, the RMSE growth trend shows that the RMSE of existing models increases exponentially with the increase in the prediction time domain, while the error of the proposed method increases very slowly, remaining at a low level. The longer the prediction time domain, the higher the prediction accuracy of the model. This demonstrates that the overall performance of the proposed method is superior to that of existing models.

[0152] At the same time, in order to obtain qualitative results, Figure 5 、 Figure 6 Some typical prediction results are shown. Figure 5The predicted result diagram shows that the vehicle is moving forward and deviating to the left. Figure 6 The prediction results of the vehicle driving in the opposite direction and offset to the right are shown to reflect the performance of the model in different driving scenarios. The upper left corner of the figure is the origin of the coordinate system, the horizontal axis is the direction of highway driving, the vertical axis is perpendicular to the direction of highway driving, the black dotted line is the historical trajectory coordinate of the target vehicle, and the prediction results of the three subnets (three driving operation categories) of the model are represented by triangles with three different orientations, and the predicted trajectory with the highest probability of the current category distribution is marked with a five-pointed star, which is compared with the real future trajectory marked with a circle. Figure 5 and Figure 6 ,It can be seen that the model driving classification is accurate and the predicted trajectory with the highest distribution probability basically coincides with the real future trajectory.

[0153] In summary, this application has the following beneficial effects:

[0154] 1. Using a data-driven approach, we fully consider different driving intentions, combine the vehicle's historical motion trajectory state information and the characteristics of different driving intentions, and perform multimodal trajectory prediction through a sparse weight sharing decoder to generate multiple reasonable predicted trajectories.

[0155] 2. Design an iterative weight pruning module to perform parameter sparsification on the self-attention mechanism in the Transformer. Through multiple iterative pruning steps, different subnet masks are generated for data with different driving intentions, making their structure adaptable to the high-dimensional feature space of each category's dataset. This allows the extraction of hidden information from different categories of trajectories and enables the accurate prediction of future trajectories to a high degree.

[0156] 3. A parallel training strategy is adopted to feed datasets of different driving operation intentions into corresponding subnets for training. This ensures that some parameters in the network are updated only in specific subnets. This reduces the computational overhead of the model while ensuring model efficiency, improves the accuracy of trajectory prediction, and enables accurate and reliable vehicle trajectory prediction in complex traffic scenarios.

[0157] The above descriptions are merely examples of various embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multimodal vehicle trajectory prediction model training method, characterized in that: include: Dividing the vehicle motion trajectory training data into multiple categories, the categories including left deviation, right deviation and straight travel; The multimodal vehicle trajectory prediction model includes an encoder module and a decoder module; Performing iterative weight pruning on the training data of each category based on the decoder module to obtain multiple mask matrices corresponding to each category; The encoder module and the decoder module are trained based on the training data of each category and the multiple mask matrices corresponding to each category to obtain a trained multimodal vehicle trajectory prediction model, wherein the trained multimodal vehicle trajectory prediction model can output multiple trajectory prediction results corresponding to the multiple categories; The training data of each category is subjected to iterative weight pruning based on the decoder module to obtain multiple mask matrices corresponding to each category, including: Initializing the network parameters of the decoder module, and performing multiple rounds of pre-training on the decoder module using the vehicle motion trajectory training data to obtain a base network; Input the training data of each category into the base network respectively, perform multiple rounds of pre-training, and obtain the subnet corresponding to each category; Iterative weight pruning is performed on the weight parameters of multiple fully connected layers of the subnet corresponding to each category to determine multiple mask matrices corresponding to each category.

2. The method according to claim 1, wherein in, The encoder module and the decoder module are trained based on the training data of each category and the multiple mask matrices corresponding to each category to obtain a trained multimodal vehicle trajectory prediction model, including: The training process is performed multiple times, and each training includes: inputting the training data of each category into the encoder module to obtain the high-dimensional space vector corresponding to each category; inputting the training data of each category, the corresponding multiple mask matrices and the corresponding high-dimensional space vector into the decoder module to obtain the trajectory prediction result corresponding to each category; After each training session, the multimodal vehicle trajectory prediction model is tested based on vehicle motion trajectory test data. When the multimodal vehicle trajectory prediction model meets the test requirements, the training is completed and a trained multimodal vehicle trajectory prediction model is obtained.

3. The method according to claim 2, wherein in, The training data of each category is input into the encoder module to obtain the high-dimensional space vector corresponding to each category, including: The encoder module includes a multi-head attention module, a residual connection layer, a normalization layer and a feedforward neural network connected in sequence; For each category, the training data is input into the multi-head attention module to obtain the weight distribution of the Value vector; Inputting the weight distribution of the Value vector and the training data into the residual connection layer for residual connection, and inputting the result of the residual connection into the normalization layer for normalization processing to obtain a normalized vector; Inputting the normalized vector into the feedforward neural network to obtain a vector after dimensionality reduction; The reduced-dimensional vector and the normalized vector are input into the residual connection layer for residual connection, and the result of the residual connection is input into the normalization layer for normalization processing to obtain a high-dimensional space vector corresponding to the category.

4. The method according to claim 2, wherein in, Inputting the training data of each category, the corresponding multiple mask matrices and the corresponding high-dimensional space vector into the decoder module to obtain the trajectory prediction result corresponding to each category, including: The decoder module includes a masked multi-head attention module, a multi-head attention module, a residual connection layer, a normalization layer and a feedforward neural network, and a fully connected layer network; For each category, input the training data and the multiple mask matrices into the masked multi-head attention module to obtain a first weight distribution of the Value vector; Inputting the first weight distribution of the Value vector and the training data into the residual connection layer for residual connection, and inputting the result of the residual connection into the normalization layer for normalization processing to obtain a first normalized vector; Inputting the first normalized vector, the high-dimensional space vector, and the multiple mask matrices into the multi-head attention module to obtain a second weight distribution of the Value vector; Inputting the second weight distribution of the Value vector and the first normalized vector into the residual connection layer for residual connection, and inputting the result of the residual connection into the normalization layer for normalization processing to obtain a second normalized vector; Inputting the second normalized vector into the feedforward neural network to obtain a vector after dimensionality reduction; Inputting the dimension-reduced vector and the second normalized vector into the residual connection layer for residual connection, and inputting the result of the residual connection into the normalization layer for normalization processing to obtain a third normalized vector; The third normalized vector is input into the fully connected layer network to obtain a trajectory prediction result corresponding to the category.

5. The method according to claim 2, wherein The multimodal vehicle trajectory prediction model is tested based on vehicle motion trajectory test data. When the multimodal vehicle trajectory prediction model meets the test requirements, the training ends, including: The vehicle motion trajectory test data includes test data of multiple categories; the test data of each category is input into the encoder module respectively to obtain a high-dimensional space vector corresponding to each category; Inputting the test data of each category, the corresponding multiple mask matrices and the corresponding high-dimensional space vector into the decoder module to obtain the trajectory prediction result corresponding to each category; Performing linear transformation and Softmax function processing on the high-dimensional space vector corresponding to each category to obtain the distribution probability of each category; Calculate the error between the trajectory prediction result corresponding to the category with the largest distribution probability and the actual trajectory result; If the error is within the set range, the multimodal vehicle trajectory prediction model meets the test requirements and the training ends.

6. A multimodal vehicle trajectory prediction model training device, characterized in that: include: A data classification unit is used to classify the vehicle motion trajectory training data into a plurality of categories, wherein the categories include left deviation, right deviation and straight travel; A mask matrix acquisition unit for the multimodal vehicle trajectory prediction model including an encoder module and a decoder module; Performing iterative weight pruning on the training data of each category based on the decoder module to obtain multiple mask matrices corresponding to each category; a training unit, configured to train the encoder module and the decoder module based on the training data of each category and the multiple mask matrices corresponding to each category, to obtain a trained multimodal vehicle trajectory prediction model, wherein the trained multimodal vehicle trajectory prediction model is capable of outputting multiple trajectory prediction results corresponding to the multiple categories; The mask matrix acquisition unit is further configured to: Initializing the network parameters of the decoder module, and performing multiple rounds of pre-training on the decoder module using the vehicle motion trajectory training data to obtain a base network; Input the training data of each category into the base network respectively, perform multiple rounds of pre-training, and obtain the subnet corresponding to each category; Iterative weight pruning is performed on the weight parameters of multiple fully connected layers of the subnet corresponding to each category to determine multiple mask matrices corresponding to each category.

7. The device according to claim 6, characterized in that The training unit is further configured to: The training process is performed multiple times, and each training includes: inputting the training data of each category into the encoder module to obtain the high-dimensional space vector corresponding to each category; inputting the training data of each category, the corresponding multiple mask matrices and the corresponding high-dimensional space vector into the decoder module to obtain the trajectory prediction result corresponding to each category; After each training session, the multimodal vehicle trajectory prediction model is tested based on vehicle motion trajectory test data. When the multimodal vehicle trajectory prediction model meets the test requirements, the training is completed and a trained multimodal vehicle trajectory prediction model is obtained.

8. A multimodal vehicle trajectory prediction method, characterized in that: include: Input the trajectory sequence to be predicted into the multimodal vehicle trajectory prediction model and output multiple trajectory prediction results; The multimodal vehicle trajectory prediction model is obtained according to the multimodal vehicle trajectory prediction model training method according to any one of claims 1 to 5.

Images