A trajectory prediction method based on self-perception mask

Abstract

This invention discloses a trajectory prediction method based on self-aware masks, comprising: constructing a Transformer-based deep neural network; replicating a deep neural network with the same structure to form a teacher-student model; adding a fully connected layer to the top layer of the model and training it for self-aware mask generation on a trajectory prediction dataset; the trained student model subsequently serving as a mask generation module to generate high-quality masks; constructing an attention neural network based on a Transformer encoder-decoder structure; adding the trained mask generation module to the attention neural network and pre-training the attention neural network for trajectory reconstruction on the trajectory prediction dataset; after pre-training, training the attention neural network for trajectory prediction on the trajectory dataset; and applying the trained trajectory prediction network to an autonomous driving system to predict the trajectories of other traffic participants. This invention can improve the training efficiency of the model and make trajectory prediction more accurate.

Description

Technical Field

[0001] This invention belongs to the field of trajectory prediction, and in particular relates to a trajectory prediction method based on self-perception mask. Background Technology

[0002] With the discovery and in-depth research of deep learning theory, many fields and tasks under autonomous driving have seen rapid breakthroughs and significant improvements. Among them, trajectory prediction, due to the high requirements of autonomous driving for accuracy and real-time performance, has become one of the most pressing prediction tasks and a current hot research direction. The trajectory prediction task requires the prediction system to predict the coordinates of future trajectories based on a known trajectory sequence over a certain period. Currently, mainstream trajectory prediction solutions are mainly divided into methods based on recurrent neural networks, methods based on graph neural networks, and methods based on attention mechanisms.

[0003] The most well-known method based on recurrent neural networks is the work "Argoverse: 3D Tracking and Forecasting With Rich Maps" published by Chang et al. at the IEEE / CVFConference on Computer Vision and Pattern Recognition conference in 2019. The method introduced in this work is an important solution in this field. By calculating the distance between the vehicle and the center line of the lane, it proposes a Long Short Term Memory (LSTM) encoder-decoder network that takes into account map information and surrounding environment information. It uses recurrent neural networks to fully aggregate the features of the map environment and the vehicle.

[0004] A notable graph neural network-based approach is the work "LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting" presented by Zeng et al. at the 2021 IEEE / RSJ International Conference on Intelligent Robots and Systems. This work proposes LaneRCNN, which obtains a local lane map representation for each traffic participant. This traffic map encodes the participant's past trajectories and local map topological features, and finally uses an interaction module to complete the interaction of the local lane map. Attention-based methods are currently the mainstream approach. A notable example is the work "GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving" presented by Huang et al. at the 2023 IEEE / CVF International Conference on Computer Vision. This work proposes a Transformer-based interaction prediction and planning framework, GameFormer, which uses a hierarchical decoding structure to capture vehicle interactions. In each decoding layer, the decoder uses the prediction results from the previous layer and shared environmental features to iteratively refine the interaction process between vehicles, similar to a game between vehicles.

[0005] However, existing trajectory prediction methods suffer from limited data dimensionality, making it difficult to learn rich feature representations, thus limiting the accuracy improvement of these methods. In recent years, pre-training methods have emerged in the fields of natural language processing and computer vision. By pre-training the network using proxy tasks before formal training, the model can learn some general features, improving model performance and reducing training time based on pre-trained parameters. Therefore, pre-training work has also appeared in the trajectory prediction field. For example, Cheng et al.'s work, "Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoders," presented at the IEEE / CVF International Conference on Computer Vision in 2023, utilizes random masks to hide some trajectory information and then reconstructs the complete trajectory using a Transformer network based on the unmasked trajectory information. However, the random masking approach can result in the preserved trajectory information being either too simple or too difficult for the reconstruction task, leading to the network failing to learn useful features. Therefore, a better masking method is needed to improve the pre-training effect and produce more accurate trajectory prediction models. Summary of the Invention

[0006] This invention provides a trajectory prediction method based on self-aware masking, which can effectively help trajectory prediction models obtain key features, improve model training efficiency, and make trajectory prediction more accurate.

[0007] A trajectory prediction method based on self-aware masks includes the following steps:

[0008] (1) Construct a deep neural network based on Transformer;

[0009] (2) Duplicate a deep neural network with the same structure as in step (1), and the two form a teacher-student model;

[0010] (3) Add a fully connected layer network to the top layer of the teacher-student model, and train the self-aware mask generation on the trajectory prediction dataset. The trained student model is then used as a mask generation module to generate high-quality masks.

[0011] (4) Construct an attention neural network based on the Transformer encoder-decoder structure;

[0012] (5) Add the trained mask generation module to the attention neural network, generate high-quality masks to enable the attention neural network to reconstruct the trajectory, and pre-train the attention neural network on the trajectory prediction dataset for trajectory reconstruction.

[0013] (6) After pre-training, the attention neural network is trained on the trajectory dataset for trajectory prediction to obtain a high-quality trajectory prediction network.

[0014] (7) The trained trajectory prediction network is applied to the autonomous driving system to predict the trajectories of other traffic participants.

[0015] The proposed method first constructs a Transformer-based deep neural network to generate high-quality masks. Then, a replica of this network with the same structure is used to form a teacher-student model, and a deep learning network incorporating the teacher-student model is trained. Next, the trained mask generation module produces high-quality masks, and an attention neural network based on a Transformer encoder-decoder structure is constructed to reconstruct the masked trajectory features. This network is pre-trained on a real-world trajectory dataset. Finally, a trajectory prediction decoder structure is added to the top layer of the pre-trained attention network, and the network is trained on the training dataset (trajectory prediction task) to predict the future trajectory sequence corresponding to the input historical trajectory. This method effectively reduces the impact of random masks on pre-training performance. By improving the network's ability to capture features and the quality of learned features during pre-training, it optimizes the efficiency and accuracy of model training, resulting in a more robust trajectory prediction model. This model can then be used in autonomous driving systems to predict the trajectories of other vehicles, pedestrians, and other traffic participants to plan a safe driving path.

[0016] In step (1), a deep neural network based on Transformer is constructed. By encoding the input trajectory sequence, the predicted reconstruction loss of the corresponding position of the trajectory sequence is output, and a high-quality mask is generated by the predicted reconstruction loss.

[0017] In step (2), a teacher-student model is constructed to allow the network to continuously optimize the mask generation quality through self-aware training. The student network is responsible for adjusting its own prediction parameters from the real reconstruction loss, and then using the parameters of the student network to update the teacher network through a moving average update strategy, thereby reducing the fluctuation of each update and making the model more stable. The moving average update strategy is defined as follows:

[0018] θ t =mθ t +(1-m)θ s

[0019] Where, θt θ represents the weight parameters of the teacher network. s This represents the weight parameter of the student network, where m controls the magnitude of the weight change during updates, and is set to 0.9 in this invention.

[0020] To alleviate the problem of insufficient trajectory prediction data, the dataset used is the large-scale real trajectory prediction dataset Argoverse1, which can improve the training quality of the mask generation network. The trained student network is then used as a mask generation module.

[0021] The specific process of step (2) is as follows:

[0022] (2-1) Construct a deep neural network identical to that in step (1). The two networks constitute a teacher-student model. The student network is responsible for predicting the reconstruction loss and is trained by calculating the loss with the actual reconstruction error. The teacher network updates itself using the parameters of the student model by means of moving average. Then, the teacher network predicts the reconstruction loss of the trajectory sequence during training and uses this loss to construct a high-quality mask.

[0023] (2-2) The trajectory sequence data of shape (K, T, 2) is input into the teacher network. The teacher network first encodes the trajectory sequence data into trajectory features of shape (K, T, H) through the Encoder structure, and then inputs the trajectory features into the Decoder structure to output a prediction and reconstruction loss of shape (K, T); where K represents the number of moving objects, T represents the time length, and H represents the feature dimension.

[0024] (2-3) Mask the trajectory sequence data at a ratio of 60%; to facilitate network training, a mask sampling method from simple to difficult is adopted. In each training round, a certain proportion of high reconstruction loss locations are dynamically selected as difficult masks, and random masks are used to supplement the 60% quantity requirement. The proportion of difficult samples in the current round is defined as:

[0025]

[0026] Where α0 represents the initial proportion of difficult samples, which is set to 0 in this invention; n represents the current training round; N represents the total number of training rounds; α N The number of difficult samples in the last training round is represented by 0.5 in this invention; the difficulty mask corresponding to each training round is the position of the trajectory sequence corresponding to the M largest values ​​before the reconstruction loss, where:

[0027] M = 0.6 × α n T

[0028] Additionally, a random sample of 0.6×(1-α) nThe positions of the T trajectory sequences are masked, and the features of the masked trajectories are all set to 0.

[0029] (2-4) Input the trajectory features after masking into the student network, reconstruct the trajectory features through the student Encoder structure, and then input the reconstructed features into the student Decoder structure to output the prediction reconstruction loss corresponding to each mask position.

[0030] In step (3), self-aware mask generation training is performed on the trajectory prediction dataset Argoverse1 to help the mask generation module perceive the trajectory importance distribution of traffic participants in the autonomous driving environment.

[0031] The specific process of step (3) is as follows:

[0032] (3-1) A fully connected layer network is added to the top layer of the teacher-student model to decode the trajectory features reconstructed by the student model into trajectory sequence coordinates. For example, if the trajectory feature shape reconstructed by the student model is (K, T, H), the Decoder structure will use this feature to predict the trajectory sequence coordinates with shape (K, T, 2), where N represents the number of moving objects, T represents the time length, H represents the feature dimension size, and finally the 2 in the output dimension represents the x and y coordinates.

[0033] (3-2) Calculate the true reconstruction loss using the decoded trajectory sequence coordinates and the original trajectory sequence coordinates, and train the student network using the MSE loss function;

[0034] (3-3) The constructed deep learning model is trained on GPUs. Each GPU calculates the reconstruction loss of 32 trajectory sequence data in a single run, and 8 GPUs are trained in parallel.

[0035] (3-4) The Adam optimization algorithm was used, with parameters β1 and β2 set to 0.9 and 0.999 respectively, and the initial learning rate was 0.001. The learning rate was reduced to 50% after every 20 cycles for a total of 60 cycles until the model converged.

[0036] In step (4), an attention neural network based on the Transformer encoder-decoder structure is constructed to better capture feature information in the complex environment of autonomous driving using the Transformer global attention mechanism, and to complete the pre-training of the trajectory reconstruction task with high quality. This is because after the student model is trained in step (3), a high-quality mask can be generated to cover part of the trajectory sequence data. The Transformer attention mechanism can ignore this masked data and complete the trajectory reconstruction by fusing other unmasked data features.

[0037] The specific process of step (5) is as follows:

[0038] (5-1) Use a fully connected network to convert the input trajectory sequence data into an H-dimensional vector; for example, if the shape of the input trajectory sequence data is (K, T, 2), after processing by the fully connected network, it becomes a trajectory feature of (K, T, H). Where K represents the number of moving objects, T represents the time length, and H represents the feature dimension.

[0039] (5-2) Input the trajectory sequence data into the mask generation module to generate a high-quality mask with shape (K, T), set the trajectory features at the corresponding positions of the mask to 0, input them into the Encoder structure of the attention neural network for feature fusion, and obtain the trajectory features with shape (K, T, H). Input the features into the Decoder structure to decode the trajectory coordinates with shape (K, T, 2).

[0040] (5-3) Use the MSE function to calculate the prediction loss of the predicted trajectory coordinates and the true trajectory coordinates of the mask position, and train the attention neural network;

[0041] (5-4) The constructed attention neural network was trained on GPUs. Each GPU calculated the reconstructed coordinates of 32 trajectory sequence data in a single run, and 8 GPUs were trained in parallel.

[0042] (5-5) The Adam optimization algorithm was used, with parameters β1 and β2 set to 0.9 and 0.999 respectively, and the initial learning rate was 0.005. The learning rate was reduced to 50% after every 10 cycles for a total of 50 cycles until the model converged.

[0043] In step (6), the historical trajectory data in the trajectory dataset Nuscenes is input into the pre-trained attention neural network. The network outputs future trajectory data through the Decoder structure and uses the MSE loss function to supervise the training of the trajectory prediction task, thereby obtaining a high-quality trajectory prediction model.

[0044] The resulting high-quality trajectory prediction model can be used in autonomous driving systems to plan a safe driving path by predicting the trajectories of other traffic participants such as vehicles and pedestrians.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. This invention proposes a trajectory prediction method based on self-perceived masks. Through self-perceived training of a teacher-student model, the method completes the prediction of trajectory reconstruction loss, making the selection of trajectory masks more interpretable and reliable.

[0047] 2. This invention designs a mask generation strategy ranging from simple to difficult. In the early stage of network reconstruction training, using simple masks is beneficial for network convergence, while increasing the proportion of difficult masks in the later stage of training is beneficial for the network to learn valuable and more general features.

[0048] 3. This invention has strong applicability. By adding a self-aware mask generation module to the attention neural network, the pre-training of the attention neural network can be completed directly and effectively, helping the network to capture common feature information in the autonomous driving environment and improving the accuracy and training efficiency of the final trajectory prediction. Attached Figure Description

[0049] Figure 1 This is a flowchart of a trajectory prediction method based on self-perceived mask according to an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the model structure of the mask generation module of the present invention;

[0051] Figure 3 This is a schematic diagram of the pre-training process for trajectory prediction in this invention. Detailed Implementation

[0052] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0053] like Figure 1 As shown, a trajectory prediction method based on self-aware masks includes the following steps:

[0054] S01, construct a deep neural network based on Transformer.

[0055] The deep neural network used is designed with reference to the Transformer network, keeping parameters such as network depth and width unchanged, and the feature encoding size is set to 128 dimensions.

[0056] S02, replicate a deep neural network with the same structure, and the two form a teacher-student model.

[0057] The student network is responsible for learning and adjusting its own prediction parameters from the real reconstruction loss. Then, the teacher network is updated using the parameters of the student network through a moving average update strategy, thereby reducing the fluctuation of each update and making the model training more stable.

[0058] S03 adds a fully connected layer network to the top layer of the teacher-student model and trains it on the large trajectory prediction dataset Argoverse1.

[0059] Argoverse1 is a massive dataset of autonomous driving scenarios, presented in the paper "Argoverse: 3D Tracking and Forecasting with Rich Maps" by Chang et al. at the Conference on Computer Vision and Pattern Recognition 2019. This dataset collected over 30,000 scenarios in Pittsburgh and Miami. Each scenario is a trajectory sequence sampled at 10 Hz. Each sequence contains a moving object, and the task is to predict the object's trajectory within the next 3 seconds, given the object's historical x and y coordinates over 2 seconds. The sequences are divided into training, validation, and test sets, with 205,942, 39,472, and 78,143 trajectory sequences respectively. This dataset originates from real-world driving environments, enabling the model to better learn from real-world features.

[0060] The training process for the generation module is as follows: Figure 2 As shown, the trajectory sequence data is first input into the teacher network. The teacher network encodes the trajectory sequence data into 128-dimensional trajectory features through the Encoder structure. Then, the trajectory features are input into the Decoder structure to output the prediction and reconstruction loss.

[0061] Next, using a mask sampling method ranging from simple to difficult, different proportions of high reconstruction loss locations are selected for masking in each training round, and the mask quantity requirement of 60% is supplemented by random sampling of masks, and the features of the mask locations are set to 0.

[0062] Then, the trajectory features after the mask are input into the student network to reconstruct the trajectory features, and the reconstructed features are input into the student Decoder structure to output the predicted reconstruction loss corresponding to each mask position.

[0063] Finally, a fully connected network is used to convert the reconstructed features into the final output trajectory coordinates, and the true reconstruction loss is calculated by combining the true trajectory coordinates with the true reconstruction loss. This true loss is then combined with the reconstruction loss predicted by the student network to calculate the MSE loss, thereby completing the training of the mask generation module.

[0064] S04. Construct an attention neural network based on the Transformer encoder-decoder structure. The attention neural network used is designed with reference to the Transformer network, keeping parameters such as network depth and network width unchanged, and setting the feature dimension to 128 dimensions.

[0065] S05, the student network part in step S03 is extracted and used as a mask generation module to train the attention neural network in S04. The parameters of the attention neural network are randomly initialized.

[0066] Training is performed on the Argoverse1 dataset. The training process in this stage is as follows: Figure 3 As shown, based on the input trajectory data, the mask generation module generates masks according to a strategy from simple to difficult, thereby obtaining the masked trajectory features. The Encoder structure of the attention neural network fuses the unmasked trajectory features, and finally the Decoder structure decodes the masked trajectory coordinates to complete the trajectory reconstruction task.

[0067] S06. Extract the attention neural network from step S05 and start training the model. The training data uses the NuScene dataset, which was published by Holger et al. at the Conference on Computer Vision and Pattern Recognition 2020 as "nuScenes: A multimodal dataset for autonomous driving". The model is input with historical trajectory data and outputs predicted future trajectory data.

[0068] The trained model is used to perform trajectory prediction tasks in autonomous driving systems. By predicting the trajectories of other traffic participants such as vehicles and pedestrians, a safe and reliable driving path can be planned.

[0069] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A trajectory prediction method based on self-perceived masking, characterized in that, Includes the following steps: (1) Construct a deep neural network based on Transformer; (2) Duplicate a deep neural network with the same structure as in step (1), and the two form a teacher-student model; (3) Add a fully connected layer network to the top layer of the teacher-student model and train the self-aware mask generation on the trajectory prediction dataset. The trained student model is then used as a mask generation module to generate high-quality masks. (4) Construct an attention neural network based on the Transformer encoder-decoder structure; (5) The trained mask generation module is added to the attention neural network to generate high-quality masks for trajectory reconstruction. The attention neural network is then pre-trained on the trajectory prediction dataset for trajectory reconstruction. Specifically, this includes: (5-1) Use a fully connected layer network to convert the input trajectory sequence data into an H-dimensional vector; (5-2) Input the trajectory sequence data into the mask generation module to generate a high-quality mask with shape (K, T), set the trajectory features at the corresponding positions of the mask to 0, input the Encoder structure of the attention neural network for feature fusion, and obtain the trajectory features with shape (K, T, H). Input the features into the Decoder structure to decode the trajectory coordinates with shape (K, T, 2). (5-3) Use the MSE function to calculate the prediction loss of the predicted trajectory coordinates and the true trajectory coordinates of the mask position, and train the attention neural network; (6) After pre-training, the attention neural network is trained on the trajectory dataset for trajectory prediction to obtain a high-quality trajectory prediction network; (7) The trained trajectory prediction network is applied to the autonomous driving system to predict the trajectories of other traffic participants.

2. The trajectory prediction method based on self-perceived mask according to claim 1, characterized in that, In step (1), a deep neural network based on Transformer is constructed. By encoding the input trajectory sequence, the predicted reconstruction loss of the corresponding position of the trajectory sequence is output, and a high-quality mask is generated by the predicted reconstruction loss.

3. The trajectory prediction method based on self-perceived mask according to claim 1, characterized in that, In step (2), a teacher-student model is constructed to allow the network to continuously optimize the mask generation quality through self-aware training. The student network is responsible for adjusting its own prediction parameters from the real reconstruction loss, and then using the parameters of the student network to update the teacher network through a moving average update strategy, thereby reducing the fluctuation of each update and making the model more stable. The moving average update strategy is defined as follows: in, The weight parameters represent the teacher network. This represents the weight parameters of the student network, where m controls the magnitude of the weight changes during updates.

4. The trajectory prediction method based on self-perceived mask according to claim 1, characterized in that, The specific process of step (2) is as follows: (2-1) Construct a deep neural network identical to that in step (1). The two networks constitute a teacher-student model. The student network is responsible for predicting the reconstruction loss and is trained by calculating the loss with the actual reconstruction error. The teacher network updates itself using the parameters of the student model by means of moving average. Then, the teacher network predicts the reconstruction loss of the trajectory sequence during training and uses this loss to construct a high-quality mask. (2-2) The trajectory sequence data with shape (K, T, 2) is input into the teacher network. The teacher network first encodes the trajectory sequence data into trajectory features of shape (K, T, H) through the Encoder structure, and then inputs the trajectory features into the Decoder structure to output a prediction and reconstruction loss with shape (K, T); where K represents the number of moving objects, T represents the time length, and H represents the feature dimension. (2-3) Mask the trajectory sequence data at a ratio of 60%; to facilitate network training, a masking sampling method from simple to difficult is adopted. In each training round, a certain proportion of high reconstruction loss locations are dynamically selected as difficult masks, and random masks are used to supplement the 60% quantity requirement. The proportion of difficult samples in the current round is defined as: in, This represents the initial proportion of difficult samples; n represents the current training epoch; N represents the total number of training epochs. This represents the number of difficult samples in the last training epoch; the difficulty mask corresponding to each training epoch is the position of the M largest trajectory sequence corresponding to the reconstruction loss, where: In addition, random sampling The positions of each trajectory sequence are masked, and the features of the masked trajectories are all set to 0; (2-4) Input the trajectory features after masking into the student network, reconstruct the trajectory features through the student Encoder structure, and then input the reconstructed features into the student Decoder structure to output the prediction reconstruction loss corresponding to each mask position.

5. The trajectory prediction method based on self-perceived mask according to claim 1, characterized in that, In step (3), self-aware mask generation training is performed on the trajectory prediction dataset Argoverse1 to help the mask generation module perceive the trajectory importance distribution of traffic participants in the autonomous driving environment.

6. The trajectory prediction method based on self-perceived mask according to claim 1, characterized in that, The specific process of step (3) is as follows: (3-1) A fully connected layer network is added to the top layer of the teacher-student model to decode the trajectory features reconstructed by the student model into trajectory sequence coordinates; (3-2) Calculate the true reconstruction loss using the decoded trajectory sequence coordinates and the original trajectory sequence coordinates, and train the student network using the MSE loss function; (3-3) The constructed deep learning model is trained on GPUs. Each GPU calculates the reconstruction loss of 32 trajectory sequence data in a single run, and 8 GPUs are trained in parallel. (3-4) Using the Adam optimization algorithm, parameters and The values ​​were set to 0.9 and 0.999 respectively, with an initial learning rate of 0.

001. The learning rate was reduced to 50% after every 20 iterations for a total of 60 iterations, until the model converged.

7. The trajectory prediction method based on self-perceived mask according to claim 1, characterized in that, Step (5) further includes: (5-4) The constructed attention neural network was trained on GPUs. Each GPU calculated the reconstructed coordinates of 32 trajectory sequence data in a single run, and 8 GPUs were trained in parallel. (5-5) Using the Adam optimization algorithm, parameters and The values ​​were set to 0.9 and 0.999 respectively, with an initial learning rate of 0.

005. The learning rate was reduced to 50% after every 10 iterations for a total of 50 iterations, until the model converged.

8. The trajectory prediction method based on self-perceived mask according to claim 1, characterized in that, In step (6), the historical trajectory data in the trajectory dataset Nuscenes is input into the pre-trained attention neural network. The network outputs future trajectory data through the Decoder structure and uses the MSE loss function to supervise the training of the trajectory prediction task, thereby obtaining a high-quality trajectory prediction model.

Images