An autonomous vehicle multi-modal trajectory prediction method fusing Perceiver-IO and DERT-Mamba

Abstract

In view of the problems of large feature interaction operation pressure and poor cross-scene adaptability of existing trajectory prediction models in complex scenes, the application provides a trajectory prediction method and model fusing pre-training and fine-tuning strategies. The model is composed of an improved Perceiver IO encoder and a DERT-Mamba hybrid decoder: in the pre-training stage, the feature representation capability is enhanced through a mask self-supervised task; in the fine-tuning stage, the encoder is frozen, a multi-modal decoder is accessed, and a diversified future trajectory and its confidence score are generated by using a multi-modal query and an expert routing mechanism. The application can effectively improve the accuracy and diversity of prediction and provide more robust and reliable decision support for an automatic driving system.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving trajectory prediction, and more specifically to an autonomous driving trajectory prediction method that integrates a Perceiver IO encoder and a DERT-Mamba hybrid decoder. Background Technology

[0002] Currently, autonomous driving technology and related industries are entering a phase of rapid development. In the modular predictive planning architecture of autonomous driving algorithms, the trajectory prediction module provides the necessary spatiotemporal priors for trajectory planning and is one of the core modules that ensures the output of safe and comfortable trajectories by the autonomous driving algorithm.

[0003] Existing trajectory prediction models are often built on an encoder-decoder architecture, with scene information input including both vehicle and environment information. Scene element features are fused in the model's interaction layer through three main types of interactions: temporal interaction, spatial interaction, and environmental interaction (vehicle-environment). The model is then decoded and outputs the future trajectory in the specified prediction time domain. Existing models suffer from the following problems: First, scene information typically contains a large amount of redundant information. When conventional attention-based models model interactions across the entire scene, this leads to a significant increase in computational and storage requirements, thus reducing the real-time performance of the prediction model. Second, models trained entirely using supervised learning are highly dependent on the training scenario; their performance significantly degrades under different operating scenarios or distributional shifts.

[0004] Self-supervised learning is a technique that uses auxiliary tasks to train models. It typically involves building self-supervised signals on large-scale pre-training data to learn an encoder (feature extractor) with good generalization capabilities. In downstream tasks, this encoder is combined with task-specific modules and fine-tuned on the target dataset, enabling the model to quickly adapt to different data distributions. Compared to full training directly on downstream tasks, the method of large-scale pre-training followed by fine-tuning in the target scenario demonstrates better generalization and convergence speed when samples are scarce or tasks frequently switch.

[0005] To address the aforementioned problems, this invention proposes a trajectory prediction model and method based on a two-stage training strategy. The core of this method consists of an improved Perceiver IO encoder and a DERT-Mamba hybrid decoder.

[0006] During the pre-training phase, the model is trained using large-scale trajectory data to enhance its ability to represent input features. Specifically, the input features are first masked to a certain proportion; then, through the bottleneck structure of Perceiver IO, combined with a learnable query vector pool, key information under different traffic scenarios is selectively retained, and full interaction of features is achieved in the latent space. The model completes the self-supervised learning task by reconstructing the masked and compressed original features, thereby deeply exploring the intrinsic correlations between features.

[0007] During the fine-tuning phase, to improve the model's adaptability to specific downstream tasks, the encoder parameters obtained in the pre-training phase are frozen, modules related to the reconstruction task are disabled, and the DERT-Mamba multimodal trajectory decoder is integrated. This decoder introduces multimodal query vectors that interact with the encoded features to model diverse future predictions. The interacted features are then routed to the corresponding expert feedforward networks for processing via a routing module. Finally, the outputs of all features are fused to decode and generate multimodal future trajectories and corresponding modality confidence scores.

[0008] The method proposed in this invention effectively balances the accuracy and diversity of trajectory prediction, and can provide more robust and reliable decision support for autonomous driving. Summary of the Invention

[0009] This invention relates to a prediction module in an end-to-end autonomous driving system, specifically providing a model and method for predicting the future trajectories of surrounding vehicles. The core architecture of this model consists of an improved Perceiver IO encoder and a DERT-Mamba multimodal trajectory decoder, aiming to achieve high-precision, multimodal trajectory prediction of the behavior of surrounding vehicles in complex driving scenarios. The model input includes historical trajectory information of the target vehicle and surrounding vehicles, vehicle state features, and road environment features. After feature embedding processing, the above inputs are combined with position encoding and fed into the model's backbone network for further processing. The model training process is divided into two stages: a pre-training stage and a fine-tuning stage. In the pre-training stage, the focus is on improving the encoder's ability to extract, compress, and interact with information from complex scenarios. To this end, some features in the input scenario data are first masked, and then the bottleneck structure of the improved Perceiver IO encoder, combined with a query vector pool, is used to selectively retain and fully interact with key scenario features. The supervised encoder based on scene reconstruction results preserves key spatial structure and dynamic semantic information in the input, ensuring sufficient recoverability of the compressed representation and improving the encoder's ability to mine the intrinsic feature correlations of the input data. In the fine-tuning stage, based on the pre-trained model parameters, decoding components related to the reconstruction task are disabled, and an upsampling head and a DERT-Mamba multimodal trajectory decoder suitable for downstream tasks are added. This decoder further performs deep fusion and context modeling of high-level semantic features, ultimately outputting multimodal trajectories and corresponding modality scores, and updating the model parameters through the corresponding loss. The specific implementation steps of this method are as follows:

[0010] Step 1: Select a vehicle as your ego vehicle, and then select surrounding vehicles and environmental elements within a 150-meter radius of your ego vehicle. For vehicle elements, select up to nine vehicles closest to your ego vehicle and obtain the information for each vehicle. The trajectory at each time step; the trajectory points at each time step should include the following features: vehicle position coordinates (x, y), orientation (x, y), and direction (y, y). ), lateral and longitudinal speeds ( , The model is designed to include the vehicle's length and width (L, W); for environmental elements, a maximum of 32 lane elements closest to the vehicle are selected, and 20 points are retained for each polyline using interpolation downsampling; the selected lane polyline information includes centerline coordinates, road boundary coordinates, road speed limits, and traffic light information, and the points in the polyline are vectorized using a difference method; when scene elements are insufficient, padding should be used to ensure format consistency, and a mask should be used to ensure that the padding data does not interfere with normal interaction in subsequent processing; in the pre-training stage, the required data is extracted from multiple datasets for training, and in the fine-tuning stage, the required data is extracted from the target dataset for training the model.

[0011] To better measure the relative spatial position information between vehicles, the positional relationships of elements in the scene are represented by relative coordinates and relative orientation angles. Specifically, based on the rigid body secondary transformation method, the coordinates of all elements are transformed to a coordinate system with the current pose of the vehicle as the origin, and their orientation angles are rotated. The above features can be represented as the following two categories: vehicle trajectory features. Lane characteristics .

[0012] Step 2 involves feeding the two types of features from Step 1 into an embedding layer to obtain a high-dimensional feature vector, which is then fed into an improved Perceiver IO encoder. In the encoder, each scene element is first randomly masked at a certain ratio, then sequentially passed through a downsampling layer, a latent space interaction layer, and an upsampling layer. Finally, the upsampled features are decoded through their respective feedforward layers and the reconstruction decoder to output the corresponding features. The specific implementation steps of Step 2 are as follows:

[0013] Step 21: The two types of scene feature vectors mentioned in Step 1 are first input into the vehicle and environment embedding layers respectively to obtain high-dimensional feature representations, and then position encoding is introduced, as shown in the following formula:

[0014] ;

[0015] ;

[0016] (Embedding) represents the operation of projecting relevant features onto a specified dimension through a linear layer, where the location encoding corresponding to vehicle features includes the absolute position of the vehicle. Embedded and history timestep vectors, the position encoding corresponding to the lane features is the vehicle absolute position embedding; where the vehicle absolute position vector is obtained by processing the absolute position of the vehicle in the current scene through MLP, and the history time vector is obtained by processing the sinusoidal position encoding of each time step through an MLP.

[0017] ;

[0018] ;

[0019] ;

[0020] Step 22 involves constructing a query vector pool for feature dimensionality reduction and designing a routing mechanism for the query vector pool to adapt to the key information extraction needs in different application scenarios. Specifically, the scene features S concatenated in Step 23 are first subjected to layer normalization and then mapped to corresponding layers of a multilayer perceptron (MLP). The processing order is as follows:

[0021] ;

[0022] ;

[0023] Where Linear represents a linear layer, ReLU represents the activation function, and Layernorm represents the layer normalization operation, as shown in the formula:

[0024] ;

[0025] For the output Gaussian noise with variance scheduled during training is added to obtain This allows for the full exploration of each query vector using a large variance in the early stages of training, and the use of a small variance to stabilize the training in the later stages.

[0026] Regarding the 'and Perform the SoftMax operation to obtain the selection probability of each candidate vector in the vector pool. and weighted weights ;

[0027] ;

[0028] ;

[0029] During training, each dimensionality reduction query vector in the vector pool is selected and updated using a straight-through strategy: specifically, the vector with the highest probability is selected through argmax operation during forward computation. Dimensionality reduction vector of the scene ,based on The loss is calculated during forward propagation, and then during backward propagation... The weighted vector obtained by the weights This updates all vectors in the vector pool to prevent mode collapse; where sg indicates stopping gradient calculation.

[0030] ;

[0031] ;

[0032] Step 23: Randomly create mask matrices for the two types of features, wherein the mask matrix is ​​created by randomly selecting features from a certain time step for the vehicle features. A mask matrix is ​​created by randomly selecting a certain number of points based on the lane features. For the two scene features and Perform dimensional merging and stitching, specifically: merge vehicle historical features into the vehicle quantity... With the time dimension The above merging will incorporate environmental characteristics in the dimension of lane segment quantity. Dimension of Number of Points Merge above, and similarly apply to and Perform similarity dimension merging and concatenation operations:

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] The merged new dimension is uniformly defined as the token dimension, resulting in a new scene vector. and mask matrix ,in ;

[0038] Step 24: In the pre-training stage, the encoder mainly consists of a downsampling layer, a latent space interaction layer, and two upsampling layers; the query vector obtained in step 22 is then used... and the scene feature vector obtained in step 23 The input downsampling layer is first normalized; then... Obtained through linear mapping matrix, After two different linear mappings, we obtain... The matrix is ​​used to perform dot product attention operations, and the scene features are aggregated and reduced in dimensionality based on the masking information. The features obtained by the upsampling operation are then decoded by the corresponding MLP decoder to output the scene reconstruction results. The specific steps are as follows:

[0039] First, perform layer normalization:

[0040] ;

[0041] ;

[0042] The original query vector and scene features are then mapped to corresponding query (Q), key (K), and value (V) matrices:

[0043] ;

[0044] Calculate the scaled dot-product similarity between the query and key vectors, where... It's a scaling item. The additive mask in the mask matrix represents the key position; its value is the value when the key-corresponding element is masked. Its value is 0 when it is not masked.

[0045] ;

[0046] The score for each query is converted into weights using a softmax operation:

[0047] ;

[0048] in, This represents the attention weight of the i-th token to the j-th token. The result of the attention operation is:

[0049] ;

[0050] The output of the attention operation and Perform residual connections to obtain dimensionality-reduced features. ;

[0051] ;

[0052] The latent space interaction layer consists of N pre-normalized self-attention sublayers connected sequentially, with the output of the previous self-attention block serving as the input of the next self-attention block; the feature vector obtained by downsampling... Interaction features are obtained after latent space interaction processing. ;

[0053] ;

[0054] The upsampling layer is responsible for transforming the latent space vectors into appropriate dimensions; specifically, it transforms the two types of vectors described in step 25. Two query matrices are obtained after mapping the corresponding query matrices. Output features of the latent space interaction layer After key-value matrix mapping, K and V matrices are obtained; cross-attention operations are then performed on each matrix to obtain the features corresponding to the environment reconstruction task. and the characteristics of the corresponding vehicle historical trajectory reconstruction task ;

[0055] ;

[0056] and First, the components are processed through their respective feedforward neural networks (FNNs), and then reconstructed using their respective MLP decoders to obtain the environmental element reconstruction results. Reconstruction results of vehicle historical trajectory points ;

[0057] Step 25: To adapt to the upsampling requirements of different tasks, this method constructs two types of vectors: vectors for environment reconstruction tasks. The location encoding is obtained by adding the learnable vector and the lane element range embedding; the location encoding is obtained by extracting the coordinate information of four extreme values ​​(top, bottom, left, and right) from each route and embedding them.

[0058] ;

[0059] Vectors used for historical trajectory reconstruction tasks It is obtained by adding the learnable vector and the temporal encoding of the historical time step;

[0060] ;

[0061] Step 26: The pre-training loss consists of historical trajectory reconstruction loss and environment reconstruction loss; both historical trajectory reconstruction loss and environment reconstruction loss are calculated using the l2 norm error between the decoded position coordinates and the true coordinates; It is the loss weight coefficient, used to adjust the relative importance of the reconstruction loss of the masked part and the reconstruction loss of the non-masked part during the model optimization process;

[0062] ;

[0063] Step 3: After the pre-trained model converges, deactivate the pre-trained components and freeze the parameters of the remaining components; add the necessary decoder structure for fine-tuning, as follows:

[0064] Step 31: Deactivate all components from the upsampling head to the reconstruction decoder in the converged pre-trained model, freeze the parameters of the remaining components, remove the masking of the encoder input features and the noise addition to logits in the query vector scorer; add an upsampling head and a feedforward layer (FFN) to serve the fine-tuning task. Then, concatenate and fuse the multimodal trajectory decoding module with the DERT and Mamba blocks after the FFN layer.

[0065] Step 32: This upsampling head is also based on cross-attention, and its query vector is a learnable vector. After the query vector and latent space features are mapped accordingly, cross-attention operation is performed, and after processing by the feedforward layer, the decoder input features are obtained.

[0066] ;

[0067] ;

[0068] Step 33: The decoder consists of a pre-normalized multimodal cross-attention block, a hybrid expert feedforward layer (MoE), a Mamba block, and a score decoding MLP;

[0069] The multimodal cross-attention block is implemented by initializing a series of learnable query vectors. Each is transformed by its own linear mapping. After processing, a query matrix corresponding to M modes is obtained. And concatenate multiple queries along the modal axis to obtain The dimensions were adjusted to meet the standard dimensional requirements for attention operations.

[0070] ;

[0071] Will After key-value mapping, a key-value matrix is ​​obtained. Then, in conjunction with the aforementioned query matrix Perform cross-attention calculations to obtain multimodal features. ;

[0072] ;

[0073] Then The data is fed into the MoE feedforward layer for processing to obtain multimodal features for decoding. Experts selected a TOP-K strategy; specifically, First, the selection probability of each expert in the FFN layer is obtained through the linear routing module, and the K experts with the highest probability are selected based on the softmax result. Noise is also added to the logits here, and noise variance scheduling is used.

[0074] ;

[0075] ;

[0076] Based on the probability without noise, perform a weight renormalization operation on the selected K experts:

[0077] ;

[0078] ;

[0079] ;

[0080] The corresponding tokens are allocated to the corresponding K expert FFN layers, and the output results are weighted according to the above renormalized weights;

[0081] ;

[0082] Step 34: The output feature residuals of the weighted MoE layer are concatenated and normalized, and then input into the Mamba-MLP block and the trajectory score MLP decoder for decoding, respectively.

[0083] Directly in the trajectory scoring decoder The decoding map is used to obtain scores for the next M modalities;

[0084] In the Mamba-MLP block, the data is expanded to include a time dimension and copied an equal number of times to future time steps:

[0085] ;

[0086] Future time-encoded features are obtained using the embedding method described in step 21. and through MLP The scaling factor tensor and bias tensor required for mapping to Film transform .right Perform affine transformation:

[0087]

[0088] Will The data is fed into a Mamba block for processing, and the output is mapped to a specified dimension through an MLP layer.

[0089] Step 35, the fine-tuning loss consists of three parts, including the optimal modal regression loss. Optimal mode score cross-entropy loss and balance load loss; It is a hybrid expert layer that balances load loss. The frequency at which experts are selected in a batch. For expert routing probability, where It is a weighting coefficient used to adjust the proportion of load balancing losses;

[0090] ;

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] ;

[0096] ; Attached Figure Description

[0097] Figure 1 This is a model framework diagram of the present invention. Detailed Implementation

[0098] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be understood that this example is for illustrative purposes only and is not intended to limit the scope of the invention. A model diagram of a multimodal trajectory prediction method for autonomous vehicles integrating Perceiver IO and DERT-Mamba provided by the present invention is shown below. Figure 1 As shown, the specific implementation method includes the following steps:

[0099] Step 1: Select a vehicle as your ego vehicle, and then select surrounding vehicles and environmental elements within a 150-meter radius of your ego vehicle. For vehicle elements, select up to nine vehicles closest to your ego vehicle and obtain the information for each vehicle. The trajectory at each time step; the trajectory points at each time step should include the following features: vehicle position coordinates (x, y), orientation (x, y), and direction (y, y). ), lateral and longitudinal speeds ( , The model is designed to include the vehicle's length and width (L, W); for environmental elements, a maximum of 32 lane elements closest to the vehicle are selected, and 20 points are retained for each polyline using interpolation downsampling; the selected lane polyline information includes centerline coordinates, road boundary coordinates, road speed limits, and traffic light information, and the points in the polyline are vectorized using a difference method; when scene elements are insufficient, padding should be used to ensure format consistency, and a mask should be used to ensure that the padding data does not interfere with normal interaction in subsequent processing; in the pre-training stage, the required data is extracted from multiple datasets for training, and in the fine-tuning stage, the required data is extracted from the target dataset for training the model.

[0100] To better measure the relative spatial position information between vehicles, the positional relationships of elements in the scene are represented by relative coordinates and relative orientation angles. Specifically, based on the rigid body secondary transformation method, the coordinates of all elements are transformed to a coordinate system with the current pose of the vehicle as the origin, and their orientation angles are rotated. The above features can be represented as the following two categories: vehicle trajectory features. Lane characteristics .

[0101] Step 2 involves feeding the two types of features from Step 1 into an embedding layer to obtain a high-dimensional feature vector, which is then fed into an improved Perceiver IO encoder. In the encoder, each scene element is first randomly masked at a certain ratio, then sequentially passed through a downsampling layer, a latent space interaction layer, and an upsampling layer. Finally, the upsampled features are decoded through their respective feedforward layers and the reconstruction decoder to output the corresponding features. The specific implementation steps of Step 2 are as follows:

[0102] Step 21: The two types of scene feature vectors mentioned in Step 1 are first input into the vehicle and environment embedding layers respectively to obtain high-dimensional feature representations, and then position encoding is introduced, as shown in the following formula:

[0103] ;

[0104] ;

[0105] (Embedding) represents the operation of projecting relevant features onto a specified dimension through a linear layer, where the location encoding corresponding to vehicle features includes the absolute position of the vehicle. Embedded and history timestep vectors, the position encoding corresponding to the lane features is the vehicle absolute position embedding; where the vehicle absolute position vector is obtained by processing the absolute position of the vehicle in the current scene through MLP, and the history time vector is obtained by processing the sinusoidal position encoding of each time step through an MLP.

[0106] ;

[0107] ;

[0108] ;

[0109] Step 22 involves constructing a query vector pool for feature dimensionality reduction and designing a routing mechanism for the query vector pool to adapt to the key information extraction needs in different application scenarios. Specifically, the scene features S concatenated in Step 23 are first subjected to layer normalization and then mapped to corresponding layers of a multilayer perceptron (MLP). The processing order is as follows:

[0110] ;

[0111] ;

[0112] Where Linear represents a linear layer, ReLU represents the activation function, and Layernorm represents the layer normalization operation, as shown in the formula:

[0113] ;

[0114] For the output Gaussian noise with variance scheduled during training is added to obtain This allows for the full exploration of each query vector using a large variance in the early stages of training, and the use of a small variance to stabilize the training in the later stages.

[0115] Regarding the 'and Perform the SoftMax operation to obtain the selection probability of each candidate vector in the vector pool. and weighted weights ;

[0116] ;

[0117] ;

[0118] During training, each dimensionality reduction query vector in the vector pool is selected and updated using a straight-through strategy: specifically, the vector with the highest probability is selected through argmax operation during forward computation. Dimensionality reduction vector of the scene ,based on The loss is calculated during forward propagation, and then during backward propagation... The weighted vector obtained by the weights This updates all vectors in the vector pool to prevent mode collapse; where sg indicates stopping gradient calculation.

[0119] ;

[0120] ;

[0121] Step 23: Randomly create mask matrices for the two types of features, wherein the mask matrix is ​​created by randomly selecting features from a certain time step for the vehicle features. A mask matrix is ​​created by randomly selecting a certain number of points based on the lane features. For the two scene features and Perform dimensional merging and stitching, specifically: merge vehicle historical features into the vehicle quantity... With the time dimension The above merging will incorporate environmental characteristics in the dimension of lane segment quantity. Dimension of Number of Points Merge above, and similarly apply to and Perform similarity dimension merging and concatenation operations:

[0122] ;

[0123] ;

[0124] ;

[0125] ;

[0126] The merged new dimension is uniformly defined as the token dimension, resulting in a new scene vector. and mask matrix ,in ;

[0127] Step 24: In the pre-training stage, the encoder mainly consists of a downsampling layer, a latent space interaction layer, and two upsampling layers; the query vector obtained in step 22 is then used... and the scene feature vector obtained in step 23 The input downsampling layer is first normalized; then... Obtained through linear mapping matrix, After two different linear mappings, we obtain... The matrix is ​​used to perform dot product attention operations, and the scene features are aggregated and reduced in dimensionality based on the masking information. The features obtained by the upsampling operation are then decoded by the corresponding MLP decoder to output the scene reconstruction results. The specific steps are as follows:

[0128] First, perform layer normalization:

[0129] ;

[0130] ;

[0131] The original query vector and scene features are then mapped to corresponding query (Q), key (K), and value (V) matrices:

[0132] ;

[0133] Calculate the scaled dot-product similarity between the query and key vectors, where... It's a scaling item. The additive mask in the mask matrix represents the key position; its value is the value when the key-corresponding element is masked. Its value is 0 when it is not masked.

[0134] ;

[0135] The score for each query is converted into weights using a softmax operation:

[0136] ;

[0137] in, This represents the attention weight of the i-th token to the j-th token. The result of the attention operation is:

[0138] ;

[0139] The output of the attention operation and Perform residual connections to obtain dimensionality-reduced features. ;

[0140] ;

[0141] The latent space interaction layer consists of N pre-normalized self-attention sublayers connected sequentially, with the output of the previous self-attention block serving as the input of the next self-attention block; the feature vector obtained by downsampling... Interaction features are obtained after latent space interaction processing. ;

[0142] ;

[0143] The upsampling layer is responsible for transforming the latent space vectors into appropriate dimensions; specifically, it transforms the two types of vectors described in step 25. Two query matrices are obtained after mapping the corresponding query matrices. Output features of the latent space interaction layer After key-value matrix mapping, K and V matrices are obtained; cross-attention operations are then performed on each matrix to obtain the features corresponding to the environment reconstruction task. and the characteristics of the corresponding vehicle historical trajectory reconstruction task ;

[0144] ;

[0145] and First, the components are processed through their respective feedforward neural networks (FNNs), and then reconstructed using their respective MLP decoders to obtain the environmental element reconstruction results. Reconstruction results of vehicle historical trajectory points ;

[0146] Step 25: To adapt to the upsampling requirements of different tasks, this method constructs two types of vectors: vectors for environment reconstruction tasks. The location encoding is obtained by adding the learnable vector and the lane element range embedding; the location encoding is obtained by extracting the coordinate information of four extreme values ​​(top, bottom, left, and right) from each route and embedding them.

[0147] ;

[0148] Vectors used for historical trajectory reconstruction tasks It is obtained by adding the learnable vector and the temporal encoding of the historical time step;

[0149] ;

[0150] Step 26: The pre-training loss consists of historical trajectory reconstruction loss and environment reconstruction loss; both historical trajectory reconstruction loss and environment reconstruction loss are calculated using the l2 norm error between the decoded position coordinates and the true coordinates; It is the loss weight coefficient, used to adjust the relative importance of the reconstruction loss of the masked part and the reconstruction loss of the non-masked part during the model optimization process;

[0151] ;

[0152] Step 3: After the pre-trained model converges, deactivate the pre-trained components and freeze the parameters of the remaining components; add the necessary decoder structure for fine-tuning, as follows:

[0153] Step 31: Deactivate all components from the upsampling head to the reconstruction decoder in the converged pre-trained model, freeze the parameters of the remaining components, remove the masking of the encoder input features and the noise addition to logits in the query vector scorer; add an upsampling head and a feedforward layer (FFN) to serve the fine-tuning task. Then, concatenate and fuse the multimodal trajectory decoding module with the DERT and Mamba blocks after the FFN layer.

[0154] Step 32: This upsampling head is also based on cross-attention, and its query vector is a learnable vector. After the query vector and latent space features are mapped accordingly, cross-attention operation is performed, and after processing by the feedforward layer, the decoder input features are obtained.

[0155] ;

[0156] ;

[0157] Step 33: The decoder consists of a pre-normalized multimodal cross-attention block, a hybrid expert feedforward layer (MoE), a Mamba block, and a score decoding MLP;

[0158] The multimodal cross-attention block is implemented by initializing a series of learnable query vectors. Each is transformed by its own linear mapping. After processing, a query matrix corresponding to M modes is obtained. And concatenate multiple queries along the modal axis to obtain The dimensions were adjusted to meet the standard dimensional requirements for attention operations.

[0159] ;

[0160] Will After key-value mapping, a key-value matrix is ​​obtained. Then, in conjunction with the aforementioned query matrix Perform cross-attention calculations to obtain multimodal features. ;

[0161] ;

[0162] Then The data is fed into the MoE feedforward layer for processing to obtain multimodal features for decoding. Experts selected a TOP-K strategy; specifically, First, the selection probability of each expert in the FFN layer is obtained through the linear routing module, and the K experts with the highest probability are selected based on the softmax result. Noise is also added to the logits here, and noise variance scheduling is used.

[0163] ;

[0164] ;

[0165] Based on the probability without noise, perform a weight renormalization operation on the selected K experts:

[0166] ;

[0167] ;

[0168] ;

[0169] The corresponding tokens are allocated to the corresponding K expert FFN layers, and the output results are weighted according to the above renormalized weights;

[0170] ;

[0171] Step 34: The output feature residuals of the weighted MoE layer are concatenated and normalized, and then input into the Mamba-MLP block and the trajectory score MLP decoder for decoding, respectively.

[0172] Directly in the trajectory scoring decoder The decoding map is used to obtain scores for the next M modalities;

[0173] In the Mamba-MLP block, the data is expanded to include a time dimension and copied an equal number of times to future time steps:

[0174] ;

[0175] Future time-encoded features are obtained using the embedding method described in step 21. and through MLP The scaling factor tensor and bias tensor required for mapping to Film transform .right Perform affine transformation:

[0176]

[0177] Will The data is fed into a Mamba block for processing, and the output is mapped to a specified dimension through an MLP layer.

[0178] Step 35, the fine-tuning loss consists of three parts, including the optimal modal regression loss. Optimal mode score cross-entropy loss and balance load loss; It is a hybrid expert layer that balances load loss. The frequency at which experts are selected in a batch. For expert routing probability, where It is a weighting coefficient used to adjust the proportion of load balancing losses;

[0179] ;

[0180] ;

[0181] ;

[0182] ;

[0183] ;

[0184] ;

[0185] .

Claims

1. A multimodal trajectory prediction method for autonomous vehicles integrating Perceiver-IO and DERT-Mamba, characterized in that, Includes the following steps: (1) Collect scene information; perform feature embedding processing on the converted scene elements, add position encoding information, generate the corresponding mask matrix, and merge and splice the dimensions of each modality feature to form a unified input representation; (2) The feature vector obtained in step 1 is fed into the Perceiver IO encoder and passed through the downsampling layer, the latent space interaction layer and the upsampling layer in sequence to realize the compression, interaction and reconstruction of the input features. In the downsampling layer, in order to improve the learning effect, masked cross attention is adopted, and the corresponding dimension reduction query vector is matched based on the input features to realize targeted information compression. In the upsampling layer, two types of query vectors, trajectory and lane, are constructed to reconstruct the features. The model parameters are optimized by minimizing the historical trajectory reconstruction loss and the lane reconstruction loss to complete the pre-training. (3) Freeze the parameters of the pre-trained model, set the decoding branch of historical trajectory and lane reconstruction to an inactive state, and cancel the masking of the data and the addition of noise in the query vector pool; add an upsampled cross attention block for the future trajectory prediction of fine-tuning; introduce an improved DERT-Mamba multimodal decoder to further interact and fuse features, and finally decode and generate multiple candidate trajectories and their corresponding trajectory scores; update the model based on the optimal modal regression loss and score loss during the fine-tuning stage.

2. The trajectory prediction method according to claim 1, characterized in that, The two types of scene feature vectors are first input into two embedding layers, one for vehicles and one for the environment, to obtain high-dimensional feature representations, and then position encoding (PE) is introduced into them: ； ； The location encoding of vehicle features includes the vehicle's absolute location embedding and the timestamp embedding of all historical time steps. The location encoding of lane features is achieved by embedding the absolute position of the vehicle. ； ; ； Randomly select a certain number of elements from each token to mask them, generating the corresponding mask matrix. and the features of the two scenarios mentioned above and Perform dimension merging and concatenation, specifically: ； ； ； ； The merged new dimension is uniformly defined as the token dimension, and the two types of feature vectors are concatenated along this token dimension to obtain a new scene vector. ,in .

3. The trajectory prediction method according to claim 1, characterized in that, An updatable dimensionality reduction query vector pool is designed for dimensionality reduction operations; scene features are first mapped to corresponding logits through a routing module composed of a multilayer perceptron (MLP). ； And Gaussian noise with variance scheduled as the training process is added to logits: ; Then, a SoftMax operation is performed on the two types of logits to obtain the selection probability of each candidate vector. and weighted weights And select the corresponding vector based on probability: ； ； ； in accordance with Weighted fusion of vectors to obtain The query vector propagation formula is: ； During forward propagation, the optimal query vector is selected based on the noisy SoftMax calculation results. The loss is calculated based on this vector; during backpropagation, all vectors in the vector pool are weighted according to the noise-free SoftMax result, thereby updating all vector parameters simultaneously. The query matrix Q is obtained by mapping y through a query, representing the scene characteristics. After key-value mapping, key-value matrices K and V are obtained. Then, dot-product cross-attention operations with additive masks are performed on the Q, K, and V matrices to obtain low-dimensional feature vectors. ; 。 4. The trajectory prediction method according to claim 1, characterized in that, For the scene reconstruction and historical trajectory reconstruction tasks in the pre-training process, a query vector is constructed; Vectors used for environmental reconstruction tasks It is formed by adding the learnable vector and the lane element range embedding; The location encoding is obtained by extracting the coordinate information of four extreme points (top, bottom, left, and right) from the route and embedding them; Vectors used for historical trajectory reconstruction tasks It is obtained by adding the learnable vector and the time code of the historical time step.

5. The trajectory prediction method according to claim 1, characterized in that, In the upsampling layer of the pre-training phase, the query vector is respectively interacted with the features that have undergone latent space interaction. Perform cross-attention operations to obtain upsampled feature vectors for corresponding environment reconstruction and historical trajectory reconstruction, and then send them to their respective decoders after passing through the corresponding interaction layers to obtain the decoding output; ； and First, the elements are processed through their respective feedforward layers (FNN), and then reconstructed using their respective MLP decoders to obtain the environmental element reconstruction results. Results of vehicle historical trajectory point reconstruction ; The pre-training loss consists of historical trajectory reconstruction loss and environmental feature reconstruction loss, both of which are calculated using the l2 norm error between the decoded location coordinates and the true coordinates. ; in It is the loss weight coefficient, used to adjust the relative importance of the reconstruction loss of the masked part and the reconstruction loss of the non-masked part during the model optimization process.

6. The trajectory prediction method according to claim 1, characterized in that, During the fine-tuning phase, the upsampling layer of the encoder of the model deletes two reconstruction branches. Specifically, all architectures from the upsampling head to the reconstruction decoder are set to an inactive state; the masking of the encoder input data is removed, and the parameters of the remaining architectures are frozen; based on the above-mentioned frozen architectures, a cross-attention-based upsampling head is added for fine-tuning, wherein the query vector of the upsampling head is a learnable vector. ； 。 7. The trajectory prediction method according to claim 1, characterized in that, Initialize a series of learnable query vectors Each is transformed through its own linear mapping ( After processing, a query matrix corresponding to M modes is obtained. And concatenate multiple queries along the modal axis to obtain ; 。 8. The trajectory prediction method according to claim 1, characterized in that, Following the aforementioned upsampling module, a fused DERT-Mamba decoder is connected. Specifically, the feedforward layer of the DERT decoder is modified into a hybrid expert layer (MoE) composed of multiple FFNs, and Mamba blocks are connected after the MoE layer to fuse interactive features. The routing module of the MoE layer also incorporates scheduled noise during the training phase to fully explore each expert. The features output by the MoE layer... First, expand the time dimension and replicate it T times; ; Future time coding features are obtained using the embedding method described in claim 2. and through MLP Perform mapping to obtain the scaling factor tensor required for Film transformation. and bias tensor ,right Perform an affine transformation to transform the features Input Mamba block processing; The output of the Mamba block is mapped to a specified dimension by an MLP, and the output of the MoE layer is decoded into modality scores by another MLP. The fine-tuning loss consists of three parts, including the optimal modal regression loss. Optimal mode score cross-entropy loss and balance load loss; in These are weighting coefficients used to adjust the importance of the single-modal regression loss in the total loss. It is a hybrid expert layer that balances load loss. The frequency at which experts are selected in a batch. For expert routing probability; ； ； ； ； ; ; 。

Images