An autonomous driving trajectory prediction method and system based on trajectory decoupling and Mamba

Abstract

The present invention discloses an automatic driving trajectory prediction method based on trajectory decoupling and Mamba, comprising the following steps: obtaining a high-definition map and traffic participants in a driving scene, encoding the high-definition map and traffic participants in the driving scene using a pre-trained encoder to obtain scene context features composed of the encoded high-definition map and traffic participants, and inputting the obtained scene context features into a pre-trained query decoupling decoder to obtain a multimodal future trajectory corresponding to each concerned traffic participant among the traffic participants. The present invention can solve the technical problem that the trajectory prediction of the existing automatic driving trajectory prediction method is not accurate enough, and may produce suboptimal results, thereby limiting the accuracy and practicality of trajectory prediction during driving, and the technical problem that the method lacks an effective solution to combine the respective advantages of Mamba and attention mechanism.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and sequence prediction, and more specifically, to an autonomous driving trajectory prediction method and system based on trajectory decoupling and Mamba. Background Art

[0002] Trajectory prediction of traffic participants plays a key role in autonomous driving research, and accurate trajectory prediction is crucial for downstream ego-vehicle path planning. Therefore, trajectory prediction has become one of the cores of deep learning research in the field of autonomous driving, showing great research potential and application value.

[0003] The trajectory prediction task aims to predict multiple possible future movement paths of traffic participants based on high-precision maps and their historical trajectories. In recent years, research progress in this field has not only focused on scene representation and modeling techniques, but also achieved significant breakthroughs in accurate prediction. With the development of state-space models, a model called Mamba has become a promising alternative to Transformer in sequence modeling, especially due to its effectiveness and efficiency in capturing long-term dependencies.

[0004] Existing trajectory prediction methods mainly follow a specific paradigm inspired by the field of object detection, mainly based on the Transformer model, namely the single query-single trajectory paradigm. This paradigm utilizes several queries to represent a multimodal estimated trajectory covering different directional intents.

[0005] However, the above existing autonomous driving trajectory prediction methods have some non-negligible defects:

[0006] First, the above method adopts a single query-single trajectory paradigm, where each query corresponds to a specific trajectory to predict multimodal trajectories. This paradigm only generally represents each trajectory that needs to be predicted, lacks a detailed representation of future trajectories, and lacks modeling of dynamic changes in the trajectory in the time dimension. In real-world applications, trajectory prediction should accurately predict the direction information of each multimodal trajectory and the dynamic change information of each trajectory. Therefore, the trajectory prediction of this method is not accurate enough and may produce suboptimal results, thereby limiting the accuracy and practicality of trajectory prediction during driving.

[0007] Second, since the above method is mainly based on the Transformer model, it lacks the exploration of effective solutions to combine the respective advantages of Mamba and the attention mechanism. The Mamba model has become a promising alternative to the Transformer due to its effectiveness and efficiency in sequence modeling. Therefore, this method lacks an effective solution to combine the respective advantages of Mamba and the attention mechanism. Summary of the invention

[0008] In view of the above defects or improvement needs of the prior art, the present invention provides an autonomous driving trajectory prediction method and system based on trajectory decoupling and Mamba, which aims to solve the technical problem that the trajectory prediction of the existing autonomous driving trajectory prediction method is not accurate enough, which may produce suboptimal results, thereby limiting the accuracy and practicality of trajectory prediction during driving, and the technical problem that the method lacks an effective solution to combine the respective advantages of Mamba and attention mechanism.

[0009] To achieve the above object, according to one aspect of the present invention, a method for automatic driving trajectory prediction based on trajectory decoupling and Mamba is provided, comprising the following steps:

[0010] (1) Obtain a high-definition map and traffic participants in a driving scene, and use a pre-trained encoder to encode the high-definition map and traffic participants in the driving scene to obtain scene context features composed of the encoded high-definition map and traffic participants.

[0011] (2) The scene context features obtained in step (1) are input into a pre-trained query-decoupled decoder to obtain the multimodal future trajectory corresponding to each of the traffic participants of interest among the traffic participants obtained in step (1).

[0012] Preferably, the scene encoder includes a map encoder, a history track encoder and an interaction encoder.

[0013] The map encoder uses the PointNet network, whose input is of dimension N m ×L×2 high-precision map vector, where N m Represents the total number of HD map vectors, L represents the number of points contained in each HD map vector, and feature dimension 2 represents the coordinate value of the map under the bird's-eye view; its dimension is N m ×L×2 high-precision map vectors are processed locally and globally, and the output dimension is N m ×C map feature tensor F m , where C represents the dimension of the map feature tensor.

[0014] The history track encoder uses multiple unidirectional Mamba modules, whose input is N-dimensional a ×H×4 historical trajectory vectors of traffic participants, where N a represents the total number of traffic participants, H represents the number of steps of the historical trajectory vector, and the feature dimension 4 represents the horizontal coordinate, vertical coordinate, speed on the horizontal coordinate, and speed on the vertical coordinate of the traffic participant, which is N a ×H×4 traffic participants’ historical trajectory vectors are encoded and processed, and the output is a vector with dimension Na ×C object feature tensor F a .

[0015] The interactive encoder uses a standard Transformer network, whose input is of dimension N m ×C map feature tensor F m With dimension N a ×C object feature tensor F a , after connecting the two, the output dimension is (N a +N m )×C current scene context feature tensor F s .

[0016] Preferably, the decoder based on query decoupling includes a modal positioning module, a state continuity module and a hybrid integration module, and its specific structure is:

[0017] The modality localization module is implemented based on the attention mechanism, and its input is initialized with a dimension of N fa ×K×C modal query Q m , represents the multimodal trajectory of the traffic participant of interest, and the dimension of the interaction encoder output is (N a +N m )×C current scene context feature tensor F s , where N fa represents the number of concerned traffic participants, K represents the number of multimodal trajectories, and the modal localization module converts the modal query Q into m and the current scene context feature tensor F s Process and output the preliminary multimodal trajectory Y m and probability P m , used to assist supervision;

[0018] The state continuous module is implemented based on the attention mechanism and Mamba, and its input is initialized to a dimension of N fa ×T s ×C status query Q s , represents the dynamic state trajectory of the traffic participant of interest, and the dimension of the interaction encoder output is (N a +N m )×C current scene context feature tensor F s , where T s is the time step represented by the state query. The state continuous module uses the attention mechanism to s and the current scene context feature tensor F s Processing, and querying the status Q through the Mamba model s The self-sequence is scanned interactively and a preliminary single trajectory Y is outputs , used to assist supervision;

[0019] The hybrid integration module is implemented based on the attention mechanism and Mamba, and its input is the state query Q s , Modal Query Q m The dimension of the output of the interactive encoder is (N a +N m )×C current scene context feature tensor F s , which combines state query and modal query into a dimension of N fa ×K×T s ×C mixed query Q h , and the mixed query Q h and the current scene context feature tensor F s Processing and using the Mamba model to query the mixed query Q h The self-sequence is scanned interactively to output the final multimodal trajectory Y h and probability P h As the trajectory prediction result.

[0020] Preferably, the autonomous driving trajectory prediction model based on trajectory decoupling Mamba is trained by the following steps:

[0021] (2-1) Obtain an autonomous driving dataset for trajectory prediction, which includes historical and future trajectories of traffic participants and high-precision maps, preprocess the autonomous driving dataset to obtain a preprocessed autonomous driving dataset, and use the vectorization method in the VectorNet model to convert the preprocessed autonomous driving dataset into a vector format to obtain a format-converted autonomous driving dataset, and divide the format-converted autonomous driving dataset into a training set and a validation set in a ratio of 9:1.

[0022] (2-2) For each autonomous driving scene segment in the training set obtained in step (2-1), the high-precision map vector M (whose dimension is N) in the autonomous driving scene segment is m ×L×2) and the historical trajectory vector A of the traffic participants (whose dimension is N a ×H×4) are input into the map encoder and the history track encoder respectively to obtain the dimensions of N m ×C map feature tensor F m , dimension is N a ×C object feature tensor F a , and combine the two through the interactive encoder to obtain a dimension of (N a +N m )×C scene context feature tensor F s .

[0023] (2-3) Initialization status query Q S , query Q for this state S And the scene context feature tensor F obtained in step (2-2) s Perform interactive processing to obtain the optimized state query Q S , and query the status Q s Decoded as future trajectory Y s (It serves as auxiliary supervision).

[0024] (2-4) Initialize modal query And query Q for this modality m And the scene context feature tensor F obtained in step (2-2) s Perform interactive processing to obtain the optimized modal query Q m , and query Q for the optimized modal m Decode to obtain the future multimodal trajectory Y m and probability P m , which is used as auxiliary supervision.

[0025] (2-5) Query the status Q obtained in step (2-3) s and the modal query Q obtained in step (2-4) m Combined processing to obtain hybrid queries containing spatiotemporal information And use the hybrid integration module composed of attention mechanism and Mamba to deal with the mixed query Q h Further processing is performed and the processing results are decoded using MLP to obtain the final multimodal trajectory Y h and probability P h .

[0026] (2-6) The future trajectory Y obtained according to step (2-3) s , the future multimodal trajectory Y obtained in steps (2-4) m and probability P m , and the final multimodal trajectory Y obtained in steps (2-5) h and probability P h The loss function is calculated and the model is iteratively trained using the loss function until the model converges, thereby obtaining a trained autonomous driving trajectory prediction model based on trajectory decoupling and Mamba.

[0027] Preferably, the autonomous driving datasets used in step (2-1) are Argoverse 2 and nuScenes datasets. The Argoverse 2 dataset includes 250,000 scenes with a sampling frequency of 10 Hz, each of which has a 5-second historical trajectory length and predicts the trajectory length of the next 6 seconds. The nuScenes dataset contains 1,000 2 Hz scenes and predicts the future 6-second trajectory from the past 2-second history. Both datasets use high-precision maps with a radius of 150 meters and traffic participants.

[0028] Preferably, in step (2-2), the map encoder adopts a PointNet network, which first maps the high-precision map vector M into a high-dimensional feature through a multi-layer perceptron, and then processes the high-precision map vector M using a maximum pooling operation to obtain a global feature corresponding to the high-precision map vector M. Then, the global feature is connected to each point feature in the high-precision map vector M to obtain a point feature containing global information. Then, the above mapping operation and maximum pooling operation are repeated for all point features, thereby obtaining a high-dimensional feature with a dimension of N. m ×C map feature tensor F m .

[0029] The historical trajectory encoder uses several unidirectional Mamba modules to encode the historical trajectory vectors of traffic participants, and the output is a vector with dimension N. a ×C object feature tensor F a .

[0030] The interactive encoder uses a standard Transformer network, and the dimension of the map encoder output is N m ×C map feature tensor F m And the dimension of the historical trajectory encoder output is N a ×C object feature tensor F a Perform concatenation to obtain a dimension of (N a +N m )×C scene context feature tensor F s .

[0031] Preferably, step (2-3) is specifically as follows: first, initialize the status query Where R represents the real number domain, N fa The number of traffic participants who expressed concern, T s Indicates the status query step length;

[0032] Then, the scene context feature tensor F is transformed using the state continuity module. s Status Query S Perform interactive processing to obtain the optimized state query Q S .

[0033] Afterwards, use bidirectional Mamba to query the status Q S Perform forward and reverse scanning to obtain the optimized state query Q S .

[0034] Finally, the multi-layer perceptron MLP module is used to transform the state query Q S Decoded into a single future trajectory Y s ;

[0035] The process of step (2-3) can be described as follows:

[0036]

[0037] Q s =MultiHeadAttention(Q=Q s ,K,V=F s )

[0038] Q s =BiMamba(Q s )

[0039] Y s =MLP(Q s )

[0040] in Indicates the state query step length T s The corresponding real time step, Q represents the query of the attention mechanism, K represents the key of the attention mechanism, V represents the value of the attention mechanism, MultiHeadAttention represents the multi-head attention mechanism, and BiMamba represents the bidirectional Mamba.

[0041] Preferably, step (2-4) is specifically as follows: first, initialize the modal query Where N fa represents the number of traffic participants of interest, and K represents the number of multimodal trajectories.

[0042] Then, the modality query Q is localized using the modality localization module (which uses two attention mechanism modules). m and scene context F s Perform interactive processing to obtain the optimized modal query Q m .

[0043] Finally, the modal query Q is transformed into m Decoded into future multimodal trajectory Y m and probability P m .

[0044] The process of this step can be described as follows:

[0045] Q m =MultiHeadAttention(Q=Q m ,K,V=F s )

[0046] Q m =MultiHeadAttention(Q=Q m ,K,V=Q m )

[0047] Y m ,P m =MLP(Q m )

[0048] Preferably, step (2-5) is specifically as follows: first, an attention mechanism module is used to query the state Q obtained in step (2-3) s and the modal query Q obtained in step (2-4) m Combine the query Q with temporal and spatial information h .

[0049] Then, the hybrid query Q h Perform cross-time-step and cross-modal hybrid interaction processing to obtain the optimized hybrid query Q h ;

[0050] Then, an attention mechanism module is used to optimize the mixed query Q h Interact between different modalities to obtain a further optimized hybrid query Q h .

[0051] Then, the bidirectional Mamba module is used to further optimize the hybrid query Q h Perform forward and reverse scanning to obtain the final optimized hybrid query Q h ;

[0052] Finally, MLP is used to optimize the final mixed query Q h Decode and finally output the multimodal trajectory Y h and probability P h .

[0053] The process of this step can be described as follows:

[0054] Q h =MultiHeadAttention(Q=Q h ,K,V=F s )

[0055] Q h=HybridMultiHeadAttention(Q=Q h ,K,V=Q h )

[0056] Q h =ModeMultiHeadAttention(Q=Q h ,K,V=Q h )

[0057] Q h =BiMamba(Q h )

[0058] Y h ,P h =MLP(Q h )

[0059] Among them, HybridMultiHeadAttention represents the multi-head attention mechanism of the hybrid self-attention module, and ModeMultiHeadAttention represents the multi-head attention mechanism for modal interaction.

[0060] The total loss function Loss in step (2-6) is:

[0061] Loss=Loss reg +Loss cls +Loss state +Loss mode

[0062] in:

[0063] Loss reg Represents the loss function used to supervise the prediction trajectory, which uses the SMOOTH-L1 function, Loss cls Represents the loss function used to predict probability, which uses the cross entropy function, Loss mode and Loss state represents the loss function for auxiliary output, and the above loss functions are expressed as:

[0064]

[0065] in Indicates the loss function using the SMOOTH-L1 function, L CE represents the loss function using the cross entropy function, represents the ground truth of the future trajectory, Represents the true value of the probability of future multimodal trajectories.

[0066] According to another aspect of the present invention, there is provided an autonomous driving trajectory prediction system based on trajectory decoupling and Mamba, comprising:

[0067] The first module is used to obtain high-definition maps and traffic participants in driving scenes, and encode the high-definition maps and traffic participants in driving scenes using a pre-trained encoder to obtain scene context features composed of the encoded high-definition maps and traffic participants.

[0068] The second module is used to input the scene context features obtained by the first module into a pre-trained query decoupled decoder to obtain the multimodal future trajectory corresponding to each traffic participant of interest among the traffic participants obtained by the first module.

[0069] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0070] (1) The present invention adopts step (2), which decouples the multimodal trajectory query into a modal query and a state query to respectively represent the directional intention and the dynamic state, thereby respectively optimizing the multimodal characteristics and dynamic evolution characteristics of the trajectory. Therefore, the present invention can solve the problem of the existing methods lacking modeling of dynamic change information of the trajectory in the time dimension.

[0071] (2) The present invention adopts steps (2-3) to (2-5), which uses a modal localization module based on the attention mechanism and a state continuity module based on the combination of the attention mechanism and Mamba to respectively optimize the modal query and the state query, and uses a hybrid integration module based on the combination of the attention mechanism and Mamba to optimize the hybrid query obtained by combining the modal query and the state query, thereby obtaining a comprehensive and detailed representation of the trajectory. Therefore, the present invention can make a more accurate prediction of future multimodal trajectories, thereby solving the problem that the existing methods lack an effective solution to effectively combine the respective advantages of Mamba and the attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a flow chart of the automatic driving trajectory prediction method based on trajectory decoupling and Mamba of the present invention.

[0073] Figure 2 It is a structural schematic diagram of the autonomous driving trajectory prediction model based on trajectory decoupling and Mamba of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0075] In view of the problem that existing autonomous driving trajectory prediction methods lack specific representation of the dynamic change information of the trajectory in continuous time steps, the present invention proposes an autonomous driving trajectory prediction method based on trajectory decoupling and Mamba. Through the trajectory decoupling method proposed in the present invention, the predicted trajectory query is decoupled into two types: modal query that captures the directionality of multimodal trajectories and dynamic state query of trajectories in continuous time steps. By utilizing this paradigm, the model optimizes the multimodal characteristics and dynamic evolution characteristics of the trajectory respectively. Subsequently, the modal and state queries are combined to obtain a comprehensive and detailed representation of the trajectory. In order to achieve these operations, the present invention introduces a technology combining the attention mechanism with Mamba for global information aggregation and state sequence modeling to take advantage of their respective advantages.

[0076] like Figure 1 As shown, the present invention provides an automatic driving trajectory prediction method based on trajectory decoupling and Mamba, comprising the following steps:

[0077] (1) Obtain a high-definition map and an agent in a driving scene, and encode the high-definition map and the agent in the driving scene using a pre-trained encoder to obtain scene context features composed of the encoded high-definition map and the agent.

[0078] The goal of motion prediction is to predict the future trajectory of an agent of interest. A high-definition map consists of several polylines, while agents are traffic participants such as vehicles and pedestrians. In order to convert these elements into easily processable and learnable inputs, the present invention uses the popular vectorized representation based on VectorNet.

[0079] Specifically, the scene encoder includes a map encoder, a history trajectory encoder, and an interaction encoder.

[0080] The map encoder uses the PointNet network, whose input is of dimension N m ×L×2 high-precision map vector, where N m Represents the total number of HD map vectors, L represents the number of points contained in each HD map vector, and feature dimension 2 represents the coordinate value of the map under the bird's-eye view; its dimension is N m ×L×2 high-precision map vectors are processed locally (Local) and globally (Global) in sequence, and the output dimension is Nm ×C map feature tensor F m , where C represents the dimension of the map feature tensor.

[0081] The history track encoder uses multiple unidirectional Mamba modules, whose input is N-dimensional a ×H×4 historical trajectory vectors of traffic participants, where N a represents the total number of traffic participants, H represents the number of steps of the historical trajectory vector, and the feature dimension 4 represents the horizontal coordinate, vertical coordinate, speed on the horizontal coordinate, and speed on the vertical coordinate of the traffic participant, which is N a ×H×4 traffic participants’ historical trajectory vectors are encoded and processed, and the output is a vector with dimension N a ×C object feature tensor F a .

[0082] The interactive encoder uses a standard Transformer network, whose input is of dimension N m ×C map feature tensor F m With dimension N a ×C object feature tensor F a , after connecting the two, the output dimension is (N a +N m )×C current scene context feature tensor F s .

[0083] The advantage of this step (1) is that the encoded scene context features (i.e., HD map and agents in the driving scene) interact with the trajectory query in the decoder to generate accurate future trajectory prediction results.

[0084] (2) Input the scene context features obtained in step (1) into the pre-trained query decoupled decoder (such as Figure 2 As shown in step (1), the multimodal future trajectory corresponding to each agent of interest in the agents obtained in step (1) is obtained.

[0085] Specifically, the query decoupling-based decoder in this step includes a modal localization module, a state continuity module, and a hybrid integration module, and its specific structure is as follows:

[0086] The modality localization module is implemented based on the attention mechanism, and its input is initialized with a dimension of N fa ×K×C modal query Q m , represents the multimodal trajectory of the attention agent, and the dimension of the interaction encoder output is (N a +N m )×C current scene context feature tensor F s , where N farepresents the number of focused agents, K represents the number of multimodal trajectories, and the modal localization module uses the attention mechanism to convert the modal query Q m and the current scene context feature tensor F s Process and output the preliminary multimodal trajectory Y m and probability P m , used to assist supervision;

[0087] The state continuous module is implemented based on the attention mechanism and Mamba, and its input is initialized to a dimension of N fa ×T s ×C status query Q s , represents the dynamic state trajectory of the attention agent, and the dimension of the interaction encoder output is (N a +N m )×C current scene context feature tensor F s , where T s is the time step represented by the state query (e.g., 1 to 60 milliseconds in the future). The state continuous module uses the attention mechanism to s and the current scene context feature tensor F s Processing, and querying the status Q through the Mamba model s The self-sequence is scanned interactively and a preliminary single trajectory Y is output s , used to assist supervision;

[0088] The hybrid integration module is implemented based on the attention mechanism and Mamba, and its input is the state query Q s , Modal Query Q m The dimension of the output of the interactive encoder is (N a +N m )×C current scene context feature tensor F s , which combines state query and modal query into a dimension of N fa ×K×T s ×C mixed query Q h , and the mixed query Q h and the current scene context feature tensor F s Processing and using the Mamba model to query the mixed query Q h The self-sequence is scanned interactively to output the final multimodal trajectory Y h and probability P h As the trajectory prediction result.

[0089] The advantage of this step (2) is that, through the query decoupling paradigm, modal query and state query are used to optimize the multimodal characteristics and dynamic evolution characteristics of the trajectory respectively, and the modal and state queries are combined to obtain a comprehensive and detailed representation of the trajectory; by introducing the technology of combining the attention mechanism with Mamba, they are used for global information aggregation and state sequence modeling respectively to utilize their respective advantages.

[0090] Specifically, the autonomous driving trajectory prediction model based on trajectory decoupling Mamba of the present invention is trained by the following steps:

[0091] (2-1) Obtain an autonomous driving dataset for trajectory prediction (which includes historical and future trajectories of traffic participants and high-precision maps), preprocess the autonomous driving dataset to obtain a preprocessed autonomous driving dataset, and use the vectorization method in the VectorNet model to convert the preprocessed autonomous driving dataset into a vector format to obtain a format-converted autonomous driving dataset, and divide the format-converted autonomous driving dataset into a training set and a validation set in a ratio of 9:1.

[0092] Specifically, the autonomous driving datasets used in this step are the Argoverse 2 and nuScenes datasets. The Argoverse 2 dataset includes 250,000 scenes with a sampling frequency of 10Hz, each of which has a 5-second historical trajectory length and predicts the trajectory length of the next 6 seconds. The nuScenes dataset contains 1,000 2Hz scenes and predicts the future 6s trajectory from the past 2s history. Both datasets use high-precision maps with a radius of 150 meters and traffic participants.

[0093] (2-2) For each autonomous driving scene segment in the training set obtained in step (2-1), the high-precision map vector M (whose dimension is N) in the autonomous driving scene segment is m ×L×2) and the historical trajectory vector A of the traffic participants (whose dimension is N a ×H×4) are input into the map encoder and the history track encoder respectively to obtain the dimensions of N m ×C map feature tensor F m , dimension is N a ×C object feature tensor F a , and combine the two through the interactive encoder to obtain a dimension of (N a +N m )×C scene context feature tensor F s .

[0094] Specifically, the map encoder uses the PointNet network, which first maps the HD map vector M into high-dimensional features through a multi-layer perceptron. Then, the HD map vector M is processed by the maximum pooling operation to obtain the global features corresponding to the HD map vector M. Then, the global features are connected to each point feature in the HD map vector M to obtain point features containing global information. Then, the above mapping operation and maximum pooling operation are repeated for all point features to obtain a high-dimensional feature with a dimension of N. m ×C map feature tensor F m .

[0095] The historical trajectory encoder uses several unidirectional Mamba modules to encode the historical trajectory vectors of traffic participants, and the output is a vector with dimension N. a ×C object feature tensor F a .

[0096] The interactive encoder uses a standard Transformer network, and the dimension of the map encoder output is N m ×C map feature tensor F m And the dimension of the historical trajectory encoder output is N a ×C object feature tensor F a Perform concatenation to obtain a dimension of (N a +N m )×C scene context feature tensor F s .

[0097] (2-3) Initialization status query Q S , query Q for this state S And the scene context feature tensor F obtained in step (2-2) s Perform interactive processing to obtain the optimized state query Q S , and query the status Q s Decoded as future trajectory Y s (It serves as auxiliary supervision).

[0098] This step is specifically as follows: First, in order to maintain accurate time information, the status query (where R represents the real number domain, N fa Indicates the number of agents to follow, T s represents the state query step) to obtain real-time differences. This step represents the future trajectory as a series of dynamic state queries across different time steps, which are different but interrelated. It is worth noting that in terms of balancing effectiveness and efficiency, the state query time step T s Can be different from the future time step T f, especially when predicting long-term future trajectories or higher-frequency future trajectories.

[0099] Then, the scene context feature tensor F is transformed using the state continuity module. s Status Query S Perform interactive processing to obtain the optimized state query Q S .

[0100] Afterwards, use Bidirectional Mamba to query the status Q S Perform forward and reverse scanning to obtain the optimized state query Q S .

[0101] Finally, a multilayer perception (MLP) module is used to transform the state query Q s Decoded into a single future trajectory Y s , which serves as an auxiliary output to explicitly supervise the timing continuity.

[0102] The process of this step can be described as follows:

[0103]

[0104] Q s =MultiHeadAttention(Q=Q s ,K,V=F s )

[0105] Q s =BiMamba(Q s )

[0106] Y s =MLP(Q s )

[0107] in Indicates the state query step length T s The corresponding real time step, Q represents the query of the attention mechanism, K represents the key of the attention mechanism, V represents the value of the attention mechanism, MultiHeadAttention represents the multi-head attention mechanism, and BiMamba represents the bidirectional Mamba.

[0108] The advantage of this sub-step (2-3) is that it specifically represents the dynamic evolution characteristics of the future trajectory, optimizes the consistency and continuity of the state query, and enhances the temporal continuity of the trajectory.

[0109] (2-4) Initialize modal query And query Q for this modalitym And the scene context feature tensor F obtained in step (2-2) s Perform interactive processing to obtain the optimized modal query Q m , and query Q for the optimized modal m Decode to obtain the future multimodal trajectory Y m and probability P m , which is used as auxiliary supervision.

[0110] Specifically, this step is to first initialize the modal query Each query is responsible for decoding one of K trajectories, with different directional modes representing future trajectories, where R represents the real number domain and N fa represents the number of agents of interest and K represents the number of multimodal trajectories.

[0111] Then, the modality query Q is localized using the modality localization module (which uses two attention mechanism modules). m and scene context F s Perform interactive processing to obtain the optimized modal query Q m .

[0112] Finally, the modal query Q is transformed into m Decoded into future multimodal trajectory Y m and probability P m , as an auxiliary output to give modal queries different directional intentions.

[0113] The process of this step can be described as follows:

[0114] Q m =MultiHeadAttention(Q=Q m ,K,V=F s )

[0115] Q m =MultiHeadAttention(Q=Q m ,K,V=Q m )

[0116] Y m ,P m =MLP(Q m )

[0117] The advantage of this step (2-4) is that the potential directional intention represented by the modal query is obtained, and the multimodal characteristics of the trajectory are optimized. Through steps (2-3) and (2-4), the design of the present invention combines the advantages of Mamba with the attention mechanism to store the dynamic evolution characteristics and multimodal characteristics of the future trajectory in the state query and the modal query respectively, creating conditions for subsequent combination to obtain a comprehensive and detailed representation of the trajectory.

[0118] (2-5) Query the status Q obtained in step (2-3) s and the modal query Q obtained in step (2-4) m Combined processing to obtain hybrid queries containing spatiotemporal information (to combine dynamic state and directional intent), and use the hybrid integration module composed of attention mechanism and Mamba to deal with mixed queries Q h Further processing is performed and the processing results are decoded using MLP to obtain the final multimodal trajectory Y h and probability P h .

[0119] Specifically, this step is as follows: first, an attention mechanism module is used to query the state Q obtained in step (2-3) s and the modal query Q obtained in step (2-4) m Combine the query Q with temporal and spatial information h .

[0120] Then, the hybrid query Q h Perform cross-time-step and cross-modal hybrid interaction processing to obtain the optimized hybrid query Q h , increasing the diversity of predicted trajectories.

[0121] Then, an attention mechanism module is used to optimize the mixed query Q h Interact between different modalities to obtain a further optimized hybrid query Q h .

[0122] Then, the bidirectional Mamba module is used to further optimize the hybrid query Q h Perform forward and reverse scanning to obtain the final optimized hybrid query Q h , realizing the interaction between states across time steps.

[0123] The changes in feature dimensions in this module are as follows Figure 2 shown.

[0124] Finally, MLP is used to optimize the final mixed query Qh Decode and finally output the multimodal trajectory Y h and probability P h .

[0125] The process of this step can be described as follows:

[0126] Q h =MultiHeadAttention(Q=Q h ,K,V=F s )

[0127] Q h =HybridMultiHeadAttention(Q=Q h ,K,V=Q h )

[0128] Q h =ModeMultiHeadAttention(Q=Q h ,K,V=Q h )

[0129] Q h =BiMamba(Q h )

[0130] Y h ,P h =MLP(Q h )

[0131] Among them, HybridMultiHeadAttention represents the multi-head attention mechanism of the hybrid self-attention module, and ModeMultiHeadAttention represents the multi-head attention mechanism for modal interaction.

[0132] The advantage of steps (2-3) to (2-5) is that by decoupling the trajectory into state query and modal query, the dynamic evolution characteristics and multimodal characteristics are stored separately, and finally combined into a hybrid query to obtain a comprehensive and detailed representation of the trajectory, ultimately obtaining a better trajectory prediction result.

[0133] (2-6) The future trajectory Y obtained according to step (2-3) s , the future multimodal trajectory Y obtained in steps (2-4) m and probability P m , and the final multimodal trajectory Y obtained in steps (2-5) h and probability P h The loss function is calculated and the model is iteratively trained using the loss function until the model converges, thereby obtaining a trained autonomous driving trajectory prediction model based on trajectory decoupling and Mamba.

[0134] The total loss function Loss is:

[0135] Loss=Loss reg +Loss cls +Loss state +Loss mode

[0136] Among them, Loss reg Represents the loss function used to supervise the prediction trajectory, which uses the SMOOTH-L1 function, Loss cls Represents the loss function used to predict probability, which uses the cross entropy function, Loss mode and Loss state represents the loss function for auxiliary output, and the above loss functions are expressed as:

[0137]

[0138] in Indicates the loss function using the SMOOTH-L1 function, L CE represents the loss function using the cross entropy function, represents the ground truth of the future trajectory, Represents the true value of the probability of future multimodal trajectories.

[0139] Test Results

[0140] In order to illustrate the effectiveness of the method of the present invention and the improvement of trajectory prediction accuracy, comparative experiments were conducted with several mainstream network models on the Argoverse 2 and nuScenes autonomous driving datasets. The evaluation indicators involved in the test process of this method include: (1) Minimum Average Displacement Error (minADE): This value evaluates the error distance between the predicted trajectory and the actual trajectory. The smaller the value, the better the method effect; (2) Minimum Final Displacement Error (minFDE): This value evaluates the error distance between the predicted trajectory and the actual trajectory. The smaller the value, the better the method effect.

[0141] Tables 1 and 2 below show the comparison between the present invention and other methods on the Argoverse 2 and nuScenes datasets:

[0142] Table 1

[0143] Model minFDE@1 minADE@1 minFDE@6 minADE@6 THOMAS 4.71 1.95 1.51 0.88 GANet 4.48 1.77 1.34 0.72 ProphNet 4.74 1.80 1.33 0.68 QCNet 4.30 1.69 1.29 0.65 The present invention 3.74 1.49 1.17 0.61

[0144] Table 2

[0145]

[0146]

[0147] It can be seen from Tables 1 and 2 above that the method of the present invention far exceeds most previous autonomous driving trajectory prediction methods. Specifically, the method of the present invention is significantly better than other methods in terms of minFDE@1 and minADE@1 indicators, which are 13.02% and 11.83% higher than the previous best method QCNet, respectively. In addition, the method of the present invention also achieves good performance in other indicators.

[0148] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An autonomous driving trajectory prediction method based on trajectory decoupling and Mamba, characterized in that: The following steps are involved: (1) Obtaining a high-definition map and traffic participants in a driving scene, and encoding the high-definition map and traffic participants in the driving scene using a pre-trained encoder to obtain scene context features composed of the encoded high-definition map and traffic participants; (2) Inputting the scene context features obtained in step (1) into a pre-trained query decoupled decoder to obtain the multimodal future trajectory corresponding to each of the traffic participants of interest among the traffic participants obtained in step (1); the decoder based on query decoupling includes a modal localization module, a state continuity module and a hybrid integration module, and its specific structure is as follows: The modality localization module is implemented based on the attention mechanism, and its input is initialized with a dimension of N fa ×K×C modal query Q m , represents the multimodal trajectory of the traffic participant of interest, and the dimension of the interaction encoder output is (N a +N m )×C current scene context feature tensor F s , where N fa represents the number of concerned traffic participants, K represents the number of multimodal trajectories, and the modal localization module converts the modal query Q into m and the current scene context feature tensor F s Process and output the preliminary multimodal trajectory Y m and probability P m , used to assist supervision; The state continuous module is implemented based on the attention mechanism and Mamba, and its input is initialized to a dimension of N fa ×T s ×C status query Q s , represents the dynamic state trajectory of the traffic participant of interest, and the dimension of the interaction encoder output is (N a +N m )×C current scene context feature tensor F s , where T s is the time step represented by the state query. The state continuous module uses the attention mechanism to s and the current scene context feature tensor F s Processing, and querying the status Q through the Mamba model s The self-sequence is scanned interactively and a preliminary single trajectory Y is output s , used to assist supervision; The hybrid integration module is implemented based on the attention mechanism and Mamba, and its input is the state query Q s , Modal Query Q m The dimension of the output of the interactive encoder is (N a +N m )×C current scene context feature tensor F s , which combines state query and modal query into a dimension of N fa ×K×T s ×C mixed query Q h , and the mixed query Q is transformed into h and the current scene context feature tensor F s Processing and using the Mamba model to query the mixed query Q h The self-sequence is scanned interactively to output the final multimodal trajectory Y h and probability P h As the trajectory prediction result; The trajectory prediction model for autonomous driving based on trajectory decoupling Mamba is trained through the following steps: (2-1) obtaining an autonomous driving dataset for trajectory prediction, which includes historical and future trajectories of traffic participants and a high-precision map, preprocessing the autonomous driving dataset to obtain a preprocessed autonomous driving dataset, and using a vectorization method in a VectorNet model to perform vector format conversion on the preprocessed autonomous driving dataset to obtain a format-converted autonomous driving dataset, and dividing the format-converted autonomous driving dataset into a training set and a validation set at a ratio of 9:1; (2-2) For each autonomous driving scene segment in the training set obtained in step (2-1), the high-precision map vector M and the historical trajectory vector A of the traffic participant in the autonomous driving scene segment are respectively input into the map encoder and the historical trajectory encoder to obtain vectors with dimensions N m ×C map feature tensor F m , dimension is N a ×C object feature tensor F a , and combine the two through the interactive encoder to obtain a dimension of (N a +N m )×C scene context feature tensor F s , the dimension of the high-precision map vector M is N m ×L×2, the dimension of the historical trajectory vector A is N a ×H×4; (2-3) Initialization status query Q S , query Q for this state S And the scene context feature tensor F obtained in step (2-2) s Perform interactive processing to obtain the optimized state query Q S , and query the status Q s Decoded as future trajectory Y s , the future trajectory Y s Serves as auxiliary supervisor; (2-4) Initialize modal query And query Q for this modality m And the scene context feature tensor F obtained in step (2-2) s Perform interactive processing to obtain the optimized modal query Q m , and query Q for the optimized modal m Decode to obtain the future multimodal trajectory Y m and probability P m , which is used as auxiliary supervision; (2-5) Query the status Q obtained in step (2-3) s and the modal query Q obtained in step (2-4) m Combined processing to obtain hybrid queries containing spatiotemporal information And use the hybrid integration module composed of attention mechanism and Mamba to deal with the mixed query Q h Further processing is performed and the processing results are decoded using MLP to obtain the final multimodal trajectory Y h and probability P h ; (2-6) The future trajectory Y obtained according to step (2-3) s , the future multimodal trajectory Y obtained in steps (2-4) m and probability P m , and the final multimodal trajectory Y obtained in steps (2-5) h and probability P h The loss function is calculated and the model is iteratively trained using the loss function until the model converges, thereby obtaining a trained autonomous driving trajectory prediction model based on trajectory decoupling and Mamba.

2. The method for automatic driving trajectory prediction based on trajectory decoupling and Mamba according to claim 1, characterized in that: The scene encoder includes a map encoder, a history track encoder, and an interaction encoder; The map encoder uses the PointNet network, whose input is of dimension N m ×L×2 high-precision map vector, where N m Represents the total number of HD map vectors, L represents the number of points contained in each HD map vector, and feature dimension 2 represents the coordinate value of the map under the bird's-eye view; its dimension is N m ×L×2 high-precision map vectors are processed locally and globally, and the output dimension is N m ×C map feature tensor F m , where C represents the dimension of the map feature tensor; The history track encoder uses multiple unidirectional Mamba modules, whose input is N-dimensional a ×H×4 historical trajectory vectors of traffic participants, where N a represents the total number of traffic participants, H represents the number of steps of the historical trajectory vector, and the feature dimension 4 represents the horizontal coordinate, vertical coordinate, speed on the horizontal coordinate, and speed on the vertical coordinate of the traffic participant, which is N a ×H×4 traffic participants’ historical trajectory vectors are encoded and processed, and the output is a vector with dimension N a ×C object feature tensor F a ; The interactive encoder uses a standard Transformer network, whose input is of dimension N m ×C map feature tensor F m With dimension N a ×C object feature tensor F a , after connecting the two, the output dimension is (N a +N m )×C current scene context feature tensor F s .

3. The method for automatic driving trajectory prediction based on trajectory decoupling and Mamba according to claim 2, characterized in that: The autonomous driving datasets used in step (2-1) are the Argoverse 2 and nuScenes datasets. The Argoverse 2 dataset includes 250,000 scenes with a sampling frequency of 10 Hz. Each scene has a historical trajectory length of 5 seconds and predicts the trajectory length of the next 6 seconds. The nuScenes dataset contains 1,000 2 Hz scenes and predicts the future 6-second trajectory from the past 2-second history. Both datasets use high-precision maps with a radius of 150 meters and traffic participants.

4. The method for automatic driving trajectory prediction based on trajectory decoupling and Mamba according to claim 3, characterized in that: In step (2-2), the map encoder uses the PointNet network, which first maps the high-precision map vector M into a high-dimensional feature through a multi-layer perceptron. Then, the high-precision map vector M is processed by the maximum pooling operation to obtain the global feature corresponding to the high-precision map vector M. Then, the global feature is connected to each point feature in the high-precision map vector M to obtain a point feature containing global information. Then, the above mapping operation and maximum pooling operation are repeated for all point features to obtain a high-dimensional feature with a dimension of N. m ×C map feature tensor F m ; The historical trajectory encoder uses several unidirectional Mamba modules to encode the historical trajectory vectors of traffic participants, and the output is a vector with dimension N. a ×C object feature tensor F a ; The interactive encoder uses a standard Transformer network, and the dimension of the map encoder output is N m ×C map feature tensor F m And the dimension of the historical trajectory encoder output is N a ×C object feature tensor F a Perform concatenation to obtain a dimension of (N a +N m )×C scene context feature tensor F s .

5. The method for automatic driving trajectory prediction based on trajectory decoupling and Mamba according to claim 4, characterized in that: Step (2-3) is as follows: First, initialize the status query Where R represents the real number domain, N fa The number of traffic participants who expressed concern, T s Indicates the status query step length; Then, the scene context feature tensor F is transformed using the state continuity module. s Status Query S Perform interactive processing to obtain the optimized state query Q S ; Afterwards, use bidirectional Mamba to query the status Q S Perform forward and reverse scanning to obtain the optimized state query Q S ; Finally, the multi-layer perceptron MLP module is used to transform the state query Q s Decoded into a single future trajectory Y s ; The process of step (2-3) is described as follows: Q s =MultiHeadAttention(Q=Q s ,K,V=F s ) Q s =BiMamba(Q s ) AND s =MLP(Q s ) in Indicates the state query step length T s The corresponding real time step, Q represents the query of the attention mechanism, K represents the key of the attention mechanism, V represents the value of the attention mechanism, MultiHeadAttention represents the multi-head attention mechanism, and BiMamba represents the bidirectional Mamba.

6. The method for automatic driving trajectory prediction based on trajectory decoupling and Mamba according to claim 5, characterized in that: Steps (2-4) are as follows: First, initialize the modal query Where N fa represents the number of traffic participants of interest, K represents the number of multimodal trajectories; Then, the modal query Q is localized using the modal localization module. m and scene context F s Perform interactive processing to obtain the optimized modal query Q m ; Finally, the modal query Q is transformed into m Decoded into future multimodal trajectory Y m and probability P m ; The process of this step is described as follows: Q m =MultiHeadAttention(Q=Q m ,K,V=F s ) Q m =MultiHeadAttention(Q=Q m ,K,V=Q m ) AND m ,P m =MLP(Q m )。 7. The method for automatic driving trajectory prediction based on trajectory decoupling and Mamba according to claim 6, characterized in that: Step (2-5) is as follows: first, an attention mechanism module is used to query the state Q obtained in step (2-3) s and the modal query Q obtained in step (2-4) m Combine the query Q with temporal and spatial information h ; Then, the hybrid query Q h Perform cross-time-step and cross-modal hybrid interaction processing to obtain the optimized hybrid query Q h ; Then, an attention mechanism module is used to optimize the mixed query Q h Interact between different modalities to obtain a further optimized hybrid query Q h ; Then, the bidirectional Mamba module is used to further optimize the hybrid query Q h Perform forward and reverse scanning to obtain the final optimized hybrid query Q h ; Finally, MLP is used to optimize the final mixed query Q h Decode and finally output the multimodal trajectory Y h and probability P h ; The process of this step is described as follows: Q h =MultiHeadAttention(Q=Q h ,K,V=F s ) Q h =HybridMultiHeadAttention(Q=Q h ,K,V=Q h ) Q h =ModeMultiHeadAttention(Q=Q h ,K,V=Q h ) Q h =BiMamba(Q h ) AND h ,P h =MLP(Q h ) Among them, HybridMultiHeadAttention represents the multi-head attention mechanism of the hybrid self-attention module, and ModeMultiHeadAttention represents the multi-head attention mechanism for modal interaction; The total loss function Loss in step (2-6) is: Loss=Loss reg +Loss cls +Loss state +Loss mode in: Loss reg Represents the loss function used to supervise the prediction trajectory, which uses the SMOOTH-L1 function, Loss cls Represents the loss function used to predict probability, which uses the cross entropy function, Loss mode and Loss state represents the loss function for auxiliary output, and the above loss functions are expressed as: in Indicates the loss function using the SMOOTH-L1 function, L CE represents the loss function using the cross entropy function, represents the true value of the future trajectory, Represents the true value of the probability of future multimodal trajectories.

8. An autonomous driving trajectory prediction system based on trajectory decoupling and Mamba, characterized in that: include: The first module is used to obtain a high-definition map and traffic participants in a driving scene, and encode the high-definition map and traffic participants in the driving scene using a pre-trained encoder to obtain scene context features composed of the encoded high-definition map and traffic participants; The second module is used to input the scene context features obtained by the first module into a pre-trained query decoupled decoder to obtain the multimodal future trajectory corresponding to each of the traffic participants of interest among the traffic participants obtained by the first module; the decoder based on query decoupling includes a modal positioning module, a state continuity module and a hybrid integration module, and its specific structure is as follows: The modality localization module is implemented based on the attention mechanism, and its input is initialized with a dimension of N fa ×K×C modal query Q m , represents the multimodal trajectory of the traffic participant of interest, and the dimension of the interaction encoder output is (N a +N m )×C current scene context feature tensor F s , where N fa represents the number of concerned traffic participants, K represents the number of multimodal trajectories, and the modal localization module converts the modal query Q into m and the current scene context feature tensor F s Process and output the preliminary multimodal trajectory Y m and probability P m , used to assist supervision; The state continuous module is implemented based on the attention mechanism and Mamba, and its input is initialized to a dimension of N fa ×T s ×C status query Q s , represents the dynamic state trajectory of the traffic participant of interest, and the dimension of the interaction encoder output is (N a +N m )×C current scene context feature tensor F s , where T s is the time step represented by the state query. The state continuous module uses the attention mechanism to s and the current scene context feature tensor F s Processing, and querying the status Q through the Mamba model s The self-sequence is scanned interactively and a preliminary single trajectory Y is output s , used to assist supervision; The hybrid integration module is implemented based on the attention mechanism and Mamba, and its input is the state query Q s , Modal Query Q m The dimension of the output of the interactive encoder is (N a +N m )×C current scene context feature tensor F s , which combines state query and modal query into a dimension of N fa ×K×T s ×C mixed query Q h , and the mixed query Q is transformed into h and the current scene context feature tensor F s Processing and using the Mamba model to query the mixed query Q h The self-sequence is scanned interactively to output the final multimodal trajectory Y h and probability P h As the trajectory prediction result; The trajectory prediction model for autonomous driving based on trajectory decoupling Mamba is trained through the following steps: (2-1) obtaining an autonomous driving dataset for trajectory prediction, which includes historical and future trajectories of traffic participants and a high-precision map, preprocessing the autonomous driving dataset to obtain a preprocessed autonomous driving dataset, and using a vectorization method in a VectorNet model to perform vector format conversion on the preprocessed autonomous driving dataset to obtain a format-converted autonomous driving dataset, and dividing the format-converted autonomous driving dataset into a training set and a validation set at a ratio of 9:1; (2-2) For each autonomous driving scene segment in the training set obtained in step (2-1), the high-precision map vector M and the historical trajectory vector A of the traffic participant in the autonomous driving scene segment are respectively input into the map encoder and the historical trajectory encoder to obtain vectors with dimensions N m ×C map feature tensor F m , dimension is N a ×C object feature tensor F a , and combine the two through the interactive encoder to obtain a dimension of (N a +N m )×C scene context feature tensor F s , the dimension of the high-precision map vector M is N m ×L×2, the dimension of the historical trajectory vector A is N a ×H×4; (2-3) Initialization status query Q S , query Q for this state S And the scene context feature tensor F obtained in step (2-2) s Perform interactive processing to obtain the optimized state query Q S , and query the status Q s Decoded as future trajectory Y s , the future trajectory Y s Serves as auxiliary supervisor; (2-4) Initialize modal query And query Q for this modality m And the scene context feature tensor F obtained in step (2-2) s Perform interactive processing to obtain the optimized modal query Q m , and query Q for the optimized modal m Decode to obtain the future multimodal trajectory Y m and probability P m , which is used as auxiliary supervision; (2-5) Query the status Q obtained in step (2-3) s and the modal query Q obtained in step (2-4) m Combined processing to obtain hybrid queries containing spatiotemporal information And use the hybrid integration module composed of attention mechanism and Mamba to deal with the mixed query Q h Further processing is performed and the processing results are decoded using MLP to obtain the final multimodal trajectory Y h and probability P h ; (2-6) The future trajectory Y obtained according to step (2-3) s , the future multimodal trajectory Y obtained in steps (2-4) m and probability P m , and the final multimodal trajectory Y obtained in steps (2-5) h and probability P h The loss function is calculated and the model is iteratively trained using the loss function until the model converges, thereby obtaining a trained autonomous driving trajectory prediction model based on trajectory decoupling and Mamba.

Images