Future feature enhanced vehicle trajectory prediction generative adversarial method

Abstract

The invention belongs to the field of intelligent transportation, and relates to a future feature enhanced vehicle trajectory prediction generative adversarial method, which comprises the following steps: a conditional information learning step: completing joint modeling of historical and future features; a generative adversarial training step: performing multi-modal trajectory prediction by adopting a CVAE-GAN hybrid architecture; wherein the CVAE-GAN hybrid architecture comprises an encoder, a generator and a discriminator; the encoder comprises a regression encoder and a prior encoder, and the regression encoder is responsible for mapping a real track to a submerged space, learning compact representation of the track and only being used during model training; the priori encoder performs network sampling to obtain a latent variable; the generator reconstructs a multi-modal future trajectory according to latent variables and condition information; the discriminator receives the trajectory and the condition information, and evaluates the authenticity of the trajectory and the consistency of the trajectory and the condition information. By designing a double-stage strategy, stable generation and optimization of multi-modal trajectory distribution are realized, and the problems that traditional GAN training is unstable and CVAE output is too smooth are solved.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and autonomous driving technology, and in particular to a generative adversarial method for predicting vehicle trajectories with enhanced future features. Background Technology

[0002] The safe operation of autonomous vehicles relies on accurate prediction of the future trajectories of surrounding traffic participants. Traditional trajectory prediction methods primarily rely on historical observation data to model spatiotemporal dependencies, but often neglect the potential evolution of future scenarios and its impact on motion intentions. This results in predictions lacking diversity and realism, making them difficult to adapt to complex open traffic environments. Currently, mainstream multimodal trajectory prediction methods mainly include Generative Adversarial Networks (GANs), Conditional Variational Autoencoders (CVAEs), and Diffusion Models (DMs). Among them, CVAEs can generate diverse trajectories, but often lead to overly smooth predictions lacking physical plausibility; Diffusion Models perform well in tasks such as image generation, but suffer from excessively long inference times; GANs can effectively model motion uncertainty, but still have limitations in conditional guidance. Especially in autonomous driving scenarios, existing models generally struggle to explicitly integrate future feature information, thus failing to characterize the potential behavioral intentions and interaction trends of traffic participants.

[0003] Therefore, there is an urgent need for a new trajectory prediction framework that can simultaneously utilize historical and potential future features to construct a robust multimodal trajectory generation model and achieve stable convergence and interpretable result generation in adversarial optimization. Summary of the Invention

[0004] This application is proposed based on the above-mentioned situation of the prior art. The technical problem to be solved by this application is to provide a generative adversarial method for vehicle trajectory prediction with future feature enhancement, which overcomes the problems of unstable training of traditional GAN ​​and overly smooth output of CVAE, thereby improving the accuracy of trajectory prediction and scene adaptability.

[0005] To address the aforementioned technical problems, the technical solution provided in this application includes:

[0006] A generative adversarial method for vehicle trajectory prediction with enhanced future features includes: Step 1, a conditional information learning step, which completes the joint modeling of historical and future features, including: 1) a temporal feature encoder extracting temporal dependencies from the historical states of each agent; 2) a spatial feature encoder learning the relationship between the agent and the semantics of its neighboring scenes, as well as the interaction between agents, based on observation data; 3) a latent future feature encoder learning possible future scene features through a future representation modeling module, an interactive entity recognition module, and a future feature extraction module; Step 2, a generative adversarial training step, which uses a CVAE-GAN hybrid architecture for multimodal trajectory prediction; wherein the CVAE-GAN hybrid architecture includes an encoder, a generator, and a discriminator; the encoder includes a regression encoder and a prior encoder, the regression encoder is responsible for mapping the real trajectory to the latent space and learning a compact representation of the trajectory, which is only used during model training, and the prior encoder samples the latent variable z; the generator reconstructs the multimodal future trajectory with the latent variable z and conditional information C as input; the discriminator receives the trajectory and conditional information C and evaluates the authenticity of the trajectory and its consistency with the conditional information.

[0007] Preferably, the regression encoder includes a trajectory feature extraction layer that uses a one-dimensional convolutional network to capture local temporal patterns of the trajectory; a conditional fusion layer that realizes the interaction between trajectory features and conditional information through a cross-attention mechanism; a parameter estimation layer that outputs the distribution parameters of latent variables through a multilayer perceptron; and a regression encoder φ given the true trajectory Y and conditional information C with strong semantic constraints. CVAE (·) Output the mean and variance of the latent variables: μ, σ 2 =φ CVAE (Y,C), where, and Let d represent the mean and variance of the latent variable, respectively. z For the dimension of latent variables; latent variables are sampled through reparameterization techniques: where ⊙ represents element-wise multiplication, ε represents standard Gaussian noise; z=μ+σ⊙ε,ε∈N(0,I).

[0008] Preferably, the prior encoder and the regression encoder share a similar architecture, both using a multilayer perceptron (MLP) as the backbone network. However, the prior encoder network only relies on conditional information to model the distribution and samples the latent variable z. The generator G(·) uses the latent variable z and conditional information C as input to reconstruct the multimodal future trajectory. In the formula, Indicates the generated K f 3 future trajectories, each containing T fut The coordinates of each time step;

[0009] The generator employs a hierarchical decoding structure, divided into a conditional latent variable fusion layer and a time-series trajectory generation layer. First, the conditional latent variable fusion layer fuses conditional information with latent variables. In the formula, φ MLP (·) represents an MLP network; the temporal trajectory generation layer generates trajectory sequences step by step based on gated recurrent units: In the formula, h t Let θ represent the hidden state vector at time step t. g φ represents the set of learnable parameters of a GRU network. GRU (·) represents a GRU network, φ Linear (·) indicates a linear layer.

[0010] Preferably, the discriminator receives the real trajectory Y or the generated trajectory. And based on conditional information C, evaluate the authenticity of the trajectory and its consistency with the conditional information: In the formula, σ(·) represents the Sigmoid function, whose output range [0,1] represents the true probability, and φ CNN Represents a convolutional neural network, φ MLP (·) represents an MLP network, φ Attn (·) indicates a multi-head attention network.

[0011] Preferably, the discriminator includes a trajectory feature extraction module, a conditional attention fusion module, and a authenticity discrimination module; the trajectory feature extraction module uses a one-dimensional temporal convolutional network to extract the dynamic features F of the trajectory. Traj =φ TCN (Y), where φ TCN This represents a temporal convolutional network; the conditional attention fusion module achieves deep interaction between trajectory features and conditional information through a multi-head attention mechanism: φ Attn (Y,C)=φ MHA (Q=F Traj K = C, V = C), where φ Attn (·) and φ MHA (·) indicates a multi-head attention mechanism; the authenticity judgment module integrates trajectory features and conditional interaction information to output the judgment result:

[0012]

[0013] D(Y,C)=σ(φ Linear (F fus )),

[0014] In the formula, F fus φ represents the characteristics after fusion. MLP (·) represents an MLP network, φ Attn(·) denotes a multi-head attention mechanism, D(·) denotes a discriminator network, and φ Linear (·) indicates a linear layer.

[0015] The beneficial effects of this invention are as follows:

[0016] This invention proposes a generative adversarial method for vehicle trajectory prediction with enhanced future features. By designing a two-stage strategy, namely a conditional information learning step and a generative adversarial training stage, it achieves stable generation and optimization of multimodal trajectory distribution, overcoming the problems of unstable training in traditional GANs and overly smooth CVAE output.

[0017] This invention proposes an enhanced conditional guidance mechanism that explicitly models the spatial interaction relationships of traffic participants at future moments by constructing a latent future feature encoder, thereby enhancing the model's semantic understanding of future scenarios and improving the accuracy and scene adaptability of trajectory prediction. Detailed Implementation

[0018] The method proposed in this invention sequentially performs two key phases: a conditional information learning step and a generative adversarial training step. In the conditional information learning step, the proposed method aims to comprehensively understand the complex historical and future characteristics of traffic scenarios. In the generative adversarial training step, this method employs a CVAE-GAN hybrid model to generate multimodal future trajectories and optimizes the prediction results through adversarial training.

[0019] Step 1: Conditional Information Learning Steps

[0020] In traffic scenarios, the diverse interactions between intelligent agents largely determine their future motion states. Therefore, the goal of the conditional information learning step is to accurately model the interaction patterns in the environment.

[0021] This step comprises three sub-steps: 1) A temporal feature encoder extracts temporal dependencies from the historical states of each agent; 2) A spatial feature encoder embeds historical spatial relationships between agents and between agents and the lane; 3) A potential future feature encoder learns possible future scene features through a future representation modeling module, an interactive entity recognition module, and a future feature extraction module. These steps provide information about how the agents' motion states will evolve and how other agents will influence the target vehicle in future time steps, thereby enhancing conditional guidance capabilities.

[0022] 1) The temporal feature encoder extracts temporal dependencies from the historical states of each agent.

[0023] Since the historical motion trend of a vehicle influences its subsequent motion state, capturing the dependencies in the time dimension is crucial. To reveal the motion state correlations between agents at different time steps, this invention employs a more computationally efficient framework: the state-space model. Specifically, firstly, through linear transformation and layer normalization operations, the historical state x at time step t is obtained. t Extracting temporal embedding features z t :

[0024] z t =δ ReLU (φ LN (φ FC (x t W emd ))),(2.1)

[0025] In the formula, φ FC (·) indicates that the learnable matrix W is derived from the matrix W. emd Parameterized fully connected layer, φ LN (·) denotes the layer normalization operation, δ ReLU (·) represents the ReLU activation function.

[0026] After obtaining the embedded features, they are input into a Mamba block, which iteratively optimizes the input sequence features. Mamba is a state-space model with a unique selection mechanism, enabling the proposed model to adaptively capture the interrelationships between segments in the sequence. Furthermore, this invention introduces gated linear units to derive the final time-aware features H of the target vehicle and its surrounding agents. a The above process can be formalized as follows:

[0027]

[0028] In the formula, φ MB (·) and φ GLU (·) represent Mamba blocks and gated linear units, respectively.

[0029] 2) The spatial feature encoder learns the relationship between the agent and the semantics of its neighboring scene, as well as the interactions between agents, based on the observed data.

[0030] This step integrates inputs from multiple data streams (i.e., the agent's temporal features H). a And high-definition map data (M), a unified representation is generated through a Transformer decoder architecture. Specifically, this invention first uses a PointNet-based encoder φ PointNet (·) Process the vectorized high-resolution map to obtain high-dimensional scene features H m Furthermore, feature H a and Hm Input two Transformer decoders φ TFDec (·) Through interactive learning, the local historical spatial features H are ultimately formed. s This process can be described as follows:

[0031] H m =φ PointNet (M), (2.3)

[0032] H′ m =φ TFDec (H m H a ), (2.4)

[0033] H s =φ TFDec (H a ,H′ m (2.5)

[0034] Specifically, the standard Transformer decoder comprises three core components: a self-attention layer, a cross-attention layer, and a feedforward neural network. Following this architectural design, in the second Transformer decoder, feature H... a First, a self-attention layer enables the model to capture the interaction relationships between multiple agents. The resulting features are then processed through a cross-attention layer and H′. m Interacting with the system integrates scene context information into the representation learning process. To further enhance context modeling capabilities, this invention ultimately integrates the agent's features H... s and map features H′ m Input another multi-head attention layer to extract global interaction features.

[0035] 3) The latent future feature encoder learns potential future scene features through the future representation modeling module, interactive entity recognition module, and future feature extraction module.

[0036] In obtaining historical scene context features H s and Subsequently, most existing studies directly decode multimodal future trajectories. However, observational data only contains raw historical input information, essentially a deterministic record of past events. Predicting multimodal future trajectories solely based on historical spatiotemporal features extracted from observational data often overlooks potential causal interactions that may occur in the future. This limitation may lead to generated trajectories that are incompatible with real-world scenarios. To address this issue, this invention proposes a future-aware paradigm that explicitly simulates potential future scenarios and models agent interactions within these virtual scenarios. Specifically, this invention proposes a three-step approach to modeling the potential future features of driving scenarios: first, predicting the future representations of each agent in the scenario; second, identifying pairs of interacting agents; and third, estimating future features. Therefore, the potential future feature encoder consists of three core modules: a future representation modeling step, an interaction entity identification step, and a future feature extraction step.

[0037] ① Future representation modeling steps

[0038] To characterize future driving scenarios, this invention first constructs representations of specific agents containing potential future information. Given the inherent temporal and modal causality in these future representations, this invention represents these features as dynamic state sequences across different time steps, and multiple motion modalities across different agents. First, the state query is initialized. Where N is the number of agents in the scene, and C is the feature dimension. This invention uses a time step T... s Set to less than the total prediction duration T fut This is because overly dense time step settings can introduce redundant information and noise interference. Furthermore, a multi-head attention module φ is utilized. MHA (·) Derive the updated dynamic state from historical features:

[0039] F s =φ MHA (Q=F s K = H s V=H s (2.6)

[0040] To capture the multimodal motion characteristics of surrounding agents, a method is proposed to generate diverse future modal representations F from historical features using a multilayer perceptron. m Subsequently, the dynamic state representation is fused with multiple motion modal representations and input into a multi-head attention block along with historical features H. s Through interaction, the future representation F is ultimately generated. f This process can be described as follows:

[0041] F m =φ MLP (H s ),F ms =Fm +F s (2.7)

[0042] F f =φ MHA (Q=F ms K = H s V=H s ), (2.8)

[0043] In the formula, φ MLP (·) denotes the multilayer perceptron function, F ms This represents the fused characterization.

[0044] ② Interactive entity recognition steps

[0045] The open nature of traffic scenarios necessitates the accurate distinction between interactive and non-interactive agents. This invention identifies agents interacting with a target vehicle by learning the relationships between implicit future representations.

[0046] This invention selects causal interaction agents from all agents based on feature similarity scores, and their future representations are denoted as...

[0047] Top J =∏(Similarity,J), (2.9)

[0048]

[0049] In the formula, ∏(·,·) is used to obtain the indices of the first J representations, Index(·) represents the function to retrieve the corresponding values ​​based on these indices, and Similarity(·) represents the feature similarity function.

[0050] ③ Future Feature Extraction Steps

[0051] By successfully modeling future representations and accurately identifying interacting entities, this invention has derived the agents and their information that may be affected in future time steps. The next challenge lies in capturing the complex causal future interactions between the target vehicle and other agents to construct future scene features. To this end, this invention proposes a dual-fusion method that integrates inter-agent and inter-modal dependencies through a multi-head differential attention mechanism to construct future features. The differential attention mechanism is chosen as the core framework because it can reduce attention noise through differential denoising techniques, allowing the proposed model to focus on key features. Specifically, given the future representation of the target agent… Future representations of interactive entities This invention inputs it into the multi-head differential attention module φ MHADiff (·):

[0052]

[0053] In the formula, Z a This represents the future features of the target agent after its update, which aggregates the representational information of the interacting agents.

[0054] Furthermore, this invention enhances the Z-based [process] through another multi-head differential attention block. a Intermodal interactions are explored to delve deeper into dependencies and facilitate modal associations in representation learning. Future features of modality perception. It can be obtained through the following process:

[0055]

[0056] In the formula, φ MHADiff (·) represents another multi-head differential attention function, K f This represents the number of modes. Furthermore, The data is fed into a GRU layer to capture the final future features.

[0057] Step 2: Generate adversarial training steps

[0058] After the conditional information learning step completes the joint modeling of historical and future features, the generative adversarial training stage aims to further optimize the realism and diversity of generated trajectories. This stage adopts a CVAE-GAN hybrid architecture, combining the distribution modeling capabilities of CVAE with the generation quality advantages of GAN to achieve stable and diverse multimodal trajectory prediction.

[0059] The CVAE-GAN hybrid architecture comprises three core components: encoder, generator, and discriminator.

[0060] 1) Encoder (CVAE encoding section)

[0061] The encoders proposed in this invention include a regression encoder and a prior encoder. The regression encoder is responsible for mapping the real trajectory to the latent space, learning a compact representation of the trajectory, and is only used during model training. The regression encoder specifically consists of three core modules: a trajectory feature extraction layer (using a one-dimensional convolutional network to capture the local temporal patterns of the trajectory); a conditional fusion layer (achieving the interaction between trajectory features and conditional information through a cross-attention mechanism); and a distribution parameter estimation layer (the distribution parameters of the latent variables are output by the multilayer perceptron).

[0062] Given the true trajectory and conditional information with strong semantic constraints The regression encoder outputs the mean and variance of the latent variables:

[0063] μ,σ 2 =φ CVAE(Y,C), (2.14)

[0064] In the formula, and Let d represent the mean and variance of the latent variable, respectively. z For latent variable dimensions.

[0065] Sampling latent variables using reparameterization techniques:

[0066] z=μ+σ⊙ε,ε∈N(0,I), (2.15)

[0067] In the formula, ⊙ represents element-wise multiplication, and ε represents standard Gaussian noise.

[0068] Meanwhile, the prior encoder and the regression encoder share a similar architecture, but the prior encoder network only relies on conditional information to model the distribution and sample the latent variable z.

[0069] 1) Generator

[0070] The generator takes latent variable z and conditional information C as input to reconstruct the multimodal future trajectory:

[0071]

[0072] In the formula, Indicates the generated K f 3 future trajectories, each containing T fut The coordinates of each time step.

[0073] The generator employs a hierarchical decoding structure, consisting of a conditional latent variable fusion layer and a time-series trajectory generation layer. First, the conditional latent variable fusion layer fuses conditional information with latent variables:

[0074]

[0075] Furthermore, the time-series trajectory generation layer generates trajectory sequences step by step based on gated recurrent units (GRUs):

[0076]

[0077]

[0078] In the formula, h t Let θ represent the hidden state vector at time step t. g This represents the set of learnable parameters for a GRU network.

[0079] 3) Discriminator

[0080] The discriminator receives the trajectory (real trajectory Y or generated trajectory). And conditional information C, evaluate the authenticity of the trajectory and its consistency with the conditional information:

[0081]

[0082] In the formula, σ(·) represents the Sigmoid function, whose output range [0,1] represents the true probability, and φ CNN This represents a convolutional neural network.

[0083] The discriminator consists of three key modules: a trajectory feature extraction module, a conditional attention fusion module, and a authenticity discrimination module.

[0084] First, the trajectory feature extraction module uses a one-dimensional temporal convolutional network (TCN) to extract the dynamic features of the trajectory:

[0085] F Traj =φ TCN (Y), (2.21)

[0086] In the formula, φ TCN This represents a temporal convolutional network.

[0087] Furthermore, the conditional attention fusion module achieves deep interaction between trajectory features and conditional information through a multi-head attention mechanism:

[0088] φ Attn (Y,C)=φ MHA (Q=F Traj (2.22)

[0089] In the formula, φ MHA This represents a multi-head attention mechanism.

[0090] Finally, the authenticity determination module integrates trajectory features and conditional interaction information to output the determination result:

[0091]

[0092] D(Y,C)=σ(φ Linear (F fus (2.24)

[0093] This invention proposes a generative adversarial method for predicting vehicle multimodal trajectories with enhanced future features. By integrating two key stages—conditional information learning and generative adversarial training—it achieves efficient modeling and optimization of the multimodal distribution of future motion. In the conditional information learning stage, the proposed model accurately captures the complex spatiotemporal dependencies in historical and future traffic scenarios. In the generative adversarial training stage, a CVAE-GAN hybrid architecture is constructed to generate multimodal future trajectories, and adversarial training is used to obtain reliable and diverse final results. The two-stage strategy designed in this invention not only improves the accuracy of prediction but also enhances the model's ability to handle uncertainty, making the prediction results more consistent with the dynamic changes in traffic scenarios.

[0094] The enhanced conditional guidance mechanism proposed in this invention endows the prediction model with sophisticated insights by meticulously characterizing future features in driving scenarios. Traditional prediction models often focus only on historical information, neglecting potential future changes. However, this invention, by introducing a future feature encoder, enables the model to envision and encode potential future driving scenarios, thereby gaining a more comprehensive understanding of the dynamic behavioral characteristics of traffic participants. This enhanced conditional guidance mechanism allows the model to consider more future possibilities during the prediction process, significantly improving prediction accuracy. Simultaneously, this mechanism also promotes a deeper understanding of the uncertainties in traffic scenarios, making the prediction results more robust and reliable, providing strong assurance for the safe operation of autonomous vehicles.

Claims

1. A generative adversarial method for vehicle trajectory prediction with enhanced future features, characterized in that... include: Step 1, Conditional Information Learning Step, completes the joint modeling of historical and future features, which includes: 1) Temporal feature encoder extracts temporal dependencies from the historical states of each agent; 2) Spatial feature encoder learns the relationship between the agent and the semantics of its neighboring scenes, as well as the interaction between agents, based on observation data; 3) Potential future feature encoder learns possible future scene features through the future representation modeling module, the interactive entity recognition module, and the future feature extraction module. Step 2: Generative Adversarial Training Step. A CVAE-GAN hybrid architecture is used for multimodal trajectory prediction. This architecture includes an encoder, a generator, and a discriminator. The encoder comprises a regression encoder and a prior encoder. The regression encoder maps the real trajectory to the latent space, learns a compact representation of the trajectory, and is used only during model training. The prior encoder samples the latent variable z. The generator reconstructs the multimodal future trajectory using the latent variable z and conditional information C as input. The discriminator receives the trajectory and conditional information C and evaluates the authenticity of the trajectory and its consistency with the conditional information.

2. The generative adversarial method for predicting vehicle trajectories with enhanced future features according to claim 1, characterized in that, The regression encoder comprises a trajectory feature extraction layer that uses a one-dimensional convolutional network to capture local temporal patterns of the trajectory; a conditional fusion layer that enables the interaction between trajectory features and conditional information through a cross-attention mechanism; a parameter estimation layer that outputs the distribution parameters of latent variables through a multilayer perceptron; and a regression encoder φ given the true trajectory Y and conditional information C with strong semantic constraints. CVAE (·) Output the mean and variance of the latent variables: μ, σ 2 =φ CVAE (Y,C), where, and Let d represent the mean and variance of the latent variable, respectively. z For the dimension of latent variables; latent variables are sampled through reparameterization techniques: where ⊙ represents element-wise multiplication, ε represents standard Gaussian noise; z=μ+σ⊙ε,ε∈N(0,I).

3. The generative adversarial method for predicting vehicle trajectories with enhanced future features according to claim 2, characterized in that, The prior encoder and the regression encoder share a similar architecture, both using a multilayer perceptron (MLP) as the backbone network. However, the prior encoder network only relies on conditional information to model the distribution and sample the latent variable z.

4. The generative adversarial method for predicting vehicle trajectories with enhanced future features according to claim 3, characterized in that, The generator G(·) reconstructs the multimodal future trajectory using latent variable z and conditional information C as input: In the formula, Indicates the generated K f 3 future trajectories, each containing T fut The coordinates of each time step; The generator employs a hierarchical decoding structure, divided into a conditional latent variable fusion layer and a time-series trajectory generation layer. First, the conditional latent variable fusion layer fuses conditional information with latent variables. In the formula, φ MLP (·) indicates an MLP network; The time-series trajectory generation layer generates trajectory sequences step by step based on gated loop units: In the formula, h t Let θ represent the hidden state vector at time step t. g φ represents the set of learnable parameters of a GRU network. GRU (·) represents a GRU network, φ Linear (·) indicates a linear layer.

5. The generative adversarial method for predicting vehicle trajectories with enhanced future features according to claim 4, characterized in that, The discriminator receives the true trajectory Y or the generated trajectory And based on conditional information C, evaluate the authenticity of the trajectory and its consistency with the conditional information: In the formula, σ(·) represents the Sigmoid function, whose output range [0,1] represents the true probability, and φ CNN Represents a convolutional neural network, φ MLP (·) represents an MLP network, φ Attn (·) indicates a multi-head attention network.

6. The generative adversarial method for predicting vehicle trajectories with enhanced future features according to claim 5, characterized in that, The discriminator includes a trajectory feature extraction module, a conditional attention fusion module, and an authenticity discrimination module; The trajectory feature extraction module uses a one-dimensional temporal convolutional network to extract the dynamic features F of the trajectory. Traj =φ TCN (Y), where φ TCN Represents a temporal convolutional network; The conditional attention fusion module achieves deep interaction between trajectory features and conditional information through a multi-head attention mechanism: φ Attn (Y,C)=φ MHA (Q=F Traj K = C, V = C), where φ Attn (·) and φ MHA (·) indicates a multi-head attention mechanism; The authenticity determination module integrates trajectory features and conditional interaction information to output the determination result: D(Y,C)=σ(φ Linear (F fus )), In the formula, F fus φ represents the characteristics after fusion. MLP (·) represents an MLP network, φ Attn (·) denotes a multi-head attention mechanism, D(·) denotes a discriminator network, and φ Linear (·) indicates a linear layer.

Images