A conditional trajectory generation and intelligent simulation system with domain generalization capability

Abstract

The application discloses a conditional trajectory generation and intelligent simulation system with field generalization capability, solves the problems of insufficient generalization capability of multi-modal behavior processing, time-varying agent quantity and unobserved scene in a complex scene. The system is based on a dynamic space-time graph probability generation model, models multi-agent interaction as a graph structure, and combines a recursive sequence with a variational deep generation model to learn a future trajectory distribution. By encoding the trajectory into a latent space and outputting probability distribution parameters, a physically reasonable future trajectory is generated under a given observation history. The system uses a structured representation of the scene and interaction, significantly improves the generalization capability of the model in unobserved field trajectory generation, and effectively supports intelligent simulation and automatic driving decision.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, robotics, computer vision, and intelligent transportation. In particular, it relates to a conditional trajectory generation and intelligent simulation system with domain generalization capabilities, which is especially suitable for complex dynamic environments such as autonomous driving, robot navigation, urban traffic simulation, and human-machine collaboration. Background Technology

[0002] In fields such as autonomous driving, robot navigation, and virtual simulation, accurately predicting and generating the future trajectories of multiple agents (such as pedestrians and vehicles) is crucial for achieving safe and efficient decision-making. However, existing technologies face multiple challenges:

[0003] First, the future behavior of humans or vehicles is often highly multimodal. Given the same historical context, an agent may exhibit multiple plausible future movement patterns. Traditional deterministic prediction models struggle to capture this inherent uncertainty and diversity.

[0004] Secondly, the number of agents in real-world scenarios changes constantly, and agents can enter or leave the scenario at any time, which poses a challenge to fixed-structure neural network models.

[0005] Furthermore, existing data-driven models often rely heavily on training data, leading to poor performance in unseen scenarios or different domains. These models may fail to adapt to new environments or interaction patterns, thus limiting their practical application.

[0006] Current researchers have attempted to address multimodality using Conditional Variational Autoencoders (CVAEs) or Generative Adversarial Networks (GANs), but these methods still have limitations in effectively modeling complex interactions between multiple agents, handling dynamically changing scene structures, and improving cross-domain generalization capabilities. Some models neglect the structured relationships between agents or handle interactions solely through simple aggregation, which limits their ability to understand and generate complex social dynamics. Therefore, developing a conditional trajectory generation and intelligent simulation system that can simultaneously address multimodality, the problem of time-varying agent numbers, and possess strong domain generalization capabilities is an urgent need for current technological development. Summary of the Invention

[0007] The present invention aims to overcome the shortcomings of the prior art and provide a conditional trajectory generation and intelligent simulation system with domain generalization ability, which can effectively handle multimodal future behavior and time-varying number of agents, and exhibit excellent generalization performance in unseen scenarios.

[0008] To achieve the above objectives, this invention provides a conditional trajectory generation and intelligent simulation system with domain generalization capabilities, specifically comprising: a. Dynamic Spatiotemporal Graph Construction Module: This module is used to model multi-agent scenarios as a graph structure that changes dynamically over time. The agents are the nodes of the graph, and the interactions between agents are the edges of the graph. The graph structure can flexibly represent the time-varying number of agents and complex interaction relationships. b. Multi-level graph encoder module: used to extract context information from the dynamic spatiotemporal graph, the multi-level graph encoder module includes: i. Node encoder: Used to encode the historical trajectory of each agent using a recurrent neural network to extract individual features; ii. Edge encoder: Used to encode the interaction features such as relative position and velocity between nodes (i.e., between agents) to capture interaction information; iii. Graph Convolutional Network (GCN) or Graph Attention Network (GAT): Used to aggregate the node and edge features through a graph neural network at each time step to generate a context-relevant representation of each agent at the current time step, thereby understanding complex social dynamics; c. Conditional Variational Autoencoder (CVAE) Framework: Used for probabilistic modeling of the multimodal distribution of future trajectories. The CVAE framework includes: i. Encoder: Used to learn the observation history and the context information Encoding into latent variables The distribution of the latent variables Capturing the multimodal uncertainty of future trajectories; ii. Generator Network (Decoder): Used to generate the latent variables and the context information Under the condition of [condition], the probability distribution parameters of the future trajectory are generated, and the probability distribution is usually the mean and covariance of a Gaussian distribution; d. Trajectory Generator Module: Used to generate the trajectory based on the latent variables. and the context information The system generates future trajectories in an autoregressive manner. The generation process is implemented through a recurrent neural network, and a parameterized probability distribution of the future position is output at each time step. The predicted position is obtained by sampling from the probability distribution. e. Joint optimization objective: used to train the system, including reconstruction loss and KL divergence loss, wherein the total loss function minimizes the weighted sum of reconstruction loss and KL divergence loss.

[0009] Preferably, the edges in the dynamic spatiotemporal graph construction module are established when the distance between two agents is less than a preset interaction threshold.

[0010] Preferably, the recurrent neural network used in the node encoder and the trajectory generator module is a long short-term memory network (LSTM) or a gated recurrent unit (GRU).

[0011] Preferably, the inference network outputs the latent variables. The mean of the Gaussian distribution Sum of logarithmic variance .

[0012] Preferably, when the trajectory generator module generates the future trajectory through autoregression, it uses the last real position of the observed trajectory as the input for the first prediction time step.

[0013] Preferably, the reconstruction loss in the joint optimization objective is the Huber loss or the negative log-likelihood loss.

[0014] Preferably, the KL divergence loss in the joint optimization objective constrains the distribution of the latent variables. Compared with the standard normal distribution The similarity.

[0015] Preferably, the total loss function in the joint optimization objective is: ,in Hyperparameters are used to balance reconstruction accuracy and potential spatial diversity.

[0016] Preferably, the system is capable of domain generalization in generating trajectories across different scenarios or datasets by learning abstract interaction patterns, utilizing probabilistic modeling, and decoupling from the latent space.

[0017] Preferably, the system can be used in an intelligent simulation platform to generate large-scale, diverse virtual traffic flow data, test the performance of the decision-making and planning modules of autonomous vehicles in complex traffic flows, and support the simulation of extreme and edge cases.

[0018] The conditional trajectory generation and intelligent simulation system with domain generalization capability provided by this invention has the following beneficial effects:

[0019] (1) Powerful multimodal trajectory generation capability: It can capture and generate highly multimodal future behavior of intelligent agents. Under the same historical background, the system can output a variety of reasonable and diverse future trajectories, solving the problem that traditional deterministic prediction models are difficult to capture inherent uncertainty.

[0020] (2) Effectively handle the time-varying number of agents: The system adopts dynamic spatiotemporal graph construction, models agents as graph nodes and interactions as graph edges, so that the system can flexibly represent and handle the time-varying number of agents in the scene, overcoming the limitations of fixed structure models in dynamic environments.

[0021] (3) Significantly improve domain generalization ability: By learning the structured representation of agent interaction and utilizing the probabilistic modeling ability of conditional variational autoencoder (CVAE), the system can better adapt to different or unseen scenarios and datasets, and can generate reasonable and high-quality trajectories in domains outside the training data distribution, greatly expanding the application scope of the model.

[0022] (4) Generate high-quality, realistic and physically reasonable trajectories: The trajectory data generated by the system is not only diverse, but also conforms to physical laws and has a high degree of realism, providing a reliable data foundation for downstream applications.

[0023] (5) Effectively avoid mode collapse: The introduction of KL divergence loss constrains the distribution of latent variables, ensuring the effective use of the latent space, thereby avoiding the mode collapse problem common in generative models and ensuring the richness and diversity of generated data.

[0024] (6) Strong support for intelligent simulation applications: The system can provide high-fidelity and high-reliability simulation data, which can be used to generate large-scale and diverse virtual traffic flows, support the performance testing of the decision planning module of autonomous vehicles in various complex and edge situations, and significantly reduce the cost of real-world data collection and testing.

[0025] (7) Promote the safety and efficiency of autonomous driving: Provide high-precision multimodal future trajectory prediction for the planning and prediction modules of autonomous vehicles, enhance the safety and robustness of autonomous vehicles in complex traffic flows, and help to discover potential safety hazards and optimize decision-making.

[0026] (8) Supports simulation of extreme and edge cases: It can simulate low-probability but high-risk interaction scenarios, which is crucial for fields with extremely high safety requirements such as autonomous driving, and helps to comprehensively evaluate the robustness of the system.

[0027] (9) Improve data generation efficiency and customization capabilities: reduce the need for expensive real data collection, and generate customized trajectory data according to user-specified scenario conditions (such as density, type, and interaction rules) to meet diverse simulation testing needs.

[0028] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the scope of protection of the appended claims. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the overall system architecture of a conditional trajectory generation and intelligent simulation system with domain generalization capability according to the present invention. Detailed Implementation

[0031] The following description, in conjunction with the accompanying drawings, illustrates a specific implementation of the conditional trajectory generation and intelligent simulation system with domain generalization capability provided by the present invention.

[0032] A conditional trajectory generation and intelligent simulation system with domain generalization capabilities, such as Figure 1 As shown, it includes: a. Dynamic Spatiotemporal Graph Construction Module: This module is used to model multi-agent scenarios as a graph structure that changes dynamically over time. Agents are nodes in the graph, and interactions between agents are edges. The graph structure can flexibly represent the number of time-varying agents and complex interaction relationships. Edges in the Dynamic Spatiotemporal Graph Construction Module are established when the distance between two agents is less than a preset interaction threshold.

[0033] This system at each time step The intelligent agents in the scene are observed, and these intelligent agents and their relationships are dynamically constructed into a spatiotemporal graph. .

[0034] node Construction: The set of nodes in a graph This represents the time step All intelligent agents present in the scene. Each node. Includes the agent's characteristic information at the current time step, such as its two-dimensional position. And velocity. In practical applications, node features can also be extended to include agent type, size, orientation, acceleration, etc. These features are mapped to higher-dimensional feature vectors through an initial embedding layer MLP.

[0035] side Construction: The set of edges of the graph This represents the interaction between intelligent agents. This system employs a dynamic connection strategy: when two intelligent agents... and The Euclidean distance between them is less than the preset interaction threshold. When a potential interaction is considered to exist between nodes, an undirected or directed edge is created between their corresponding nodes. Each edge... Including these two agents at time step Relative characteristics, such as relative position The relative velocities and the distances between them are also considered. These relative features are processed through an initial embedding layer.

[0036] This dynamic graph-building mechanism enables the system to flexibly handle scenarios with a time-varying number of agents, as the nodes and edges of the graph are dynamically updated based on the real-time appearance and disappearance of agents in the scenario. The graph's topology is dynamically adjusted with each time step to accurately reflect the interaction relationships at the current moment.

[0037] b. Multi-level graph encoder module: Used to extract context information from dynamic spatiotemporal graphs. The multi-level graph encoder module includes: i. Node Encoder: Used to encode the historical trajectory of each agent using a recurrent neural network, extracting individual features; for each agent in the scene... In the past Observation trajectory at each time step It is considered as a time series. This system employs a recurrent neural network (RNN), preferably a long short-term memory network (LSTM) or a gated recurrent unit (GRU), for each agent. The historical trajectory is independently encoded. The LSTM receives the agent's position and velocity at each time step as input and processes them through hidden states. Capture its accumulated motion information.

[0038] Input: Agent At time step The feature vectors (such as position and velocity).

[0039] deal with: unit.

[0040] Output: At each time step At the end, a representation of the intelligent agent is generated. The hidden state of an individual's movement history Finally, at the observation time step At the end, each agent is obtained. Final node encoding features .

[0041] Formula illustration: ,in It is an intelligent agent Steps in history The original characteristics.

[0042] ii. Edge encoder: Used to encode the interaction features such as relative position and velocity between nodes, capturing interaction information; for graphs... There exists an edge connecting any pair of agents. This system will calculate them at the current time step. Interaction features, such as their relative positions and relative velocity These relative features are then encoded by a multilayer perceptron (MLP) to capture the semantic information of their interactions.

[0043] Input: agent and At time step The relative characteristics.

[0044] deal with: unit.

[0045] Output: edges At time step Edge encoding features .

[0046] Formula illustration: .

[0047] iii. Graph Convolutional Network (GCN) or Graph Attention Network (GAT): Used to aggregate node and edge features through graph neural networks at each time step to generate a context-relevant representation of each agent at the current time step, thereby understanding complex social dynamics;

[0048] At each time step , Node encoding features and edge encoding features The input is fed into one or more graph neural network (GNN) layers. The GNN layers are responsible for information passing and aggregation on the graph structure, thereby generating a context-sensitive representation for each agent at the current time step. This indicates that it incorporates its own history and interaction information with neighboring intelligent agents.

[0049] If a Graph Convolutional Network (GCN) is used: GCN updates node features by weighted summation of the features of its neighboring nodes. For a node... Its updated features It can be represented as:

[0050]

[0051] in It is a node The neighborhood group, It is a node The degree, It is a learnable weight matrix. It is an activation function (such as ReLU). Edge features can be merged with node features before aggregation, or used as input to an attention mechanism.

[0052] If a Graph Attention Network (GAT) is used: GAT introduces an attention mechanism, allowing nodes to... When aggregating neighbor information, different weights are assigned based on the importance of neighbor nodes. This allows the model to dynamically focus on the neighbors that have the greatest impact on the agent's future behavior. Attention coefficient. It can be calculated as follows:

[0053] Through multi-layered GNNs, the system can capture multi-hop interaction information. Ultimately, all agents within the observation period... The graph-encoded features within the data will be further processed through a context aggregation LSTM to generate the final context information for each agent. .this The vectors integrate the agent's own history and its interactions with the surrounding environment and other agents throughout the observation period, serving as conditional inputs for subsequent CVAEs. For the entire scene, all... Aggregate into scene-level context .

[0054] c. Conditional Variational Autoencoder (CVAE) framework: used for probabilistic modeling of the multimodal distribution of future trajectories. CVAE introduces a low-dimensional latent variable. To explicitly represent multiple possibilities for future trajectories and thus overcome the limitation of deterministic models in capturing multimodality, the CVAE framework includes: i. Encoder: Used to learn the observation history and context information Encoding into latent variables The distribution of the latent variables Capturing the multimodal uncertainty of future trajectories; the inference network outputs the latent variables. The mean of the Gaussian distribution Sum of logarithmic variance .

[0055] Network Structure: Inference networks typically consist of one or more fully connected layers (MLPs) or a recurrent neural network (LSTM / GRU) to process sequential inputs. It maps input features to a latent space.

[0056] Output: The inference network outputs latent variables. The probability distribution parameters. In this invention, we assume... Since it follows a Gaussian distribution, the network outputs the mean of this Gaussian distribution. Sum of logarithmic variance .

[0057] Formula illustration:

[0058] in It is to observe history The feature vector is obtained after preprocessing (such as LSTM encoding).

[0059] Reparameterization technique: To enable backpropagation of gradients during training, we employ a reparameterization technique from... Mid-sampling yields latent variables :

[0060] ii. Generator Network (Decoder): Used for processing latent variables and context information Under the condition of [condition], the probability distribution parameters of the future trajectory are generated, and the probability distribution is usually the mean and covariance of a Gaussian distribution;

[0061] Network structure: Generative networks typically consist of one or more fully connected layers or recurrent neural networks (LSTM / GRU) used to transform latent features into parameters for future trajectories.

[0062] Output: The generator network outputs the future trajectory at each time step. The parameterized probability distribution of the position. In this invention, we assume that the position at each time step follows a two-dimensional Gaussian distribution. Therefore, the generator network will output the mean of this Gaussian distribution. Covariance Matrix The parameters.

[0063] Formula illustration: covariance matrix The following parameters are reconstructed:

[0064] In this way, the generative network can output the complete probability distribution of future trajectories, thus supporting multimodal trajectory generation.

[0065] d. Trajectory Generator Module: Used to generate trajectories based on latent variables. and context information The system generates future trajectories autoregressively. The generation process is implemented through a recurrent neural network, and the system outputs a parameterized probability distribution of the future position at each time step. The predicted position is obtained by sampling from the probability distribution. When generating future trajectories autoregressively, the trajectory generator module takes the last real position of the observed trajectory as the input of the first prediction time step.

[0066] Network Structure: The trajectory generator uses a recurrent neural network (RNN), typically implemented as a long short-term memory network (LSTM) or a gated recurrent unit (GRU).

[0067] Initialization: The initial hidden state of the decoder is typically determined by latent variables. and context information The combination is initialized through a linear layer or MLP, and the first input for trajectory generation (i.e., the first position of the predicted sequence) is usually the last true position of the observed trajectory. This helps the model make continuous predictions from a known starting point.

[0068] Autoregressive generation: The decoder generates future trajectories step-by-step in an autoregressive manner. At each time step... :

[0069] Input: The decoder receives the predicted position from the previous time step (or receives the predicted position from the first time step). ), latent variables and context information As input.

[0070] Processing: The LSTM unit processes the input and updates its internal hidden state.

[0071] Output: The hidden state of the STM is then mapped to the current time step through a fully connected layer (MLP). Predicted location The parameterized probability distribution. As mentioned earlier, this is typically the mean of a two-dimensional Gaussian distribution. Covariance Matrix The parameters.

[0072] Sampling: In intelligent simulation and actual generation tasks, the specific predicted location can be sampled from the probability distribution of this output. This sampling yielded This will be used as input for the next time step to continue autoregressive prediction.

[0073] e. Joint optimization objective: Used for training the system, including reconstruction loss and KL divergence loss. The total loss function minimizes the weighted sum of reconstruction loss and KL divergence loss. The reconstruction loss is either Huber loss or negative log-likelihood loss, and the KL divergence loss constrains the distribution of the latent variables. Compared with the standard normal distribution The similarity is given by the total loss function. ,in To balance reconstruction accuracy and potential spatial diversity, hyperparameters are used for each agent. The reconstruction losses are:

[0074]

[0075] KL divergence loss is used to measure the distribution of latent variables in the output of the inference network. Compared with the pre-set prior distribution The differences between them. Typically, the prior distribution... Set as standard normal distribution The existence of KL divergence loss is key to the CVAE framework, as it forces the latent space... It has good regularization properties, making sampling from the prior distribution... It can also generate meaningful trajectories, thus ensuring the diversity and generalization ability of model generation and effectively avoiding pattern collapse. (Formula illustration follows.)

[0076]

[0077] The KL divergence of a Gaussian distribution has a closed-form solution when At that time, for each dimension of the latent variable :

[0078] The entire CVAE model is trained by minimizing the following joint optimization objective function to obtain the total loss function:

[0079] in It is an adjustable hyperparameter (often referred to as KL divergence weights or "KL annealing") used to balance the importance of reconstruction accuracy with latent space regularization (i.e., generative diversity). In the early stages of training, The value may gradually increase from a small value. This system uses the Adam optimizer to iteratively update the model parameters in order to minimize... To prevent gradient explosion during training, especially when using recurrent neural networks, this system also implements gradient clipping, which limits the norm of the gradient to a preset maximum value, thereby ensuring the stability and convergence of the training process.

[0080] The system can learn abstract interaction patterns, utilize probabilistic modeling, and decouple from the latent space, thereby achieving domain-specific generalization capabilities in generating trajectories across different scenarios or datasets. Its implementation mechanism includes: a. Structured Representation Learning: Through dynamic spatiotemporal graphs and multi-level graph encoders, the model learns abstract interaction patterns between agents and structured information about the scene, rather than just pixels or simple coordinate sequences. This structured representation can be better transferred to new scenes with different specific values ​​but similar topological structures. For example, learned obstacle avoidance rules remain effective at new intersections.

[0081] b. Decoupling of Probabilistic Modeling and Latent Space: The CVAE framework encodes the inherent uncertainty of future trajectories into latent variables. In, and will determine the contextual information Decoupling from random information. This means latent variables... What is learned is a general representation of "behavioral style" or "intent," unrestricted by specific scenarios. In new domains, this can be achieved by sampling different... This allows for the generation of diverse and plausible behaviors, even those specific patterns not present in the training data. This decoupling enables the model to adapt more flexibly to new domains.

[0082] c. End-to-end training and big data learning: By performing end-to-end training on large-scale, diverse datasets, models can learn robust feature representations and general behavioral patterns, thereby improving their adaptability to domain changes.

[0083] In addition, the system can be used in intelligent simulation platforms to generate large-scale, diverse virtual traffic flow data, test the performance of the decision-making and planning modules of autonomous vehicles in complex traffic flows, and support the simulation of extreme and edge cases.

[0084] This invention is not limited to the specific embodiments described above. All modifications and variations made by those skilled in the art within the scope and essence of this invention should fall within the protection scope of this invention.

Claims

1. A conditional trajectory generation and intelligent simulation system with domain generalization capability, characterized in that, include: a. Dynamic Spatiotemporal Graph Construction Module: This module is used to model multi-agent scenarios as a graph structure that changes dynamically over time. The agents are the nodes of the graph, and the interactions between agents are the edges of the graph. The graph structure can flexibly represent the time-varying number of agents and complex interaction relationships. b. Multi-level graph encoder module: used to extract context information from the dynamic spatiotemporal graph, the multi-level graph encoder module includes: i. Node encoder: Used to encode the historical trajectory of each agent using a recurrent neural network to extract individual features; ii. Edge encoder: Used to encode the interaction features such as relative position and velocity between nodes (i.e., between agents) to capture interaction information; iii. Graph Convolutional Network (GCN) or Graph Attention Network (GAT): Used to aggregate the node and edge features through a graph neural network at each time step to generate a context-relevant representation of each agent at the current time step, thereby understanding complex social dynamics; c. Conditional Variational Autoencoder (CVAE) Framework: Used for probabilistic modeling of the multimodal distribution of future trajectories. The CVAE framework includes: i. Encoder: Used to learn the observation history and the context information Encoding into latent variables The distribution of the latent variables Capturing the multimodal uncertainty of future trajectories; ii. Generator Network (Decoder): Used to generate the latent variables and the context information Under the condition of [condition], the probability distribution parameters of the future trajectory are generated, and the probability distribution is usually the mean and covariance of a Gaussian distribution; d. Trajectory Generator Module: Used to generate the trajectory based on the latent variables. and the context information The system generates future trajectories in an autoregressive manner. The generation process is implemented through a recurrent neural network, and a parameterized probability distribution of the future position is output at each time step. The predicted position is obtained by sampling from the probability distribution. e. Joint optimization objective: used to train the system, including reconstruction loss and KL divergence loss, wherein the total loss function minimizes the weighted sum of reconstruction loss and KL divergence loss.

2. The system according to claim 1, characterized in that, The edges in the dynamic spatiotemporal graph construction module are established when the distance between two agents is less than a preset interaction threshold.

3. The system according to claim 1, characterized in that, The recurrent neural networks used in the node encoder and the trajectory generator module are either Long Short-Term Memory (LSTM) networks or Gated Recurrent Units (GRUs).

4. The system according to claim 1, characterized in that, The inference network outputs the latent variables. The mean of the Gaussian distribution Sum of logarithmic variance .

5. The system according to claim 1, characterized in that, When generating future trajectories via autoregression, the trajectory generator module uses the last real position of the observed trajectory as the input for the first prediction time step.

6. The system according to claim 1, characterized in that, The reconstruction loss in the joint optimization objective is either Huber loss or negative log-likelihood loss.

7. The system according to claim 1, characterized in that, The KL divergence loss constraint in the joint optimization objective defines the distribution of the latent variables. Compared with the standard normal distribution The similarity.

8. The system according to claim 1, characterized in that, The total loss function in the joint optimization objective is: ,in Hyperparameters are used to balance reconstruction accuracy and potential spatial diversity.

9. The system according to claim 1, characterized in that, The system is capable of domain generalization in generating trajectories across different scenarios or datasets by learning abstract interaction patterns, utilizing probabilistic modeling, and decoupling from the latent space.

10. The system according to claim 1, characterized in that, The system can be used as an intelligent simulation platform to generate large-scale, diverse virtual traffic flow data, test the performance of the decision-making and planning modules of autonomous vehicles in complex traffic flows, and support the simulation of extreme and edge cases.

Images