A Multi-Agent Trajectory Co-generation and Modeling System for Urban Scenarios

Abstract

This invention discloses a multi-agent trajectory co-generation and modeling system for urban scenarios, aiming to solve the problems of difficulty in collecting real-world datasets and the limited capacity of traditional algorithm models in complex interactive scenarios. This system proposes a learning-based automated trajectory generation model, which can explain the interactive behavior generated by the target through explicit modeling. It employs a conditional variational autoencoder architecture and a self-supervised training scheme. The core of this system lies in using a Gaussian mixture model to model the implicit distribution of multimodal temporal destinations, generating actions through residual connections, and introducing mode collapse loss to ensure data diversity. The realistic trajectories generated by this system can be used as a data augmentation technique to effectively improve the performance of downstream motion planners.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, robotics, and computer vision, and specifically relates to behavior modeling, trajectory generation and prediction in multi-agent systems, especially a multi-agent trajectory co-generation and modeling system for urban scenarios. Background Technology

[0002] Data-driven approaches offer significant advantages in modeling complex human behavioral dynamics and handling human-computer interaction applications. However, collecting and labeling large-scale real-world human datasets is a time-consuming and labor-intensive task, especially in highly interactive scenarios. On the other hand, while some algorithmic data generation methods can generate various human trajectory data based on deterministic or stochastic motion generators, these rule-based methods typically perform well in limited domains but struggle to generate realistic and smooth trajectory data for general purposes.

[0003] With advancements in deep learning methods, the demand for large-scale human behavior datasets has intensified. Existing researchers have constructed various generative models by encoding highly multimodal and uncertain human motion into latent states. However, these models are typically based on the assumption that the latent randomness in human behavior follows a conditional Gaussian distribution, limiting their performance to the latent space's ability to model the agent's hidden features and failing to guarantee the diversity of generated data. For example, some methods focus on implicitly modeling future patterns, while others choose to discretize the output space as the target or anchor for prediction, which may increase computational burden or depend on the quality of predefined target anchors. Therefore, how to generate realistic, diverse, and interpretable multi-agent trajectory data is a critical problem that urgently needs to be solved. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technologies and provide a multi-agent trajectory co-generation and modeling system for urban scenarios, which can generate interactive human behaviors with diverse contexts, and the data has high authenticity and strong interpretability.

[0005] The technical solution provided by this invention is: a multi-agent trajectory co-generation and modeling system for urban scenarios, the system comprising:

[0006] a. Spatiotemporal encoder: used to extract historical and interactive perception context information from multi-agent scenarios and represent it as encoded history e;

[0007] b. Gaussian Mixture Model (GMM): Receives the encoded history e and explicitly models the agent's temporal destination. The multimodal distribution of the GMM, wherein the probability density function of the GMM is:

[0008] ;

[0009] in, , and They are the first The weights, mean, and covariance matrices of Gaussian kernels, where K is the number of Gaussian kernels;

[0010] The parameters of the GMM are obtained from the encoding history e through a fully connected layer:

[0011] , , ;

[0012] The covariance matrix From variance and correlation The reconstruction yields:

[0013] ;

[0014] c. Gated Recurrent Unit (GRU) Decoder: Receives the encoded history e and the time destination (which is the ground truth during training). During testing, samples were taken from the GMM. The predicted trajectory is generated by autoregressively outputting the predicted trajectory through a residual layer. The process of generating the predicted trajectory includes: , ; in, Let q represent a GRU cell and q represent a residual layer. This represents the predicted state of agent i at time step t. It is the residual displacement;

[0015] d. Loss function: used to train the system, including:

[0016] 1) Destination loss: based on the ground truth time objective Maximum log-likelihood objective;

[0017] 2) Mode collapse loss: Regularize the weights, variances, and distances between the centers of the GMM to address the mode collapse problem.

[0018] ;

[0019] 3) Reconstruction loss: Minimize the predicted trajectory ground true trajectory Huber error between: ;

[0020] The system is optimized by minimizing the total loss function:

[0021] ;

[0022] in, , and It's a hyperparameter.

[0023] Furthermore, the system is able to generate multi-agent trajectory data with high diversity and realism.

[0024] Furthermore, the system can be used as a data augmentation tool in downstream imitation-based motion planning tasks to improve planning performance.

[0025] Furthermore, the spatiotemporal encoder includes node history LSTM and edge history LSTM.

[0026] Furthermore, the number of Gaussian kernels K in the GMM is set to 4.

[0027] Furthermore, the parameterized fully connected layer of the GMM consists of four independent layers, which are used to output weights, mean, log-variance, and correlation, respectively.

[0028] Furthermore, the system is initialized using the actual initial state during training.

[0029] Furthermore, during the training phase, the data is segmented into batches, including observed trajectories and future trajectories, with the observed trajectories consisting of 8 time steps and the future trajectories consisting of 12 time steps.

[0030] Furthermore, the system is trained using the Adam optimizer, and the gradients are pruned.

[0031] Furthermore, the system can accurately match the distribution of motion primitives such as velocity, acceleration, angular velocity, and angular acceleration, thereby ensuring the physical authenticity of the generated data.

[0032] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the scope of protection of the appended claims. Attached Figure Description

[0033] Figure 1 This is a flowchart of the technical solution of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0035] like Figure 1 As shown, a multi-agent trajectory co-generation and modeling system for urban scenarios is disclosed. The system includes:

[0036] a. Spatiotemporal encoder: used to extract historical and interactive perception context information from multi-agent scenarios and represent it as encoded history e; the spatiotemporal encoder includes node history LSTM and edge history LSTM.

[0037] 1) Node History LSTM: This module independently processes the trajectory data of each agent over the past H time steps. For each agent... Its time step The two-dimensional position is represented as The LSTM node history is responsible for learning and encoding the intelligent agent. from arrive Historical position sequence Generate individual historical features for each agent.

[0038] 2) Edge History LSTM: This module captures the interaction information between agents over time. It constructs a graph structure between agents, using interaction features such as relative positions and relative velocities as input, and uses LSTM to encode these dynamic interaction patterns. In this way, the system can understand complex social behaviors such as avoidance, following, and cooperation between agents.

[0039] Through the combined operation of the two LSTM methods described above, the spatiotemporal encoder can effectively extract information from the past data of all agents. Trajectory data at each time step Rich and interactive contextual information is extracted. This encoded past trajectory history information is uniformly represented as encoded history. The encoding history It includes the individual behavioral patterns and interactions of all agents in the scene, providing comprehensive conditions for subsequent trajectory generation.

[0040] b. Gaussian Mixture Model (GMM): Receives the encoded history e and explicitly models the agent's temporal destination. The multimodal distribution of the GMM, wherein the probability density function of the GMM is:

[0041] ;

[0042] in, , and They are the first The weights, mean, and covariance matrices of Gaussian kernels, where K is the number of Gaussian kernels;

[0043] The parameters of the GMM are obtained from the encoding history e through a fully connected layer:

[0044] , , ;

[0045] The covariance matrix From variance and correlation The reconstruction yields:

[0046] ;

[0047] A prominent characteristic of human behavior is its purposefulness, driven by one or more potential destinations. Since future destinations can exhibit multimodal characteristics (e.g., at an intersection, a pedestrian might go straight, turn left, or turn right), this invention innovatively introduces a Gaussian Mixture Model (GMM) to explicitly model the temporal destination distribution of multiple agents. This enables the system to... In this context, diverse future behaviors are generated.

[0048] The GMM receives the encoding history output from the spatiotemporal encoder. As input, and based on this, inferences are made about each agent. Time destination The probability distribution of the intelligent agent. Time destination The probability density function (PDF) is defined as:

[0049]

[0050] in: * K represents the number of Gaussian kernels; in this embodiment, K is set to 4. Selecting 4 Gaussian kernels can effectively capture four common main destination or behavior patterns (such as going straight, turning left or right, stopping, etc.), achieving a balance between expressive power and model complexity.

[0051] * It is the first The weights (mixing coefficients) of a Gaussian kernel represent the probability of that destination pattern.

[0052] * It is the first The two-dimensional mean of a Gaussian kernel represents the center position of the destination pattern.

[0053] * It is the first The covariance matrix of Gaussian kernels describes the shape and orientation of the destination pattern in two-dimensional space.

[0054] In order to learn from coding history To obtain the parameters of the aforementioned GMM, this system employs four independent parameterized fully connected layers. :

[0055]

[0056] in: * Output normalized weights ( In actual implementation, it is usually processed by Softmax to ensure that the sum is 1).

[0057] * Output the mean of each Gaussian kernel ( ).

[0058] * Output the logarithmic variance of each Gaussian kernel, through The function ensures that the variance is positive.

[0059] * Output the correlation of each Gaussian kernel ( ),pass The function limits its value to between.

[0060] Subsequently, the The covariance matrix of each component Based on its variance and correlation Refactor:

[0061]

[0062] During the training phase, to encourage GMM to learn meaningful and diverse destination patterns and to prevent pattern collapse (i.e., GMM focusing only on a few Gaussian kernels while ignoring others), this invention introduces pattern collapse regularization loss. The loss function constrains the weights, variance, and distance between the centers of the Gaussian kernel.

[0063]

[0064] in, It's a hyperparameter. This is the hinge loss function. The first term encourages the means of different Gaussian kernels to maintain sufficient distance, the second term encourages uniform weight distribution, and the third term encourages that the variance is not too small.

[0065] c. Gated Recurrent Unit (GRU) Decoder: Receives the encoded history e and the time destination (which is the ground truth during training). During testing, samples were taken from the GMM. The predicted trajectory is generated by autoregressively outputting the predicted trajectory through a residual layer. The process of generating the predicted trajectory includes: , ; in, Let q represent a GRU cell and q represent a residual layer. This represents the predicted state of agent i at time step t. It is the residual displacement, * It is by using the current predicted state Add residual displacement The predicted state for the next time step is obtained. During training, the system is initialized using the true initial state (i.e., The first value is the last true location of the observed trajectory.

[0066] d. Loss function: used to train the system, including:

[0067] 5) Destination loss: Based on the ground truth time target Maximum log-likelihood objective;

[0068] 6) Mode collapse loss: Regularize the weights, variances, and distances between the centers of the GMM to address the mode collapse problem.

[0069] ;

[0070] 3) Reconstruction loss: Minimize the predicted trajectory ground true trajectory Huber error between: ;

[0071] The system is optimized by minimizing the total loss function:

[0072] ;

[0073] in, , and It's a hyperparameter.

[0074] In a preferred embodiment, the system was trained and evaluated on the ETH and UCY datasets, which provide rich interactive pedestrian trajectory data. During training, the data was segmented into batches, with each batch containing 8 time steps (3.2 seconds) of observed trajectories as input and the next 12 time steps (4.8 seconds) of future trajectories as prediction targets. The model was implemented using the PyTorch framework and trained on a GPU. The parameters of the GMM were inferred from four fully connected layers, and the GRU decoder had a hidden layer size of 128. The system was trained using the Adam optimizer with an initial learning rate of 0.001 and an exponential learning rate decay strategy. To ensure training stability, gradient pruning was also performed to prevent gradient explosion.

[0075] The system can generate highly diverse and realistic multi-agent trajectory data, which can be used as a data augmentation technique for downstream imitation-based motion planning tasks to improve planning performance. Simultaneously, the system can accurately match the distribution of motion primitives such as velocity, acceleration, angular velocity, and angular acceleration, thereby ensuring the physical authenticity of the generated data.

Claims

1. A system for multi-agent trajectory co-generation and modeling in urban scenarios, characterized in that, include: a. Spatiotemporal encoder: used to extract historical and interactive perception context information from multi-agent scenarios and represent it as encoded history e; b. Gaussian Mixture Model (GMM): Receives the encoded history e and explicitly models the agent's temporal destination. The multimodal distribution of the GMM, wherein the probability density function of the GMM is: ； in, , and They are the first The weights, mean, and covariance matrices of Gaussian kernels, where K is the number of Gaussian kernels; The parameters of the GMM are obtained from the encoding history e through a fully connected layer: , , ； The covariance matrix From variance and correlation The reconstruction yields: ； c. Gated Recurrent Unit (GRU) Decoder: Receives the encoded history e and the time destination (which is the ground truth during training). During testing, samples were taken from the GMM. The predicted trajectory is generated by autoregressively outputting the predicted trajectory through a residual layer. The process of generating the predicted trajectory includes: ， ； in, Let q represent a GRU cell and q represent a residual layer. This represents the predicted state of agent i at time step t. It is the residual displacement; d. Loss function: used to train the system, including: 1) Destination loss: based on the ground truth time objective Maximum log-likelihood objective; 2) Mode collapse loss: Regularize the weights, variances, and distances between the centers of the GMM to address the mode collapse problem. ； 3) Reconstruction loss: Minimize the predicted trajectory ground true trajectory Huber error between: ; The system is optimized by minimizing the total loss function: ； in, , and It's a hyperparameter.

2. The system according to claim 1, characterized in that, The system is capable of generating multi-agent trajectory data with high diversity and realism.

3. The system according to claim 1, characterized in that, The system can be used as a data augmentation tool in downstream imitation-based motion planning tasks to improve planning performance.

4. The system according to claim 1, characterized in that, The spatiotemporal encoder includes a node history LSTM and an edge history LSTM.

5. The system according to claim 1, characterized in that, The number of Gaussian kernels K in the GMM is set to 4.

6. The system according to claim 1, characterized in that, The parameterized fully connected layer of the GMM consists of four independent layers, which are used to output weights, mean, log-variance, and correlation, respectively.

7. The system according to claim 1, characterized in that, The system is initialized using real initial states during training.

8. The system according to claim 1, characterized in that, During the training phase, the system's data is segmented into batches, including observed trajectories and future trajectories, with the observed trajectory consisting of 8 time steps and the future trajectory consisting of 12 time steps.

9. The system according to claim 1, characterized in that, The system is trained using the Adam optimizer and the gradients are clipped.

10. The system according to claim 1, characterized in that, The system can accurately match the distribution of motion primitives such as velocity, acceleration, angular velocity, and angular acceleration, thereby ensuring the physical authenticity of the generated data.

Images