A digital city simulation method fusing structured semantic modeling and trajectory sequence generation
By constructing a digital city simulation method that integrates structured semantic modeling and trajectory sequence generation, and utilizing ConvLSTM and CVAE deep learning models, the problems of insufficient training data diversity and high inference time complexity are solved. This method achieves real-time, multimodal, and scene-consistent trajectory prediction for multi-agent behavior, improving prediction accuracy and adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PEKING UNIV SHENZHEN GRADUATE SCHOOL
- Filing Date
- 2026-01-17
- Publication Date
- 2026-05-29
AI Technical Summary
Existing trajectory prediction methods suffer from insufficient diversity in training data and high inference time complexity, making it difficult to achieve real-time, multimodal, and scenario-consistent trajectory prediction for multi-agent behaviors in urban traffic environments.
By constructing a digital city simulation method that integrates structured semantic modeling and trajectory sequence generation, and utilizing ConvLSTM and CVAE deep learning models, combined with intelligent driver models and multi-lane ensemble models, multimodal future trajectories are generated, achieving constant-time inference and high-scene-consistency prediction.
It improves the diversity of training data and the robustness of the model, achieves real-time prediction with O(1) time complexity, improves prediction accuracy and scene consistency, and adapts to dynamic scene changes.
Smart Images
Figure CN122113578A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of digital city simulation, and in particular to a digital city simulation method that integrates structured semantic modeling and trajectory sequence generation. Background Technology
[0002] With the rapid development of autonomous driving and intelligent transportation systems, accurate prediction of the future trajectories of multiple agents (such as vehicles and pedestrians) in urban environments has become crucial. However, existing trajectory prediction methods generally face the following challenges: 1. Insufficient diversity of training data: Existing trajectory prediction methods are limited by insufficient diversity of training data, making it difficult to fully cover the various motion patterns that may occur in real-world scenarios. Although a large number of autonomous driving datasets have been proposed, such as KITTI, NGSIM, and Argoverse, they often have limitations in terms of scene diversity, traffic behavior patterns, or trajectory length, making it difficult for the models to generalize to unseen complex situations.
[0003] 2. High Inference Time Complexity: Many existing trajectory prediction methods, especially those considering multi-agent interactions, typically operate on a single-agent basis during inference. This means that each agent in the scene needs to perform an independent forward propagation. Consequently, the prediction time complexity increases linearly or even quadratically with the number of agents in the scene, making it difficult to meet the demands of real-time applications. The inherent randomness of vehicle motion and the need to predict multimodal trajectories consistent with scene semantics further significantly increase the time complexity.
[0004] 3. Existing methods often struggle to fully capture the multimodal characteristics of motion behavior when predicting future trajectories. This means that an agent may have multiple plausible future paths within a given scenario. This is primarily due to the lack of sufficient diversity coverage in the training data and the difficulty of effectively modeling this inherent uncertainty. Although some generative models have been applied to trajectory prediction to introduce diversity, they may sometimes produce trajectories that are inconsistent with the semantics of the scene or exhibit abnormal behavior. Summary of the Invention
[0005] This invention aims to overcome the aforementioned deficiencies in existing technologies and provide a digital city simulation method that integrates structured semantic modeling and trajectory sequence generation to achieve real-time, multimodal, and scenario-consistent trajectory prediction of multi-agent behaviors in urban traffic environments. This invention addresses two key challenges—insufficient training data diversity and high inference time complexity—by introducing an innovative dataset simulation strategy and a multi-agent trajectory prediction method. The specific technical solution is as follows.
[0006] A digital city simulation method integrating structured semantic modeling and trajectory sequence generation includes the following steps:
[0007] a. Constructing a simulation environment: Obtain road map information of the scene and represent it as a graph data structure; extract vehicle trajectory information from the dataset and project it onto the road map to reconstruct the scene;
[0008] b. Generate diverse behaviors: For each vehicle in the scene, identify multiple possible driving maneuvers (e.g., going straight, turning left, turning right, and changing lanes) based on traffic conditions and road structure, and uniformly sample different vehicle behaviors from them;
[0009] c. Sampling reference speed profile: Based on the sampled vehicle behavior, obtain the speed profile that is closest to the behavior from the reference speed profile library of the real dataset as a reference;
[0010] d. Dynamics simulation: Using the road map, sampled vehicle behavior and reference speed profile, vehicle dynamics and lane change decisions are modeled using the Intelligent Driver Model (IDM) and Multi-Lane Integrated Model (MOBIL) to generate a sequence of vehicle position, speed, heading, acceleration and steering angle states during the simulation time.
[0011] e. Constructing a trajectory prediction model: Construct a deep learning model based on a convolutional long short-term memory network (ConvLSTM) and a conditional variational autoencoder (CVAE). This model takes the scene context graph, the agent's current state graph, the position mask graph, the trajectory label graph, and the latent vector graph as input.
[0012] f. Achieve constant-time inference: State pooling is introduced into the ConvLSTM to ensure the availability of previous state information when predicting the next position, thereby enabling synchronous prediction for all agents;
[0013] g. Generate multimodal predicted trajectories: Model different behavior types of trajectories using the CVAE, and generate multimodal future trajectories during inference by conditionally sampling the latent vectors;
[0014] h. Model training: The deep learning model is trained by minimizing the reconstruction loss between the predicted trajectory and the true trajectory, and by regularizing the KL divergence loss of the latent space distribution.
[0015] Preferably, in step a, the road map information includes lane center lines, number of lanes, and one-way information.
[0016] Preferably, in step a, the vehicle trajectory information includes the vehicle's attitude information, which is used to reconstruct an accurate driving scene.
[0017] Preferably, in step b, the diversified behaviors include going straight, turning left, turning right, and changing lanes.
[0018] Preferably, in step c, the reference speed profile is obtained by a nearest neighbor method based on the distance before the turn and the average speed.
[0019] Preferably, in step d, the acceleration of the intelligent driver model (IDM) The calculation formula is:
[0020] in, To determine the actual distance to the bumper of the vehicle in front, The vehicle's current speed. To maintain the speed difference with the vehicle in front, For comfortable acceleration, For the expected reference speed, To influence the exponent of acceleration decreasing with velocity, This represents the desired minimum gap.
[0021] Preferably, in step d, the lane change decision of the Multi-Lane Integrated Model (MOBIL) is controlled by the following formula:
[0022] in, For the current acceleration, The new acceleration after the lane change; subscript These respectively represent the current vehicle, the vehicle in the new lane, and the vehicle in the old lane; As a polite factor, To change the acceleration threshold for the lane, For maximum safe deceleration, Offset for a specific lane.
[0023] Preferably, in step e, the scene context graph for The bird's-eye view shows that the channel dimension represents a hot-coded information for each pixel corresponding to a road, lane, or unknown road element.
[0024] The digital city simulation method integrating structured semantic modeling and trajectory sequence generation provided by this invention has the following advantages:
[0025] (1) Improve the diversity of training data: Through automated simulation methods, training data that is larger in scale and more diverse than existing real datasets can be generated, effectively making up for the shortcomings of real data in covering possible motion patterns, thereby improving the generalization ability and robustness of trajectory prediction models.
[0026] (2) Achieve constant-time inference: By adopting global scene representation and state pooling ConvLSTM, the model can predict the future trajectories of all agents in the scene at once, and the inference time complexity reaches O(1), which significantly improves the real-time performance in large-scale multi-agent scenarios and meets the stringent requirements of applications such as autonomous driving.
[0027] (3) Enhance multimodal prediction capability: By combining CVAE and behavioral labels, the model can generate multimodal prediction trajectories that are consistent with scene semantics and multi-agent interactions, capture the uncertainty of the agent's future behavior, and provide more comprehensive information for the decision-making system.
[0028] (4) Significantly improved prediction accuracy: The method of this invention is superior to the state-of-the-art methods in terms of average displacement error (ADE), final displacement error (FDE) and negative log-likelihood (NLL) after training and testing on simulated datasets and existing real datasets, demonstrating its superior prediction performance.
[0029] (5) High scene consistency: The proposed method can generate trajectories that are highly consistent with the scene context (such as lane lines and road structure), avoiding semantic inconsistencies or abnormal behaviors that may occur in traditional methods.
[0030] (6) Adapting to dynamic scenarios: It adopts a pixel-level prediction method, which can flexibly handle the dynamic entry and exit of agents in the scenario and has good adaptability. Attached Figure Description
[0031] Figure 1 This is an overall architecture diagram of the present invention;
[0032] Figure 2 It is a flowchart of the steps to generate diverse behaviors;
[0033] Figure 3 This is a flowchart of the steps involved in dynamic simulation; Detailed Implementation
[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0035] A digital city simulation method that integrates structured semantic modeling and trajectory sequence generation, such as Figure 1 As shown, it includes the following steps:
[0036] a. Constructing the Simulation Environment: Acquire road map information of the scene, including lane centerlines, number of lanes, and one-way information, and represent it as a graph data structure; extract vehicle trajectory information from the dataset, including vehicle attitude information, to reconstruct an accurate driving scene, and project it onto the road map to reconstruct the scene; specifically, road map construction involves first creating a road map using publicly available map information (such as OpenStreetMaps (OSM)) or datasets (such as Argoverse) providing lane centerline information. The road map is constructed as a graph data structure where nodes represent lane endpoints and edges represent lane centerlines. For two-way roads, segmentation is performed based on a specified number of lanes and one-way information. Object Instantiation: Specifically, using the vehicle trajectory and attitude information from the dataset, accurately project them onto the generated road map. This is achieved by defining a coordinate system with the vehicle as the reference and finding the nearest lane edge occupied by the object in the scene. This step completes the scene reconstruction, laying the foundation for subsequent behavior simulation.
[0037] b. Generate diverse behaviors: such as Figure 2 As shown, for each vehicle in the scenario, multiple possible driving maneuvers (e.g., going straight, turning left, turning right, and changing lanes) are identified based on traffic conditions and road structure, and different vehicle behaviors are uniformly sampled from these. Specifically, for each instantiated vehicle in the scenario, based on the current traffic conditions and road structure, the system explores various maneuvers it can perform on the local road map. This is achieved through depth-first search (DFS) to reach the K leaf nodes within a certain distance threshold. These feasible maneuvers are categorized into three basic types: "turn left," "turn right," and "go straight," with lane-changing behavior also included. The system uniformly samples different vehicle behaviors from these possible maneuvers for simulation.
[0038] c. Sampling Reference Speed Profile: Based on the sampled vehicle behavior, and using a nearest neighbor method based on the distance before turning and average speed, the system obtains the speed profile closest to the behavior from a reference speed profile library in the real dataset as a reference. Specifically, to ensure the generated behavior is as realistic as possible, a reference speed profile library is created from the real dataset before the simulation begins. During simulation, after sampling the desired behavior, the system obtains the nearest neighbor speed profile closest to the expected behavior in the current scenario as a reference, based on features such as the distance before turning (for turning maneuvers) and average speed (for straight-line maneuvers).
[0039] d. Dynamic simulation: such as Figure 3As shown, using the road map, sampled vehicle behavior, and reference speed profiles, vehicle dynamics and lane change decisions are modeled using an Intelligent Driver Model (IDM) and a Multi-Lane Integrated Model (MOBIL) to generate a sequence of vehicle position, speed, heading, acceleration, and steering angle states over the simulation time, where acceleration... The calculation formula is:
[0040] in, To determine the actual distance to the bumper of the vehicle in front, The vehicle's current speed. To maintain the speed difference with the vehicle in front, For comfortable acceleration, For the expected reference speed, To influence the exponent of acceleration decreasing with velocity, This represents the desired minimum gap.
[0041] The lane-changing decision of the Multi-Lane Integrated Model (MOBIL) is controlled by the following formula:
[0042] in, For the current acceleration, The new acceleration after the lane change; subscript These respectively represent the current vehicle, the vehicle in the new lane, and the vehicle in the old lane; As a polite factor, To change the acceleration threshold for the lane, For maximum safe deceleration, Offset for a specific lane.
[0043] By combining IDM and MOBIL, the simulation engine can generate a 7-second sequence of vehicle states. The simulation includes the vehicle's position, speed, heading, acceleration, and steering angle, with a simulation frequency of 10Hz, and generates up to three diverse behaviors per scene.
[0044] e. Constructing a trajectory prediction model: A deep learning model based on a Convolutional Long Short-Term Memory (ConvLSTM) network and a Conditional Variational Autoencoder (CVAE) is constructed. This model takes the scene context graph, the agent's current state graph, the position mask graph, the trajectory label graph, and the latent vector graph as input. for The bird's-eye view represents a type of hot-coded information where each pixel corresponds to a road, lane, or unknown road element. The agent's current state diagram. for The representation includes all agents in the scene at time step. The relative spatial position. Track label map. for The representation includes one-hot encoded trajectory labels projected onto specific locations of the agent. Latent Vector Graph for The representation includes information from the identification module. Acquired latent vectors .
[0045] Agent Trajectory and State Diagram : set up Indicates the first An agent from time step 1 to... The trajectory information, among which This represents the agent's spatial position within the scene. The network input is in the form of relative coordinates relative to the agent's starting position. For the first in the scene An intelligent agent, its Projected onto the corresponding Location, to construct a spatial location map of the agent's state. , making Including the An agent at time step The relative coordinates.
[0046] Real trajectory : Represents the true trajectory, i.e., from the observation time step To the future The relative trajectory.
[0047] Location mask map : This represents the location mask of the agent configuration in the scene, used to indicate the presence of the agent in the spatial grid.
[0048] Vehicle ID map To track vehicles across time steps, a vehicle ID graph is constructed. ,in .
[0049] Track label map Each trajectory is associated with a label. The related categories indicate the type of behavior of the trajectory, such as "straight ahead", "left turn", and "right turn". Trajectories that change lanes also fall into one of these three categories. Encoded The grid diagram representation makes Vehicle trajectories differ from the randomness of human movement; they depend on the behavior of other vehicles on the road. Therefore, classifying trajectories according to different maneuvers is of great significance.
[0050] The network remains in its previous state. Scene context graph Track label map Location mask and noise graph As input, predict the position of each agent in the scene within its corresponding grid. future trajectory It is worth noting that this invention does not set a separate prediction head for each agent, but rather the network predicts a single future state graph. Each agent attempts to Matching its time .
[0051] The latent encoder serves as the recognition module within the CVAE framework. It is only used during the training phase.
[0052] Input: Receive past and future trajectory information .
[0053] Embedding layer and LSTM network: The input trajectory information is passed through the embedding layer, and then the embedding vector is input into the LSTM network to output the code at each time step.
[0054] Vector concatenation and MLP: Encoding the output of all time steps and one-hot encoded trajectory labels. Concatenate into a single vector .
[0055] Mean and variance prediction: This vector The mean is then obtained using a multilayer perceptron (MLP). and standard deviation This outputs a distribution. .
[0056] formal:
[0057]
[0058]
[0059] in, It is a latent vector that represents the abstract features of the trajectory and is used to capture multimodal information.
[0060] Encoder module: from time step arrive The above input is passed through an encoder module. This module consists of strided convolutions, which encode information into a smaller spatial dimension.
[0061] Decoder Module: The encoder's output is then passed to the decoder. The decoder consists of a ConvLSTM and a transposed convolution (for upsampling), with skip connections from the encoder module, ultimately outputting a... The map.
[0062] State Pooling ConvLSTM: The decoder output is further fed into another ConvLSTM layer with state pooling operations. This network is shared during the observation and prediction phases.
[0063] Output layer: Finally, add one The convolutional layer outputs a 2-channel map containing the relative predicted coordinates of the agent in the scene at the next time step. .
[0064] During the trajectory observation phase, the real agent locations are used to deploy the ConvLSTM-based network. In the prediction phase (from... arrive The network's output is not directly fed back as input; instead, the agent's state is updated to its next position in the scene based on the prediction results. (Relative predicted position) Updated to absolute predicted position To obtain an updated scene state graph containing the update positions of all agents in the scene. This scene representation is independent of the number of agents, and because the next state of an agent is predicted at its corresponding pixel position, it can handle situations where agents dynamically enter and exit the scene.
[0065] f. Achieving constant-time inference: A state pooling operation is introduced into the ConvLSTM to ensure the availability of previous state information when predicting the next position, thereby achieving synchronous prediction for all agents; the state pooling operation stores the state information of the previous ConvLSTM layer. Pooling is used to initialize the hidden state and cell state for the next time step.
[0066] g. Generate multimodal predicted trajectories: Model different behavior types of trajectories using the CVAE, and generate multimodal future trajectories during inference by conditionally sampling the latent vectors;
[0067] h. Model Training: The deep learning model is trained by minimizing the reconstruction loss between the predicted and true trajectories, and by regularizing the KL divergence loss of the latent space distribution. Reconstruction Loss The calculation formula is:
[0068] in, For the number of agents, For the first The true relative trajectory of each intelligent agent For the first The predicted relative trajectory of each agent.
[0069] KL divergence loss The calculation formula is:
[0070] in, To identify the distribution of the module output, For the prior distribution, i.e. .
[0071] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.
Claims
1. A digital city simulation method integrating structured semantic modeling and trajectory sequence generation, characterized in that, include: a. Constructing a simulation environment: Obtain road map information of the scene and represent it as a graph data structure; extract vehicle trajectory information from the dataset and project it onto the road map to reconstruct the scene; b. Generate diverse behaviors: For each vehicle in the scene, identify multiple possible driving maneuvers (e.g., going straight, turning left, turning right, and changing lanes) based on traffic conditions and road structure, and uniformly sample different vehicle behaviors from them; c. Sampling reference speed profile: Based on the sampled vehicle behavior, obtain the speed profile that is closest to the behavior from the reference speed profile library of the real dataset as a reference; d. Dynamics simulation: Using the road map, sampled vehicle behavior and reference speed profile, vehicle dynamics and lane change decisions are modeled using the Intelligent Driver Model (IDM) and Multi-Lane Integrated Model (MOBIL) to generate a sequence of vehicle position, speed, heading, acceleration and steering angle states during the simulation time. e. Constructing a trajectory prediction model: Construct a deep learning model based on a convolutional long short-term memory network (ConvLSTM) and a conditional variational autoencoder (CVAE). This model takes the scene context graph, the agent's current state graph, the position mask graph, the trajectory label graph, and the latent vector graph as input. f. Achieve constant-time inference: State pooling is introduced into the ConvLSTM to ensure the availability of previous state information when predicting the next position, thereby enabling synchronous prediction for all agents; g. Generate multimodal predicted trajectories: Model different behavior types of trajectories using the CVAE, and generate multimodal future trajectories during inference by conditionally sampling the latent vectors; h. Model training: The deep learning model is trained by minimizing the reconstruction loss between the predicted trajectory and the true trajectory, and by regularizing the KL divergence loss of the latent space distribution.
2. The method according to claim 1, characterized in that, In step a, the road map information includes lane center lines, number of lanes, and one-way information.
3. The method according to claim 1, characterized in that, In step a, the vehicle trajectory information includes the vehicle's attitude information, which is used to reconstruct an accurate driving scene.
4. The method according to claim 1, characterized in that, In step b, the diverse behaviors include going straight, turning left, turning right, and changing lanes.
5. The method according to claim 1, characterized in that, In step c, the reference speed profile is obtained by a nearest neighbor method based on the distance before turning and the average speed.
6. The method according to claim 1, characterized in that, In step d, the acceleration of the intelligent driver model (IDM) The calculation formula is: in, To determine the actual distance to the bumper of the vehicle in front, The vehicle's current speed. To maintain the speed difference with the vehicle in front, For comfortable acceleration, For the expected reference speed, To influence the exponent of acceleration decreasing with velocity, This represents the desired minimum gap.
7. The method according to claim 1, characterized in that, In step d, the lane change decision of the Multi-Lane Integrated Model (MOBIL) is controlled by the following formula: in, For the current acceleration, The new acceleration after the lane change; subscript These respectively represent the current vehicle, the vehicle in the new lane, and the vehicle in the old lane; As a polite factor, To change the acceleration threshold for the lane, For maximum safe deceleration, Offset for a specific lane.
8. The method according to claim 1, characterized in that, In step e, the scene context graph for The bird's-eye view shows that the channel dimension represents a hot-coded information for each pixel corresponding to a road, lane, or unknown road element.