A dynamic traffic scene generation method based on a pre-trained trajectory large model
By generating high-fidelity dynamic traffic scenes using pre-trained trajectory large models, the problems of insufficient realism, diversity, and controllability in existing technologies are solved, achieving efficient and diversified traffic scene generation, which is suitable for training autonomous driving perception algorithms, traffic flow simulation, and intelligent transportation system evaluation.
Patent Information
- Application Number
- CN202610011843.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-06
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies lack realism, diversity, controllability, and generation efficiency when generating dynamic traffic scenes. They are unable to effectively capture complex traffic participant behaviors and multi-agent interactions, and their generalization ability is limited.
By employing a pre-trained trajectory model, a high-fidelity dynamic traffic scene is generated by constructing a spatiotemporal scene graph and utilizing deep learning architectures such as Transformer and GNN. This scene includes predicted trajectory sequences of multiple agents, and is then instantiated and rendered in a 3D environment.
The generated scenes are highly realistic, diverse, and controllable, and can quickly and efficiently meet different testing and simulation needs, improving generation efficiency and overcoming the limitations of traditional methods.
Smart Images
Figure CN122116625A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of traffic scene simulation and generation technology, and in particular relates to a dynamic traffic scene generation method based on a pre-trained trajectory large model, which can be widely used in autonomous driving perception algorithm training, decision planning testing, traffic flow simulation, intelligent transportation system evaluation and other fields. Background Technology
[0002] With the rapid development of autonomous driving technology and intelligent transportation systems, increasingly higher demands are being placed on the generation of high-fidelity, diverse, and controllable dynamic traffic scenes. Traditional traffic scene generation methods mainly rely on manual rule modeling or statistical models. When generating complex, multi-agent interactive dynamic scenes, these methods often face problems such as insufficient realism, lack of diversity, low generation efficiency, and limited generalization ability. For example, simple physical models struggle to capture the complex behavioral patterns of human drivers and the subtle interactions between different traffic participants. Even models trained on large-scale real-world data typically require a large amount of labeled data, and their generalization ability remains limited for unseen complex scenes or edge cases, resulting in a "content gap" between synthetic and real-world scenes.
[0003] In recent years, large-scale model technology has made significant progress in fields such as natural language processing and computer vision, with its powerful generalization ability and ability to learn complex patterns attracting widespread attention. In the transportation field, the emergence of large-scale models pre-trained based on trajectory data has provided a new approach to solving the problems in generating dynamic traffic scenes. However, how to effectively apply these pre-trained trajectory models to the generation of dynamic traffic scenes, ensuring the realism, diversity, and controllability of the generated scenes, while overcoming the limitations of traditional methods in modeling the complexity and diversity of real traffic, remains a pressing technical challenge. Summary of the Invention
[0004] This invention aims to address the shortcomings of existing dynamic traffic scene generation methods in terms of realism, diversity, controllability, and generation efficiency, and provides a dynamic traffic scene generation method and system based on a pre-trained large-scale trajectory model. By leveraging a deep understanding of traffic behavior patterns through a large-scale pre-trained trajectory model, this invention can generate dynamic traffic scenes with high realism that conform to real-world traffic rules and agent behavior habits.
[0005] To achieve the above objectives, this invention provides a method for generating dynamic traffic scenes based on a pre-trained large trajectory model, comprising the following steps:
[0006] S1: Acquire real traffic data and preprocess the real traffic data to construct a spatio-temporal scene graph representation. The spatio-temporal scene graph includes the topological structure of the scene, static element information, and the initial state and historical trajectory information of multiple dynamic traffic agents.
[0007] S2: Using the spatiotemporal scene map as a conditional input, a predicted trajectory sequence of the multiple dynamic traffic agents is generated through a pre-trained trajectory model. The predicted trajectory sequence reflects the temporal and spatial interaction relationship between the dynamic traffic agents.
[0008] S3: Convert the predicted trajectory sequence into high-fidelity dynamic traffic scene data, which can be used for simulation or downstream tasks.
[0009] Preferably, the construction of the spatiotemporal scene graph in step S1 includes: extracting static element information such as road topology information (e.g., lane lines, intersections, traffic signs, etc.) and traffic light positions from the real traffic data, as well as state information such as the position, speed, acceleration, and heading angle of dynamic traffic agents such as vehicles, pedestrians, and bicycles at multiple historical time steps, and encoding the information into a graph structure. In the graph structure, nodes can represent traffic agents or scene elements, and edges can represent the spatiotemporal relationships between them, such as the relative positions between agents, lane connections, and the relationship between agents and lanes. This spatiotemporal scene graph can effectively encode complex spatial interactions and temporal dependencies.
[0010] Preferably, the pre-trained trajectory model is trained through self-supervised or semi-supervised learning on large-scale real traffic trajectory data. The training process aims to learn the behavioral patterns, driving habits, and complex multi-agent interaction rules of different traffic participants. The training dataset may include real traffic data from different geographical regions, with varying traffic densities and weather conditions.
[0011] Preferably, the pre-trained trajectory model can employ Transformer, Graph Neural Network (GNN), Recurrent Neural Network (RNN), or a combination thereof. For example, a Transformer-based model can capture long-distance spatiotemporal dependencies and interactions between agents through its multi-head attention mechanism; GNN can effectively process graph-structured data and capture spatial relationships between agents; and RNN is suitable for processing temporal information of trajectories. These architectures can jointly learn and capture the temporal dependencies of trajectory data and the complex spatiotemporal interactions between multiple agents.
[0012] Preferably, the generation of the predicted trajectory sequence in step S2 further includes: conditionally controlling the generation process of the large trajectory model based on preset scene attributes, such as traffic density (e.g., congestion, smooth traffic), weather conditions (e.g., sunny, rainy, snowy, foggy), time period (e.g., daytime, nighttime, peak hours), or specific events (e.g., traffic accidents, road construction, emergency vehicle passage). By adjusting these conditional parameters, the model can be guided to generate dynamic traffic scenes with specific characteristics, thereby meeting different testing and simulation needs, such as generating traffic scenes under extreme weather conditions or specific traffic events.
[0013] Preferably, step S3, converting the predicted trajectory sequence into high-fidelity dynamic traffic scene data, includes mapping each trajectory point (containing position, attitude, and other information) in the predicted trajectory sequence to a pre-constructed 3D simulation environment. The 3D simulation environment contains detailed static elements such as road models, buildings, vegetation, and traffic facilities. Then, based on the position and attitude information of the trajectory points, vehicle models, pedestrian models, etc., are instantiated in the 3D scene and made to move along the generated trajectory. Simultaneously, lighting, weather effects (such as rain and snow), traffic light status, and environmental sound effects can be simulated to further enhance the realism of the scene. Finally, the rendered dynamic traffic scene data is output in a standard format (such as OpenDRIVE, OpenSCENARIO, FBX, GLTF, etc.) for use by autonomous driving simulators, traffic flow simulation software, or other downstream applications.
[0014] Preferably, the method can generate dynamic traffic scenarios that include complex interactions between at least two different types of traffic agents (e.g., vehicles, pedestrians, and bicycles), including but not limited to behaviors such as following other vehicles, changing lanes, turning, yielding, and passing through intersections.
[0015] The present invention also provides a dynamic traffic scene generation system, comprising:
[0016] Data Processing Module: This module is responsible for interacting with external data sources (such as high-precision map databases, real-time traffic flow sensors, historical traffic datasets, etc.) to acquire raw traffic data. Its core function is to preprocess and parse the data, ultimately outputting a structured spatiotemporal scene graph representation. This module may include sub-modules such as a sensor data parser, a map data processor, and a graph builder.
[0017] Trajectory Generation Module: This module is the core computational component of the system, internally deploying a pre-trained large trajectory model. It receives the spatiotemporal scene map output from the data processing module as input and generates predicted trajectory sequences for multiple dynamic traffic agents based on model inference and sampling mechanisms. This module supports various generation modes, such as single-shot generation, batch generation, and conditional generation. To improve computational efficiency, the large trajectory model can be deployed on high-performance computing clusters, GPU servers, or cloud computing platforms.
[0018] Scene Rendering Module: This module is responsible for instantiating and rendering the abstract trajectory sequence output by the trajectory generation module in a 3D simulation environment. It includes a 3D scene engine capable of loading 3D models, materials, and lighting information. This module converts the trajectory sequence into the actual motion of entities such as vehicles and pedestrians in 3D space and renders high-fidelity images, videos, or interactive virtual environments. This module typically provides interfaces with mainstream simulation platforms (such as Carla, Unity3D, and Unreal Engine) or outputs scene files that conform to simulation standards.
[0019] The dynamic traffic scene generation method and system based on a pre-trained trajectory large model provided by this invention has the following beneficial effects:
[0020] (1) High realism and diversity: Through the pre-trained trajectory large model, the present invention can learn and reproduce the complex and diverse traffic behavior patterns and intelligent agent interaction rules in the real world. The generated scene has a high degree of realism and diversity, which overcomes the problem of single and unnatural behavior patterns in traditional methods and effectively makes up for the "content gap" between the synthetic scene and the real scene.
[0021] (2) Strong controllability: It can precisely control the generation process through condition input (such as spatiotemporal scene map, traffic density, weather, etc.) to generate dynamic traffic scenes of specific types and complexities, meet different testing and simulation needs, such as generating traffic scenes under extreme weather or specific traffic events.
[0022] (3) Efficiency improvement: Once the trajectory model has been pre-trained, it can quickly and efficiently generate a large number of dynamic traffic scenes, which significantly improves the efficiency of scene generation and reduces the cost of manual creation or data collection.
[0023] (4) Generalization ability: The pre-trained large model has a strong generalization ability and can handle and generate complex traffic scenarios that have not appeared in the training data, which is of great significance for the long-tail problem testing of autonomous driving systems.
[0024] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the scope of protection of the appended claims. Attached Figure Description
[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 This is a flowchart illustrating a dynamic traffic scene generation method based on a pre-trained trajectory large model according to the present invention.
[0027] Figure 2 This is a schematic diagram of the structure of a dynamic traffic scene generation system according to the present invention. Detailed Implementation
[0028] The following description, in conjunction with the accompanying drawings, illustrates a specific implementation of the dynamic traffic scene generation method based on a pre-trained trajectory large model provided by the present invention.
[0029] Example 1: Method Flow
[0030] refer to Figure 1 The present invention provides a method for generating dynamic traffic scenes based on a pre-trained trajectory large model, which specifically includes the following steps:
[0031] S1: Acquisition of Real Traffic Data and Construction of Spatiotemporal Scene Maps
[0032] First, real-world traffic data is acquired from various sources, such as vehicle-mounted sensors (cameras, lidar), traffic monitoring systems, and floating car data. This data includes road geometry, lane topology, traffic light status, and temporal motion data (position, speed, acceleration, heading angle, etc.) of various traffic participants (vehicles, pedestrians, bicycles, etc.).
[0033] The raw data undergoes preprocessing, including data cleaning, noise reduction, missing value imputation, coordinate system transformation, and format standardization. Then, a spatio-temporal scene graph is constructed based on the preprocessed data. The spatio-temporal scene graph is a graph structure, where:
[0034] Nodes represent entities in the scene, such as road elements (lanes, intersections), traffic lights, and dynamic agents (vehicles, pedestrians) at each moment.
[0035] Edges represent relationships between entities, such as a vehicle in a lane, the relative positional relationship between vehicles, and the effect of traffic lights on vehicles.
[0036] For example, a scene diagram can be represented as ,in It is a set of nodes. It is a set of edges. Each dynamic agent at the initial moment... The state can be represented as Static scene information, such as lane lines and intersection topology, is also encoded into the scene graph.
[0037] S2: Trajectory sequence generation based on pre-trained large trajectory model
[0038] This step is the core of the invention. We use the scene graph constructed in S1 as input and feed it into a pre-trained trajectory model. This trajectory model has been fully trained on a large-scale real traffic trajectory dataset. Its training objective can be to predict future trajectories or generate trajectories that conform to given conditions, thereby learning rich traffic participant behavior patterns (such as following, changing lanes, turning, and avoiding) and complex inter-agent interaction logic.
[0039] The large trajectory model can employ a Transformer architecture. For example, the information of nodes and edges in the scene graph can be encoded into embedding vectors, which serve as the input sequence for the Transformer. Through a multi-head attention mechanism, the model can capture long-distance dependencies and temporal correlations between different agents. The model outputs the predicted trajectory sequence of all dynamic agents within the next T time steps. ,in It is the first A sequence of trajectory points for an agent.
[0040] To increase the diversity of generated scenarios, random noise can be introduced into the model or generative models such as variational autoencoders (VAEs) can be used, enabling the model to generate multiple possible future trajectories given the same input conditions. Furthermore, conditional control parameters can be introduced, such as inputting desired traffic density or rain / snow weather parameters, to guide the model in generating trajectory sequences that conform to these attributes.
[0041] S3: Conversion and Output of High-Fidelity Dynamic Traffic Scenes
[0042] After obtaining the predicted trajectory sequence generated in S2, it needs to be converted into high-fidelity dynamic traffic scene data. This typically involves 3D reconstruction and rendering techniques.
[0043] First, each trajectory point in the trajectory sequence is mapped to a pre-built 3D simulation scene. This 3D scene contains detailed static elements such as road models, buildings, and trees.
[0044] Then, based on the position and posture information of the trajectory points, vehicle models, pedestrian models, etc., are instantiated in the 3D scene and made to move along the generated trajectory. At the same time, lighting, weather effects (such as rain and snow), traffic lights, and environmental sound effects can be simulated to further enhance the realism of the scene.
[0045] Finally, the rendered dynamic traffic scene data will be output in standard formats (such as OpenDRIVE, OpenSCENARIO, FBX, GLTF, etc.) so that it can be used by autonomous driving simulators, traffic flow simulation software or other downstream applications.
[0046] Example 2: System Structure
[0047] refer to Figure 2 The present invention also provides a dynamic traffic scene generation system based on a pre-trained trajectory large model, comprising:
[0048] Data Processing Module: This module is responsible for interacting with external data sources (such as high-precision map databases, real-time traffic flow sensors, historical traffic datasets, etc.) to acquire raw traffic data. Its core function is to preprocess and parse the data, transforming the raw data into a structured spatiotemporal scene graph representation. This module may include sub-modules such as sensor data parser, map data processor, and graph builder to ensure data accuracy and consistency.
[0049] Trajectory Generation Module: This module is the core computational component of the system, internally deploying a pre-trained large-scale trajectory model. It receives the spatiotemporal scene map output from the data processing module as input and generates predicted trajectory sequences for multiple dynamic traffic agents based on model inference and sampling mechanisms. This module supports various generation modes, such as single-shot generation, batch generation, and conditional generation. To improve computational efficiency, the large-scale trajectory model can be deployed on high-performance computing clusters, GPU servers, or cloud computing platforms to meet the needs of large-scale scene generation.
[0050] Scene Rendering Module: This module is responsible for instantiating and rendering the abstract trajectory sequence output by the trajectory generation module in a 3D simulation environment. It includes a 3D scene engine capable of loading 3D models, materials, and lighting information. This module converts the trajectory sequence into the actual motion of entities such as vehicles and pedestrians in 3D space and renders high-fidelity images, videos, or interactive virtual environments. This module typically provides interfaces with mainstream simulation platforms (such as Carla, Unity3D, and Unreal Engine) or outputs scene files conforming to simulation standards for seamless integration.
[0051] Example 3: Training of a pre-trained trajectory large model
[0052] The key to this invention lies in the pre-trained large trajectory model. This model can be trained in the following ways:
[0053] Datasets: Collect large-scale, diverse real-world traffic trajectory data, such as publicly available datasets like Waymo Open Dataset, nuScenes, and Argoverse, as well as proprietary datasets. The datasets should include data on different traffic types (urban, highways), road conditions (intersections, straightaways), weather conditions, and time periods to ensure the model's general applicability.
[0054] Model architecture: Employs deep learning architectures such as Transformer, GNN-RNN combination, or Diffusion Model. For example, a large trajectory model based on Transformer can be designed as follows: inputting spatiotemporal scene graph embeddings of the current and historical moments, and outputting the joint trajectory distribution of all agents at multiple future time steps through a multi-layer encoder-decoder structure. The training objective of the model can be to minimize the error between the predicted trajectory and the real trajectory (such as L1 / L2 loss), or to improve the realism of the generated trajectory through adversarial training (GAN).
[0055] Training tasks can employ self-supervised learning tasks, such as masked trajectory prediction, where a portion of the trajectory is randomly masked and the model predicts the masked trajectory; or sequence-to-sequence trajectory generation tasks, where given historical trajectories and a scene map, the model predicts future trajectories. Adversarial learning (such as GANs) can also be combined to improve the realism of generated trajectories, where the discriminator distinguishes between real and generated trajectories, while the generator strives to generate more realistic trajectories to deceive the discriminator.
[0056] The method and system of this invention can be widely applied to:
[0057] Autonomous driving simulation testing: Provides an unlimited number of diverse and high-fidelity dynamic traffic scenarios for autonomous driving algorithms (perception, decision-making, control), accelerating algorithm development and verification, and reducing the cost and risk of real vehicle testing.
[0058] Traffic flow simulation and optimization: Generate macroscopic and microscopic traffic flow scenarios that conform to real-world traffic patterns for use in urban traffic planning, traffic signal optimization, and traffic congestion prediction.
[0059] Driver Training and Assessment: Provides a realistic traffic environment for virtual driving simulators, used for training new drivers and assessing the skills of professional drivers.
[0060] Intelligent Transportation Systems (ITS) Evaluation: Used to evaluate the performance and effectiveness of new ITS solutions in different traffic scenarios.
[0061] This invention is not limited to the specific embodiments described above. All modifications and variations made by those skilled in the art within the scope and essence of this invention should fall within the protection scope of this invention.
Claims
1. A method for generating dynamic traffic scenes based on a pre-trained large trajectory model, characterized in that, Includes the following steps: S1: Acquire real traffic data and preprocess the real traffic data to construct a spatio-temporal scene graph representation. The spatio-temporal scene graph includes the topological structure of the scene, static element information, and the initial state and historical trajectory information of multiple dynamic traffic agents. S2: Using the spatiotemporal scene map as a conditional input, a predicted trajectory sequence of the multiple dynamic traffic agents is generated through a pre-trained trajectory model. The predicted trajectory sequence reflects the temporal and spatial interaction relationship between the dynamic traffic agents. S3: Convert the predicted trajectory sequence into high-fidelity dynamic traffic scene data, which can be used for simulation or downstream tasks.
2. The method according to claim 1, characterized in that, The construction of the spatiotemporal scene graph in step S1 includes: extracting static element information such as road topology information, lane boundaries, and traffic light positions from the real traffic data, as well as state information such as the position, speed, and heading angle of dynamic traffic intelligent agents such as vehicles and pedestrians at multiple historical time steps, and encoding the information into a graph structure, where nodes represent intelligent agents or scene elements, and edges represent the spatiotemporal relationships between them.
3. The method according to claim 1, characterized in that, The pre-trained trajectory model is trained by self-supervised or semi-supervised learning on large-scale real traffic trajectory data to learn the behavioral patterns and complex interaction rules of different traffic participants.
4. The method according to claim 3, characterized in that, The pre-trained trajectory model employs Transformer, Graph Neural Network (GNN), Recurrent Neural Network (RNN), or a combination thereof, and is capable of capturing the temporal dependencies of trajectory data and the complex spatiotemporal interaction relationships between multiple agents.
5. The method according to claim 1, characterized in that, The generation of the predicted trajectory sequence in step S2 further includes: controlling the generation process of the trajectory model according to preset scene attributes, such as traffic density, weather conditions, time period or specific event, so as to generate a dynamic traffic scene with specific characteristics.
6. The method according to claim 1, characterized in that, The step S3 of converting the predicted trajectory sequence into high-fidelity dynamic traffic scene data includes: mapping the predicted trajectory sequence onto a three-dimensional simulation environment and rendering a visual scene containing vehicles, pedestrians, traffic facilities and environmental background, wherein the visual scene has a realistic visual effect.
7. The method according to claim 1, characterized in that, The method can generate dynamic traffic scenarios that include interactions between at least two different types of traffic agents (e.g., vehicles, pedestrians, and bicycles).
8. A dynamic traffic scene generation system, characterized in that, include: The data processing module is used to acquire real traffic data and construct a spatiotemporal scene representation; The trajectory generation module is used to generate a predicted trajectory sequence of the multiple dynamic traffic agents using the spatiotemporal scene graph through a pre-trained trajectory model; and the scene rendering module is used to convert the predicted trajectory sequence into high-fidelity dynamic traffic scene data.
9. The system according to claim 8, characterized in that, The pre-trained trajectory model in the trajectory generation module is deployed on a cloud computing platform or a high-performance computing device.
10. The system according to claim 8, characterized in that, The scene rendering module can interface with autonomous driving simulation platforms or traffic flow simulation software and output scene files in a standard format.