Automatic driving simulation scene construction system and method based on 3DGS

Abstract

The invention relates to the technical field of automatic driving simulation testing, and discloses an automatic driving simulation scene construction system and method based on 3DGS, and the system comprises a multi-source data fusion module which is used for preprocessing collected original data and fusing the data into environment perception information; the scene mining module is used for automatically extracting scene elements; recognizing a typical or extreme driving scene according to the scene elements, and converting the typical or extreme driving scene into simulation scene description; the 3D scene generation module is used for jointly constructing a scene through massive three-dimensional Gaussian points to perform three-dimensional scene reconstruction; the visual editing module is used for adjusting the three-dimensional model; the scene generalization module is used for performing transformation and parameterization operation on a single scene to generate scene variants; and the multi-format scene library construction module is used for storing the scenes in multiple formats and organizing the scenes into a scene library. According to the method, the automation degree and diversity of scene construction are effectively improved, the generated scene has higher geometric and illumination fidelity, and the coverage capability of the scene library is further enhanced.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving simulation testing technology, specifically to an autonomous driving simulation scene construction system and method based on 3DGS. Background Technology

[0002] In autonomous driving R&D, simulation testing is a crucial means of verifying system safety and reliability, enabling efficient and low-cost coverage of multiple operating conditions and supporting system iteration. Its value depends not only on massive amounts of diverse scenario data but also on highly realistic 3D scenario models as a foundation. Currently, the scenario data and model construction supporting simulation testing mainly originate from two paths, together forming the basic data and model system for autonomous driving simulation.

[0003] One type is the pre-defined virtual scenario construction mode, where designers explicitly define core elements such as road layout, obstacle distribution, and traffic signals, and build models based on these elements. This type of scenario has a significant efficiency advantage in testing the traffic rule compliance of the system and can specifically verify the system's performance under standard operating conditions. However, due to the limitations of the pre-defined logic, on the one hand, it is difficult to incorporate the random and dynamic elements of the real traffic environment, resulting in deviations between the test scenario and actual road conditions, and failing to fully simulate the interactive behavior in complex traffic flows; on the other hand, the construction of the 3D scene model is highly dependent on human design and debugging, and it takes a lot of time from element definition to model rendering. Not only is the generation speed slow, but manually built models also have limitations in the degree of detail reproduction, making it difficult to achieve a high degree of simulation matching with real roads.

[0004] Another approach is based on scene extraction and modeling from vehicle road test logs. This method extracts scene information from real road test data and reverse-engineers a 3D scene model. This type of scene fully preserves the traffic characteristics of real roads, accurately reflecting various interaction details under complex road conditions. It provides a realistic environment for simulation testing, and models based on real data have a greater advantage in scene realism. However, its scene content is relatively fixed, making it difficult to flexibly adjust and generate according to testing needs. Furthermore, its modeling efficiency is low, failing to meet the demands of rapid iteration testing. It also cannot proactively uncover unknown "corner cases," and these unpredictable extreme scenarios are precisely the key hidden dangers that could trigger system security risks.

[0005] Studies have shown that simulation testing based on real road data is more likely to capture unknown scenarios. However, relying solely on road test logs for modeling cannot overcome data limitations to achieve rapid scenario expansion, nor can it solve the problem of high-efficiency simulation model construction. This dual contradiction puts simulation testing in a dilemma: relying on preset modes cannot cover complex working conditions, and the models are slow and have low simulation accuracy; relying on road test modes makes it difficult to uncover extreme risks, and the modeling efficiency cannot match the testing requirements. Summary of the Invention

[0006] The present invention aims to provide a system and method for constructing autonomous driving simulation scenarios based on 3DGS, so as to solve the problem that existing simulation test scenarios cannot meet the requirements for rapidly constructing highly realistic 3D models.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: an autonomous driving simulation scene construction system based on 3DGS, including a multi-source data fusion module: used to perform time synchronization, filtering and registration of sensor data, high-precision map data and other auxiliary data collected from real vehicles, and fuse them into consistent environmental perception information; Scene mining module: used to automatically extract scene elements from environmental perception information; the scene elements include vehicle trajectory, traffic participant behavior and triggering events; based on the scene elements, typical or extreme driving scenarios are identified and converted into simulation scene descriptions; 3D Scene Generation Module: Based on acquired multi-view images and sensor data, it reconstructs the geometric shape and lighting characteristics of the scene using a large number of parameter-carrying 3D Gaussian points to quickly generate a 3D scene model. Visual editing module: Used to adjust and enrich the generated 3D model through interactive editing; Scene generalization module: used to perform transformation and parameterization operations on a single scene, and automatically generate scene variants; Multi-format scene library building module: Used to store and organize the processed scenes into a scene library in multiple formats.

[0008] Meanwhile, this solution also provides a method for constructing autonomous driving simulation scenarios based on 3DGS, which is applied to the aforementioned 3DGS-based autonomous driving simulation scenario construction system, including the following steps: S1 collects raw environmental data through various sensors and high-precision map equipment installed on the test vehicle, and generates dense environmental point cloud and image data, as well as accurate vehicle positioning information after preprocessing. S2 automatically searches for and identifies key scenarios through the scenario mining module, extracts scenario elements according to the required test objectives, and generates structured scenario descriptions based on scenario requirements. S3 initializes a massive number of 3D Gaussian point lattices carrying parameters to represent the static structure in the scene, and then optimizes and adjusts the parameters of the Gaussian points through backpropagation to render a 3D scene model containing geometric and lighting information. S4: Edit the generated scene model according to the scene requirements to form a complete test environment; S5 performs parameterization on a single scene, generating multiple scene variants by combining different scene element parameters; and saves the processed scenes to the scene library in multiple formats.

[0009] The principles and advantages of this scheme are: First, this solution is based on multi-source data, replacing the traditional model that relies on single data or manual construction, which greatly reduces the need for manual intervention and significantly improves the automation level of scene construction; the fusion characteristics of multi-source data break the limitations of single data sources, effectively enrich the types and forms of scene presentation, and enhance the diversity of the scene library.

[0010] Secondly, a technical approach combining 3D reconstruction and visual editing is adopted. 3D reconstruction technology ensures that the geometric structure of the scene accurately restores the characteristics of the real environment, while visual editing can optimize the details such as lighting and materials. The synergy of the two makes the generated scene have higher fidelity in both geometric shape and lighting effect, and is closer to the real road and traffic environment.

[0011] Third, the built-in scenario generalization function of this solution can generate more new scenario types based on existing data and scenario templates, especially covering complex or extreme working conditions that are difficult to reach by traditional methods. It effectively breaks through the limitations of fixed scenarios, further enhances the coverage of the scenario library, and provides more comprehensive support for simulation testing.

[0012] Fourth, the multi-format scene library built in this solution supports the format requirements of various mainstream simulation platforms, solving the problem of scene data not being interoperable between different platforms. This not only ensures good compatibility of scenes between different simulation platforms, but also enables efficient reuse of scene resources, reducing the cost of scene development and migration. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the structure of the autonomous driving simulation scene construction system based on 3DGS according to the present invention; Figure 2 This is a flowchart illustrating the autonomous driving simulation scene construction method based on 3DGS of the present invention. Detailed Implementation

[0014] The following detailed description illustrates the specific implementation method: The 3DGS-based autonomous driving simulation scene construction system and method in this embodiment can integrate information from multiple sources such as real vehicle data collection, maps and simulation data, automatically mine safety-critical scenes from road test logs, and quickly generate high-fidelity virtual scenes using advanced 3D reconstruction and visualization editing technologies. At the same time, it can parameterize and generalize individual scenes to form diverse scenes.

[0015] Option 1 A 3DGS-based autonomous driving simulation scene construction system is provided, as shown in the attached document. Figure 1 As shown, including, Multi-source data fusion module: used to perform time synchronization, filtering and registration of sensor data, high-precision map data and other auxiliary data collected from the actual vehicle, and fuse them into consistent environmental perception information.

[0016] In this embodiment, the system first collects raw environmental data using various sensors (such as LiDAR, cameras, millimeter-wave radar, GPS / IMU, etc.) and high-precision map equipment mounted on the test vehicle. The multi-source data fusion module is responsible for collecting and preprocessing multi-source data from the vehicle's sensors and maps. Sensor data includes LiDAR point clouds, camera images, millimeter-wave radar, GPS / IMU, etc., while other auxiliary data includes map data. By synchronizing and fusing the multi-source data in time, including calibrating the extrinsic parameters of each sensor, performing coordinate transformation, and data filtering, the system ensures close registration and consistency of the data sources.

[0017] In this embodiment, the specific data fusion implementation process includes: 1. Data Preprocessing Stage: The raw data from each sensor is preprocessed, including LiDAR point cloud denoising (using a combination of voxel filtering and statistical filtering to remove outliers), camera image distortion correction (based on Zhang's calibration method), millimeter-wave radar target clustering (using the DBSCAN algorithm), and GPS / IMU data smoothing (Kalman filtering preprocessing). Adaptive filtering strategies are employed for different sensor characteristics; for example, LiDAR uses intensity threshold filtering for ground points, and the camera uses the Retinex algorithm to enhance images in backlit scenes.

[0018] 2. Time Synchronization Phase: Sensor time synchronization is achieved through hardware triggering (IEEE 1588 PTP protocol), with timestamp accuracy controlled within ±0.1ms. Spatial synchronization uses a coordinate transformation matrix to unify the coordinate system of multiple sensors. Specifically, sensor extrinsic parameter calibration is completed through a combination of offline and online calibration: In the offline phase, the initial extrinsic parameters are obtained using a checkerboard-based camera calibration method and a LiDAR-camera joint calibration algorithm; in the online phase, extrinsic parameter drift is optimized in real time through SLAM, and the ICP algorithm is used to achieve inter-frame matching between the LiDAR and the high-precision map.

[0019] 3. Multimodal fusion algorithm: A hierarchical fusion architecture is adopted. At the data layer, the original data is stitched together by aligning timestamps. At the feature layer, an attention-based feature fusion network (such as FusionNet) is used to extract cross-modal features. At the decision layer, an improved DS evidence theory is used to process the target detection results from heterogeneous sensors. For dynamic targets, a multi-sensor tracking and association algorithm (such as JPDAF) is used, while for static environments, Bayesian estimation is used to achieve map fusion and updating.

[0020] 4. Quality Verification and Optimization: Establish a multi-dimensional fusion quality evaluation index system, including point cloud density uniformity (coefficient of variation < 5%), target detection accuracy (mAP > 95%), and positioning accuracy (RTK-GPS contrast error < 0.5m). Accumulated errors are eliminated through closed-loop detection and loop closure optimization (BA algorithm based on graph optimization), ultimately generating dense environmental point cloud and image data, as well as accurate vehicle positioning information, ensuring close registration and consistency across data sources.

[0021] Scene mining module: used to automatically extract scene elements from environmental perception information; scene elements include vehicle trajectory, traffic participant behavior and triggering events; based on scene elements, typical or extreme driving scenarios are identified and transformed into simulation scene descriptions.

[0022] In this embodiment, the scene mining module employs Log2World technology (log scene mining). After obtaining processed real driving data, it automatically searches for and identifies key scenes from a large amount of fused real road log data. Key features are extracted to meet scene mining needs, including vehicle motion features (speed, acceleration, yaw rate, etc.), traffic participant interaction features (relative distance, relative speed, TTC / THW safety indicators, etc.), and environmental state features (weather code, light intensity, road surface type, etc.). Scene elements are extracted to identify typical or extreme driving scenarios, and the real vehicle log data is converted into a simulation scene description. The simulation scene description includes information such as road geometry, vehicle status, and obstacle configuration.

[0023] To ensure the accuracy of the mining, in this embodiment, the module has built-in quantifiable judgment criteria, that is, the scene mining module includes a typical scene definition unit and an extreme scene definition unit.

[0024] Among them, the typical scenario definition unit identifies common driving conditions with statistical representativeness, such as "urban congestion following" and "highway stable cruising", by performing cluster analysis on massive data (such as K-means algorithm) based on parameters such as traffic flow density, average vehicle speed, and road type.

[0025] Specifically, typical scenario definition achieves standardized scenario classification through multi-dimensional feature analysis and clustering algorithms. By constructing dynamic feature sets (such as following distance standard deviation and lane change frequency) and static feature sets (such as road curvature and speed limit information), key features are selected through mutual information entropy (retaining the top 15 features with a cumulative contribution of >90%). An improved K-means++ algorithm is used, and the optimal number of clusters (usually 8-12) is determined through silhouette score and Davies-Bouldin index. Subclassing is then performed using hierarchical clustering (Agglomerative clustering).

[0026] The clustering results are manually labeled and semantically mapped to form typical scenario libraries such as "urban congestion following" (features: vehicle speed < 20 km / h, following distance < 50 m, traffic flow density > 30 vehicles / km) and "highway stable cruise" (features: vehicle speed 90-110 km / h, lane keeping rate > 95%, no emergency acceleration or deceleration), and a scenario feature vector index (dimensional 20) is established.

[0027] The extreme scenario definition unit makes judgments based on specific physical quantity thresholds. For example, when the time-to-collision (TTC) is less than 1.5 seconds, the post-encroachment time (PET) is less than 1 second, or the vehicle's lateral / longitudinal acceleration / deceleration or impact exceeds a preset safety threshold, the system automatically captures the scene segment and classifies it as an extreme or dangerous scenario.

[0028] Specifically, the extreme scenario definition unit achieves accurate identification through a multi-parameter fusion threshold system and a dynamic event triggering mechanism. For example, based on industry safety standards (such as ISO 21448) and accident data analysis, a three-level threshold model is established: Level 1 warning threshold (TTC=2.5s, PET=1.5s), Level 2 danger threshold (TTC=1.5s, PET=1.0s), and Level 3 emergency threshold (TTC<1.0s, longitudinal deceleration>8m / s²). Lateral parameters include steering wheel angular rate>120° / s and lateral acceleration>0.8g. The system integrates lidar point cloud data (target distance accuracy ±0.1m), millimeter-wave radar data (velocity measurement error <0.5km / h), and visual recognition results (target classification accuracy>98%), employing DS evidence theory for data credibility weighting (lidar weight 0.6, vision weight 0.3, millimeter-wave weight 0.1).

[0029] Once the threshold condition is triggered, a complete data segment from 5 seconds before to 3 seconds after the event is automatically extracted, including the vehicle's status (position, speed, acceleration), the trajectory of traffic participants (sampling frequency 100Hz), and environmental parameters (weather, light intensity), to extract scene segments. The scene is then classified in two ways: based on risk level (L1-L5) and scene rarity (based on frequency per million kilometers), such as "sudden entry of a high-speed vehicle" (L4 level, frequency <0.1 times / 10,000 kilometers) and "pedestrian suddenly appearing" (L5 level, frequency <0.01 times / 10,000 kilometers). Structured scene metadata containing the trigger time, duration, and scope of impact is generated.

[0030] 3D Scene Generation Module: Based on acquired multi-view images and sensor data, it reconstructs the geometric shape and lighting characteristics of the scene using a large number of parameter-carrying 3D Gaussian points to quickly generate a 3D scene model.

[0031] In this embodiment, the 3D scene generation module employs 3D Gaussian scene reconstruction (3DGS) technology for 3D reconstruction. This module utilizes Gaussian point clouds to represent the geometry and lighting characteristics of a static scene, quickly generating a 3D model of the scene. To ensure high fidelity, this module adopts a representation method centered on 3D Gaussian "splats," using a massive set of 3D Gaussian points carrying parameters. Together, they construct the geometry and lighting characteristics of the scene, ultimately achieving high-fidelity 3D model generation and photorealistic rendering effects. Each 3D Gaussian point is the basic unit carrying scene information, and its parameter system fully covers the two core dimensions of geometry and lighting. Through the synergistic effect of the parameters, scene details are accurately reproduced.

[0032] Traditional 3D scene generation often uses a "mesh + texture" approach. This method requires first constructing a polygonal mesh structure for the object, then adding texture maps to supplement the appearance details. Lighting effects rely on complex lighting models calculated in the rendering engine. This solution completely abandons the dependence on mesh structures. It uses discrete Gaussian points as the basic unit, directly encapsulating the geometric shape and lighting characteristics within the parameters of each Gaussian point. Without additional textures and complex lighting models, it can achieve more refined geometric fitting and more realistic lighting simulation, while also having the advantage of rapid scene generation.

[0033] In this embodiment, the parameters of a 3D Gaussian point include geometric parameters and lighting characteristics. That is, each Gaussian point contains a set of parameters that collectively represent the geometric and lighting characteristics of the scene. These two types of parameters together constitute the complete features of the Gaussian point. Therefore, the core parameter relationship can be expressed as follows: (1) In the formula, G is a single complete three-dimensional Gaussian point, which is the basic information unit for constructing the scene and integrates all geometric and lighting features; The set of geometric characteristic parameters of a Gaussian point determines its shape, position, and size in space; This is the set of illumination characteristic parameters for Gaussian points, which determines the color, transparency, and illumination response characteristics of Gaussian points.

[0034] The geometry of a scene, i.e., the geometry of a Gaussian point, is determined by the spatial position (x, y, z) of the 3D Gaussian point and its 3×3 covariance matrix. The position parameter determines the point's center in space, while the covariance matrix defines the shape, size, and orientation of its 3D ellipsoid. Through their combined effect, the spatial state of the 3D ellipsoid is defined. A vast number of tiny ellipsoids are used to fit the fine geometric surfaces of complex objects such as roads, buildings, and vegetation, thus fitting the scene's object surfaces. This can be represented as follows: , (2) In the formula, Given the spatial coordinates (x, y, z), define the center position of the ellipsoid corresponding to the Gaussian point in space; It is a 3×3 covariance matrix, specifically ,in, For rotation matrix, The scale matrix has diagonal elements corresponding to the scaling scales of the three principal axes of the Gaussian ellipsoid. It is used to define the shape (such as sphere, ellipsoid), size (volume scale), and spatial orientation of the ellipsoid, and to fit the fine surface of scene objects by using a large number of ellipsoids.

[0035] Based on these geometric parameters, the distribution of ellipsoids formed by Gaussian points in three-dimensional space can be described by a Gaussian function, and its probability density distribution, i.e., the distribution of Gaussian points in three-dimensional space, can be expressed as: (3) In the formula, The coordinates of any point in three-dimensional space are used to describe the distribution range of the Gaussian ellipsoid in space.

[0036] The lighting characteristics of a Gaussian point are represented by its color (R, G, B), opacity (Alpha), and spherical harmonics coefficients. Color and opacity directly determine the point's basic appearance and transparency. The spherical harmonics coefficients simulate view-dependent lighting effects. Using these coefficients, each point can simulate view-dependent lighting effects such as material reflections, ambient occlusion, and shadows. This allows for photorealistic rendering without the complex lighting models of traditional rendering engines, highly replicating the physical properties of real-world lighting. The relationship between the overall set of lighting parameters and the rendering effect can then be expressed as follows: ; (4) In the formula, Define the basic color attributes of the Gaussian point for the color values ​​(R,G,B) of the lighting parameters; The opacity parameter has a value range of [0,1] and determines the transparency of the Gaussian point (0 is completely transparent, and 1 is completely opaque). These are the coefficients of the spherical harmonic function, used to encode lighting information so that Gaussian points can simulate view-dependent lighting effects (such as reflections and shadows). The observation angle parameter reflects the spatial angle of the observer relative to the Gaussian point. The spherical harmonic function coefficients can dynamically adjust the illumination output according to V. The illumination output value of a single Gaussian point at a viewing angle V is a result of the combined effects of color, opacity, and spherical harmonic function.

[0037] A massive number of 3D Gaussian points are spatially distributed to fit the geometric surface of the scene. The lighting effects of each point are superimposed to achieve the rendering of the overall scene. The final pixel color of the scene is the sum of the lighting contributions of all Gaussian points within the field of view. In other words, the scene model is represented as follows: (5) In the formula, The final color value of the target pixel in the scene under the viewpoint V is the sum of the contributions of all relevant Gaussian points; N is the total number of three-dimensional Gaussian points in the field of view that contribute to the target pixel, to reflect the richness of scene details. The coordinates of the target pixel in three-dimensional space are used to calculate the contribution weight of each Gaussian point to the pixel.

[0038] By using the above three-dimensional Gaussian point-based generation method, in terms of geometric representation, an ellipsoid defined by the covariance matrix replaces the traditional mesh, which can more accurately fit the surfaces of complex objects such as roads and buildings; in terms of lighting simulation, by using spherical harmonic function coefficients, it gets rid of the dependence on complex rendering engines and directly restores the real lighting characteristics such as material reflection and ambient light occlusion, ultimately achieving the goal of quickly generating high-fidelity 3D scenes.

[0039] In this embodiment, when reconstructing the 3D scene model, the loss function between the rendered output value and the original image pixel value is also minimized. The backpropagation algorithm is used to iteratively optimize the parameters of the 3D Gaussian points to update the 3D Gaussian point set. The loss function is then expressed as: ; (6) In the formula, The mean absolute error between the rendered image and the original image; These are preset weighting coefficients; The structural similarity loss is used to constrain the geometric integrity and viewpoint lighting consistency of static structures in the reconstructed scene.

[0040] By calculating the gradient And update the set of Gaussian points along the negative gradient direction. ,in Including spatial location Covariance matrix Opacity parameter spherical harmonic coefficients .

[0041] Visual editing module: Used to adjust and enrich the generated 3D model through interactive editing.

[0042] In this embodiment, the visualization editing module provides an interactive editing interface and automated tools to adjust and enrich the generated 3D scene. Users or algorithms can add or modify static elements (road signs, traffic signals, buildings, vegetation, etc.) and dynamic elements (traffic flow, pedestrians, etc.) in the scene, as well as configure environmental parameters such as lighting and weather, to improve the visualization effect and diversity of the scene.

[0043] Scene generalization module: used to transform and parameterize a single scene, and automatically generate scene variants.

[0044] In this embodiment, parameterization operations include adjusting road layout, changing traffic density, randomly moving obstacles, and modifying traffic rules to obtain a wider range of scene distributions and higher test coverage.

[0045] Multi-format scene library building module: Used to store and organize the processed scenes into a scene library in multiple formats.

[0046] In this embodiment, the processed scenes are stored in multiple formats and organized into a scene library, supporting cross-platform use. The scene library includes scene data (such as 3DGS format, WorldSim format, OpenDRIVE format, etc.) and auxiliary resources (such as map data) for different simulation platforms. By combining various scene elements, almost unlimited scenes can be constructed, providing a rich variety of scene inputs for downstream simulation testing.

[0047] Option 2 A method for constructing autonomous driving simulation scenes based on 3DGS is provided, which is applied to the aforementioned 3DGS-based autonomous driving simulation scene construction system, as shown in the attached figure. Figure 2 As shown, it includes the following steps: S1 collects raw environmental data through various sensors and high-precision map equipment installed on the test vehicle. After preprocessing, it generates dense environmental point clouds and image data, as well as accurate vehicle positioning information.

[0048] In this embodiment, the system first collects raw environmental data using various sensors (such as LiDAR, cameras, millimeter-wave radar, GPS / IMU, etc.) and high-precision map equipment mounted on the test vehicle. Then, the raw environmental data is preprocessed, including synchronizing and fusing the multi-sensor data according to timestamps, removing noise and redundant information, and fusing them into consistent environmental perception information.

[0049] In addition, this step may also include technical processing such as calibrating the extrinsic parameters of each sensor, performing coordinate transformation and data filtering, to ensure close registration and consistency of each data source.

[0050] S2 automatically searches for and identifies key scenarios through the scenario mining module, extracts scenario elements according to the required test objectives, and generates structured scenario descriptions based on scenario requirements.

[0051] In this embodiment, after obtaining processed real-world driving data, the scene mining module automatically searches for and identifies key scenes from a large amount of road logs. These scenes can be moments when the autonomous driving system takes over, collision warnings occur, or other safety-critical events take place. The module extracts scene parameters based on the required test objectives (such as traffic conditions, vehicle interaction patterns, etc.). For specific (typical and extreme) scene requirements, a structured scene description file (such as OpenSCENARIO format) is generated to meet simulation needs. In this embodiment, the scene description file records information including the static environment, dynamic objects, triggering conditions and events, and environmental parameters.

[0052] The key scenario detection and classification employs a dual-engine detection mechanism of "rules + learning." The rule engine filters potential risk scenarios based on preset safety thresholds (such as TTC < 2s, collision warning activation, system takeover request, etc.); the learning engine automatically identifies implicit dangerous scenarios without explicit triggering conditions (such as pedestrians suddenly appearing from behind obstacles, slow traffic in construction areas, etc.) by training a Transformer-based event classification model (input is a 10s time-series feature sequence, output is scenario category probability). Scenario classification uses a multi-level labeling system, with the first-level classification including 6 major categories such as "vehicle interaction," "vulnerable road users," and "road anomalies," and the second-level classification further refined into 32 typical scenario types.

[0053] For the detected key scenarios, standardized scenario parameters are generated. The static environment uses high-precision map data (such as OpenDRIVE files), automatically parsing the road topology, defining road geometry, lane lines, traffic signs, etc., and extracting geometric parameters such as the number of lanes, radius of curvature, and slope. The dynamic objects accurately describe the initial state (position, velocity, acceleration) and time-varying complete trajectory of all traffic participants (vehicles, pedestrians, etc.) in the scenario. Kalman filtering is used to smooth the traffic participant trajectories, and the RANSAC algorithm is used to remove trajectory noise, generating spatiotemporal sequence data (sampling frequency 10Hz) containing position (WGS84 coordinate system), velocity, and acceleration. Triggering conditions and events explicitly record key indicators for identifying the scenario (such as a cut-in event with TTC=1.2s) and define the start and end times of the scenario. Environmental parameters record information such as weather and lighting (e.g., noon, dusk) to provide a basis for high-fidelity reconstruction.

[0054] The scenario compiler converts parameterized data into multi-format scenario files. The core outputs include: OpenSCENARIO format (for dynamic behavior description), OpenDRIVE format (for road network definition), and JSON format metadata (containing scenario ID, risk level, data source, etc.).

[0055] Using the above methods, real driving situations can be accurately replayed in a closed-loop environment.

[0056] S3 initializes a massive number of 3D Gaussian point lattices carrying parameters to represent the static structure in the scene, and then optimizes and adjusts the parameters of the Gaussian points through backpropagation to render a 3D scene model containing geometric and lighting information.

[0057] In this embodiment, for complex scenes or environments requiring high realism, the 3DGS module uses camera and radar data for 3D reconstruction.

[0058] First, the camera pose is recovered and a sparse point cloud is generated using visual measurement technology (SfM). In this embodiment, SuperPoint is used to extract image feature points (≥2000 feature points per frame), and a FLANN matcher is used to achieve cross-frame feature association (matching accuracy >95%). The camera pose and 3D point coordinates are solved using Ceres Solver, with reprojection error controlled within 1.0 pixel. Triangulation calculations are performed on the matched feature points to generate a sparse point cloud containing ≥1 million points, with a point cloud density ≥50 points / m². 2 .

[0059] Then, based on the sparse point cloud distribution, a large number of 3D Gaussian point ellipsoids carrying position, covariance, color, opacity, and spherical harmonic coefficients are stacked to represent the static structure in the scene. An iterative optimization strategy can be used to adjust the Gaussian point parameters, and a virtual view can be generated using a differentiable renderer (such as 3DGS-Renderer). This involves projecting the 3D Gaussian ellipsoids onto a two-dimensional plane after affine transformation and calculating the photometric loss (MSE < 10) compared to the real image. -3 ) and structural loss (SSIM > 0.95).

[0060] The parameters of the Gaussian points are then continuously adjusted using optimization algorithms such as backpropagation. For example, the Adam optimizer (learning rate 1e-4) iteratively optimizes the position, covariance, color, and opacity parameters. Every 100 iterations, Gaussian points are split (when the eigenvalue ratio of the covariance matrix is ​​>3) and pruned (removed when the opacity is <0.01), gradually improving the detail reproduction. This results in a realistic rendering of environmental details such as roads, buildings, and road surface textures. The output is a 3D scene model containing rich geometric and lighting information, which can be used for subsequent simulations and rendering.

[0061] S4 allows you to edit the generated scene model according to the scene requirements, forming a complete test environment.

[0062] In this embodiment, the visual editor (WorldSim editor) allows users to edit or add / remove static elements, dynamic elements, and environmental parameters of the generated scene model to enhance visual effects and diversity. Users can add static objects to the scene, set lighting and weather conditions, and arrange dynamic traffic flow.

[0063] Static elements include road signs, traffic signals, buildings, and vegetation. Dynamic elements include traffic flow and pedestrians. Environmental parameters include lighting and weather (such as sunshine, rain, snow, daytime, and nighttime).

[0064] In this embodiment, additional vehicles or pedestrian agents can be inserted according to testing requirements. This enriches the original reconstructed scene into a complete test environment usable in simulation. The editing process supports visual interaction and can automatically generate scene variations that comply with traffic rules using a rule engine, ensuring the effectiveness and safety of the scene.

[0065] S5 performs parameterization on a single scene, generating multiple scene variants by combining different scene element parameters; and saves the processed scenes to the scene library in multiple formats.

[0066] In this embodiment, the scenario is generalized to generate more variations. Parametric processing includes adjusting road layout, changing traffic density, randomly moving obstacles, and modifying traffic rules. The system can automatically process these variations, such as adjusting road connections, changing vehicle density, moving obstacle positions, or altering vehicle behavior patterns (e.g., acceleration, lane-changing strategies), to construct a series of "extreme" or "boundary" scenarios. This generalization further expands the scenario library's coverage, enhancing the robustness testing of autonomous driving algorithms. By combining different scenario element parameters, a near-infinite number of scenarios can be constructed, including more extreme cases and safety boundary scenarios, improving the reliability of simulation testing.

[0067] The system then saves the scenes, after being mined, reconstructed, edited, and generalized, into scene libraries in multiple formats to support the needs of different simulation platforms. For example, the 3DGS scene library retains high-fidelity raster representations; the WorldSim scene library stores executable scene files for direct use by the simulation editor; the Log2World scene library stores scene configurations based on actual logs; and a map library composed of high-definition map data is also maintained. Based on this, the scene library provides rich resources for downstream simulation testing modules and can output scene information to different simulation engines (such as Unreal Engine, Unity, CARLA, etc.) through a unified interface, ensuring cross-platform usability.

[0068] By constructing high-fidelity scenarios, it can be applied to test the robustness of perception algorithms such as cameras and LiDAR in complex lighting conditions (e.g., sunrise / sunset glare, tunnel entrances / exits), severe weather conditions (e.g., road surface glare in rainy weather), and complex texture environments, thus validating the perception algorithms. Furthermore, by comparing the output of virtual sensors in the simulation with real-world data at the pixel level or point cloud level, sensor model parameters can be precisely calibrated and optimized to achieve sensor simulation model calibration. Additionally, in highly realistic hazardous scenarios (e.g., sudden pedestrian appearances or emergency braking by the vehicle in front), it can test the extreme reaction capabilities of planning and control algorithms, or provide drivers with an immersive visual experience for research on autonomous driving system takeover, human-machine interaction, and other issues, realizing human-machine co-driving and driver-in-the-loop simulation.

[0069] In this embodiment, a closed-loop generation process from multi-source data to high-quality simulation scenes is achieved, effectively improving the automation and diversity of scene construction. By combining 3D reconstruction and visual editing, the generated scenes have higher geometric and lighting fidelity. The scene generalization function further enhances the coverage of the scene library. The multi-format library ensures the compatibility and reusability of scenes across different simulation platforms.

[0070] The implementation of this solution also overcame many challenging limiting factors and obstacles. First, the stringent requirements of 3DGS models for high-performance GPUs and large amounts of video memory constitute a computational power challenge from the outset. However, this solution did not rely on hardware upgrades. Instead, it explored the potential of computational power within a reasonable hardware configuration through model optimization, task splitting, and innovative parallel scheduling. After hundreds of algorithm iterations, it overcame the challenge of matching computational power with application requirements.

[0071] Secondly, the quality of the raw data directly determines the fidelity of the reconstruction, but the collected data often suffers from problems such as blurriness and missing perspectives. This solution constructs a multi-dimensional data preprocessing system, introducing image enhancement, perspective completion, and sensor fusion correction technologies. Combined with pre-control of data quality and optimization of the model, it achieves a stable transformation from non-ideal data to high-fidelity scenes.

[0072] The fusion of static and dynamic elements is the core barrier, as 3DGS excels at static reconstruction but struggles with dynamic objects. This solution creatively designs a "dynamic-static collaborative rendering" mechanism. After extracting trajectories using Log2World, it constructs a real-time lighting model for dynamic objects, allowing the lighting effects to blend naturally with the static background. Multiple scene tests have been conducted to avoid fusion defects.

[0073] Furthermore, the difficulty of geometric editing in 3DGS scenes once limited the practicality of the solution. This solution addresses this by developing and designing a "Gaussian point cloud and lightweight mesh hybrid editing interface," enabling indirect geometric adjustments based on mesh interaction. This enhances editing capabilities while retaining high fidelity, and its ease of use and reliability have been ensured through multi-tool compatibility testing. These technological breakthroughs and innovations have facilitated the implementation and realization of this solution.

[0074] The above descriptions are merely embodiments of the present invention, and common knowledge such as specific technical solutions and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solutions of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A 3DGS-based autonomous driving simulation scene construction system, characterized in that, include, Multi-source data fusion module: used to perform time synchronization, filtering and registration of sensor data, high-precision map data and other auxiliary data collected by the actual vehicle, and fuse them into consistent environmental perception information; Scene mining module: used to automatically extract scene elements from environmental perception information; the scene elements include vehicle trajectory, traffic participant behavior and triggering events; Identify typical or extreme driving scenarios based on scene elements and convert them into simulation scene descriptions; 3D Scene Generation Module: Based on acquired multi-view images and sensor data, it reconstructs the 3D scene by constructing the geometric shape and lighting characteristics of the scene through a massive set of 3D Gaussian points carrying parameters, and quickly generates a 3D scene model; it optimizes the parameters of the 3D Gaussian points through backpropagation to reconstruct a 3D scene model containing geometric shape and view-related lighting characteristics. Visual editing module: Used to adjust and enrich the generated 3D model through interactive editing; Scene generalization module: used to perform transformation and parameterization operations on a single scene, and automatically generate scene variants; Multi-format scene library building module: Used to store and organize the processed scenes into a scene library in multiple formats.

2. The autonomous driving simulation scene construction system based on 3DGS according to claim 1, characterized in that: The parameters of the three-dimensional Gaussian points include geometric parameters and lighting characteristics; the geometry of the scene is determined by its spatial location and covariance matrix, and fitted to the surface of the scene objects, the distribution of Gaussian points in three-dimensional space is represented as follows: ； In the formula, Let be the coordinates of any point in three-dimensional space; Spatial location coordinates; It is a 3×3 covariance matrix. , For rotation matrix, The scale matrix, with its diagonal elements corresponding to the scaling scales of the three principal axes of the Gaussian ellipsoid, is used to define the shape, size, and orientation of the 3D ellipsoid; it fits the fine surface of scene objects using a large number of ellipsoids.

3. The autonomous driving simulation scene construction system based on 3DGS according to claim 2, characterized in that: The lighting characteristics of a Gaussian point are represented by color, opacity, and spherical harmonic function coefficients. The spherical harmonic function coefficients are used to simulate view-dependent lighting effects. Therefore, the relationship between the overall lighting parameter set and the rendering effect is as follows: ； In the formula, The color value is a parameter of the lighting. This is the opacity parameter; These are the coefficients of the spherical harmonic function; These are the parameters for the observation viewpoint; This represents the illumination output value of a single Gaussian point at a viewing angle V.

4. The autonomous driving simulation scene construction system based on 3DGS according to claim 3, characterized in that: If a scene is rendered by overlaying lighting effects on a massive number of 3D Gaussian points, then the scene model is represented as follows: ； In the formula, N is the total number of three-dimensional Gaussian points within the field of view that contribute to the target pixel; The coordinates of the target pixel in three-dimensional space.

5. The autonomous driving simulation scene construction system based on 3DGS according to claim 3, characterized in that: When reconstructing a 3D scene model, the loss function between the rendered output value and the original image pixel value is minimized. The parameters of the three-dimensional Gaussian points are iteratively optimized using the backpropagation algorithm; the loss function is expressed as: ； In the formula, The mean absolute error between the rendered image and the original image; These are preset weighting coefficients; The structural similarity loss is used to constrain the geometric integrity and viewpoint lighting consistency of static structures in the reconstructed scene. By calculating the gradient And update the set of Gaussian points along the negative gradient direction. ,in Including spatial location Covariance matrix Opacity spherical harmonic coefficients .

6. The autonomous driving simulation scene construction system based on 3DGS according to claim 1, characterized in that: The scene mining module includes a typical scene definition unit and an extreme scene definition unit; the typical scene definition unit identifies representative common driving conditions based on traffic flow density, average vehicle speed, and road type; the extreme scene definition unit makes judgments based on set physical quantity thresholds.

7. A method for constructing autonomous driving simulation scenarios based on 3DGS, characterized in that, The 3DGS-based autonomous driving simulation scene construction system applied to any one of claims 1-6 includes the following steps: S1 collects raw environmental data through various sensors and high-precision map equipment installed on the test vehicle, and generates dense environmental point cloud and image data, as well as accurate vehicle positioning information after preprocessing. S2 automatically searches for and identifies key scenarios through the scenario mining module, extracts scenario elements according to the required test objectives, and generates structured scenario descriptions based on scenario requirements. S3 initializes a massive number of 3D Gaussian point lattices carrying parameters to represent the static structure in the scene, and then optimizes and adjusts the parameters of the Gaussian points through backpropagation to render a 3D scene model containing geometric and lighting information. S4: Edit the generated scene model according to the scene requirements to form a complete test environment; S5 performs parameterization on a single scene, generating multiple scene variants by combining different scene element parameters; and saves the processed scenes to the scene library in multiple formats.

8. The method for constructing an autonomous driving simulation scene based on 3DGS according to claim 7, characterized in that: In S2, scene descriptions include static environment, dynamic objects, triggering conditions and events, and environment parameters.

9. The method for constructing an autonomous driving simulation scene based on 3DGS according to claim 7, characterized in that: In S4, the generated scene model can be edited or have its static elements, dynamic elements, and environmental parameters added or removed; the static elements include road signs, traffic signals, buildings, and vegetation; the dynamic elements include traffic flow and pedestrians; and the environmental parameters include lighting and weather.

10. The method for constructing an autonomous driving simulation scene based on 3DGS according to claim 7, characterized in that: In S5, parametric processing includes adjusting road layout, changing traffic density, randomly moving obstacles, and modifying traffic rules.

Images