A prompt word driven method for generating and reconstructing dangerous scenarios

Abstract

The application discloses a dangerous scene generation and reconstruction method based on prompt word driving, which comprises the following steps: firstly, receiving a conventional driving scene data, and separating the scene information into a static background feature, a dynamic target feature and an original motion trajectory through a front-end feature decoupling module; then, combining a natural language prompt word input by a user, reconstructing a dangerous behavior of the target motion trajectory, and generating a generative repair for a background missing area caused by the target movement; finally, constructing a three-dimensional Gaussian representation of the background and the dynamic target respectively by using a 3D Gaussian sputtering technology, and generating a high-fidelity three-dimensional scene image conforming to the target dangerous behavior through a differentiable rendering. The application solves the problems that existing reconstructed scene entities are highly coupled in space, a forced change of a motion trajectory of a scene entity leads to serious texture missing and artifacts, and a traditional method is difficult to generate diversified and natural dynamics-compliant complex dangerous behaviors.

Description

Technical Field

[0001] This invention belongs to the field of safe driving simulation testing technology, specifically, it relates to a method for generating and reconstructing dangerous scenarios based on prompt words. Background Technology

[0002] Generating hazardous scenarios for autonomous driving relies on highly realistic and editable 3D environment modeling techniques. While traditional scene construction methods based on game engines (such as CALA, Unreal Engine, and Unity) can achieve large-scale environment simulation, they fall short in terms of visual realism and the representation of complex dynamic environments. In recent years, the development of neural rendering technologies such as Neural Radiation Field (NeRF) and 3D Gaussian Splatting (3DGS) has provided new solutions for recovering photorealistic 3D scenes from realistically acquired images. NeRF represents scenes using implicit neural networks and combines this with threeiable volume rendering to achieve high-quality reconstruction; methods such as Block-NeRF and Mega-NeRF further expand their application capabilities in urban-level autonomous driving scenarios.

[0003] However, autonomous driving environments not only include static backgrounds such as roads and buildings, but also complex dynamic targets such as vehicles and pedestrians. Existing dynamic NeRF methods achieve temporal modeling through deformation fields, but struggle to accurately describe the motion relationships of multiple objects in a driving scene. Neural Scene Graphs (NSG) utilizes 3D bounding box priors to decompose the scene into static backgrounds and dynamic targets, achieving object-based independent modeling; Panoptic Neural Fields (PNF) further integrates semantic information, improving scene expressiveness. However, because NeRF relies on a large number of neural network queries and volume rendering computations, its training and inference efficiency is low, making it difficult to meet the real-time simulation requirements of autonomous driving.

[0004] To address the unique characteristics of driving scenarios, Neural Scene Graphs (NSG) proposes a graph-based decoupled representation method. It utilizes 3D bounding boxes as priors to explicitly decompose the scene into static background nodes and multiple dynamic object nodes, each represented by an independent NeRF. This decomposition mechanism allows for rigid body transformations of the vehicle during rendering, providing preliminary scene editing capabilities. Panoptic Neural Fields (PNF) further integrates panoramic segmentation semantic information, using a two-stream MLP structure to encode objects and background separately, improving semantic consistency. While these methods address the static-dynamic separation problem to some extent, the query and integration processes of implicit MLPs are computationally intensive, resulting in inference speeds often below 1 FPS, which is insufficient to meet the real-time requirements of autonomous driving closed-loop simulation.

[0005] To improve the efficiency of neural scene modeling, 3D Gaussian Splatting replaces traditional implicit networks with explicit Gaussian representations. It achieves rapid training and real-time rendering through efficient rasterization and has been further applied to dynamic scene reconstruction for autonomous driving. Methods such as DrivingGaussian and Street Gaussians combine dynamic target modeling, object decoupling, and normalized space representation to achieve high-fidelity representations of vehicle motion and appearance changes.

[0006] Existing NeRF and 3DGS methods are still mainly geared towards scene reconstruction or fixed trajectory playback. In the process of generating dangerous scenes, they still face problems such as high coupling of scene entities, texture loss after target motion editing, and difficulty in generating complex dangerous behaviors naturally. Summary of the Invention

[0007] To address the problems of existing methods for reconstructing scene entities, such as high spatial coupling, forced changes to the motion trajectory of scene entities leading to severe texture loss and artifacts, and the difficulty of generating diverse and complex dangerous behaviors that conform to natural dynamics, this invention provides a cue word-driven method for generating and reconstructing dangerous scenes.

[0008] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows: A method for generating and reconstructing dangerous scenes based on cue words includes the following steps: S1. Obtain data for typical driving scenarios; S2. The feature receiving module receives regular driving scenario data and decouples and separates the regular driving scenario data into a set of latent variables. The set of latent variables includes static background features, dynamic object features, and the original motion trajectory of dynamic objects in the real world. S3. Send the set of latent variables into the prompt word-driven generation module, introduce natural language prompt words issued by the user or test system, guide the trajectory diffusion model to perform dynamic resampling of the original motion trajectory, and output a dangerous trajectory that conforms to the semantics. S4. To address the geometric and textural occlusion holes in the static background caused during the generation of dangerous trajectories, a spatiotemporally consistent background restoration model is introduced to complete the background and output the restored background features. S5. The repaired background features, dangerous trajectories and dynamic object features are sent to the dual-layer 3D Gaussian physical reconstruction module. The scene is instantiated by 3D Gaussian sputtering, and two independent sets of three-dimensional Gaussian primitives are constructed, namely the background 3D Gaussian set and the object 3D Gaussian set. Then, the micro-rasterization renderer performs spatial projection and blending calculations. S6, outputs the final high-fidelity dangerous scene footage.

[0009] Furthermore, the output data of the feature receiving module in step S2 consists of three independent feature subsets, including the pure static background feature map after removing dynamic objects, the geometric and appearance feature set of K independent dynamic objects in the scene, and the original motion trajectory of the object in the real world containing temporal three-dimensional coordinates, attitude and velocity information.

[0010] Furthermore, in step S3, the natural language prompts issued by the user or the testing system are used to extract semantic embedding vectors based on a pre-trained text encoder, and then deeply fused into each step of trajectory generation through a denoising network based on the Transformer architecture.

[0011] Furthermore, in step S3, the forward noise addition process of the trajectory diffusion generation model is a fixed Markov chain that gradually adds Gaussian noise to the real trajectory until it becomes a pure noise trajectory with a standard normal distribution. The trajectory diffusion generation model uses a reverse denoising process to gradually recover meaningful trajectories from pure noise. The reverse denoising process is limited by the text prompts input by the user and the prior static environmental context.

[0012] Furthermore, in step S3, dynamic resampling is performed during the back-inference stage of the diffusion model. A vehicle kinematics bicycle model is explicitly introduced as a hard boundary constraint, and a nonlinear dynamic penalty loss is constructed. At each step of back-sampling, the gradient of this physical loss is used to forcibly correct the predicted mean, ensuring that dangerous behaviors strictly adhere to the laws of dynamics.

[0013] Furthermore, the spatiotemporally consistent background inpainting model in step S4 is a background inpainting generation model based on latent diffusion. In the U-Net denoising network for latent space inpainting, a cross-frame interleaved attention mechanism is introduced. For the damaged latent features of the current frame, the network is forced to perform cross-frame attention calculation in the previous and next frames to achieve spatiotemporal consistency and multi-view geometric coherence. Using the undamaged context region as a spatial prior condition, multi-step denoising inference is performed in the latent space. In each step of denoising inference, a mask mixing strategy is adopted to ensure the absolute fidelity of the known region.

[0014] Furthermore, in step S5, the static background 3D Gaussian field is used to reconstruct the physical environment unaffected by dangerous events, utilizing the repaired background features output in step S4 and the corresponding pseudo-LiDAR point cloud for dense initialization. During the optimization phase, since the background is strictly defined as static, the mean and rotation of all its Gaussian elements remain constant temporally. The dynamic object 3D Gaussian field initializes the object Gaussian based on the appearance features of the selected traffic participants. The dangerous trajectory generated in step S3 is injected into this Gaussian field, and the physical properties of the object Gaussian will undergo corresponding rigid body transformations according to the trajectory state at each time step, achieving dynamic evolution in the time dimension.

[0015] Furthermore, in step S5, the differentiable rasterization rendering employs a global depth sorting and drivable region hard truncation mechanism. Using the static background drivable region mask as a spatial hard boundary, the background Gaussian boundary between the background field and the dynamic field, along with the pose-transformed object Gaussian boundary, is uniformly mapped onto the virtual camera's truncated cone. A joint continuous cardinality sort is then performed based on the depth from the center of each Gaussian unit to the camera's optical center. This avoids clipping issues during new perspective synthesis.

[0016] Furthermore, in step S5, the differential rasterization rendering introduces a gradient compaction truncation mechanism. By setting a gradient threshold, it actively suppresses erroneous Gaussian cell splitting caused by inconsistencies in the generated image's perspective, ensuring the cleanliness of the physical reconstruction.

[0017] Compared with the prior art, the present invention has the following advantages: Instead of directly generating the entire driving scene as a whole, this method explicitly decouples the static environment, dynamic targets, and motion trajectories within the scene, and controls the behavior changes of dynamic targets through semantic cues. Addressing the issue of background occlusion caused by target position changes during dangerous behavior generation, this method utilizes a generative model to recover the texture and geometric information of the affected areas, thereby achieving a controllable transformation from a normal driving scene to a dangerous edge scene, improving the realism and editability of the generated scene. Attached Figure Description

[0018] Figure 1This is an overall flowchart of the method for generating and reconstructing dangerous scenes based on prompt words in an embodiment of the present invention; Figure 2 This is a flowchart of the trajectory diffusion generation model guided by prompt words in an embodiment of the present invention; Figure 3 This is a flowchart of the background repair model in an embodiment of the present invention; Figure 4 This is a flowchart of the physical reconstruction and rendering process based on two-layer decoupled 3D Gaussian in an embodiment of the present invention; Figure 5 This is a visualization diagram illustrating the generation of a dangerous edge scene in an embodiment of the present invention; Figure 6 This is a schematic diagram of hyperparameter sensitivity analysis in an embodiment of the present invention. Detailed Implementation

[0019] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0020] like Figure 1 As shown, this embodiment provides a method for generating and reconstructing dangerous scenes based on prompt words, including the following steps: S1. Obtain data for typical driving scenarios; S2. The feature receiving module receives regular driving scenario data and decouples and separates the regular driving scenario data into a set of latent variables. The set of latent variables includes static background features, dynamic object features, and the original motion trajectory of dynamic objects in the real world. S3. Send the set of latent variables into the prompt word-driven generation module, introduce natural language prompt words issued by the user or test system, guide the trajectory diffusion model to perform dynamic resampling of the original motion trajectory, and output a dangerous trajectory that conforms to the semantics. S4. To address the geometric and textural occlusion holes in the static background caused during the generation of dangerous trajectories, a spatiotemporally consistent background restoration model is introduced to complete the background and output the restored background features. S5. The repaired background features, dangerous trajectories and dynamic object features are sent to the dual-layer 3D Gaussian physical reconstruction module. The scene is instantiated by 3D Gaussian sputtering, and two independent sets of three-dimensional Gaussian primitives are constructed, namely the background 3D Gaussian set and the object 3D Gaussian set. Then, the micro-rasterization renderer performs spatial projection and blending calculations. S6, outputs the final high-fidelity dangerous scene footage.

[0021] In step S2, the output data of the feature receiving module consists of three independent feature subsets: a clean static background feature map after removing dynamic objects, a geometric and appearance feature set of K independent dynamic objects in the scene, and the original motion trajectory of the objects in the real world, which includes temporal three-dimensional coordinates, attitude and velocity information.

[0022] In step S3, the natural language prompts issued by the user or the testing system are used to extract semantic embedding vectors based on a pre-trained text encoder. These vectors are then deeply fused into each step of trajectory generation through a denoising network based on the Transformer architecture.

[0023] In step S3, the forward noise addition process of the trajectory diffusion generation model is a fixed Markov chain that gradually adds Gaussian noise to the real trajectory until it becomes a pure noise trajectory with a standard normal distribution. The trajectory diffusion generation model uses a reverse denoising process to gradually recover meaningful trajectories from pure noise. The reverse denoising process is limited by the text prompts input by the user and the prior static environmental context.

[0024] In step S3, dynamic resampling is performed during the back inference stage of the diffusion model. The vehicle kinematics bicycle model is explicitly introduced as a hard boundary constraint to construct a nonlinear dynamic penalty loss. At each step of back sampling, the gradient of this physical loss is used to forcibly correct the predicted mean, ensuring that dangerous behaviors strictly follow the laws of dynamics.

[0025] like Figure 2 As shown, the historical motion trajectory of the dynamic target within the time window is first obtained and used as the initial input to the trajectory generation module. Traditional rule-based trajectory planning methods typically rely on fixed mathematical constraints, making it difficult to generate dangerous driving behaviors with complex interactions and long-tail distribution characteristics. Therefore, this method adopts a trajectory generation approach based on a conditional diffusion model, controlling the target's motion pattern through natural language prompts to achieve the transformation from normal driving behavior to dangerous behavior trajectories.

[0026] First, obtain the historical motion state of the dynamic target within the time window and represent it as follows: (1) in, Indicates the length of time. The trajectory state dimension includes information such as the target's three-dimensional position, attitude angle, and velocity. The trajectory diffusion model first progressively adds Gaussian noise to the real trajectory, transforming it into a random noise state; then, in the reverse denoising process, it progressively recovers the semantically constrained target trajectory based on the input conditions.

[0027] In the reverse denoising stage, the system gradually reconstructs the target trajectory based on the user-input description of dangerous behavior and environmental context information. (Input natural language prompts) First, semantic encoding is performed using a pre-trained text encoder:

[0028] (2) Simultaneously, environmental context information such as road structure and surrounding vehicle status is also considered. Construct a conditional denoising network based on the Transformer structure. A cross-attention mechanism is used to fuse textual semantics with trajectory features, ultimately generating a new trajectory that matches the description of dangerous behavior.

[0029] (3) in, This represents a conditional trajectory diffusion generation network. This network fuses textual semantic features with trajectory motion features through a cross-attention mechanism, enabling the generated results to respond to descriptions of different dangerous behaviors.

[0030] Since purely data-driven trajectory generation may produce abnormal results that do not conform to the laws of vehicle motion, this method further introduces vehicle kinematic constraints. During trajectory generation, the system corrects the generated trajectory based on physical limitations such as the vehicle's maximum acceleration and maximum steering angle to avoid motion states that the vehicle cannot achieve. Specifically, the model calculates the deviation between the generated trajectory and the vehicle dynamics constraints during backsampling and uses gradient guidance to adjust the trajectory results, ensuring that the final generated trajectory simultaneously satisfies both hazard semantics and kinematic feasibility.

[0031] Furthermore, since generated targets are typically derived from image space, direct mapping to a 3D scene is susceptible to depth scale uncertainties. To address this issue, this method matches the background region of the generated image with Gaussian depth information of the real scene, and uses the actual camera mounting height as a scale reference to perform scale correction on the 3D point cloud corresponding to the generated target, enabling the hazardous target to be accurately embedded in the real physical coordinate space.

[0032] Through the above process, the trajectory generation module can generate diverse target motion trajectories that conform to the laws of vehicle dynamics based on different dangerous behavior prompts, and provide controllable motion priors for subsequent dynamic target 3D Gaussian reconstruction.

[0033] like Figure 3As shown, the spatiotemporally consistent background inpainting model in step S4 is a background inpainting generation model based on latent diffusion. In the U-Net denoising network for latent space inpainting, a cross-frame interleaved attention mechanism is introduced. For the damaged latent features of the current frame, the network is forced to perform cross-frame attention calculation in the previous and next frames to achieve spatiotemporal consistency and multi-view geometric coherence. The undamaged context region is used as a spatial prior condition, and multi-step denoising inference is performed in the latent space. In each step of denoising inference, a mask mixing strategy is used to ensure the absolute fidelity of the known region.

[0034] The background restoration and generation module is used to solve the problem of background loss caused by changes in the trajectory of dynamic targets. When a dynamic target changes from its original trajectory... Adjust to dangerous trajectory Subsequently, the position of the target object in three-dimensional space changes, and the static area that was originally occluded by the target will be exposed again, resulting in texture loss and geometric holes in the image space.

[0035] First, generate a mask for the occluded / missing region based on the dynamic changes in the target's position. The region with a value of 1 represents the background area that needs repair. Based on this mask, the original static background features are converted into a damaged background representation:

[0036] (4) in, Represents the original static background features. This indicates the background features to be repaired after removing the missing areas.

[0037] To ensure continuous scene textures and the absence of visual artifacts during subsequent 3D Gaussian reconstruction, this method designs a spatiotemporally consistent background inpainting network based on a latent diffusion model. Unlike traditional single-frame image restoration methods, this module not only needs to restore the texture information of the missing areas, but also needs to ensure the geometric consistency of the restoration results across consecutive time frames and under different viewing angles.

[0038] During the repair process, the system first uses a variational autoencoder (VAE) to map high-resolution background features and missing masks to a low-dimensional latent space to reduce computational complexity. (5) in, Indicates encoder, This represents the compressed background latent features. This represents the mask for the missing region after downsampling. Subsequently, the latent diffusion model recovers the complete background features through a multi-step denoising process, under the contextual constraints provided by the known background region.

[0039] In the latent space, the diffusion model performs multi-step denoising inference based on the contextual information provided by the undamaged region, gradually restoring the background features of the missing region. Unlike traditional single-frame image inpainting methods, this method further considers the continuous temporal variations and multi-view consistency issues in autonomous driving scenarios, introducing a cross-frame interleaved attention mechanism into the diffusion denoising network. This mechanism utilizes the feature correlation between adjacent time frames to jointly model the current frame inpainting process with the information of previous and subsequent frames, and combines dense optical flow information for spatial compensation, ensuring that the generated result remains consistent in both the temporal and spatial dimensions.

[0040] During the diffusion denoising process, the system employs a masking hybridization strategy to constrain known regions to retain their original information, while only generating and repairing missing regions. (6) in, This represents the latent state of a known background region after forward noise addition. This strategy avoids the diffusion model from repeatedly generating the real background region, ensuring a natural transition between the restored area and the original scene.

[0041] After multiple denoising steps, the system uses a VAE decoder to restore the repaired latent features to the original feature space, obtaining a complete background representation: (7) in, This represents the final generated restored background feature. This feature will serve as an important input for the subsequent initialization of the background 3D Gaussian model, ensuring that the static environment maintains continuous and stable geometric structure and texture information during rendering from different viewpoints.

[0042] like Figure 4 As shown, in step S5, the static background 3D Gaussian field is used to reconstruct the physical environment unaffected by dangerous events. Dense initialization is performed using the repaired background features output in step S4 and the corresponding pseudo-LiDAR point cloud. During the optimization phase, since the background is strictly defined as static, the mean and rotation of all its Gaussian elements remain constant over time. The dynamic object 3D Gaussian field initializes the object Gaussian based on the appearance features of the selected traffic participants. The dangerous trajectory generated in step S3 is injected into this Gaussian field, and the physical properties of the object Gaussian undergo corresponding rigid body transformations according to the trajectory state at each time step, achieving dynamic evolution in the time dimension.

[0043] In step S5, the differentiable rasterization rendering employs a global depth sorting and drivable region hard truncation mechanism. Using the static background drivable region mask as a spatial hard boundary, the background Gaussian boundary between the background field and the dynamic field, along with the pose-transformed object Gaussian boundary, is uniformly mapped onto the virtual camera's truncated cone. Furthermore, a joint continuous cardinality sort is performed based on the depth from the center of each Gaussian primitive to the camera's optical center. This avoids clipping issues during new perspective synthesis.

[0044] In step S5, the differential rasterization rendering introduces a gradient compaction truncation mechanism. By setting a gradient threshold, it actively suppresses erroneous Gaussian cell splitting caused by inconsistencies in the generated image's perspective, ensuring the cleanliness of the physical reconstruction.

[0045] After completing the generation of dangerous trajectories and the restoration of background features, the system has obtained the semantic information and structured priors required to construct the target dangerous scene. In order to further achieve 3D scene reconstruction with realistic geometric structure, multi-view consistency and physical rationality, this method adopts an explicit neural representation based on 3D Gaussian Splatting (3DGS) to replace the traditional implicit neural radiation field method.

[0046] This module further decouples the scene into independent 3D Gaussian fields for the background and 3D Gaussian fields for dynamic objects. The background Gaussian field represents static environmental information such as roads and buildings, while the dynamic object Gaussian field represents moving entities such as vehicles and pedestrians. By modeling these two types of Gaussian fields separately, the movement of dynamic targets can be independently controlled in 3D space, and physically-level scene editing can be achieved based on hazardous trajectories. Finally, a high-fidelity visual effect of the hazardous scene is generated through a differential rasterization rendering module.

[0047] Mathematical representation of 3D Gaussian units: In this embodiment, 3DGS uses a large number of three-dimensional Gaussian primitives to explicitly represent the scene. Each Gaussian primitive contains attributes such as spatial location, shape scale, rotation direction, opacity, and color. Compared with traditional point cloud or mesh representation methods, it can more flexibly describe the geometric structure and view-related appearance changes in complex environments.

[0048] For any three-dimensional Gaussian element Its spatial distribution is determined by the mean location Covariance Matrix Jointly determined: (8) in, This indicates the position of the Gaussian element in three-dimensional space. It indicates its spatial shape and orientation information.

[0049] To ensure the stability of Gaussian elements during the optimization process, the system decomposes the covariance matrix into a rotation matrix and a scaling matrix: (9) in, Used to control the scale of the Gaussian ellipsoid in different directions. It is used to describe its spatial rotation state. Through this parameterization method, Gaussian elements can adapt to different object structures and environmental morphological changes.

[0050] In addition, each Gaussian primitive contains optical properties for rendering, including opacity and color features. Opacity describes the primitive's influence on light; color features are represented using spherical harmonic functions, enabling the system to generate corresponding lighting and color changes based on different viewing angles.

[0051] Construction and fusion of two-layer decoupled Gaussian fields: In this embodiment, based on the aforementioned scene decoupling results, the system constructs a background 3D Gaussian field and a dynamic object 3D Gaussian field respectively, thereby realizing the independent representation of the static environment and dynamic entities.

[0052] First, for a static 3D Gaussian field background, the system utilizes the complete background features output by the background restoration module. The system first performs Gaussian initialization based on the corresponding 3D spatial information. This Gaussian field is mainly used to describe environmental areas such as roads, buildings, and vegetation that do not change over time. During the optimization process, the position and spatial structure of the background Gaussian primitives remain stable to ensure the consistency of the static environment under different times and viewpoints. Secondly, for the 3D Gaussian field of dynamic objects, the system independently models traffic participants for whom hazardous behaviors need to be generated. First, the Gaussian representation of the object is initialized based on the target's appearance features, and then the hazardous trajectory output by the trajectory generation module is generated. As a motion control condition, the object undergoes a Gaussian rigid body transformation.

[0053] For a dynamic target at time step t, its spatial position and attitude are updated according to the danger trajectory: (10) in, This represents the target's spatial displacement at the current time step. In this way, dynamic targets can achieve motion changes that conform to the logic of dangerous behavior while maintaining their geometric and appearance characteristics.

[0054] By using a two-layer decoupled representation of the background Gaussian field and the dynamic object Gaussian field, the system avoids the problem of high coupling between dynamic targets and the environment in traditional overall scene reconstruction methods, enabling dangerous events to be edited independently in real 3D space.

[0055] Microrasterization rendering and hazardous scene construction After completing the construction of the two-layer Gaussian field, the system uses the 3DGS differential rasterization rendering module to project the three-dimensional Gaussian primitives onto the virtual camera's viewpoint, generating the final image of the dangerous scene.

[0056] Specifically, the system first transforms the 3D Gaussian primitives into the image space based on the virtual camera pose, and calculates their projection range and affected area on the 2D plane. Then, based on the opacity, color, and depth relationships of the Gaussian primitives, pixel-level fusion is achieved through Alpha Blending, realizing the rendering of the 3D scene into a 2D image. Since dynamic targets undergo spatial position changes under dangerous trajectories, without proper depth relationship processing, physical inconsistencies such as vehicles penetrating buildings and incorrect target occlusion relationships can easily occur. Therefore, this method introduces a global depth ranking and drivable area constraint mechanism in the rendering stage. On the one hand, the system uses road area information in the static background to constrain the movement range of dynamic targets, preventing targets from entering positions that do not conform to the road structure; on the other hand, the background Gaussian field and the dynamic object Gaussian field are uniformly projected into the camera's view frustum, and jointly ranked according to the depth relationship between the Gaussian primitives and the camera, ensuring that the target maintains the correct occlusion relationship with the environment. In addition, to reduce the problem of spatial floating Gaussian points caused by inconsistencies in the generated scene from multiple perspectives, this method introduces a Gaussian density control mechanism during the optimization process. By limiting the generation of abnormal Gaussian primitives, the stability and cross-perspective consistency of the 3D reconstruction results are improved.

[0057] Through the aforementioned differentiable rendering process, the system ultimately achieves a complete closed loop from hazard semantic description, dynamic trajectory generation, background structure repair to 3D physical scene rendering. The generated results not only possess photorealistic quality but also support free-viewpoint observation and dynamic behavior control, making them suitable for autonomous driving hazard scenario simulation and closed-loop testing.

[0058] Compared with the prior art, the present invention has the following advantages: Instead of directly generating the entire driving scene as a whole, this method explicitly decouples the static environment, dynamic targets, and motion trajectories within the scene, and controls the behavior changes of dynamic targets through semantic cues. Addressing the issue of background occlusion caused by target position changes during dangerous behavior generation, this method utilizes a generative model to recover the texture and geometric information of the affected areas, thereby achieving a controllable transformation from a normal driving scene to a dangerous edge scene, improving the realism and editability of the generated scene.

[0059] We propose a method for generating dangerous scenarios based on natural language prompts, which links high-level semantic descriptions with the 3D scene generation process. This enables the system to automatically generate corresponding dangerous scenarios based on behavioral descriptions such as "abnormal lane cutting" and "sudden braking of the vehicle in front," reducing the cost of manual rule design and improving the flexibility and controllability of generating long-tail dangerous scenarios.

[0060] A cross-modal hazardous scenario derivation framework is constructed, which combines visual semantics, dynamic trajectory and three-dimensional spatial information to achieve structured reconstruction from conventional driving data to hazardous scenarios, thereby improving the realism and generalization ability of the generated scenarios.

[0061] We propose a cue word-driven trajectory reconstruction method based on a diffusion model. This method generates and adjusts the original trajectory under textual semantic constraints, producing diverse dangerous behavior trajectories and solving the problem that traditional heuristic methods are unable to simulate complex long-tail behaviors.

[0062] Vehicle dynamics constraints are introduced during trajectory generation to limit speed changes, steering amplitude, and motion continuity, thereby avoiding the generation of anti-physics trajectories and improving the realism and simulation reliability of dangerous scenarios.

[0063] A spatiotemporally consistent background restoration mechanism is proposed, which automatically fills in the exposed area after the dynamic target moves, and maintains the stability of the background structure through cross-frame consistency constraints, effectively solving the problems of holes, missing textures and temporal flicker in traditional scene editing.

[0064] A physical reconstruction and rendering method based on two-layer decoupled 3D Gaussian is proposed. The static background and dynamic target are modeled separately to achieve independent control of dynamic entities. The method combines depth sorting and differentiable rasterization to ensure correct occlusion relationships and improve the 3D consistency and visual realism of the generated scene.

[0065] By employing explicit 3D Gaussian representation, the computational overhead of traditional implicit neural rendering is reduced, improving rendering efficiency while ensuring high-fidelity reconstruction results. This enables the generated results to be applied to simulation, testing, and evaluation of dangerous autonomous driving scenarios.

[0066] To verify the effectiveness and generalization ability of the hazardous scene generation method based on two-layer decoupled 3D Gaussian described in this invention, three publicly available autonomous driving datasets—Waymo Open Dataset, KITTI, and Virtual KITTI 2 (VKITTI2)—were selected for experiments. The Waymo dataset contains complex traffic flow, vehicle motion, and diverse environmental conditions. The experiments selected eight consecutive sequences containing dynamic targets such as vehicles and pedestrians, and used a time-uniform sampling method to divide the training and test sets. Simultaneously, the input images were uniformly adjusted to a resolution of 1066×1600, and object detection and 3D tracking methods were used to obtain prior information about dynamic entities.

[0067] Experimental Implementation: All model training, inference, and baseline method replication were performed on a deep learning workstation with a single NVIDIA RTX 4090 GPU. During optimization, the 3D Gaussian field was trained using the Adam optimizer, with a total of 30,000 iterations. For the learning rate, an exponential decay strategy was adopted for the rigid body transformation parameters of dynamic objects, with the translation parameters... The learning rate decreases from 5e−3 to 5e−5, and the rotation parameters... The learning rate of the background skybox decays from 1e−3 to 1e−5; the learning rate of the background skybox decays from 1e−2 to 1e−4 to model changes in ambient lighting. For 3D Gaussian adaptive density control, a differentiated constraint strategy is adopted for dynamic and static scenes: the spatial scale of the static background Gaussian field is limited to a 20-meter range centered on the vehicle; the Gaussian field of dynamic objects is strictly constrained by the corresponding 3D bounding box. During training, a sampling constraint mechanism is used to spatially clip the object Gaussian field; any Gaussian elements exceeding the 3D bounding box range are directly removed to avoid Gaussian drift and cross-regional visual artifacts.

[0068] Comparison of baseline model with evaluation metrics: To comprehensively evaluate the performance and advancement of the proposed hazardous scene generation and physical reconstruction architecture in autonomous driving 3D scene reconstruction tasks, representative current 3D neural rendering methods were selected as benchmarks for comparison, including NSG (Neural Scene Graphs), MARS, the original 3D Gaussian Splatting (3DGS), and EmerNeRF. NSG models static backgrounds using multi-plane images and learns independent latent variables for dynamic targets; MARS constructs a modular neural scene graph system based on the Nerfstudio framework; 3DGS uses explicit Gaussian units for scene representation to verify the limitations of purely static Gaussian methods in dynamic scenes; EmerNeRF performs implicit feature modeling through static / dynamic hierarchies and hash grids. To ensure fairness in the comparison, both NSG and MARS incorporate real target trajectories as motion priors during training and evaluation. For quantitative evaluation, PSNR, SSIM, and LPIPS are used as core image quality evaluation metrics to measure pixel error, structural similarity, and perceptual feature distance, respectively. Among them, PSNR is used to evaluate the pixel-level fidelity of the reconstruction results, SSIM is used to measure structural consistency, and LPIPS is used to reflect perceptual differences at the deep semantic level. Considering the characteristic that dynamic targets occupy a critical evaluation area in autonomous driving hazardous scene generation tasks, a local PSNR metric based on the projection of the target's 3D bounding box is further introduced. Reconstruction error is calculated only within the projected area of ​​the target instance, thus more accurately evaluating the model's ability to reconstruct dynamic vehicle motion, edge details, and geometric changes. Simultaneously, to measure the real-time performance of the model in engineering deployment, this paper statistically analyzes the frame rate (FPS) of each method under a unified hardware environment as a rendering efficiency evaluation metric.

[0069] High-precision global and local evaluation on the Waymo dataset: The Waymo dataset, renowned for its high resolution and highly dynamic complex street scenes, serves as a litmus test for the upper limits of neural rendering models for autonomous driving. In this experiment, we uniformly aligned the rendered image resolution of all participating models to 1066×1600 and rigorously recorded the performance of each method in global reconstruction quality (PSNR, SSIM, LPIPS), local reconstruction quality of dynamic objects (PSNR), and real-time rendering inference (FPS). Detailed quantitative evaluation results are shown in Table 1.

[0070] Table 1. Comparison of global and local high-precision evaluations of different methods on the Waymo dataset. ① Global image quality metrics (PSNR, SSIM, LPIPS) Global metrics are used to evaluate the fidelity of the overall scene reconstruction. As shown in Table 1, compared to the current state-of-the-art baseline model, Street Gaussians (SG), our proposed method (Ours) achieves a global PSNR improvement from 34.61 to 35.72, an SSIM improvement from 0.938 to 0.946, and a significant reduction in LPIPS from 0.079 to 0.065, all three metrics representing record-breaking performance. This is attributed to front-end optical flow denoising and multimodal feature fusion, which effectively removes environmental noise from real data, providing a cleaner and higher-fidelity geometric and texture prior for the underlying 3D Gaussian field.

[0071] ② Local reconstruction accuracy (PSNR*) PSNR* is a local accuracy metric for highly dynamic traffic participants and a core evaluation dimension for hazardous scene generation. Traditional 3D GS and the implicit hierarchical method EmerNeRF only achieve PSNR* of 21.25 and 21.67 respectively, while the high-performing SG model only reaches an upper limit of 30.23. Our proposed method, through thorough decoupling of dynamic and static features, combined with prompt-driven trajectory reconstruction and background restoration strategies, eliminates the "ghosting" and edge tearing problems during dynamic entity transfer, significantly improving PSNR* to 32.72. This directly verifies the absolute advantage of the dual-layer independent 3D Gaussian architecture in complex scene editing.

[0072] ③ Rendering real-time performance (FPS) Traditional dynamic NeRF baseline models (NSG, MARS, EmerNeRF) are limited by volume rendering integrals, resulting in inference speeds generally below 1 FPS, which is completely unacceptable for practical engineering applications. The method presented in this paper, by introducing structural decoupling and spatiotemporal feature extraction, achieves an inference frame rate of 46 FPS. While slightly lower than SG (135 FPS) and pure 3D GS (205 FPS), it consistently exceeds the 30 FPS real-time performance threshold for autonomous driving closed-loop simulation systems. This method achieves a significant improvement in scene fidelity and physical editing capabilities at a minimal and acceptable computational cost, making it the most superior overall performance solution currently available.

[0073] Evaluation of the sparse view challenge on the KITTI and VKITTI2 datasets In real-world autonomous driving scenarios, the system often faces the challenge of extremely sparse observation data due to high vehicle speeds, obstruction from other road users, and limitations in sensor perspective. To further explore the robustness of the model under extreme perspective truncation and information scarcity, we conducted more rigorous validation on the KITTI (real-world scenario) and VKITTI2 (high-fidelity clone simulation scenario) datasets.

[0074] Following industry-recognized evaluation standards, we artificially limited the prior observation data accessible to the model, training the network using only 75%, 50%, and 25% of the image frames in the sequence, respectively. Due to the significant reduction in viewpoints, this experiment accurately exposes the neural rendering model's ability to synthesize novel views and its upper limit of geometric inference in the "3D spatial blind spot." Quantitative comparison results are shown in Table 2.

[0075] Table 2. Quantitative evaluation comparison of different methods on sparse views of the KITTI and VKITTI2 datasets. ① Experimental results on the KITTI dataset: In real-world testing on the KITTI dataset, when the accessible training images are drastically reduced from 75% to 25%, traditional dynamic NeRF methods suffer significant performance degradation due to their heavy reliance on dense line-of-sight intersections for volume integration. For example, NSG's PSNR plummets from 21.53 to 20.00, resulting in severe holes and artifacts in the rendered images. In contrast, our proposed method (Ours), building upon the strongest baseline SG, introduces a dense depth / optical flow pseudo-point cloud prior and a background inpainting diffusion model, demonstrating exceptional robustness. Even under extreme conditions with only a 1 / 4 viewpoint (KITTI-25%), our method still achieves a high PSNR of 25.12, significantly surpassing SG's 24.53. Notably, our method's LPIPS (0.094) under this configuration shows only a minimal fluctuation compared to SG (0.090). This is mainly because, when faced with completely unobserved physical blind spots, the generative inpainting model introduced in this paper will "illusory" reasonable completion textures based on semantic priors. These textures are highly accurate in physical structure, but have slight stylistic differences from the real values ​​in deep perceptual features. This perceptual trade-off made to ensure "no holes in 3D geometry" is completely reasonable and more beneficial than harmful in autonomous driving physical simulation.

[0076] ② Experimental results on the VKITTI2 dataset: In the virtual VKITTI2 dataset test, our proposed method also demonstrated superior sparse viewpoint synthesis capabilities, consistently ranking first in all core metrics. While traditional methods like MARS performed slightly better than NSG overall, their PSNR still experienced a significant drop of nearly 3dB (from 29.79 to 27.01) under the 25% extremely sparse VKITTI2 partitioning. Our proposed method, thanks to its decoupling mechanism and strong geometric constraints, greatly compensated for the lack of a specific viewpoint. In the most extreme VKITTI2-25% test, our method achieved a PSNR of 28.82, superior to the baseline SG's 28.52, while also achieving an SSIM of 0.936. This further demonstrates that our method not only handles the challenges of real-world scenes but also maintains excellent rendering quality and structural integrity in synthetic scenarios.

[0077] Ablation experiment: To verify the specific contributions of each rendering and optimization module in the framework and to validate the necessity of the algorithm design, we conducted rigorous ablation studies on the Waymo dataset. Using our complete model (Ours / Complete Model) as the performance baseline, we sequentially stripped or replaced specific network modules and re-optimized them under the same hyperparameter configuration and number of training iterations.

[0078] The specific network variant configuration is as follows: Decoupling of Static and Dynamic Structures: Instead of using two independent 3D Gaussian fields, all elements in the scene (static background and dynamic vehicles) are jointly optimized and rendered in a unified 3D Gaussian field, simulating the baseline state of traditional pure 3DGS processing of dynamic scenes. Background Inpainting Generation Model (w / o Inpainting Diffusion): After changing the trajectory of dynamic objects using prompts, no latent spatial feature repair is performed on exposed occlusion holes; physical projection and rendering are directly forced. Evaluation metrics still use global rendering quality (PSNR, SSIM, LPIPS) and local high-fidelity (PSNR) for dynamic entities. The ablation results are shown in Table 3.

[0079] Table 3. Quantitative evaluation of the core components in ablation experiments on the Waymo dataset. Decoupling of static and dynamic structures: When abandoning the dual-layer decoupling mechanism and attempting to forcibly edit the trajectory of dynamic objects within a single Gaussian field, the local high-precision index PSNR experienced a precipitous drop (suddenly falling to 22.45). From a physical perspective, undecoupled Gaussian elements are highly entangled in space. Forcibly translating a Gaussian sphere in a certain region leads to extremely severe "spatial tearing" and "ghosting" phenomena, completely disrupting the physical logical continuity of editing dangerous scenes.

[0080] Background restoration generation model: After a significant vehicle trajectory redirection, without the introduction of a latent diffusion-based inpainting model, the previously occluded physical occupants would expose large areas of visual black spots or meaningless stretched textures. This directly leads to a substantial decrease in global PSNR and SSIM (PSNR drops to 32.45). The introduction of a generative model not only completes the semantic texture in the 2D image domain but also ensures the 3D geometric integrity during the new perspective synthesis process through multi-view feature mapping.

[0081] Visual analysis: To intuitively verify the structural decoupling capability, background restoration effect, and hazardous scene derivation capability of the proposed hazardous scene generation and reconstruction framework in complex autonomous driving scenarios, this section selects Waymo test sequences for qualitative visualization analysis. For example... Figure 5 As shown, the visualization results are uniformly organized and displayed in four columns, corresponding to the original input image, the static scene after background restoration, the 3D Gaussian decoupling result of dynamic objects, and the final rendering result of the dangerous scene.

[0082] Dynamic entity decoupling and feature preservation: like Figure 5 As shown, in the original driving sequence, dynamic targets such as vehicles and pedestrians are highly coupled with the complex urban background. The method in this embodiment can accurately separate traffic participants of different scales from the background and independently construct their 3D Gaussian representations. Whether it is a large truck or a small car, clear structural separation with defined boundaries can be achieved, while retaining key appearance information such as vehicle texture, specular reflection, and occlusion shadows. This demonstrates that the model has stable dynamic and static decoupling capabilities, providing a reliable geometric foundation for subsequent trajectory retargeting and 3D editing.

[0083] Static background restoration and 3D consistency completion: After dynamic targets are removed, significant occlusion holes appear in the original scene. This embodiment uses a background restoration module to fill in the missing areas, restoring static information such as road markings, building facades, and distant structures. The visualization results demonstrate this. Figure 5It can be observed that the restored background remains continuous and consistent under different viewpoints, without obvious cracks, flickering or texture jumps, indicating that the background 3D Gaussian field has good spatial geometric consistency and multi-view stability.

[0084] Hazardous scenario synthesis and physical plausibility verification: The final rendering results showcase dangerous driving scenarios generated under the guidance of text prompts and trajectory reconstruction. The system can simulate complex interactive behaviors such as multi-vehicle conflicts, emergency lane changes, and collisions, and generate motion trajectories and occlusion relationships that conform to physical intuition. For example, in lane-cutting and collision scenarios, the vehicle's motion trajectory and collision posture maintain rigid body consistency; in scenarios where vehicles lose control and run off the road, the interaction between the vehicle and the environment is natural and reasonable, while the generation of complex visual effects such as smoke enhances the realism of the scene. In complex multi-object interactions, the occlusion relationships between entities are clear and there is no obvious clipping phenomenon.

[0085] In summary, the method in this embodiment can achieve complete link reconstruction from dynamic target decoupling and background restoration to dangerous scene generation on complex real road data, verifying the effectiveness of the two-layer 3DGS framework in terms of structural expression accuracy and physical consistency, and demonstrating good modeling ability for complex long-tail dangerous scenarios of autonomous driving.

[0086] Hyperparameter sensitivity analysis: To analyze the hyperparameter sensitivity of a two-layer decoupled 3D Gaussian field during optimization and compaction, this section focuses on the position learning rate. With densification gradient threshold Experiments were conducted to evaluate the combined impact on rendering quality (PSNR, LPIPS) and real-time performance (FPS), thereby determining stable parameter configurations suitable for autonomous driving scenarios, such as... Figure 6 As shown. Experimental results indicate that, It has a significant moderating effect on the model's structural expressiveness and computational cost. When When the setting is too large (>3×10−4), the splitting and proliferation of Gaussian elements in vehicle edges and high-frequency texture regions are limited, resulting in insufficient detail representation and obvious loss of vehicle structure (such as rearview mirrors, license plates, etc.), with PSNR dropping below 35 dB; when When the threshold is set too small (e.g., 0.5 × 10⁻⁴), the Gaussian primitives grow excessively densely. While this improves local expressive power, it introduces a large amount of redundant structure and high-frequency noise, causing the PSNR to drop further to approximately 32.1 dB. Simultaneously, the computational burden increases significantly, reducing the rendering frame rate to approximately 21 FPS, which is insufficient for real-time simulation requirements. Under moderate threshold conditions ( =2×10−4), the system achieves a balance between detailed representation and computational efficiency, with a PSNR of 35.72 dB and an FPS of approximately 46 FPS, demonstrating the best overall performance. Furthermore, to analyze the coupling relationship between spatial update and compaction mechanisms, this section constructs... and The joint sensitivity analysis showed a clear matching relationship between the two: when =1×10−3 and When the value is 2 × 10⁻⁴, LPIPS reaches its optimal value of 0.065, indicating that the generated result is closest to the real image at the perceptual level, and has the best consistency between structure and illumination. Too large (e.g., above 1×10−3) and When the size is small (e.g., 0.5 × 10⁻⁴), the Gaussian meta-updates are too drastic and accompanied by frequent splits, resulting in noticeable noise and floating structures on the scene surface, and LPIPS rises to 0.088; while when Too small and When the value is too large, Gaussian updates are insufficient and structural expansion is limited, making the model prone to getting trapped in local optima and failing to recover complex photometric details such as vehicle reflections and edge highlights. LPIPS further deteriorates to 0.105. The comprehensive experimental results show a significant coupling relationship between the position learning rate and the densification threshold; both must be set collaboratively under constraints of the same order of magnitude. Otherwise, they will respectively lead to problems such as "excessive splitting noise" or "structural underfitting." The finally determined stable parameter set is as follows: =1×10−3、 =2×10−4, which ensures visual quality while taking into account real-time rendering performance, meeting the dual requirements of high fidelity and high frame rate for closed-loop simulation of autonomous driving.

[0087] The above provides a detailed description of a cue-driven method for generating and reconstructing dangerous scenes. The specific embodiments described are merely illustrative of the method and its core concepts. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims.

Claims

1. A method for generating and reconstructing dangerous scenes based on cue words, characterized in that, Including the following steps: S1. Obtain data for typical driving scenarios; S2. The feature receiving module receives regular driving scenario data and decouples and separates the regular driving scenario data into a set of latent variables. The set of latent variables includes static background features, dynamic object features, and the original motion trajectory of dynamic objects in the real world. S3. Send the set of latent variables into the prompt word-driven generation module, introduce natural language prompt words issued by the user or test system, guide the trajectory diffusion model to perform dynamic resampling of the original motion trajectory, and output a dangerous trajectory that conforms to the semantics. S4. To address the geometric and textural occlusion holes in the static background caused during the generation of dangerous trajectories, a spatiotemporally consistent background restoration model is introduced to complete the background and output the restored background features. S5. The repaired background features, dangerous trajectories and dynamic object features are sent to the dual-layer 3D Gaussian physical reconstruction module. The scene is instantiated by 3D Gaussian sputtering, and two independent sets of three-dimensional Gaussian primitives are constructed, namely the background 3D Gaussian set and the object 3D Gaussian set. Then, the micro-rasterization renderer performs spatial projection and blending calculations. S6, outputs the final high-fidelity dangerous scene footage.

2. The method for generating and reconstructing dangerous scenes based on cue words according to claim 1, characterized in that, In step S3, the natural language prompts issued by the user or the testing system are used to extract semantic embedding vectors based on a pre-trained text encoder. These vectors are then deeply fused into each step of trajectory generation through a denoising network based on the Transformer architecture.

3. The method for generating and reconstructing dangerous scenes based on cue words according to claim 2, characterized in that, In step S3, the forward noise addition process of the trajectory diffusion generation model is a fixed Markov chain that gradually adds Gaussian noise to the real trajectory until it becomes a pure noise trajectory with a standard normal distribution. The trajectory diffusion generation model uses a reverse denoising process to gradually recover meaningful trajectories from pure noise. The reverse denoising process is limited by the text prompts input by the user and the prior static environmental context.

4. The method for generating and reconstructing dangerous scenes based on cue words according to claim 3, characterized in that, In step S3, dynamic resampling is performed during the back inference stage of the diffusion model. The vehicle kinematics bicycle model is explicitly introduced as a hard boundary constraint to construct a nonlinear dynamic penalty loss. At each step of back sampling, the gradient of this physical loss is used to forcibly correct the predicted mean.

5. The method for generating and reconstructing dangerous scenes based on cue words according to claim 4, characterized in that, In step S4, the spatiotemporally consistent background inpainting model is a latent diffusion-based background inpainting generation model. In the U-Net denoising network for latent space inpainting, a cross-frame interleaved attention mechanism is introduced. For the damaged latent features of the current frame, the network is forced to perform cross-frame attention calculation in the previous and next frames to achieve spatiotemporal consistency and multi-view geometric coherence. Using the undamaged context region as a spatial prior condition, multi-step denoising inference is performed in the latent space. In each step of denoising inference, a masking mixing strategy is used to ensure the absolute fidelity of the known region.

6. The method for generating and reconstructing dangerous scenes based on cue words according to claim 5, characterized in that, In step S5, the static background 3D Gaussian field is used to reconstruct the physical environment unaffected by dangerous events. The repaired background features output from step S4 and the corresponding pseudo-LiDAR point cloud are used for dense initialization. During the optimization phase, since the background is strictly defined as static, the mean and rotation of all its Gaussian elements remain constant in time. The dynamic object 3D Gaussian field initializes the object Gaussian based on the appearance features of the selected traffic participants. The dangerous trajectory generated in step S3 is injected into this Gaussian field. The physical properties of the object Gaussian will undergo corresponding rigid body transformations according to the trajectory state at each time step, realizing dynamic evolution in the time dimension.

7. The method for generating and reconstructing dangerous scenes based on cue words according to claim 6, characterized in that, In step S5, the differential rasterization rendering adopts a global depth sorting and drivable area hard truncation mechanism. The static background drivable area mask is used as the spatial hard boundary. The background Gaussian boundary between the background field and the dynamic field and the pose-transformed object Gaussian are uniformly mapped to the virtual camera truncated cone. Joint continuous cardinality sorting is performed based on the depth from the center of each Gaussian unit to the optical center of the camera.

8. The method for generating and reconstructing dangerous scenes based on cue words according to claim 7, characterized in that, In step S5, the differential rasterization rendering introduces a gradient compaction truncation mechanism. By setting a gradient threshold, it actively suppresses erroneous Gaussian cell splitting caused by inconsistencies in the generated image's perspective, ensuring the cleanliness of the physical reconstruction.

Images