Dynamic and static scene fusion method and system based on space-time appearance and semantic logic

Abstract

The application belongs to the technical field of automatic driving data generation, and particularly relates to a dynamic and static scene fusion method and system based on space-time appearance and semantic logic. The dynamic and static scene fusion method is realized through four stages of high-fidelity dynamic and static decoupling reconstruction and initial fusion, global appearance coordination based on three-dimensional knowledge distillation, traffic signal lamp control based on semantic logic, and online rendering. The method solves the problems of poor rendering quality under sparse view and visual and logical inconsistency caused by dynamic and static fusion in the prior art, and realizes high-fidelity 4D scene data generation with visual and logical consistency under any new view.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving data generation technology, specifically involving a method and system for fusing dynamic and static scenes based on spatiotemporal appearance and semantic logic. Background Technology

[0002] Deep learning-driven autonomous driving technology is increasingly reliant on high-quality, diverse training data, but current data acquisition methods face a dilemma. First, while traditional offline real-vehicle data collection provides authentic data, it relies on large-scale sensors, resulting in high costs, long cycles, and difficulty in efficiently covering rare "long-tail scenarios" such as extreme weather and accident scenarios. Second, while traditional simulations based on game engines are inexpensive and controllable, the generated images lack realistic texture details of the physical world, and the traffic flow interaction logic exhibits significant differences between the virtual and real worlds, making it difficult to directly transfer trained models to real-world applications.

[0003] Currently, inter-vehicle (I2V) perception technology utilizes low-cost, fixed-position surveillance cameras to continuously capture real traffic flow, providing a new path for scalable data acquisition. However, existing technologies suffer from poor quality synthesis of new perspectives due to sparse viewpoints. Existing neural radiation field (NeRF) or 3D Gaussian (3DGS) reconstruction methods exhibit extremely poor generalization ability when processing roadside monitoring data from 3-4 fixed overhead viewpoints. When the viewpoint switches from high altitude to the vehicle's perspective on the ground, the rendering results show severe blurring, artifacts, and geometric distortion, failing to meet the training requirements of perception algorithms for high-fidelity images.

[0004] To address rendering quality issues under sparse perspectives, a "static-dynamic decoupling" strategy is adopted. This involves reconstructing high-precision static backgrounds and dynamic vehicles separately before merging them. While this improves the geometric quality of individual scenes, it introduces a "double inconsistency" problem in terms of spatiotemporal environment and semantic logic. On one hand, pre-reconstructed static scenes are typically captured at a single, specific moment (such as a cloudy daytime), with fixed lighting, shadows, and textures. In contrast, dynamic vehicles captured in real-time by roadside cameras may come from drastically different environments (such as heavy rain or nighttime). Forcing the rendering of these two scenes together leads to a mismatch in visual appearance. On the other hand, the traffic light states in static scenes are permanently fixed at the time of acquisition. When new dynamic traffic flows are injected, traffic logic conflicts can easily arise. For example, if a vehicle is crossing based on the current green light, a logical paradox of "vehicles legally crossing at red lights" will appear in the fused scene. This data containing incorrect causal relationships can seriously mislead the decision-making logic of autonomous driving models.

[0005] In summary, existing technologies lack an effective method to automatically coordinate and unify scene appearance and traffic semantic logic while generating high-fidelity vehicle-view images using roadside data. Summary of the Invention

[0006] To address the problems in the background technology, this invention provides a method and system for fusing dynamic and static scenes based on spatiotemporal appearance and semantic logic.

[0007] The technical solution of the present invention is as follows: This invention provides a method for fusing dynamic and static scenes based on spatiotemporal appearance and semantic logic, comprising: S1: Acquire multi-view static image data of the target intersection, and use the motion reconstruction structure algorithm and the three-dimensional Gaussian sputtering model for training to obtain a static background containing geometric parameters and appearance parameters; Real-time video stream data from the roadside is acquired, and through dynamic vehicle segmentation, 3D reconstruction, and trajectory estimation, a dynamic foreground containing the 3D structure and 3D trajectory of dynamic vehicles is obtained. The dynamic foreground and static background are blended to generate an initial blended scene; S2: After introducing a learnable appearance feature vector for each Gaussian point of the static background, a global color decoder is constructed using the original spherical harmonic coefficients of the static background, the appearance feature vector, the target environment conditions, and the viewpoint embedding. After generating appearance pseudo-real values ​​under target environmental conditions using a two-dimensional lighting prior teacher model, a three-dimensional knowledge distillation strategy is used to train the global color decoder to obtain an optimized global color decoder. S3: After constructing a state transition model based on traffic light image data, and combining it with preset semantic logic, the traffic light area in the initial fusion scene is converted into a pseudo-true value of traffic state, and the traffic light spherical harmonic coefficients are obtained through L1 loss optimization. S4: Based on real-time roadside video stream data, predict the current environmental conditions in real time, process the predicted current environmental conditions through an optimized global color decoder to obtain the target static background; after determining the traffic state according to the three-dimensional trajectory, match the traffic light ball harmonic coefficients corresponding to the traffic state; based on the target static background, the matched traffic light ball harmonic coefficients and dynamic vehicles, render to generate the target video stream.

[0008] Furthermore, as described in S3, by combining preset semantic logic, the traffic light area in the initial fusion scene is converted into traffic state pseudo-true values, and the traffic light spherical harmonic coefficients are obtained through L1 loss optimization, specifically: A comprehensive index is performed on the traffic light groups at the target intersection, and a set of mutually exclusive global traffic states is predefined. A set of dedicated spherical harmonic coefficients is created for each traffic state. Gaussian points belonging to traffic lights in the static background are identified and a set of traffic light Gaussian points is formed. In the initial fusion scene, the traffic light area is located and cropped in the corresponding rendering view. The cropped traffic light area is converted into a traffic state pseudo-real value using a state transition model. The mask L1 loss value between the traffic light image rendered with the spherical harmonic coefficients corresponding to the traffic state and the traffic state pseudo-real value is minimized to obtain the optimized traffic light spherical harmonic coefficients.

[0009] Furthermore, as described in S2, a three-dimensional knowledge distillation strategy is used to train the global color decoder to obtain an optimized global color decoder, which is achieved by minimizing the reconstruction loss between the rendered image and the appearance pseudo-true value of the global color decoder.

[0010] Furthermore, S2 also includes constructing an environmental condition recognizer using roadside overhead view image data.

[0011] Furthermore, S1 describes the process of fusing the dynamic foreground and static background to generate an initial fused scene, specifically as follows: Retrieve the corresponding vehicle model from the preset vehicle library based on the timestamp, and use the vehicle position parameters and orientation parameters in the 3D trajectory to perform Gaussian transformation on all Gaussian points in the vehicle model. Then merge the transformed dynamic Gaussian points with the static background Gaussian points to generate the initial fused scene.

[0012] Furthermore, S1 describes acquiring real-time roadside video stream data, performing dynamic vehicle segmentation, 3D reconstruction, and trajectory estimation to obtain a dynamic foreground containing the dynamic 3D structure and 3D trajectory of vehicles, specifically as follows: Dynamic vehicles are segmented from the acquired real-time roadside video stream data. A 3D reconstruction algorithm is used to restore the 3D structure of the dynamic vehicles. The 3D trajectory of the dynamic vehicles is estimated by calculating the intersection of the ray emitted from the optical center of the roadside camera through the 2D position of the dynamic vehicle and the ground plane of the static scene.

[0013] Furthermore, S1 describes acquiring multi-view static image data of the target intersection, and using a motion reconstruction structure algorithm and a 3D Gaussian sputtering model for training to obtain a static background containing geometric and appearance parameters, specifically: After acquiring multi-view static image data of the target intersection, the motion reconstruction structure algorithm is used to process the data to generate a sparse point cloud. After initializing the three-dimensional Gaussian sputtering model with the sparse point cloud, the model is trained to obtain a static background containing geometric parameters and appearance parameters.

[0014] This invention also provides a dynamic and static scene fusion system based on spatiotemporal appearance and semantic logic, comprising: Initial fusion scene generation module: used to acquire multi-view static image data of the target intersection, and use motion recovery structure algorithm and 3D Gaussian sputtering model training to obtain static background containing geometric parameters and appearance parameters; Real-time video stream data from the roadside is acquired, and through dynamic vehicle segmentation, 3D reconstruction, and trajectory estimation, a dynamic foreground containing the 3D structure and 3D trajectory of dynamic vehicles is obtained. The dynamic foreground and static background are blended to generate an initial blended scene; Global Appearance Coordination Module: After introducing a learnable appearance feature vector for each Gaussian point of the static background, it uses the original spherical harmonic coefficients of the static background, the appearance feature vector, the target environment conditions, and the viewpoint embedding to construct a global color decoder. After generating appearance pseudo-real values ​​under target environmental conditions using a two-dimensional lighting prior teacher model, a three-dimensional knowledge distillation strategy is used to train the global color decoder to obtain an optimized global color decoder. Semantic logic control module: After constructing a state transition model based on traffic light image data, it combines preset semantic logic to convert the traffic light area in the initial fusion scene into traffic state pseudo-true values, and obtains the traffic light spherical harmonic coefficients through L1 loss optimization. Online rendering module: Based on real-time roadside video stream data, it predicts the current environmental conditions in real time, processes the predicted current environmental conditions through an optimized global color decoder to obtain the target static background; after determining the traffic state based on the three-dimensional trajectory, it matches the traffic light ball harmonic coefficients corresponding to the traffic state; based on the target static background, the matched traffic light ball harmonic coefficients, and dynamic vehicles, it performs rendering to generate the target video stream.

[0015] Furthermore, the semantic logic control module, in conjunction with preset semantic logic, converts the traffic light area in the initial fusion scene into pseudo-real values ​​of traffic state, and obtains the traffic light spherical harmonic coefficients through L1 loss optimization, specifically as follows: A comprehensive index is performed on the traffic light groups at the target intersection, and a set of mutually exclusive global traffic states is predefined. A set of dedicated spherical harmonic coefficients is created for each traffic state. Gaussian points belonging to traffic lights in the static background are identified and a set of traffic light Gaussian points is formed. In the initial fusion scene, the traffic light area is located and cropped in the corresponding rendering view. The cropped traffic light area is converted into a traffic state pseudo-real value using a state transition model. The mask L1 loss value between the traffic light image rendered with the spherical harmonic coefficients corresponding to the traffic state and the traffic state pseudo-real value is minimized to obtain the optimized traffic light spherical harmonic coefficients.

[0016] Furthermore, the global appearance coordination module uses a three-dimensional knowledge distillation strategy to train the global color decoder to obtain an optimized global color decoder, which is achieved by minimizing the reconstruction loss between the global color decoder-rendered image and the appearance pseudo-true value.

[0017] Beneficial effects The dynamic and static scene fusion method provided by this invention is achieved through four stages: high-fidelity dynamic and static decoupling reconstruction and initial fusion, global appearance coordination based on 3D knowledge distillation, traffic light control based on semantic logic, and online rendering. It solves the problems of poor rendering quality under sparse perspectives and visual and logical inconsistencies caused by dynamic and static fusion in the prior art, and realizes high-fidelity 4D scene data generation with dual visual and logical consistency under any new perspective.

[0018] The static and dynamic scene fusion system provided by this invention constructs a high-quality static background model and dynamic foreground assets through an initial fusion scene generation module and completes the initial fusion. It uses a global appearance coordination module to solve the problem of visual appearance inconsistency and a semantic logic control module to eliminate traffic logic conflicts. Finally, it generates high-fidelity 4D scene data with dual consistency in vision and logic under any new perspective. This solves the problem of high-fidelity rendering under sparse perspectives, reduces data acquisition costs, and provides high-quality and diversified training data for autonomous driving models. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a dynamic and static scene fusion method based on spatiotemporal appearance and semantic logic, as described in Example 1.

[0020] Figure 2 Example 1 illustrates the impact of scene coordination on the fusion results of dynamic and static scenes under different road conditions.

[0021] Figure 3 The effects of merging dynamic and static scenes obtained using different methods are shown. Detailed Implementation

[0022] The following examples are intended to illustrate the present invention, and not to further limit the invention.

[0023] Example 1 This invention provides a method for fusing dynamic and static scenes based on spatiotemporal appearance and semantic logic, such as... Figure 1 As shown, it includes: S1: Acquire multi-view static image data of the target intersection, and use the motion reconstruction structure algorithm and the three-dimensional Gaussian sputtering model for training to obtain a static background containing geometric parameters and appearance parameters; Real-time video stream data from the roadside is acquired, and through dynamic vehicle segmentation, 3D reconstruction, and trajectory estimation, a dynamic foreground containing the 3D structure and 3D trajectory of dynamic vehicles is obtained. The dynamic foreground and static background are merged to generate an initial blended scene.

[0024] In the static background reconstruction, preferably, the acquisition of multi-view static image data of the target intersection, and the training of a motion recovery structure algorithm and a three-dimensional Gaussian sputtering model to obtain a static background containing geometric and appearance parameters, specifically involves: After acquiring multi-view static image data of the target intersection, the motion reconstruction structure algorithm is used to process the data to generate a sparse point cloud. After initializing the three-dimensional Gaussian sputtering model with the sparse point cloud, the model is trained to obtain a static background containing geometric parameters and appearance parameters.

[0025] In dynamic foreground reconstruction, preferably, the acquisition of real-time roadside video stream data, followed by dynamic vehicle segmentation, 3D reconstruction, and trajectory estimation, yields a dynamic foreground containing the 3D structure and 3D trajectory of the dynamic vehicles, specifically as follows: Dynamic vehicles are segmented from the acquired real-time roadside video stream data. A 3D reconstruction algorithm is used to restore the 3D structure of the dynamic vehicles. The 3D trajectory of the dynamic vehicles is estimated by calculating the intersection of the ray emitted from the optical center of the roadside camera through the 2D position of the dynamic vehicle and the ground plane of the static scene.

[0026] In scene fusion, preferably, the fusion of dynamic foreground and static background to generate an initial fused scene is as follows: Retrieve the corresponding vehicle model from the preset vehicle library based on the timestamp, and use the vehicle position parameters and orientation parameters in the 3D trajectory to perform Gaussian transformation on all Gaussian points in the vehicle model. Then merge the transformed dynamic Gaussian points with the static background Gaussian points to generate the initial fused scene.

[0027] In one specific embodiment, the process of high-fidelity dynamic and static scene decoupling reconstruction and initial fusion is as follows: S11: Acquire dense multi-view static image data of the target intersection (such as the RoadSight+ dataset), process the dense images using Structure-from-Motion (SfM) technology to generate sparse point clouds, initialize a 3D Gaussian Splatting (3DGS) model using the sparse point clouds, train the 3DGS model until convergence, and obtain a static scene base containing geometric and appearance parameters (denoted as...). And freeze all geometric and appearance parameters of the static scene base to keep them unchanged.

[0028] S12: Acquire real-time sparse observation video streams from roadside infrastructure cameras (e.g., RoadSight data), segment dynamic vehicle targets from the video stream using 2D segmentation models (e.g., GroundedSAM), and utilize single-image / few-view data. Figure 33D reconstruction models (such as Trellis) recover detailed 3D structural assets of dynamic vehicles. By calculating the intersection of rays emitted from the optical center of the infrastructure camera through the 2D position of the dynamic vehicle and the ground plane of the pre-reconstructed static scene, the 3D trajectory (including position parameters) of the dynamic vehicle is accurately estimated. and orientation parameters ).

[0029] S13: At any given timestamp The system retrieves the corresponding vehicle model (located in the normalized object center space) from the pre-generated 3DGS vehicle library. Based on the estimated dynamic 3D trajectory of the vehicle, it calculates the mean position of all Gaussian points in the vehicle model. Apply a rigid transformation to the covariance matrix ( The dynamic Gaussian points are rotated and placed in the world coordinate system. The transformed dynamic Gaussian points are merged with the frozen static background Gaussian points. The occlusion relationship is handled by the inherent depth sorting mechanism of the 3DGS rasterizer, and the initial blended scene is generated.

[0030] The pre-generated 3DGS vehicle library is a standardized, reusable dynamic 3D vehicle asset library built in advance. The 3D trajectory of the dynamic vehicle can provide constraints on the placement position, driving direction, and time synchronization of the 3DGS vehicle model, so that the vehicle model can adapt to the dynamic requirements of the specific scene and achieve the fusion effect of "dynamic vehicles driving on real trajectories in static scenes".

[0031] This invention adopts a reconstruction paradigm of "decoupling static and dynamic elements and dividing and conquering". By introducing a pre-collected dense static scene base and combining it with a 3D generation model for dynamic objects, it effectively solves the problems of blurring and geometric artifacts caused by traditional methods when reconstructing the vehicle view under sparse roadside view. It realizes cross-domain high-fidelity rendering from the roadside monitoring view to the vehicle view, and ensures the geometric and physical authenticity of the generated data.

[0032] S2: After introducing a learnable appearance feature vector for each Gaussian point of the static background, a global color decoder is constructed using the original spherical harmonic coefficients of the static background, the appearance feature vector, the target environment conditions, and the viewpoint embedding. A two-dimensional illumination prior teacher model is constructed using urban street view data. After generating appearance pseudo-real values ​​under the target environment conditions using the two-dimensional illumination prior teacher model, a three-dimensional knowledge distillation strategy is used to train the global color decoder, resulting in an optimized global color decoder.

[0033] To address the visual inconsistencies between static backgrounds and dynamic foregrounds due to lighting and weather (e.g., the conflict between nighttime vehicles and daytime backgrounds), a three-dimensional knowledge distillation strategy is employed to train a global color decoder, resulting in an optimized global color decoder. This optimization is achieved by minimizing the reconstruction loss between the rendered image and the appearance pseudo-true value of the global color decoder.

[0034] In addition, it includes building an environmental condition recognizer using roadside overhead view image data, which can accurately identify the current environmental conditions from a roadside overhead view without sky based solely on features such as road surface reflection and shadows.

[0035] In one specific embodiment, the global appearance coordination process based on 3D knowledge distillation is as follows: S21: Train a conditional image-to-image translation model on a large-scale urban street view dataset ( (e.g., img2img-turbo), this model supports scene images to arbitrary target environment conditions. (e.g., sunny daytime, nighttime, etc.) translation, serving as a two-dimensional lighting prior teacher model for a three-dimensional scene.

[0036] Train an infrastructure environmental condition recognizer to accurately identify current environmental conditions from a roadside overhead view without sky, based solely on features such as road surface reflections and shadows. .

[0037] S22: With the 3DGS geometry properties frozen, for each Gaussian point in the static background Introduce a learnable appearance feature vector .

[0038] Static background original spherical harmonic coefficients It can only express the appearance under one environmental condition (such as sunny daytime) and cannot adapt to the real-time environment of dynamic foregrounds (such as nighttime or rainy days). Therefore, a global color decoder based on neural rendering is constructed, which receives the original spherical harmonic coefficients of the static background. Appearance feature vector Embedding of target environmental conditions and perspective embedding As input, predict the new spherical harmonic coefficients. Obtainable based on environmental conditions Global color decoder for adjusting the appearance of static backgrounds ).

[0039] S23: Employing a 3D knowledge distillation strategy, the reconstruction loss between the global color decoder-rendered image and the appearance pseudo-true value under the target environment conditions generated by the 2D lighting prior teacher model is minimized. This is used to optimize the global color decoder, appearance feature vector, and view embedding; where view embedding... It is used to absorb random style deviations in the 2D lighting prior teacher model under different viewpoints, ensuring 3D consistency.

[0040] This invention proposes a global appearance coordination mechanism based on 3D knowledge distillation. It utilizes the prior knowledge of a 2D generative large model to guide the optimization of 3D scenes. Through an environmental condition classifier, it perceives the lighting and weather features of the dynamic foreground in real time and "soft distills" these features into the static 3DGS scene. This allows the static background to adaptively adjust its lighting and texture according to the environmental changes of the dynamic vehicle, eliminating visual incongruity and ensuring spatiotemporal consistency. This solves the problem of inconsistent visual appearance between static backgrounds and dynamic foregrounds.

[0041] S3: After constructing a state transition model based on traffic light image data, and combining it with preset semantic logic, the traffic light area in the initial fusion scene is converted into a pseudo-true value of traffic state, and the traffic light spherical harmonic coefficients are obtained through L1 loss optimization.

[0042] To address the logical conflicts caused by frozen traffic light states in static scenarios (e.g., vehicles passing under a red light), this invention proposes a traffic light control method based on semantic logic, the specific operation of which is as follows: A comprehensive index is performed on the traffic light groups at the target intersection, and a set of mutually exclusive global traffic states is predefined. A set of dedicated spherical harmonic coefficients is created for each traffic state. Gaussian points belonging to traffic lights in the static background are identified and a set of traffic light Gaussian points is formed. In the initial fusion scene, the traffic light area is located and cropped in the corresponding rendering view. The cropped traffic light area is converted into a traffic state pseudo-real value using a state transition model. The mask L1 loss value between the traffic light image rendered with the spherical harmonic coefficients corresponding to the traffic state and the traffic state pseudo-real value is minimized to obtain the optimized traffic light spherical harmonic coefficients.

[0043] In one specific embodiment, the traffic light control process based on semantic state caching is as follows: S31: Train a dedicated state transition model on a real-world set of high-brightness red / green traffic light images, denoted as... (For example, CycleGAN-turbo) is used to enhance traffic light brightness and perform high-quality red-green state transitions. This model is then used to generate pseudo-ground images of traffic conditions.

[0044] S32: Index the traffic lights at the target intersection and predefine a set of mutually exclusive global traffic states. , North-south traffic green light / East-west traffic red light; Green light for left turns (north / south) / Red light for left turns (east / west); East-west traffic green light / North-south traffic red light; Green light for left turns (east / west) / Red light for north / south turns.

[0045] Identify all 3DGS Gaussian points belonging to traffic lights in a static background and form a dedicated Gaussian point set. Subsequent adjustments will only be made to the appearance parameters of these Gaussian points, without affecting other geometric or appearance elements in the scene.

[0046] For each predefined traffic state (like , Create and independently optimize a dedicated set of spherical harmonic coefficients. (i.e., a multi-state caching strategy). This set of parameters is only responsible for describing the appearance of the traffic light in the corresponding state, and the parameters for each state are stored independently and do not interfere with each other.

[0047] S33: Render the initial fused scene to obtain a view image containing traffic lights; use an open-set object detection model (such as GroundingDINO) to accurately locate the positions of all traffic lights in this rendered view and crop out the corresponding traffic light areas. .

[0048] For the current traffic state to be optimized, a state transition model is used. cropping area Transform to pseudo-real values ​​of traffic states (i.e., the appearance of traffic lights corresponding to the traffic states); calculate the spherical harmonic coefficients corresponding to the current traffic state to be optimized. The L1 mask loss between the rendered traffic light image and the pseudo-real value of traffic conditions ( By minimizing the mask L1 loss, this process only changes the appearance of the traffic lights without affecting the geometry, and finally obtains high-quality spherical harmonic coefficients for all states.

[0049] In other words, by predefining traffic states as semantic logic instructions, indexing the corresponding spherical harmonic coefficients, and then presenting the logical appearance of traffic lights through Gaussian points, it not only achieves flexible switching of traffic logic, but also ensures high fidelity in appearance presentation and stability of geometric structure, thus solving the problem of conflict between traffic light state freezing and vehicle driving logic.

[0050] This invention addresses the logical conflicts caused by traffic light state freezing in fusion scenarios by proposing a semantic traffic light control mechanism. This mechanism can semantically locate traffic light areas in 3D scenes and perform controllable generative redrawing of traffic light states based on real-time traffic flow logic. This ensures that the generated 4D scene is strictly correct in causal logic, avoids the model learning incorrect traffic rules, and significantly improves the training value of the data.

[0051] like Figure 2 As shown, under different road conditions (daytime and nighttime), without destroying the original high-fidelity geometric structure, scene coordination is used: S2 global appearance coordination and S3 traffic light control to achieve flexible redrawing and control of the static scene appearance and semantic logic, ultimately generating 4D autonomous driving scene data that has both high visual realism and conforms to strict physical and traffic logic.

[0052] S4: Based on real-time roadside video stream data, predict the current environmental conditions in real time, process the predicted current environmental conditions through an optimized global color decoder to obtain the target static background; after determining the traffic state according to the three-dimensional trajectory, match the traffic light ball harmonic coefficients corresponding to the traffic state; based on the target static background, the matched traffic light ball harmonic coefficients and dynamic vehicles, render to generate the target video stream.

[0053] In one specific embodiment, the online data generation and rendering process is as follows: S41: For each frame of dynamic observation data, the infrastructure environmental condition identifier is used to predict environmental conditions in real time. The predicted Input global color decoder The static background instantly adjusts its global appearance (such as changing from daytime to nighttime, or from sunny to rainy) to match the dynamic foreground, achieving soft coordination.

[0054] S42: Determine the required traffic logic state based on the simulation timestamp or preset logic. Directly replace the spherical harmonic coefficients of the traffic light Gaussian points with pre-optimized cached spherical harmonic coefficients. This step achieves zero computational overhead and artifact-free traffic state switching, ensuring strict synchronization between background logic and dynamic vehicle behavior.

[0055] S43: Renders a static background with adjusted appearance, traffic lights with synchronized status, and dynamic vehicles together, and outputs a high-fidelity video stream from any new perspective.

[0056] The dynamic and static scene fusion method provided by this invention achieves its goals through four stages: high-fidelity dynamic and static decoupling reconstruction and initial fusion, global appearance coordination based on 3D knowledge distillation, traffic light control based on semantic logic, and online rendering. This solves the problems of poor rendering quality under sparse perspectives and visual-logical inconsistencies caused by dynamic-static fusion in existing technologies, achieving high-fidelity 4D scene data generation with dual visual and logical consistency under any new perspective. Furthermore, this method eliminates the need for costly, repetitive 3D reconstruction for every weather condition and traffic state; a single static scene acquisition is sufficient. Combined with low-cost roadside monitoring streams, generative editing technology can infinitely generate rich 4D scenes covering different lighting conditions and traffic states, significantly reducing data acquisition costs. It can also efficiently generate complex interactive and long-tail scene data urgently needed for autonomous driving, effectively bridging the gap between simulation and reality.

[0057] To further demonstrate the effectiveness of the method of the present invention, such as Figure 3 As shown, four scenarios were selected, and three methods were compared. The first column is the input image, the second column is the traditional method (Street-GS), the third column is the method of this invention but without using the scene coordination proposed in this invention, and the fourth column is the effect of the method of this invention.

[0058] This invention also provides a dynamic and static scene fusion system based on spatiotemporal appearance and semantic logic, comprising: Initial fusion scene generation module: used to acquire multi-view static image data of the target intersection, and use motion recovery structure algorithm and 3D Gaussian sputtering model training to obtain static background containing geometric parameters and appearance parameters; Real-time video stream data from the roadside is acquired, and through dynamic vehicle segmentation, 3D reconstruction, and trajectory estimation, a dynamic foreground containing the 3D structure and 3D trajectory of dynamic vehicles is obtained. The dynamic foreground and static background are blended to generate an initial blended scene; Global Appearance Coordination Module: After introducing a learnable appearance feature vector for each Gaussian point of the static background, it uses the original spherical harmonic coefficients of the static background, the appearance feature vector, the target environment conditions, and the viewpoint embedding to construct a global color decoder. After generating appearance pseudo-real values ​​under target environmental conditions using a two-dimensional lighting prior teacher model, a three-dimensional knowledge distillation strategy is used to train the global color decoder to obtain an optimized global color decoder. Semantic logic control module: After constructing a state transition model based on traffic light image data, it combines preset semantic logic to convert the traffic light area in the initial fusion scene into traffic state pseudo-true values, and obtains the traffic light spherical harmonic coefficients through L1 loss optimization. Online rendering module: Based on real-time roadside video stream data, it predicts the current environmental conditions in real time, processes the predicted current environmental conditions through an optimized global color decoder to obtain the target static background; after determining the traffic state based on the three-dimensional trajectory, it matches the traffic light ball harmonic coefficients corresponding to the traffic state; based on the target static background, the matched traffic light ball harmonic coefficients, and dynamic vehicles, it performs rendering to generate the target video stream.

[0059] Furthermore, the semantic logic control module, in conjunction with preset semantic logic, converts the traffic light area in the initial fusion scene into pseudo-real values ​​of traffic state, and obtains the traffic light spherical harmonic coefficients through L1 loss optimization, specifically as follows: A comprehensive index is performed on the traffic light groups at the target intersection, and a set of mutually exclusive global traffic states is predefined. A set of dedicated spherical harmonic coefficients is created for each global traffic state. Gaussian points belonging to traffic lights in the static background are identified and a set of traffic light Gaussian points is formed. In the initial fusion scene, the traffic light area is located and cropped in the corresponding rendering view. The cropped traffic light area is converted into a traffic state pseudo-real value using a state transition model. The mask L1 loss value between the traffic light image rendered with the spherical harmonic coefficients corresponding to the traffic state and the traffic state pseudo-real value is minimized to obtain the optimized traffic light spherical harmonic coefficients.

[0060] Furthermore, the global appearance coordination module uses a three-dimensional knowledge distillation strategy to train the global color decoder to obtain an optimized global color decoder, which is achieved by minimizing the reconstruction loss between the global color decoder-rendered image and the appearance pseudo-true value.

[0061] The static and dynamic scene fusion system provided by this invention constructs a high-quality static background model and dynamic foreground assets through an initial fusion scene generation module and completes the initial fusion. It uses a global appearance coordination module to solve the problem of visual appearance inconsistency and a semantic logic control module to eliminate traffic logic conflicts. Finally, it generates high-fidelity 4D scene data with dual consistency in vision and logic under any new perspective. This solves the problem of high-fidelity rendering under sparse perspectives, reduces data acquisition costs, and provides high-quality and diversified training data for autonomous driving models.

[0062] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for fusing dynamic and static scenes based on spatiotemporal appearance and semantic logic, characterized in that, include: S1: Acquire multi-view static image data of the target intersection, and use the motion reconstruction structure algorithm and the three-dimensional Gaussian sputtering model for training to obtain a static background containing geometric parameters and appearance parameters; Real-time video stream data from the roadside is acquired, and through dynamic vehicle segmentation, 3D reconstruction, and trajectory estimation, a dynamic foreground containing the 3D structure and 3D trajectory of dynamic vehicles is obtained. The dynamic foreground and static background are blended to generate an initial blended scene; S2: After introducing a learnable appearance feature vector for each Gaussian point of the static background, a global color decoder is constructed using the original spherical harmonic coefficients of the static background, the appearance feature vector, the target environment conditions, and the viewpoint embedding. After generating appearance pseudo-real values ​​under target environmental conditions using a two-dimensional lighting prior teacher model, a three-dimensional knowledge distillation strategy is used to train the global color decoder to obtain an optimized global color decoder. S3: After constructing a state transition model based on traffic light image data, and combining it with preset semantic logic, the traffic light area in the initial fusion scene is converted into a pseudo-true value of traffic state, and the traffic light spherical harmonic coefficients are obtained through L1 loss optimization. S4: Based on real-time video stream data from the roadside, predict the current environmental conditions in real time, and process the predicted current environmental conditions through an optimized global color decoder to obtain the target static background; After determining the traffic state based on the three-dimensional trajectory, the harmonic coefficients of the traffic light bulbs corresponding to the traffic state are matched. Based on the target static background, the matched traffic light spherical harmonic coefficients, and dynamic vehicles, rendering is performed to generate the target video stream.

2. The dynamic and static scene fusion method based on spatiotemporal appearance and semantic logic according to claim 1, characterized in that, S3, combined with preset semantic logic, converts the traffic light area in the initial fusion scene into pseudo-real values ​​of traffic state, and obtains the traffic light spherical harmonic coefficients through L1 loss optimization, specifically: A comprehensive index is performed on the traffic light groups at the target intersection, and a set of mutually exclusive global traffic states is predefined. A set of dedicated spherical harmonic coefficients is created for each traffic state. Gaussian points belonging to traffic lights in the static background are identified and a set of traffic light Gaussian points is formed. In the initial fusion scene, the traffic light area is located and cropped in the corresponding rendering view. The cropped traffic light area is converted into a traffic state pseudo-real value using a state transition model. The mask L1 loss value between the traffic light image rendered with the spherical harmonic coefficients corresponding to the traffic state and the traffic state pseudo-real value is minimized to obtain the optimized traffic light spherical harmonic coefficients.

3. The dynamic and static scene fusion method based on spatiotemporal appearance and semantic logic according to claim 1, characterized in that, S2 describes the use of a three-dimensional knowledge distillation strategy to train a global color decoder, resulting in an optimized global color decoder. This optimization is achieved by minimizing the reconstruction loss between the rendered image and the appearance pseudo-true value of the global color decoder.

4. The dynamic and static scene fusion method based on spatiotemporal appearance and semantic logic according to claim 1, characterized in that, S2 also includes an environmental condition recognizer built using roadside overhead view image data.

5. The dynamic and static scene fusion method based on spatiotemporal appearance and semantic logic according to claim 1, characterized in that, S1 describes the process of fusing the dynamic foreground and static background to generate an initial fused scene, specifically as follows: Retrieve the corresponding vehicle model from the preset vehicle library based on the timestamp, and use the vehicle position parameters and orientation parameters in the 3D trajectory to perform Gaussian transformation on all Gaussian points in the vehicle model. Then merge the transformed dynamic Gaussian points with the static background Gaussian points to generate the initial fused scene.

6. The dynamic and static scene fusion method based on spatiotemporal appearance and semantic logic according to claim 1, characterized in that, S1 describes acquiring real-time roadside video stream data, performing dynamic vehicle segmentation, 3D reconstruction, and trajectory estimation to obtain a dynamic foreground containing the 3D structure and trajectory of dynamic vehicles, specifically: Dynamic vehicles are segmented from the acquired real-time roadside video stream data. A 3D reconstruction algorithm is used to restore the 3D structure of the dynamic vehicles. The 3D trajectory of the dynamic vehicles is estimated by calculating the intersection of the ray emitted from the optical center of the roadside camera through the 2D position of the dynamic vehicle and the ground plane of the static scene.

7. The dynamic and static scene fusion method based on spatiotemporal appearance and semantic logic according to claim 1, characterized in that, S1 describes acquiring multi-view static image data of the target intersection, training it using a motion reconstruction structure algorithm and a 3D Gaussian sputtering model to obtain a static background containing geometric and appearance parameters, specifically: After acquiring multi-view static image data of the target intersection, the motion reconstruction structure algorithm is used to process the data to generate a sparse point cloud. After initializing the three-dimensional Gaussian sputtering model with the sparse point cloud, the model is trained to obtain a static background containing geometric parameters and appearance parameters.

8. A dynamic-static scene fusion system based on spatiotemporal appearance and semantic logic, characterized in that, include: Initial fusion scene generation module: used to acquire multi-view static image data of the target intersection, and use motion recovery structure algorithm and 3D Gaussian sputtering model training to obtain static background containing geometric parameters and appearance parameters; Real-time video stream data from the roadside is acquired, and through dynamic vehicle segmentation, 3D reconstruction, and trajectory estimation, a dynamic foreground containing the 3D structure and 3D trajectory of dynamic vehicles is obtained. The dynamic foreground and static background are blended to generate an initial blended scene; Global Appearance Coordination Module: After introducing a learnable appearance feature vector for each Gaussian point of the static background, it uses the original spherical harmonic coefficients of the static background, the appearance feature vector, the target environment conditions, and the viewpoint embedding to construct a global color decoder. After generating appearance pseudo-real values ​​under target environmental conditions using a two-dimensional lighting prior teacher model, a three-dimensional knowledge distillation strategy is used to train the global color decoder to obtain an optimized global color decoder. Semantic logic control module: After constructing a state transition model based on traffic light image data, it combines preset semantic logic to convert the traffic light area in the initial fusion scene into traffic state pseudo-true values, and obtains the traffic light spherical harmonic coefficients through L1 loss optimization. Online rendering module: Based on real-time roadside video stream data, it predicts the current environmental conditions in real time, processes the predicted environmental conditions through an optimized global color decoder, and obtains the target static background; After determining the traffic state based on the three-dimensional trajectory, the harmonic coefficients of the traffic light bulbs corresponding to the traffic state are matched. Based on the target static background, the matched traffic light spherical harmonic coefficients, and dynamic vehicles, rendering is performed to generate the target video stream.

9. The dynamic and static scene fusion system based on spatiotemporal appearance and semantic logic according to claim 8, characterized in that, The semantic logic control module, in conjunction with preset semantic logic, converts the traffic light area in the initial fusion scene into pseudo-real values ​​of traffic state, and obtains the traffic light spherical harmonic coefficients through L1 loss optimization, specifically: A comprehensive index is performed on the traffic light groups at the target intersection, and a set of mutually exclusive global traffic states is predefined. A set of dedicated spherical harmonic coefficients is created for each traffic state. Gaussian points belonging to traffic lights in the static background are identified and a set of traffic light Gaussian points is formed. In the initial fusion scene, the traffic light area is located and cropped in the corresponding rendering view. The cropped traffic light area is converted into a traffic state pseudo-real value using a state transition model. The mask L1 loss value between the traffic light image rendered with the spherical harmonic coefficients corresponding to the traffic state and the traffic state pseudo-real value is minimized to obtain the optimized traffic light spherical harmonic coefficients.

10. The dynamic and static scene fusion system based on spatiotemporal appearance and semantic logic according to claim 8, characterized in that, The global appearance coordination module uses a three-dimensional knowledge distillation strategy to train the global color decoder, resulting in an optimized global color decoder. This is achieved by minimizing the reconstruction loss between the rendered image and the appearance pseudo-true value of the global color decoder.

Images