Single-step diffusion enhanced 4d gaussian sputter reconstruction method and system based on monocular video

Abstract

A monocular video-based single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method and system, the method comprising: constructing a canonical space reference Gaussian point cloud, defining a globally shared low-dimensional SE(3) motion basis; linearly combining the low-dimensional SE(3) motion basis through motion weight coefficients to calculate the deformation of the optimizable attributes in the Gaussian primitive at any time, realizing the decoupling and regularization of camera motion and object non-rigid deformation; in the training iteration, selecting a target pose from candidate new poses based on information gain; rendering a rough image under the target pose using the current 4D Gaussian representation, generating a high-fidelity restored image based on the rough image fusion through a single-step diffusion model; using the real image of the original acquisition view and the high-fidelity restored image as pseudo-true values, constructing an extended supervised view set, constructing a weighted rendering loss function based on the extended supervised view set, and combining the temporal and geometric constraint regularization terms for joint optimization to update the 4D scene representation parameters.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and graphics, and in particular to a method and system for single-step diffusion-enhanced 4D Gaussian sputtering reconstruction based on monocular video. Background Technology

[0002] Reconstructing high-fidelity, freely viewpoint-explorable 4D representations of dynamic scenes is a cutting-edge topic in computer vision and graphics. Mainstream methods can be categorized into geometric methods based on multi-view reconstruction, implicit methods based on Neural Radiation Fields (NeRF), and emerging explicit representation methods. Multi-view geometric methods rely on highly overlapping synchronized videos and struggle to handle non-rigid deformations. In recent years, implicit neural representations, represented by NeRF and its variants, have achieved breakthroughs in view synthesis. Their 4D extensions (such as D-NeRF and HyperNeRF) can model dynamic scenes, but suffer from problems such as long optimization times, slow rendering, and sensitivity to sparse training views. 4D Gaussian Sputtering (4DGS), as the latest explicit dynamic scene representation, achieves real-time rendering through differentiable rasterization, providing a new and efficient framework for dynamic reconstruction. However, its reconstruction quality and stability under extremely sparse observations, such as monocular videos, still need improvement.

[0003] In the field of 4D dynamic reconstruction based on monocular video, the closest existing technology is a "canonical space 4DGS + sparse supervision" scheme. This scheme first obtains the initial geometry and camera pose through structure of motion reconstruction (SfM) and monocular depth estimation, defines a set of reference 3D Gaussians in canonical space, and drives their temporal SE(3) deformation through a set of learnable Gaussian-by-Gaussian vectors. This method only uses the real image frames provided by the monocular video itself as supervision signals, and optimizes the Gaussian properties and motion parameters through differentiable rendering and photometric loss. This method directly utilizes the efficient rendering characteristics of 4DGS and attempts to constrain the solution space through canonical space and basis motion decomposition.

[0004] However, the aforementioned methods and similar techniques have significant limitations. First, their supervision signals rely entirely on extremely sparse monocular view sequences. This highly underdetermined supervision method easily leads to optimization getting trapped in local optima, resulting in severe geometric distortions (such as "floating object" artifacts), temporal jitter, and texture blurring, especially in occluded and fast-moving regions. Second, the lack of an effective mechanism for injecting new view information makes it difficult for the model to generalize to unobserved viewpoints, limiting the completeness and practicality of the reconstruction. Finally, the optimization process is sensitive to initial pose and quality. In long sequences or complex non-rigid motions, errors easily accumulate, leading to a lack of long-term temporal consistency and detail fidelity in the reconstruction results. Therefore, there is an urgent need for a novel monocular 4D dynamic reconstruction method that can effectively utilize strong visual priors, enhance cross-view consistency, and achieve stable optimization and detail enhancement. Summary of the Invention

[0005] To achieve the above objectives, this invention provides a single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method and system based on monocular video. It addresses the inherent underdeterminism of monocular observation by providing efficient spatiotemporal decoupling, information gain optimization, and multi-constraint training mechanisms, thereby achieving high-fidelity and long-range stable reconstruction results. It is not only applicable to short video sequences but can also effectively handle long-sequence complex dynamic scenes.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] The first aspect is a single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method based on monocular video, which includes the following steps:

[0008] S1. Obtain the monocular video sequence and perform geometric initialization;

[0009] S2. Construct a standard spatial reference Gaussian point cloud and define a globally shared low-dimensional SE(3) motion basis. Linearly combine the low-dimensional SE(3) motion basis with motion weight coefficients to calculate the deformation of optimizable attributes in Gaussian primitives at any time, thereby achieving decoupling and regularization of camera motion and non-rigid deformation of objects.

[0010] S3. During the training iteration, the target pose is selected from the candidate new poses based on information gain;

[0011] S4. Render a rough image in the target pose using the current 4D Gaussian representation, and generate a high-fidelity repaired image based on the rough image fusion using a single-step diffusion model.

[0012] S5. Using the original captured real image and the high-fidelity restored image as pseudo-real values, construct an extended supervised view set, construct a weighted rendering loss function based on the extended supervised view set, and perform joint optimization by combining temporal and geometric constraint regularization terms to update the 4D scene representation parameters.

[0013] S6. Output the optimized 4D scene representation.

[0014] Preferably, S1 includes:

[0015] The structure-of-motion method is used to reconstruct monocular video sequences to obtain the initial camera pose and sparse 3D point cloud;

[0016] A monocular depth estimation model is used to generate a dense depth map for each frame; a long-range point tracking algorithm is used to extract a stable 2D trajectory point set across frames.

[0017] Initialize a baseline 3D Gaussian point cloud in the canonical space. Each Gaussian primitive contains an optimizable mean position, rotation quaternion, scaling vector, opacity, spherical harmonic coefficient, and motion weight coefficient.

[0018] As a preferred embodiment, initializing the baseline 3D Gaussian point cloud in the normed space in S1 includes: using the sparse 3D point cloud as the initial mean position; and using the 2D trajectory point set to estimate the initial motion coefficients through triangulation or projection constraints to provide prior guidance for dynamic decoupling.

[0019] Preferably, S2 includes:

[0020] The globally shared low-dimensional SE(3) motion basis is parameterized by a multilayer perceptron, with normalized time as input, and outputs the motion basis trajectory containing rotation angular components and translational components; the motion weight coefficient is used to represent the contribution of each Gaussian primitive to different motion basis, and satisfies the normalization constraint; a neural texture field is superimposed on each Gaussian.

[0021] As a preferred embodiment, S2 further includes obtaining the rigid body motion parameters corresponding to each low-dimensional SE(3) motion basis according to the current time input; weighting and fusing the rigid body motion parameters of multiple low-dimensional SE(3) motion basis according to the motion weight coefficient to obtain the overall motion parameters of the Gaussian primitive at the current time; updating the spatial position and rotation state of the Gaussian primitive according to the overall motion parameters, thereby realizing its dynamic deformation over time.

[0022] Preferably, S3 includes:

[0023] Sample multiple reference poses from the currently optimized camera trajectory;

[0024] Generate a set of candidate new poses starting from the initial camera pose;

[0025] Calculate the ray overlap rate of each candidate new pose and retain the candidate poses that meet the preset overlap conditions;

[0026] For the retained candidate poses, calculate and render an uncertainty map, and select the pose with higher uncertainty as the target pose;

[0027] Multiple intermediate transition poses are generated from the reference pose to the target pose, forming a temporal condition sequence.

[0028] Preferably, in S4, the diffusion model conditions include: a structural skeleton condition based on a coarse image with an appropriate amount of Gaussian noise superimposed, and an appearance reference condition based on adjacent real frames.

[0029] Preferably, in S5, the temporal and geometric constraint regularization terms include: motion smoothing loss for constraining the magnitude of parameter changes in adjacent time frames; and depth consistency loss for ensuring that the predicted depth remains consistent with the reference depth.

[0030] As a preferred approach, the joint optimization in S5 adopts a phased optimization strategy:

[0031] The first stage involves optimizing geometry-related parameters and freezing neural texture field and appearance branch parameters.

[0032] The second stage involves jointly optimizing the neural texture field, appearance attributes, and geometric parameters.

[0033] In the third stage, temporal consistency constraints are strengthened and the learning rate is decayed.

[0034] Secondly, a single-step diffusion-enhanced 4D Gaussian sputtering reconstruction system based on monocular video includes:

[0035] The data acquisition module is used to acquire monocular video sequences and perform geometric initialization;

[0036] The scene representation module is used to construct a standard spatial reference Gaussian point cloud and define a globally shared low-dimensional SE(3) motion basis. The low-dimensional SE(3) motion basis is linearly combined by motion weight coefficients to calculate the deformation of optimizable attributes in Gaussian primitives at any time, thereby achieving decoupling and regularization of camera motion and non-rigid deformation of objects.

[0037] The pose selection module is used to select the target pose from candidate new poses based on information gain during training iterations.

[0038] The diffusion model enhancement module is used to render a coarse image in the target pose using the current 4D Gaussian representation, and generate a high-fidelity repaired image based on the coarse image fusion through a single-step diffusion model.

[0039] The pseudo-truth value self-supervised training module is used to construct an extended supervised view set by using the real image from the original acquisition view and the high-fidelity restored image as pseudo-truth values. Based on the extended supervised view set, a weighted rendering loss function is constructed and jointly optimized by temporal and geometric constraint regularization terms to update the 4D scene representation parameters.

[0040] Train a robust optimization module to output an optimized 4D scene representation;

[0041] The single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method based on monocular video is used to implement the single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method and its steps based on monocular video as described in the first aspect.

[0042] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0043] 1. Unlike traditional monocular dynamic reconstruction methods that use high-dimensional implicit volume rendering networks or frame-by-frame independent optimization, this invention adopts the technical solution of "standard spatial 4D Gaussian explicit representation + globally shared SE(3) low-dimensional motion basis". Through the low-dimensional motion basis, complex dynamic deformation is structurally modeled, and the camera motion and non-rigid deformation of the object are effectively decoupled and regularized. Thus, under the condition of monocular height underdeterminacy, parameter redundancy and optimization instability are significantly reduced, long-term geometric consistency and reconstruction accuracy are improved, while maintaining the high-speed rendering advantage brought by explicit Gaussian rasterization, so as to achieve 4D dynamic reconstruction with both high quality and high efficiency.

[0044] 2. Unlike traditional methods that rely solely on real-view observations for weakly supervised training from a single perspective, this invention introduces a "single-step diffusion model cross-view repair + pseudo-ground value multi-view distillation" approach. By using generative visual priors to perform image-level completion and detail enhancement on new perspectives, and using the repair results as pseudo-ground values ​​in joint optimization, the originally sparse monocular supervision is transformed into structurally consistent multi-view strong constraints. This effectively alleviates problems such as geometric distortion, texture blurring, and missing information in occluded areas, significantly improving detail representation and cross-view consistency.

[0045] 3. Unlike traditional methods that employ random novel perspective sampling or fixed training strategies, this invention proposes a technical solution of "intelligent pose selection based on information gain + quality-weighted self-supervised mechanism + phased robust optimization". It uses ray overlap rate and rendering uncertainty to jointly screen high-value poses and implements differentiated weight control for real and pseudo-real views. Combined with motion smoothing and depth consistency regularization, it achieves stable convergence, thereby improving training efficiency, suppressing the propagation of generation errors, enhancing temporal continuity, and significantly improving generalization ability and engineering controllability in complex dynamic scenes. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the overall process of the method in Embodiment 1 of the present invention. Detailed Implementation

[0047] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0048] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0049] This invention provides a 4D Gaussian Splatting (4DGS) framework enhanced with a single-step diffusion model for reconstructing spatiotemporally continuous dynamic scenes from uncontrolled monocular videos. This framework innovatively integrates generative priors with geometric representations, providing efficient spatiotemporal decoupling, information gain optimization, and multi-constraint training mechanisms to address the inherent underdeterminism of monocular observations, thereby achieving high-fidelity and long-range stable reconstruction results. This innovative method is not only applicable to short video sequences but also effectively handles long sequences of complex dynamic scenes, such as real-world recordings containing non-rigid deformations or multi-object interactions.

[0050] Example 1:

[0051] 4D dynamic scene reconstruction of monocular video with continuously changing camera pose (taking an indoor pedestrian walking scene shot with a handheld mobile phone as an example).

[0052] This embodiment uses a typical real monocular video as input: the video is approximately 12 seconds long, totaling about 3000 frames, with a resolution of 1920×1080, and was shot handheld by a regular smartphone. During the shooting process, the camera pose continuously changes with the photographer's natural walking, including obvious translation, rotation, and slight hand tremors; in the scene, a pedestrian walks naturally from the left side of the frame to the right side, accompanied by arm swings and obvious non-rigid deformation of clothing; the background includes static objects such as tables and chairs and a small amount of natural lighting variation. This video only provides a monocular RGB image sequence, without any depth information, camera pose annotations, or multi-view synchronization data, belonging to a typical underdetermined, uncontrolled, long-term dynamic scene, which is extremely challenging.

[0053] like Figure 1 The method shown is a single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method based on monocular video, which includes the following steps:

[0054] S1. Obtain the monocular video sequence and perform geometric initialization.

[0055] S1-1, Reading the monocular video sequence ,in For the first Frame RGB images. Sparse Structure Motion Recovery (SfM) reconstruction of the entire sequence was performed using COLMAP (or its accelerated version GLOMAP). During reconstruction, feature extraction employed a strategy combining Scale Invariant Feature Transform (SIFT) and SuperPoint neural network features; image matching combined sequential matching and loop closure matching; and Ceres solver was used for bundle adjustment optimization to obtain high-precision initial camera pose. And sparse 3D point clouds; among which, It is a 3×3 rotation matrix. It is a three-dimensional translation vector.

[0056] S1-2. To obtain richer scene information, this invention integrates two powerful existing perception priors:

[0057] Dense Depth Prior: Utilizes advanced monocular depth estimation models (such as Depth Anything V2) to generate dense depth maps for each frame, providing pixel-level geometric inference.

[0058] Long-range 2D trajectories: Long-range point tracking algorithms (such as TAPIR) are used to extract stable, cross-frame 2D trajectory point sets across the entire video sequence. These trajectories are the key basis for subsequent decoupling and constraint of 3D motion.

[0059] S1-3. Initialize the baseline 3D Gaussian point cloud in canonical space. Specifically, the sparse 3D point cloud obtained in S1-1 is directly used as the initial mean position of the Gaussian primitive. To ensure the geometric skeleton aligns with the real scene, the initial motion coefficients of the Gaussian primitives are estimated using the cross-frame stable 2D trajectory point set extracted in S1-2 through triangulation or projection constraints. This provides prior guidance for subsequent dynamic decoupling. The total number of Gaussians is initialized to approximately 500,000, and each Gaussian primitive contains the following optimizable properties:

[0060] Mean position

[0061] Rotation Quaternion (Normalization)

[0062] Scaling vector (All components are positive values)

[0063] Opacity

[0064] Spherical harmonic coefficient (Highest order 3, 48 dimensions in total, used for view-dependent color modeling)

[0065] Motion weight coefficient For subsequent use In this embodiment, global motion bases are linearly combined. All Gaussian globals share 8 SE(3) (special Euclidean group) motion bases.

[0066] S2. Construct a standard spatial reference Gaussian point cloud and define a globally shared low-dimensional SE(3) motion basis. Linearly combine the low-dimensional SE(3) motion basis with motion weight coefficients to calculate the deformation of optimizable attributes in Gaussian at any time, thereby achieving decoupling and regularization of camera motion and non-rigid deformation of objects.

[0067] 4D Gaussian representation construction (core geometric representation), specifically including:

[0068] S2-1. Establish a standardized spatial benchmark Gaussian point cloud. All geometric and appearance attributes are persistently shared over time and do not change independently over time.

[0069] S2-2, Define the global SE(3) motion basis: A learnable motion base trajectory Each It is a time-varying SE(3) transform, parameterized by a lightweight MLP. The MLP takes time t as input and outputs a 6-dimensional vector containing 3-dimensional rotation axis angle components and 3-dimensional translation components. The output is parameterized by a lightweight multilayer perceptron (MLP) containing 4 hidden layers, each with 128 dimensions. The input is a normalized time. ).

[0070] S2-3. At any given time, the spatial deformation of each Gaussian primitive is determined by multiple globally shared motion bases SE(3) through motion weight coefficients. The results were calculated together. Specifically, each motion basis SE(3) represents a rigid body motion that varies with time, which includes rotational and translational components and is used to describe the basic motion patterns in the scene.

[0071] For any Gaussian primitive, its motion state at the current moment is not modeled independently, but is obtained by fusing multiple motion bases according to their corresponding weights. The weight coefficients are used to characterize the degree of response of the Gaussian primitive to different motion bases and satisfy the normalization constraint, that is, the sum of all weights is 1 and each weight is a non-negative value.

[0072] In the calculation process, the rigid body motion parameters corresponding to each motion basis are first obtained based on the current time input. Then, the motion parameters of multiple motion basis are weighted and fused according to the weight coefficient of the Gaussian primitive to obtain the overall motion parameters of the Gaussian primitive at the current time. Finally, the spatial position and rotation state of the Gaussian primitive are updated according to the overall motion parameters, thereby realizing its dynamic deformation over time.

[0073] By combining the low-dimensional motion basis and motion weight coefficients as described above, Gaussian primitives with similar motion patterns can be automatically grouped into rigid or near-rigid groups, thereby achieving structured modeling of complex non-rigid motions. Simultaneously, by using the cross-frame two-dimensional trajectory points extracted in S1-2 as geometric consistency constraints, the projected trajectory of the Gaussian primitive on the image plane remains consistent with the observed trajectory, effectively suppressing geometric drift and error accumulation during long sequence reconstruction.

[0074] S2-4, Superimpose a neural texture field onto each Gaussian. The texture field uses a 6-layer MLP, each layer with 256 dimensions, and LeakyReLU activation. The input features are [normalized view orientation]. Normalized time Gaussian Local Encoding Output the RGBA values ​​for the corresponding time and viewpoint.

[0075] S3. During the training iteration, the target pose is selected from the candidate new poses based on information gain;

[0076] Information gain-based intelligent pose selection

[0077] S3-1. In each iteration of training, the camera pose is jointly optimized based on the current optimization results of the 4D Gaussian representation, and a set of camera trajectories that are dynamically updated with training is maintained. Specifically, the initial camera pose is obtained by the structure of motion recovery method in S1; in subsequent training processes, the camera extrinsic parameters are used as one of the optimizable variables, and iteratively updated under the combined effects of photometric consistency loss, depth consistency loss, and geometric constraints, thereby obtaining progressively optimized camera trajectories. Based on this, several reference poses are randomly sampled from the currently optimized set of camera trajectories as anchor points, typically 3 to 8, for the generation of subsequent candidate new poses and information gain evaluation, to provide stable geometric and appearance priors.

[0078] S3-2. Generate a set of candidate new poses: using the initial camera pose... Starting from the spherical coordinate system, 30 candidate poses are uniformly sampled. In this embodiment, the yaw angle is ±30°, the pitch angle is ±20°, and the radial distance is ±0.3m, which constitute the candidate pool.

[0079] S3-3, Calculate the information gain index – ray overlap rate:

[0080]

[0081] in, Indicates from pose A collection of three-dimensional rays that originate from and pass through image pixels; This indicates the number of rays in the set or the number of rays after discrete sampling; This indicates the overlap between two sets of light rays in space.

[0082] In actual calculations, each ray is discretized into several points in three-dimensional space, and then based on spatial proximity (e.g., Euclidean distance less than a threshold)... Determine whether two light rays overlap, and thus approximately calculate the size of the intersection between sets of light rays.

[0083] In practical calculations, to ensure the spatial accuracy of ray sampling points, this invention employs an adaptive depth-guided strategy to determine the sampling location: In the initial training phase, the 3D coordinates are directly obtained by projecting the dense depth prior map generated by S1 to overcome the initial geometric inaccuracies; as training iterates, the sampling is dynamically switched to using the depth map rendered in real-time by the current 4D Gaussian model to accurately reflect the fine-grained structure of the scene. Based on the above depth information, each ray is discretized along the optical axis into several 3D points, and sampling is performed based on spatial proximity (i.e., the Euclidean distance between sampling points is less than a preset threshold). It determines whether light rays in different poses overlap, thereby approximating the effective intersection size between light ray sets.

[0084] Based on the above definition, only candidate poses that satisfy the following constraints are retained:

[0085]

[0086] in, and These represent the preset minimum and maximum overlap thresholds (e.g., 0.7 and 0.9).

[0087] This constraint ensures that the selected pose neither overlaps excessively with existing observations, leading to information redundancy, nor suffers from insufficient geometric constraints due to low overlap, thus maintaining the stability of the reconstruction while providing new perspective information.

[0088] Simultaneously, an uncertainty map is calculated and rendered for each candidate pose. The uncertainty map is represented by pixel-level variance or prediction confidence, and poses or regions with higher uncertainty are prioritized. These high uncertainty regions usually correspond to the most dynamic, detailed, or geometrically complex parts of the scene (such as the swinging of a pedestrian's arm or the deformation of clothing folds), thereby maximizing the information gain brought by each new viewpoint.

[0089] S3-4. Perform linear interpolation smoothing on the selected novel pose: from the reference pose... Reach the target pose Several intermediate transition poses are generated, forming a strong temporal conditional sequence. This strong temporal conditional sequence provides the diffusion model with a continuous and smooth camera motion path as conditional input when generating images from new viewpoints, thereby enhancing the temporal consistency and generation stability of cross-viewpoint inpainting. The strong temporal conditional sequence helps the diffusion model maintain the smoothness and spatiotemporal continuity of viewpoint changes when generating high-fidelity inpainted images, avoiding rendering artifacts or temporal jitter caused by abrupt viewpoint changes.

[0090] S4. Render a rough image in the target pose using the current 4D Gaussian representation, and generate a high-fidelity repaired image based on the rough image fusion using a single-step diffusion model.

[0091] Single-step diffusion model image-level cross-view restoration specifically includes:

[0092] S4-1. Utilize the complete 4D Gaussian representation parameters in the current optimization (i.e., a set of parameters including Gaussian properties, motion coefficients, motion basis MLP parameters, neural texture field, etc., denoted as...). Render a rough image at the target pose selected in S3.

[0093] S4-2, Conditions for constructing a diffusion model:

[0094] Structural skeleton: based on a rough rendered image And add an appropriate amount of Gaussian noise.

[0095] Appearance reference: Extract CLIP-ViT global and local features from adjacent ground truth frames.

[0096] S4-3. Single-step denoising is performed using a diffusion model based on Stable Diffusion 2.1 to output a high-fidelity restored image. This image significantly outperforms the direct rendering result in terms of texture detail, color realism, and view dependency. Furthermore, because it has adjacent real frames as appearance references, the high-fidelity restored image obtained through the diffusion model is superior. It is highly consistent with the original monocular video under the existing perceptual prior.

[0097] S5. Using the original captured real image and the high-fidelity restored image as pseudo-real values, construct an extended supervised view set, construct a weighted rendering loss function based on the extended supervised view set, and perform joint optimization by combining temporal and geometric constraint regularization terms to update the 4D scene representation parameters.

[0098] Pseudo-truth value self-supervised multi-view optimization (core training loop), specifically including:

[0099] S5-1, Supervision View Construction and Joint Loss Design

[0100] In each training frame First, a set of supervised views is constructed. In addition to the real images corresponding to the original acquisition viewpoints, several pseudo-truth views generated by the repair module are introduced to form an extended supervised set:

[0101]

[0102] in, Indicates time Real images, The first one generated by the S4 repair module A pseudo-true value image This indicates the number of pseudo-true viewpoints generated during the repair process. Expanding supervision through multi-viewpoints can alleviate the geometric degradation problem caused by viewpoint sparsity in monocular training and enhance the model's generalization ability to occluded and new viewpoint regions.

[0103] During the rendering phase, the model is based on the scene representation parameters at the current moment. Generate the corresponding rendered view And construct a weighted rendering loss function:

[0104]

[0105] in, It is a moment Real images, It is a moment The rendered image is represented by the parameters of the current 4D Gaussian scene. Rendered and generated for use with real images Compare them. It is a moment The first one generated by the S4 repair module A pseudo-true value image, It is a moment The parameters are represented by the current 4D Gaussian scene. The rendered generation A pseudo-ground value rendering image, used to compare with the pseudo-ground value image. Compare them. and These are the weighting coefficients for the original ground truth and the repaired view, used to balance the contribution ratios of real and pseudo-supervision. They are typically set... This ensures that real observation data dominates the training process, while also enhancing view coverage by utilizing repaired views.

[0106] Building upon this, temporal and geometric constraint regularization terms are introduced to improve reconstruction stability and structural consistency. First, the motion smoothing loss is defined:

[0107]

[0108] This is used to constrain the variation of geometric and appearance parameters between adjacent time frames, thereby avoiding jitter or deformation discontinuity issues in dynamic scenes.

[0109] Secondly, a depth consistency loss is introduced, and supervision is applied only under the original camera pose:

[0110]

[0111] This constraint ensures consistency between the model's predicted depth and the reference depth, enhancing the physical plausibility of the geometric structure. Furthermore, a Trajectory Reprojection Loss is constructed: using the 2D trajectory point set extracted in S1-2, the distance between the current 4D Gaussian scene's position projected onto the image plane at the corresponding time and the observed trajectory points is calculated and used as an additional geometric regularization term. A total loss function is added to enhance the spatiotemporal continuity of the motion of dynamic objects.

[0112] Taking into account all the above losses, the overall optimization objective is:

[0113]

[0114] in , This is the regularization weighting coefficient.

[0115] S5-2, Phased Optimization Strategy

[0116] To improve training stability and avoid gradient interference between different attributes, the overall optimization process adopts a phased strategy, with a total of 8000 iterations.

[0117] In the first stage (Stage 1), only geometry-related parameters, including attributes such as spatial location, scale, and opacity, are optimized, while the neural texture field and appearance branch parameters are frozen. The goal of this stage is to prioritize the convergence of stable geometric structures and reduce the interference of appearance information on spatial layout learning, thereby establishing a reliable initial scene representation.

[0118] In Stage 2, the texture field and appearance attribute parameters are unfrozen and jointly optimized with the geometric parameters. This stage focuses on improving the ability to express appearance details, ensuring that color, reflection, and texture distribution are fully aligned with the geometric structure to achieve high-quality rendering results.

[0119] In the third stage (Stage 3), the temporal consistency constraint is further strengthened, and the learning rate is decayed, eventually reducing it to... By refining and optimizing with smaller step sizes, the model can achieve a smoother and more stable convergence state in dynamically changing regions, thereby obtaining continuous and consistent spatiotemporal reconstruction results.

[0120] S6 outputs an optimized 4D scene representation, used for dynamic scene rendering at any time and from any new perspective.

[0121] S6-1. After optimization and convergence, output the complete 4D scene representation: normalized spatial reference Gaussian point cloud + 8 global SE(3) motion bases + neural texture field MLP + optimized camera trajectory.

[0122] S6-2, at any time Any new perspective During rendering, the deformation state of each Gaussian is first calculated using motion basis and weight coefficients, and then differentiable rasterization rendering is performed to obtain high-quality dynamic video.

[0123] Reconstruction results of this embodiment:

[0124] For the aforementioned 12-second pedestrian walking video, this method successfully reconstructed a geometrically stable, textured, and temporally consistent 4D scene. From any new perspective, the pedestrian's arm swings naturally and smoothly, and the folds of clothing change dynamically and correctly with time and perspective.

[0125] On the reserved 20-frame test set, this method achieved excellent quantitative performance: PSNR reached 28.7 dB, an improvement of 1.8 dB compared to the NeRF method; LPIPS decreased to 0.042, a reduction of 42% compared to NeRF; and Warp Error was only 0.018 pixels, a reduction of 65%. Meanwhile, in terms of rendering efficiency, this method only requires approximately 18 milliseconds for single-frame rendering on an RTX 4090, supporting real-time interaction at 50+ FPS. This significantly improves the speed bottleneck of traditional NeRF-like methods in dynamic human reconstruction, providing a high-fidelity, efficient, and real-time roaming solution for practical applications such as AR / VR content generation, virtual try-on, and film special effects preview.

[0126] This invention has been fully validated on multiple publicly available benchmark datasets and self-collected real mobile phone videos. Experiments show that removing the intelligent pose selection module increases the geometric error by 31%; removing single-step diffusion repair significantly degrades texture details. Experiments demonstrate that this invention has significant advantages in monocular video scenarios with continuously changing camera poses.

[0127] Example 2:

[0128] A single-step diffusion-enhanced 4D Gaussian sputtering reconstruction system based on monocular video includes:

[0129] The data acquisition module is used to acquire monocular video sequences and perform geometric initialization;

[0130] The scene representation module is used to construct a standard spatial reference Gaussian point cloud and define a globally shared low-dimensional SE(3) motion basis. The low-dimensional SE(3) motion basis is linearly combined by motion weight coefficients to calculate the deformation of optimizable attributes in Gaussian primitives at any time, thereby achieving decoupling and regularization of camera motion and non-rigid deformation of objects.

[0131] The pose selection module is used to select the target pose from candidate new poses based on information gain during training iterations.

[0132] The diffusion model enhancement module is used to render a rough image in the target pose using the current 4D Gaussian representation, and generate a high-fidelity repaired image by fusing multimodal conditions based on the rough image and through a single-step diffusion model.

[0133] The pseudo-truth value self-supervised training module is used to construct an extended supervised view set by using the real image from the original acquisition view and the high-fidelity restored image as pseudo-truth values. Based on the extended supervised view set, a weighted rendering loss function is constructed and jointly optimized by temporal and geometric constraint regularization terms to update the 4D scene representation parameters.

[0134] Train a robust optimization module to output an optimized 4D scene representation.

[0135] Scene Representation Module: This method uses a canonical spatial architecture to represent dynamic scenes. In the canonical frame, the scene is modeled as a set of globally distributed 3D Gaussian primitives, each containing attributes such as mean position, rotation, opacity, scaling, color, and spherical harmonic coefficients. These attributes remain persistent and shared throughout the video sequence, thus naturally ensuring geometric consistency across time. For scene motion, this method does not directly model motion independently for each Gaussian point, but introduces a small set of globally shared, learnable SE(3) motion basis trajectories. These motion basis trajectories are parameterized in the form of 6 degrees of freedom (rotation + translation), representing the basic motion patterns (translation, rotation, scaling, etc.) in the scene, and can be naturally extended to non-rigid deformation. At any time, the position and rotation of each Gaussian point are obtained by linearly weighting these global motion basis trajectories through its unique motion coefficients. This low-dimensional motion basis representation effectively regularizes the motion trajectory, automatically clustering points with similar motion into rigid or near-rigid groups, significantly reducing parameter redundancy, improving optimization stability and long-sequence generalization ability. To further enhance detail representation, this method overlays a neural texture field onto each Gaussian primitive. This texture field is parameterized by a multilayer perceptron (MLP) network, taking into account view orientation, timestamps, and the Gaussian's own attribute encoding, and outputting the corresponding RGBA color value. This neural texture field can adaptively capture complex view-dependent lighting, temporal variations, and material details, significantly surpassing traditional fixed-color Gaussian representations. In the rendering stage, this method employs explicit trajectory rasterization: first, the motion trajectory of each Gaussian point is projected onto the image plane using a differential renderer, resulting in a pixel-level 3D world coordinate map; simultaneously, monocular depth priors (such as the Depth Anything model) and long-range 2D trajectory priors (such as the TAPIR algorithm) are fused, providing strong geometric constraints for subsequent optimization.

[0136] Pose Selection Module: To avoid noise and computational waste caused by blindly generating synthetic views, this method proposes an intelligent pose selection strategy based on information gain. First, based on the currently reconstructed geometry, the ray overlap rate between candidate new poses and existing poses is calculated. Only candidate poses with an overlap rate within a reasonable range (e.g., 0.8 to 0.9) are selected to ensure that the new perspective provides sufficient new information without excessive redundancy with existing observations. Simultaneously, this method progressively interpolates the target pose from the reference pose, forming a smooth conditional path and further enhancing the generation stability of the diffusion model. Furthermore, the pose priority can be dynamically adjusted by incorporating uncertainty indicators such as rendering variance, enabling the entire selection process to adapt to the complexity of different scenes.

[0137] Diffusion Model Enhancement Module: This method introduces a single-step diffusion model for image-level cross-view restoration. The specific process is as follows:

[0138] Starting from the candidate new poses output by the pose selection module, a rough new perspective image is rendered using the current 4DGS representation, which serves as the structural skeleton for repair.

[0139] Visual features of adjacent real frames are extracted (multi-scale semantic and edge information is extracted through a pre-trained visual Transformer) as strong conditional information;

[0140] Through a single-step denoising process, a high-fidelity restored image is predicted directly from a coarse image with an appropriate amount of noise added. This single-step design significantly reduces the computational cost of traditional multi-step diffusion while retaining the powerful visual priors of the diffusion model. It can effectively repair rendering artifacts, restore texture details, and maintain consistency across viewpoints.

[0141] The pseudo-ground truth self-supervised training module treats the restored high-fidelity new viewpoint image as a "pseudo-ground truth," which, together with the original monocular ground truth image, constitutes a multi-view supervision set. For each frame, this method not only uses the original ground truth image for supervision but also introduces 2–4 carefully selected restored viewpoint images as additional supervision signals. The 4DGS parameters are jointly optimized through photometric consistency loss, SSIM loss, and perceptual loss (based on VGG features), thereby distilling the rich information from generative priors back into geometric representations, achieving a fundamental shift from single-view constraints to strong multi-view constraints.

[0142] Training a robust optimization module: To mitigate the minor 3D inconsistencies that may be introduced by the diffusion model, this method employs a quality-weighted supervision mechanism: assigning a higher weight (1.0) to the original ground truth region and an adaptively lower weight (0.3–0.5, dynamically adjusted according to confidence) to the repair region, and applying a mask to prevent noise propagation. Simultaneously, motion smoothing regularization, gradient clipping, and phased learning rate scheduling are introduced to ensure stable convergence of the optimization process. The entire training process is divided into three stages: geometric initialization, texture refinement, and temporal consistency enhancement, ultimately achieving high-quality end-to-end reconstruction.

Claims

1. A single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method based on monocular video, characterized in that, Includes the following steps: S1. Obtain the monocular video sequence and perform geometric initialization; S2. Construct a standard spatial reference Gaussian point cloud and define a globally shared low-dimensional SE(3) motion basis. Linearly combine the low-dimensional SE(3) motion basis with motion weight coefficients to calculate the deformation of optimizable attributes in Gaussian primitives at any time, thereby achieving decoupling and regularization of camera motion and non-rigid deformation of objects. S3. During training iterations, the target pose is selected from candidate new poses based on information gain; including: Sample multiple reference poses from the currently optimized camera trajectory; Generate a set of candidate new poses starting from the initial camera pose; Calculate the ray overlap rate for each candidate new pose and retain the candidate poses that meet the preset overlap conditions; For the retained candidate poses, calculate and render an uncertainty map, and select the pose with higher uncertainty as the target pose; Multiple intermediate transition poses are generated from the reference pose to the target pose, forming a temporal conditional sequence; S4. Render a rough image in the target pose using the current 4D Gaussian representation, and generate a high-fidelity repaired image based on the rough image fusion through a single-step diffusion model. S5. Using the original captured image and the high-fidelity restored image as pseudo-ground values, construct an extended supervised view set, and build a weighted rendering loss function based on the extended supervised view set: in, It is a moment Real images, It is a moment The rendered image is represented by the parameters of the current 4D Gaussian scene. Rendering generation, It is a moment The A pseudo-true value image, It is a moment The parameters are represented by the current 4D Gaussian scene. The rendered generation A pseudo-real value rendering image. and These are the weighting coefficients for the original truth value and the repaired view, respectively, used to balance the contribution ratios of real supervision and pseudo-supervision; It also combines temporal and geometric constraint regularization terms for joint optimization to update the 4D scene representation parameters; S6. Output the optimized 4D scene representation.

2. The method for single-step diffusion-enhanced 4D Gaussian sputtering reconstruction based on monocular video according to claim 1, characterized in that, S1 includes: The structure-of-motion method is used to reconstruct monocular video sequences to obtain the initial camera pose and sparse 3D point cloud; A monocular depth estimation model is used to generate a dense depth map for each frame; a long-range point tracking algorithm is used to extract a stable 2D trajectory point set across frames. Initialize a baseline 3D Gaussian point cloud in the canonical space. Each Gaussian primitive contains an optimizable mean position, rotation quaternion, scaling vector, opacity, spherical harmonic coefficient, and motion weight coefficient.

3. The method for single-step diffusion-enhanced 4D Gaussian sputtering reconstruction based on monocular video according to claim 2, characterized in that, The initialization of the baseline 3D Gaussian point cloud in the normalized space in S1 includes: using the sparse 3D point cloud as the initial mean position; and using the 2D trajectory point set to estimate the initial motion coefficients through triangulation or projection constraints to provide prior guidance for dynamic decoupling.

4. The single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method based on monocular video according to claim 1, characterized in that, S2 include: The globally shared low-dimensional SE(3) motion basis is parameterized by a multilayer perceptron, with normalized time as input, and outputs the motion basis trajectory containing rotation angular components and translational components; the motion weight coefficient is used to represent the contribution of each Gaussian primitive to different motion basis, and satisfies the normalization constraint; a neural texture field is superimposed on each Gaussian.

5. The single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method based on monocular video according to claim 4, characterized in that, S2 also includes obtaining the rigid body motion parameters corresponding to each low-dimensional SE(3) motion basis according to the current time input; weighting and fusing the rigid body motion parameters of multiple low-dimensional SE(3) motion basis according to the motion weight coefficient to obtain the overall motion parameters of the Gaussian primitive at the current time; updating the spatial position and rotation state of the Gaussian primitive according to the overall motion parameters, thereby realizing the dynamic deformation of the Gaussian primitive as time changes.

6. The method for single-step diffusion-enhanced 4D Gaussian sputtering reconstruction based on monocular video according to claim 1, characterized in that, In S4, the diffusion model conditions include: a structural skeleton condition based on a coarse image with an appropriate amount of Gaussian noise superimposed, and an appearance reference condition based on adjacent real frames.

7. The single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method based on monocular video according to claim 1, characterized in that, In S5, the temporal and geometric constraint regularization terms include: motion smoothing loss to constrain the magnitude of parameter changes in adjacent time frames; and depth consistency loss to ensure that the predicted depth remains consistent with the reference depth.

8. The method for single-step diffusion-enhanced 4D Gaussian sputtering reconstruction based on monocular video according to claim 1, characterized in that, The joint optimization in S5 adopts a phased optimization strategy: The first stage involves optimizing geometry-related parameters and freezing neural texture field and appearance branch parameters. The second stage involves jointly optimizing the neural texture field, appearance attributes, and geometric parameters. In the third stage, temporal consistency constraints are strengthened and the learning rate is decayed.

9. A single-step diffusion-enhanced 4D Gaussian sputtering reconstruction system based on monocular video, characterized in that, include: The data acquisition module is used to acquire monocular video sequences and perform geometric initialization; The scene representation module is used to construct a standard spatial reference Gaussian point cloud and define a globally shared low-dimensional SE(3) motion basis. The low-dimensional SE(3) motion basis is linearly combined by motion weight coefficients to calculate the deformation of optimizable attributes in Gaussian primitives at any time, thereby achieving decoupling and regularization of camera motion and non-rigid deformation of objects. The pose selection module is used to select the target pose from candidate new poses based on information gain during training iterations. The diffusion model enhancement module is used to render a coarse image in the target pose using the current 4D Gaussian representation, and generate a high-fidelity repaired image based on the coarse image fusion through a single-step diffusion model. The pseudo-truth value self-supervised training module is used to construct an extended supervised view set by using the real image from the original acquisition view and the high-fidelity restored image as pseudo-truth values. Based on the extended supervised view set, a weighted rendering loss function is constructed and jointly optimized by temporal and geometric constraint regularization terms to update the 4D scene representation parameters. Train a robust optimization module to output an optimized 4D scene representation; The single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method based on monocular video is used to implement the single-step diffusion-enhanced 4D Gaussian sputtering reconstruction method and its steps as described in claim 1.

Images