A two-dimensional Gaussian splashing method for cross-scale structural consistency and detail preservation in synthesis from a novel perspective
By employing multi-view training and cross-scale structural consistency supervision, the problems of aliasing and structural degradation in the 2D Gaussian splashing method under different sampling rates are solved, achieving a new perspective synthesis that preserves cross-scale structural consistency and detail.
Patent Information
- Application Number
- CN202611350096.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-02
- Publication Date
- 2026-11-17
AI Technical Summary
Existing 2D Gaussian splashing methods are prone to aliasing and structural degradation at different rendering sampling rates, and are difficult to reconstruct in areas with complex textures, fine structures and occlusion boundaries, making it difficult to maintain cross-scale structural consistency and detail representation.
By constructing multi-view training input, initializing a two-dimensional Gaussian scene representation, performing perspective-consistent rendering and multi-scale structural consistency supervision, locating local high-error regions, and generating new perspective images through cross-view consistency verification and local primitive updates.
It maintains image structural stability at different resolutions and sampling rates, enhances the ability to express details in complex regions, improves the quality of new perspective synthesis, and preserves the ability to express surface geometry.
Smart Images

Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision, computer graphics, and 3D scene representation technology, specifically to a cross-scale structural consistency and detail-preserving 2D Gaussian splashing method for new perspective synthesis, which is applicable to constructing 2D Gaussian scene representations from multi-view images with known camera poses and generating new perspective images from unobserved camera poses. Background Technology
[0002] Novel perspective synthesis constructs scene representations based on multi-view images and camera parameters, generating images even without observing camera poses. This approach has wide applications in digital twins, virtual reality, robot perception, 3D content generation, and autonomous driving. Existing methods primarily employ implicit radiative fields or explicit Gaussian primitives to represent scenes; the latter offers advantages such as fast training, high rendering efficiency, and ease of direct optimization.
[0003] 3D Gaussian splashing represents the scene using anisotropic Gaussian primitives in 3D space, achieving new perspective rendering through differentiable rasterization and transparency blending. 2D Gaussian splashing further represents the primitives as oriented 2D Gaussian disks embedded in local tangent planes in 3D space, making them easier to align with the local geometry of object surfaces, thus combining new perspective rendering with surface geometry representation capabilities.
[0004] However, existing 2D Gaussian splashing methods typically optimize pixel reconstruction errors at a fixed sampling rate. When the sampling rate, output resolution, or viewing distance of the test image differs from that during training, single-resolution supervision struggles to capture structural information at different scales, easily leading to aliasing, local blurring, and structural degradation in fine lines, regular textures, and distant regions.
[0005] Furthermore, reconstruction errors in complex textures, fine structures, occlusion boundaries, and high-frequency appearance areas exhibit significant locality. Global density control or primitive densification relying solely on gradient thresholds struggles to distinguish between truly difficult-to-reconstruct areas and pseudo-height errors caused by single-view occlusion, reflection, or noise, easily leading to insufficient local primitives, ineffective densification, or unreasonable added primitive parameters.
[0006] Some methods mitigate aliasing at different sampling rates through spatial or sampling filtering, but they primarily adjust the frequency response of Gaussian primitives. They neither directly constrain the structural consistency between the rendered and real images across multiple scales, nor can they supplement and calibrate primitives for locally high-error regions verified through multiple views. Therefore, it is necessary to combine cross-scale structural supervision and reliable local primitive management during the unified training process. Summary of the Invention
[0007] The purpose of this invention is to solve the problems of existing two-dimensional Gaussian splashing methods, which are prone to aliasing and structural degradation at different rendering sampling rates, as well as insufficient detail representation in areas that are difficult to reconstruct, such as complex textures, fine structures and occlusion boundaries. The invention provides a two-dimensional Gaussian splashing method that is cross-scale structural consistency and detail preservation for synthesis from new perspectives. To achieve the above objectives, the present invention adopts the following technical solution: S1. Construct multi-view training input and initialize a 2D Gaussian scene representation. Obtain multi-view images, camera intrinsic parameters, and camera extrinsic parameters of the same scene. Initialize a 2D Gaussian primitive set based on multi-view geometry or sparse point cloud. Each 2D Gaussian primitive contains a center position, two local tangential directions, corresponding scale, opacity, and spherical harmonic color coefficients. S2. Perform perspective-consistent 2D Gaussian differentiable rendering. For the selected training view, calculate the intersection points of pixel rays and the 2D Gaussian disk. Determine the contribution of primitives to pixels based on the Gaussian response at the intersection points. Sort the relevant primitives by depth and obtain the rendered image and rendered depth map through transparency blending. Calculate the photometric reconstruction loss based on the rendered image and the real image. S3. Construct a multi-scale structural consistency supervision system. This employs a scale set approach. Region downsampling is performed on the rendered image and the real image to construct corresponding rendered image pyramids and real image pyramids. Multi-scale structural similarity is calculated at the effective scale and a normalized multi-scale structural consistency loss is formed. S4. Locate local high-error regions. Calculate the pixel-wise average absolute color residual between the rendered image and the real image, divide the error map into multiple image blocks, filter candidate regions based on the average residual of the image blocks, and obtain a set of high-error pixels within the candidate regions using a local fractional threshold, thereby reducing the impact of differences in brightness and texture complexity in different regions on threshold selection; S5, Perform cross-view Figure 1 Consistency verification and local primitive update. Based on the camera center distance and viewing direction, a neighboring view is selected. High-error pixels are back-projected to 3D space according to the rendering depth and then projected to the neighboring view. The projection range, positive depth, depth consistency, and photometric consistency are jointly verified. Only reliable high-error regions are retained. Within the reliable high-error regions, 2D Gaussian primitives that contribute significantly to rendering are selected as source primitives. These primitives are then copied, their positions perturbed, their scales adjusted, their opacities adjusted, and their colors residuals injected. S6. Perform joint iterative optimization and generate new perspective images. A warm-up strategy is adopted to gradually increase the weight of multi-scale structural consistency loss. The photometric reconstruction loss, multi-scale structural consistency loss, depth distortion regularization term, and normal consistency regularization term constitute the overall training objective. The updated local 2D Gaussian primitives are merged into the global 2D Gaussian primitive set and participate in subsequent iterations until the preset termination condition is met. Based on the optimized 2D Gaussian primitive set and the target camera pose, a new perspective image of the target is generated.
[0008] The beneficial effects of this invention are as follows: First, the present invention constructs a multi-scale image pyramid that corresponds to the rendered image and the real image, and explicitly constrains the structural similarity of different scales in a unified objective function, so that the two-dimensional Gaussian scene representation maintains a more stable overall structure and local contour when the output resolution or sampling rate changes. Second, when the scale image size does not meet the structural similarity window requirements, the present invention skips the corresponding scale and renormalizes the weights of the remaining scales to avoid invalid scales participating in the calculation and causing unstable loss, thereby improving the adaptability of multi-scale structural supervision to different image sizes. Third, this invention locates difficult reconstruction areas by reconstructing residuals pixel by pixel and local fraction thresholds, and reduces the impact of pseudo-high error areas caused by single-view occlusion, reflection and noise on primitive management by jointly verifying the depth consistency and photometric consistency of neighboring views. Fourth, this invention replicates and calibrates high-contribution two-dimensional Gaussian primitives only within reliable high-error regions, and injects local average color residuals into the newly added primitives, enabling the newly added primitives to directly compensate for color information that has not yet been fully modeled during the initialization stage, thereby improving the expressive power of complex textures, fine structures and occluded boundary regions. Fifth, this invention employs a joint iterative mechanism of multi-scale structural consistency supervision and local primitive management. The former constrains cross-scale structural stability, while the latter enhances the expression of local details. Both improve the quality of new perspective synthesis without changing the two-dimensional Gaussian explicit surface representation, and maintain the surface geometric expression capability. Attached Figure Description
[0009] Figure 1 This is a schematic diagram of the overall process of the two-dimensional Gaussian splashing method for cross-scale structural consistency and detail preservation in synthesis from a new perspective, as proposed in this invention.
[0010] Figure 2 This is a schematic diagram illustrating the cross-scale structure preservation effect of the present invention under multi-scale training and multi-scale testing conditions.
[0011] Figure 3 This is a schematic diagram illustrating the novel perspective synthesis effect of the present invention in a real, unbounded scene.
[0012] Figure 4 This is a schematic diagram illustrating the geometric structure preservation effect of the present invention in surface normal rendering of an object. Detailed Implementation
[0013] To make the technical solution of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and embodiments. The described embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.
[0014] Example 1: Two-dimensional Gaussian splashing for cross-scale structural consistency and detail preservation in synthesis for novel perspectives; like Figure 1 As shown, this embodiment takes the multi-view images of the same static scene and the corresponding camera parameters as input to establish a two-dimensional Gaussian scene representation. The rendered image of the current training view is obtained through differentiable rendering. The rendered image and the real image are respectively sent to the multi-scale structure consistency supervision module, and the rendering residual, depth information and camera geometric information are sent to the local primitive management module. The updated two-dimensional Gaussian primitives are returned to the global primitive set and participate in the next round of iteration. In step S1, the training view set is obtained. Each training view includes a true color image. Camera intrinsic and extrinsic parameter matrices. Two-dimensional Gaussian primitive centers can be initialized using sparse point clouds obtained from multi-view geometry, and primitive orientations and initial scales can be determined based on local geometry. For the A two-dimensional Gaussian element, whose center position is denoted as . The local tangent plane is formed by mutually orthogonal tangent vectors. and Zhang Cheng, corresponding scale parameter is and Then the local plane coordinates Corresponding world coordinate position and homogeneous transformation matrix They are represented as follows: in, Let be the rotation matrix from the local coordinate system to the world coordinate system. Let be the scaling matrix. In normalized local coordinates, the two-dimensional Gaussian kernel is represented as: In step S2, for the current training view pixels The perspective-consistent intersection points of pixel rays and each 2D Gaussian disk are calculated to obtain the effective Gaussian response at the intersection points. The relevant two-dimensional Gaussian primitives are sorted according to their center depth, and pixel colors are obtained by using transparency blending: in, , and They represent the first The color, opacity, and pixel of a two-dimensional Gaussian element. The effective Gaussian response at that point, This represents the number of 2D Gaussian pixels involved in the current pixel blending. Perform this calculation on all pixels in the current view to obtain the rendered image. and rendering depth map A photometric reconstruction loss is constructed based on the rendered image and the real image: In step S3, multi-scale structural consistency supervision is performed. Let the scale set be... Here, scale 1 represents the original resolution, and the other scales represent the corresponding downsampling ratios. Region downsampling is employed. Process separately and This approach ensures that each low-resolution pixel is aggregated from its corresponding original image region, reducing the additional aliasing introduced by point sampling. This applies to the effective set of scales that meet the requirements for the multi-scale structural similarity window size. The multi-scale structural consistency loss is expressed as: in, For scale The weights are determined by the scale. When the image size at any scale is too small to meet the requirements of the multi-scale structural similarity window, that scale is skipped, and the image is renormalized based on the sum of the weights of the remaining scales to keep the loss magnitude stable between different training views. In step S4, error region localization is performed. This is done for each pixel within the image domain. Calculate the average absolute color residual for the three RGB channels: The color residuals of all pixels are used to construct the current view error map. This error map is then divided into image blocks of equal or adaptive size. Candidate regions are selected based on the average residual of each image block. Local [processing] is applied within each candidate region. The quantile threshold is selected based on high-error pixels, and the set of high-error pixels is represented as follows: in, For the local color residual within the candidate region quantile threshold, This is a set of pixels with high error. Using a local fractional threshold instead of a globally fixed threshold can reduce the filtering bias caused by differences in brightness and texture complexity in different areas of the scene; In step S5, cross-view execution is performed. Figure 1 Consistency verification. First, among the training views that satisfy the observation direction angle constraint, select the view whose camera center is closest to the current view's camera center as the nearest neighbor view. For any high-error pixel Utilizing rendering depth It is back-projected into 3D space, and then projected through the nearest view projection function. Projected onto a neighboring image plane: When the projection position When the projection point is located within the range of neighboring images, has positive depth, and satisfies the depth consistency constraint of neighboring views, bilinear interpolation is performed on the real image of the neighboring view to obtain... For pixels that satisfy the following photometric consistency condition, let the photometric consistency indicator function be: For pixels that do not meet the above conditions, let .in, This is the cross-view photometric consistency threshold. It is only applied if the cross-view photometric consistency is satisfied within the candidate region. Figure 1 When the number of pixels verified for consistency reaches a preset threshold, the candidate region is retained as a reliable high-error region. Within the reliable high-error region, a source primitive is selected based on the cumulative contribution of each 2D Gaussian primitive to the high-error pixel during the differentiable rendering process. The source primitive is then copied to generate a new 2D Gaussian primitive. The new 2D Gaussian primitive inherits the orientation parameters and higher-order spherical harmonic color coefficients of the source primitive, and its center position, scale, and opacity are initialized as follows: in, To perturb the sampling position within the local neighborhood of the source primitive, in order to prevent the newly added primitive from completely overlapping with the source primitive. and The scale and opacity of the newly added primitives are controlled separately. To compensate for local color information that is not yet fully modeled in the current 2D Gaussian scene representation during the initialization phase, the average color residual within the set of high-error pixels is calculated: Inject the average color residual into the DC spherical harmonic component of the newly added primitive: in, The constants of the DC spherical harmonic basis functions. Inject an intensity factor into the color residual. This indicates that the parameters are limited to a preset valid range; In step S6, joint training is performed. Since the primitive geometry and appearance are still unstable in the early stages of training, a warm-up strategy is used to control the weights of the multi-scale structural consistency loss. Where is the current iteration number, , where is the activation time of multi-scale structural consistency loss, and is the length of the preheating phase. This represents the maximum weight for the multi-scale structural consistency loss. In this embodiment, it can be set... The numerical values can be adjusted according to the scene image size and structural complexity. The overall training objective is expressed as: in, This is a depth distortion regularization term. For normality consistency regularization, and The corresponding weights are used. The center position, orientation, scale, opacity, and spherical harmonic color coefficients of the two-dimensional Gaussian element are updated through backpropagation.
[0015] In one specific implementation, the total number of training iterations is set to 30,000. Multi-scale structural consistency supervision is enabled starting from the 3,000th iteration. Local primitive management is executed every 500 iterations between the 10,000th and 15,000th iterations. The sampling-related parameters of the two-dimensional Gaussian primitives are updated every 100 iterations. The above iteration count and execution cycle are optional implementation parameters and can be adjusted according to the scene size, image resolution, video memory capacity, and convergence status.
[0016] After each local primitive management operation, the newly added 2D Gaussian primitives are merged with the existing global 2D Gaussian primitive set, and differentiable rendering and joint loss optimization continue. When the preset number of iterations or the overall training objective converges, the optimized 2D Gaussian scene representation is obtained. Given the target camera intrinsic and extrinsic parameters, a new perspective image of the target is generated using the same perspective-consistent rendering and transparency blending process as in the training phase.
[0017] like Figure 2 As shown, at full resolution, half resolution, quarter resolution, and eighth resolution, the present invention can maintain structural information such as object contours, fine holes, and regular textures, indicating that multi-scale structural consistency supervision can improve structural stability at different sampling rates.
[0018] like Figure 3As shown, in real-world scenes containing distant details, repetitive textures, and complex vegetation, the local structures generated by this invention are closer to the real images, demonstrating that local primitive management can enhance the detail representation of areas that are difficult to reconstruct.
[0019] like Figure 4 As shown, the surface normal distribution obtained by the present invention is more continuous in the local curved surface region, indicating that the surface geometric expression capability of two-dimensional Gaussian elements is maintained while improving the synthesis quality of new perspectives.
[0020] This invention is not limited to the embodiments described above. For those skilled in the art, without departing from the principles of this invention, variations on multi-scale sets, scale weights, image patch partitioning methods, local fractional thresholds, nearest neighbor view selection rules, and cross-view... Figure 1 Adjustments to the consistency threshold, local primitive update cycle, and various loss weights, as well as equivalent substitutions, modifications, and improvements, should all fall within the protection scope of this invention.
Claims
1. Obtain multiple view images of the same scene and the camera intrinsic and extrinsic parameters corresponding to each multiple view image. Based on the multiple view images, initialize a set of two-dimensional Gaussian elements located on a local tangent plane in three-dimensional space. The two-dimensional Gaussian elements include at least the center position, tangential direction, scale, opacity, and color parameters.
2. Select the current view from the training view, perform ray intersection calculation with the two-dimensional Gaussian disk, depth sorting and transparency blending on the two-dimensional Gaussian primitive set to obtain the rendered image and rendered depth map of the current view, and construct the photometric reconstruction loss based on the rendered image and the real image of the current view.
3. Perform region downsampling on the rendered image and the current view real image according to multiple downsampling scales to obtain the rendered image pyramid and the real image pyramid. Calculate the multi-scale structural consistency loss for the effective scale that meets the structural similarity window size requirements.
4. Calculate the pixel-by-pixel color residual between the rendered image and the current view real image and form an error map. Divide the error map into multiple image blocks and determine the candidate high error region and high error pixel set based on the average residual of the image blocks and the local fraction threshold.
5. Select a neighboring view for the current view based on the camera geometry. Back-project the high-error pixels to the three-dimensional space according to the rendering depth and project them to the neighboring view. Verify and retain reliable high-error regions through image range, positive depth, depth consistency and photometric consistency. Select two-dimensional Gaussian primitives that meet the preset conditions for rendering contribution from the reliable high-error regions as source primitives. Perform copying, position perturbation, scale adjustment, opacity adjustment and color residual injection on the source primitives to obtain an updated two-dimensional Gaussian primitive set.
6. Using structural consistency loss weights that gradually increase with the number of iterations, the photometric reconstruction loss, multi-scale structural consistency loss, depth distortion regularization term, and normal consistency regularization term are used to form the overall training objective. The updated two-dimensional Gaussian set is iteratively optimized until the preset termination condition is met. A new perspective image of the target is generated based on the optimized two-dimensional Gaussian set and the target camera pose.
