A three-dimensional gaussian representation optimization method and system for sparse view new view synthesis

Abstract

The application provides a three-dimensional Gaussian representation optimization method and system for sparse view angle new view angle synthesis, and belongs to the field of computer vision and three-dimensional reconstruction. A sparse view angle image set and camera parameters are acquired, and a three-dimensional Gaussian primitive is initialized; binocular reprojection geometric constraints are constructed, and monocular depth prior is acquired; pixel-level gating weights are generated according to at least binocular reprojection error, monocular depth gradient and Gaussian visibility; the weights are used to adaptively weight binocular consistency loss and monocular depth consistency loss, and Gaussian primitive parameters are jointly optimized in combination with color reconstruction loss; the pixel-level gating weights are aggregated into Gaussian-level reliability weights, and the splitting densification and pruning processes of the Gaussian primitive are synchronously regulated; and the optimized three-dimensional Gaussian scene representation is output for new view angle rendering.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, 3D reconstruction, neural rendering, and artificial intelligence, specifically relating to a method and system for optimizing 3D Gaussian representations for novel perspective synthesis based on sparse viewpoints. More particularly, it relates to a method and system for optimizing 3D scenes by integrating binocular consistency, online monocular depth prior, uncertainty gating, edge-guided Gaussian splitting, and annealing pruning in 3D Gaussian Splatting (3DGS) representation. Background Technology

[0002] Novel perspective synthesis aims to reconstruct a 3D representation of a scene from known viewpoint images and camera parameters, generating images from unobserved viewpoints. In existing techniques, neural radiation field methods typically utilize implicit neural networks to represent scene density and color; the 3DGS method, on the other hand, is based on sparse point clouds obtained from multi-view images and Structure from Motion (SfM), representing the scene as a large number of 3D Gaussian primitives. Each Gaussian primitive has parameters such as center position, scale, rotation, transparency, and color, and these parameters are optimized through differentiable rasterization rendering and image reconstruction loss. This type of method exhibits high rendering efficiency and visual quality in scenes with dense viewpoints or sufficient viewpoint coverage.

[0003] The first type of prior art related to this invention is based on 3DGS technology. This technology typically initializes 3D Gaussian centers based on SfM point clouds, updates Gaussian primitives through alternating optimization, densification, and pruning, and generates target viewpoint images using a visibility-aware rasterization method. It primarily relies on multi-view photometric consistency and sufficient input viewpoint coverage. When there are few input views, insufficient overlap between adjacent views, or sparse initial point clouds, relying solely on image reconstruction loss is insufficient to stably constrain the true depth and spatial distribution of Gaussian primitives.

[0004] The second type of prior art related to this invention is depth prior methods in sparse-view 3DGS. These techniques typically introduce monocular depth estimation results, depth smoothing terms, local depth normalization, or diffusion model constraints to mitigate geometric degradation under a limited number of viewpoints. For example, some methods impose correlation constraints between the monocular depth map and the rendered depth map, or utilize global-local depth normalization to reduce the impact of monocular depth scale uncertainty. While these methods can improve geometric stability under sparse viewpoints to some extent, monocular depth prediction itself may suffer from problems such as scale inconsistency, edge blurring, local distortion, and misjudgment of reflection areas.

[0005] The third type of prior art related to this invention is a binocular or multi-view consistency constraint method. This type of technology, based on camera pose and current depth estimation, back-projects reference viewpoint pixels into 3D space and then back onto the source viewpoint, constraining the 3D representation through color, features, depth, or disparity consistency. Partially sparse viewpoint 3DGS methods construct binocular pairs between adjacent viewpoints, guiding image deformation and visual perception through disparity. Figure 1 The consistency loss is optimized through self-supervised optimization, and transparency constraints can be combined to reduce Gaussian redundancy. This type of scheme can avoid relying entirely on external monocular depth priors, but the quality of binocular matching is greatly affected by low texture, repetitive texture, occlusion, viewpoint overlap ratio, and abrupt depth changes.

[0006] The closest existing technology to this invention can be summarized as follows: in sparse viewpoint 3DGS, monocular depth prior, binocular consistency loss, ordinary image gradient densification, and fixed threshold pruning are used individually or separately. The implementation of this type of scheme typically involves: first, initializing a 3D Gaussian scene representation; rendering the current viewpoint color map and depth map during training; calculating the loss by comparing the rendering results with the input image, monocular depth map, or reprojection results from adjacent viewpoints; and then densifying and pruning the Gaussian units according to image gradient, transparency, or visibility count. The above schemes have not fully addressed the reliability judgment problem of geometric supervision, nor have they systematically fed back the reliability judgment to the entire process of loss weights, Gaussian splitting, and pruning.

[0007] Existing basic 3DGS technology can form a relatively stable 3D representation when there are enough viewpoints. However, under sparse viewpoint input, the multi-viewpoint photometric consistency constraint is insufficient, and SfM point clouds may be sparse or incomplete, which can easily lead to technical problems such as Gaussian elements being in the wrong depth position, background collapse, blurred object boundaries, and floating Gaussians.

[0008] While existing depth prior methods can supplement geometric information, monocular depth estimation results suffer from absolute scale uncertainty and local unreliability. In areas with abrupt depth changes, weak textures, reflections, or occlusions, directly using monocular depth as equal-weighted supervision can easily introduce erroneous depths into the 3D Gaussian optimization process, leading to Gaussian center shifts, incorrect edge structures, or excessive smoothing of local structures.

[0009] Existing binocular consistency methods can utilize the geometric relationships between viewpoints for self-supervision, but they rely on effective overlapping regions and reliable matching. When the baselines of the reference and source viewpoints are too large or too small, textures are insufficient, or there is occlusion or duplicate textures, the binocular reprojection error itself may not accurately reflect the true geometric error. If binocular constraints are applied directly without distinguishing between reliable and unreliable regions, erroneous supervision may occur.

[0010] Existing Gaussian densification strategies are mostly triggered based on image residuals or gradient magnitudes, failing to adequately distinguish between true structural edges and pseudo-edges or floating artifacts caused by sparse viewpoints, potentially leading to excessive splitting of Gaussian units in low-confidence regions. Existing pruning strategies often employ fixed transparency, scale, or visibility thresholds, failing to adequately consider geometric consistency during the training phase and across viewpoints, easily resulting in early deletion of effective structures or later retention of incorrect Gaussians. Summary of the Invention

[0011] To address the shortcomings and deficiencies of existing technologies, this invention provides a method and system for optimizing 3D Gaussian representations for novel perspective synthesis under sparse viewpoints. Addressing the issues of insufficient geometric constraints, difficulty in ensuring the reliability of geometric supervision, and the easy propagation of errors in supervision throughout the entire optimization process under sparse viewpoints, this invention introduces two complementary types of supervision: binocular reprojection geometric constraints and monocular depth priors. A multi-factor driven uncertainty gating mechanism is constructed, using the reliability of geometric supervision itself as the core criterion to generate pixel-by-pixel gating weights and aggregate them to obtain the reliability weights for each Gaussian primitive. The same set of reliability weights is simultaneously applied to three stages: adaptive weighting of geometric loss, Gaussian splitting and densification control, and pruning process management. This strengthens constraints in reliable regions and suppresses error propagation in unreliable regions. Through multiple iterations, a geometrically accurate and low-redundancy 3D Gaussian scene representation is obtained, effectively improving the rendering quality and geometric stability of novel perspective synthesis under sparse viewpoints.

[0012] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0013] This invention provides a method for optimizing 3D Gaussian representation for synthesizing novel sparse viewpoints, comprising the following steps:

[0014] Obtain a sparse viewpoint image set of the target scene and the corresponding camera parameters;

[0015] A 3D Gaussian scene representation is initialized based on the image set and camera parameters, and the 3D Gaussian scene representation contains multiple 3D Gaussian primitives;

[0016] Construct binocular reprojection geometric constraints between a reference viewpoint and at least one source viewpoint, and obtain the monocular depth prior of the reference viewpoint;

[0017] At least based on binocular reprojection error, monocular depth gradient, and Gaussian visibility information, pixel-level gating weights are generated to characterize the reliability of geometric supervision.

[0018] The pixel-level gating weights are used to adaptively weight the binocular consistency loss constructed based on the binocular reprojection geometric constraints and the monocular depth consistency loss constructed based on the monocular depth prior. Combined with the color reconstruction loss, a joint optimization objective is constructed to update the parameters of each 3D Gaussian element.

[0019] The pixel-level gating weights are aggregated into Gaussian reliability weights corresponding to each three-dimensional Gaussian element, and the Gaussian reliability weights are used to regulate the splitting, densification and pruning process of the three-dimensional Gaussian element.

[0020] The output is an optimized 3D Gaussian scene representation used for image rendering from a new perspective of the target.

[0021] In the methods described above, binocular reprojection geometric constraints and monocular depth priors provide two complementary types of geometric supervision information for sparse viewpoint scenes. However, not all pixel locations receive equally reliable geometric supervision: binocular matching is prone to failure in low-texture regions; monocular depth estimation may be distorted in areas with abrupt depth changes or reflections; and the reprojection relationship itself does not hold in occluded regions. If equal-weighted geometric supervision is applied to all pixel locations, erroneous signals from unreliable regions will interfere with the optimization process, leading to Gaussian cell position shifts, edge blurring, or floating artifacts.

[0022] To address the aforementioned issues, this method constructs a multi-factor uncertainty gating mechanism, fusing heterogeneous information such as binocular reprojection error, monocular depth gradient, and Gaussian visibility into a unified geometric supervision reliability metric, generating pixel-level gating weights. A larger gating weight indicates more reliable geometric supervision at the corresponding location, requiring stronger constraints during optimization; a smaller gating weight suggests the region may be affected by occlusion, weak texture, abrupt depth changes, or low visibility, necessitating reduced geometric supervision or even masking. Furthermore, the pixel-level gating weights are aggregated into Gaussian-level reliability weights, enabling reliability judgments based on 2D images to be transferred to the management process of 3D Gaussian primitives. By simultaneously applying the gating weights to three stages—adaptive weighting of geometric loss, edge-guided Gaussian splitting, and annealing multi-condition pruning—a closed-loop end-to-end process from geometric supervision evaluation to model structure update is formed. This ensures that geometric constraints in reliable regions are preserved or enhanced, while erroneous supervision in unreliable regions is effectively suppressed. Simultaneously, scene-adaptive control of Gaussian primitive densification and pruning is achieved.

[0023] The specific implementation methods for each step are explained in detail below.

[0024] In one implementation, the step of generating pixel-level gating weights specifically includes: selecting at least two types of factors from binocular reprojection error, monocular depth gradient, local texture confidence, occlusion markers, and Gaussian visibility for weighted combination; mapping the weighted combination result to a pixel-level uncertainty value through a normalization function, wherein the higher the uncertainty value, the lower the reliability of geometric supervision; obtaining pixel-level gating weights based on the pixel-level uncertainty value, and performing a weighted average of the pixel-level gating weights of a single 3D Gaussian primitive in the multi-view projection region to obtain the Gaussian reliability weight of the corresponding 3D Gaussian primitive.

[0025] The meanings of the above factors and their roles in reliability assessment are as follows: Binocular reprojection error reflects the degree of geometric consistency between the reference viewpoint and the source viewpoint under the current depth estimate. A larger error usually indicates a less accurate current depth estimate, and the binocular supervision at that location should be weighted less. Monocular depth gradient reflects edges and abrupt changes in the depth map. Locations with large depth gradients (such as object boundaries or depth discontinuities) are often where monocular depth estimation is prone to errors, and the monocular supervision at these locations should be weighted less. Local texture confidence can be calculated from grayscale gradients, local variance, or feature responses. Regions with weaker textures are less reliable for binocular matching, and the binocular supervision at these locations should be weighted less. Occlusion markers are used to identify pixels invisible in the source viewpoint. Binocular constraints in occluded regions are inherently flawed and should be masked. Gaussian visibility reflects the coverage of a pixel by 3D Gaussian primitives. It can be obtained from the cumulative Gaussian transparency involved in the pixel's rendering, the number of visible occurrences, or coverage weight statistics. Higher visibility indicates greater confidence in the model's geometric estimation of the region, and the confidence of geometric supervision at this location can be appropriately increased. By combining the above factors through weighted summation and normalization function mapping, the reliability of geometric supervision at each pixel location can be comprehensively evaluated, providing a basis for subsequent adaptive weighting and structural control.

[0026] In one implementation, when obtaining the monocular depth prior of the reference viewpoint, the monocular depth estimation result and the rendered depth map are scale-aligned; the scale alignment process employs inverse depth transformation and / or local normalization; the inverse depth transformation converts the depth value into an inverse depth form; the local normalization process removes the mean and normalizes the depth value within the local window.

[0027] Since the depth maps output by monocular depth estimation networks typically only have relative scale significance, the results may differ by an unknown scaling factor or offset depending on the viewpoint or scene. Consequently, the absolute depth values ​​may deviate significantly from the scale of the real scene. Inverse depth transformation, by mapping depth values ​​to an inverse depth form, can mitigate the scale magnification effect in distant regions, giving a more balanced weight to the depth differences between near and far regions during optimization. Local normalization, by subtracting the mean or median from the depth values ​​within a local window and then dividing by the standard deviation or mean absolute deviation, can eliminate the influence of local scale drift, making the depth gradient information more reliable. These two processing methods can be used individually or in combination, allowing the monocular depth prior to maintain relative geometric structure information while avoiding the introduction of absolute scale errors into the 3D Gaussian optimization process.

[0028] In one implementation, the steps of constructing binocular reprojection geometric constraints specifically include: selecting at least one source viewpoint from a sparse viewpoint image set as a reference viewpoint based on the camera pose, viewpoint overlap ratio, and baseline angle; backprojecting the reference viewpoint pixels into 3D points according to the current rendering depth, and then projecting them onto the source viewpoints to obtain matching positions; calculating the binocular reprojection error when the matching position is within the effective image range; and setting occlusion markers for invalid regions, regions with inconsistent front and rear projections, and occlusion regions.

[0029] The viewpoint overlap ratio is determined by calculating the proportion of the shared viewing area between the reference viewpoint and the candidate source viewpoint. The baseline angle is determined by the angle between the line connecting the optical centers of the two viewpoints and the optical axis. The screening strategy tends to select viewpoints with sufficient overlap with the reference viewpoint and a moderate baseline to ensure the effectiveness of binocular matching. When the matching position exceeds the boundary of the source viewpoint image, the pixel is marked as an invalid region. Inconsistent forward and backward projection refers to a situation where the position of a pixel in the reference viewpoint, after being back-projected into 3D space, then projected back to the source viewpoint, deviates from the original pixel position by more than a preset threshold, indicating a contradiction in the depth estimation of the pixel. Occlusion regions refer to situations where pixels in the reference viewpoint are obscured by other objects in the source viewpoint and are therefore invisible. The occlusion markers for the above regions are set to indicate unreliable values ​​and are used as one of the input factors for gating weight calculation, causing the binocular supervision of these regions to be automatically downweighted or masked.

[0030] In one implementation, the joint optimization objective further includes at least one of a depth smoothing constraint term and a Gaussian regularization term; the depth smoothing constraint term is used to constrain the local smoothness of the depth or inverse depth; the Gaussian regularization term is used to constrain the scale, transparency, or spatial distribution of the three-dimensional Gaussian elements.

[0031] The depth smoothing constraint penalizes the depth difference between adjacent pixels, keeping the rendered depth map smooth in local areas and mitigating drastic fluctuations in regions of abrupt depth changes. The Gaussian regularization term constrains the size, transparency, or spatial distribution density of 3D Gaussian primitives, avoiding problems such as excessively large Gaussian primitives, too many highly transparent primitives, or excessively high Gaussian density in local areas, thereby improving the regularity and compactness of the 3D scene representation.

[0032] In one implementation, the process of controlling the splitting and densification of 3D Gaussian primitives specifically includes: statistically analyzing the Gaussian reliability weight of each 3D Gaussian primitive, the color residual of the corresponding region, the depth error, and the edge response according to a preset iteration interval; when a 3D Gaussian primitive meets the visibility requirements, the Gaussian reliability weight is greater than a threshold, the residual or gradient is greater than a threshold, and the projection position is located in the image edge or depth edge region, the 3D Gaussian primitive is split, and the splitting direction is determined according to the image edge normal, the depth change direction, the projection gradient direction, or the 3D covariance principal axis direction; for regions with low texture, occlusion, or Gaussian reliability weights lower than a threshold, the splitting of 3D Gaussian primitives is temporarily suspended or suppressed.

[0033] The image edge response can be calculated using an edge detection operator, while the depth edge response can be calculated from the first or second gradient of the depth map. The projection position is located in the image edge or depth edge region, meaning the distance between the projection position and the nearest edge pixel does not exceed a preset pixel radius. This pixel radius can be appropriately set according to the image resolution, for example, 3 pixels. The split child Gaussian units inherit some color and transparency parameters from the parent Gaussian unit, are scaled down accordingly, and their positions are slightly offset along the determined split direction. Through this edge-guided splitting strategy, the newly added Gaussian units can more accurately cover the edges of the real structure, while regions with low texture, occlusion, or low gating weights will not trigger splitting even if the image residual is large, thus reducing the generation of floating Gaussians and redundant Gaussians.

[0034] In one implementation, the pruning process for controlling 3D Gaussian primitives specifically includes: setting a pruning threshold that dynamically changes with the number of training iterations. The pruning threshold is gradually changed from an initial relaxed threshold to a final strict threshold using linear annealing, exponential annealing, cosine annealing, or piecewise annealing. Relaxed pruning conditions are used in the early stages of training, and the pruning intensity is gradually increased in the middle and later stages of training. When a 3D Gaussian primitive simultaneously satisfies the following conditions: transparency is below the threshold, visibility is below the threshold, Gaussian reliability weight is below the threshold, or cross-view consistency error is continuously higher than the threshold within consecutive iteration cycles, the 3D Gaussian primitive is deleted.

[0035] The phrase "simultaneously satisfying" means that all three conditions—transparency below a threshold, visibility below a threshold, and Gaussian reliability weight below a threshold—must be met. This indicates that the Gaussian primitive exhibits low-quality characteristics across multiple dimensions and should be deleted. Using the logic of "simultaneously satisfying" rather than "satisfying one of them" avoids mistakenly deleting effective Gaussian primitives due to a single metric being accidentally low. "Continuously exceeding the threshold within consecutive iterations" means that the binocular consistency error exceeds the preset error threshold in multiple consecutive iteration intervals. This indicates that the Gaussian primitive has a persistent geometric inconsistency problem, and its position may be fundamentally incorrect. The initial lenient threshold in the early stages of training allows the system to maintain a high tolerance for Gaussian primitives that are not yet fully optimized, avoiding the mistaken deletion of potentially effective structures. As training progresses, the threshold is gradually tightened, and the system's elimination of low-quality Gaussian primitives gradually increases, ultimately removing redundant or floating Gaussians. The annealing method can be selected from linear annealing, exponential annealing, cosine annealing or piecewise annealing according to the specific application scenario. In linear annealing, the pruning threshold changes linearly with the number of iterations. In exponential annealing, the pruning threshold changes faster in the later stage of training. In cosine annealing, the pruning threshold changes more smoothly. In piecewise annealing, different annealing rates can be set at different training stages.

[0036] In one implementation, the initialization of the 3D Gaussian scene representation is carried out in any of the following ways: when there is a sparse point cloud obtained by restoring the structure through motion, the point cloud coordinates are used as the initial position of the 3D Gaussian center; when there is no available sparse point cloud, the pixels are back-projected into the 3D space according to the monocular depth map and camera parameters to generate the initial Gaussian center.

[0037] When a motion-reconstructed point cloud exists, initialization using point cloud coordinates can provide relatively accurate initial spatial positions for 3D Gaussian primitives. When the point cloud is too sparse or completely unavailable, using a monocular depth map combined with camera intrinsic and extrinsic parameters to backproject some pixels from the image plane into 3D space can generate initial Gaussian centers with a reasonable spatial distribution, ensuring the optimization process can start normally.

[0038] In one implementation, the steps of constructing binocular reprojection geometric constraints, generating gated weights, optimizing parameters, and updating Gaussian structure are executed cyclically until a preset training termination condition is met. The preset training termination condition can be reaching a preset total number of iterations, the rendering loss decreasing below a preset threshold, or the rendering loss no longer significantly decreasing within a certain number of consecutive iterations.

[0039] Furthermore, this invention also provides a 3D Gaussian representation optimization system for sparse viewpoint synthesis, comprising: a data acquisition module for acquiring a sparse viewpoint image set of a target scene and corresponding camera parameters; a 3D Gaussian initialization module for initializing a 3D Gaussian scene representation based on the image set and camera parameters, wherein the 3D Gaussian scene representation contains multiple 3D Gaussian primitives; a binocular consistency calculation module for constructing binocular reprojection geometric constraints between a reference viewpoint and at least one source viewpoint; a monocular depth processing module for acquiring monocular depth priors of the reference viewpoint; and an uncertainty gating module for generating a representation of the sparse viewpoint based at least on binocular reprojection error, monocular depth gradient, and Gaussian visibility information. The system employs a pixel-level gating weight to supervise reliability, and aggregates these pixel-level gating weights into Gaussian-level reliability weights corresponding to each 3D Gaussian primitive. A joint optimization module adaptively weights the binocular consistency loss (constructed based on the binocular reprojection geometric constraints) and the monocular depth consistency loss (constructed based on the monocular depth prior) using the pixel-level gating weights, combining them with the color reconstruction loss to construct a joint optimization objective and update the parameters of each 3D Gaussian primitive. A Gaussian structure control module regulates the splitting, densification, and pruning process of the 3D Gaussian primitives using the Gaussian-level reliability weights. An output rendering module outputs the optimized 3D Gaussian scene representation and renders the image based on the target new perspective parameters.

[0040] The above modules can be implemented through software programs, hardware circuits, or a combination of software and hardware, and can be deployed on terminal devices, servers, or electronic devices with graphics processing capabilities. In one implementation, the system includes a processor and a memory, the memory storing a computer program for executing the above methods. When the processor runs the computer program, it calls the above modules to complete the sparse viewpoint 3D Gaussian representation optimization and new viewpoint rendering.

[0041] Compared to existing technologies, this invention and its preferred solutions can effectively alleviate depth ambiguity and geometric degradation problems under sparse viewpoints. By complementing two types of geometric supervision, it compensates for the deficiencies in multi-viewpoint photometric consistency constraints, improving the accuracy of 3D Gaussian unit spatial positions and overall geometric stability. It can reduce the risk of error propagation from monocular depth priors, eliminate absolute scale deviations in monocular depth through scale alignment processing, and dynamically adjust the supervision intensity of different regions through a gating mechanism to avoid erroneous depth information from weak textures, occlusions, etc., misleading the optimization process. Furthermore, it can improve the robustness of binocular consistency constraints by comprehensively judging the reliability of supervision based on multi-dimensional factors, avoiding... It avoids introducing incorrect matching supervision in areas with low texture, repetitive texture, or insufficient viewpoint overlap; it can improve the directionality and accuracy of Gaussian densification, prioritizing split densification at the edges of high-confidence real structures, suppressing blind densification in low-confidence regions, and reducing the generation of floating Gaussians and pseudo-structures; it can achieve dynamic adaptation of the Gaussian pruning process, gradually adjusting the pruning intensity according to the training progress, taking into account both early structure exploration and later redundancy cleanup, and optimizing the compactness of 3D Gaussian representation; at the same time, it has formed a complete technical system from geometric prior construction and reliability quantification to full-process adaptive control, which can adapt to the needs of 3D scene reconstruction and new viewpoint synthesis with various sparse viewpoint inputs. Attached Figure Description

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0043] Figure 1 This is a flowchart illustrating the optimization method for 3D Gaussian representation of a novel perspective synthesis based on sparse perspectives, as described in an embodiment of the present invention.

[0044] Figure 2 This is an architectural block diagram of a 3D Gaussian representation optimization system for sparse viewpoint synthesis according to an embodiment of the present invention. Detailed Implementation

[0045] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0046] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0047] The technical problem to be solved by this invention is: under sparse viewpoint input conditions, how to integrate binocular consistency and online monocular depth prior, and use multi-factor uncertainty gating to determine the reliability of geometric supervision, so that reliable regions are subject to strong geometric constraints and erroneous supervision in unreliable regions is suppressed; at the same time, how to further use this uncertainty information for Gaussian splitting and pruning, thereby reducing floating Gaussian, error densification, edge blurring and geometric instability problems.

[0048] (I) Overall Technical Solution

[0049] To address this, the present invention provides a method for optimizing 3D Gaussian representations for novel perspective synthesis based on sparse viewpoints. This method uses a small number of multi-view images, camera intrinsics, camera extrinsics, and an optional initial sparse point cloud as input to construct a 3D Gaussian scene representation. To address the issues of insufficient geometric constraints and poor reliability of single priors under sparse viewpoints, it simultaneously introduces two types of geometric supervision: binocular consistency constraints and online monocular depth priors, forming a multi-source complementary geometric constraint system.

[0050] Based on this, this invention constructs a multi-factor uncertainty gating mechanism, which integrates multi-dimensional features such as binocular reprojection error, monocular depth gradient, local texture intensity, occlusion relationship, and Gaussian visibility to generate pixel-level and Gaussian-level reliability weights, realizing point-by-point quantization of the reliability of geometric supervision. This uncertainty gating weight is then permeated throughout the entire 3D Gaussian optimization process: on the one hand, it is used for adaptive weighted monocular depth consistency loss and binocular consistency loss, so that reliable regions are given strong geometric constraints and erroneous supervision in unreliable regions is effectively suppressed; on the other hand, Gaussian-level reliability is combined with edge response, color residual, and depth error to achieve edge-guided Gaussian splitting and densification, while annealing pruning strategy that dynamically adjusts the threshold during the training phase is used to finely manage the lifecycle of Gaussian units.

[0051] Ultimately, through multiple rounds of iterative optimization, a 3D Gaussian scene representation with accurate geometry, low redundancy, and suitability for sparse perspective synthesis was obtained.

[0052] like Figure 1 As shown, the method flow of the present invention can be summarized as follows:

[0053] S100, acquires sparse viewpoint images and camera parameters;

[0054] S200, Initialize the three-dimensional Gaussian representation;

[0055] S300, establish a binocular consistency relationship;

[0056] S400, online acquisition of monocular depth prior and scale alignment;

[0057] S500, constructing uncertainty gating;

[0058] S600 utilizes gated weights for joint optimization;

[0059] S700 performs edge-guided Gaussian splitting;

[0060] S800, perform annealing and pruning;

[0061] S900 outputs a 3D Gaussian scene representation and performs new perspective rendering.

[0062] Steps S300 to S800 constitute a training iteration loop unit, which is executed repeatedly during the training process until the termination condition is met.

[0063] (II) Input Data and 3D Gaussian Initialization

[0064] In step S100, the same static scene is acquired. Image , The value is an integer not less than 2, and the camera intrinsic parameters corresponding to each image are obtained. and external references When a SfM point cloud exists, the point cloud coordinates are used as the initial position of the 3D Gaussian center; when the SfM point cloud is too sparse or unavailable, some pixels can be back-projected into 3D space based on the monocular depth map and camera parameters to generate the initial Gaussian center.

[0065] In step S200, the scene is represented as several three-dimensional Gaussian primitives. Each three-dimensional Gaussian element includes at least a center location. ,scale Rotation parameters ,transparency and color or spherical harmonic coefficient A three-dimensional Gaussian primitive is projected onto a reference viewpoint using a differentiable Gaussian rasterizer to obtain a rendered color map. Rendering depth map And visibility information corresponding to pixels or Gaussians.

[0066] (III) Construction of Binocular Consistency Relationship

[0067] In step S300, a reference viewpoint is selected from the sparse input viewpoints based on the camera pose, viewpoint overlap ratio, and baseline angle. Select at least one source perspective For reference viewpoint pixels Based on the current rendering depth Or the aligned depth estimate will Back projection into three-dimensional points Then, the corresponding position is obtained by projecting the source view camera parameters onto the source view. .like If the reference viewpoint is within the effective image range of the source viewpoint, then the color difference, feature difference, or depth difference between the reference viewpoint and the source viewpoint is calculated to obtain the binocular reprojection error. .

[0068] In one alternative implementation, binocular reprojection error It can be obtained by weighting color error, feature error, and forward / backward projection consistency error. When When a pixel is in an invalid region, has inconsistent projections in the front and back directions, or is occluded, an occlusion marker is set for that pixel. Otherwise set The larger the binocular reprojection error or the more obvious the occlusion mark, the less reliable the binocular supervision is at that pixel.

[0069] (iv) Online monocular depth prior and scale alignment

[0070] In step S400, the reference view image is input into the monocular depth estimation network to obtain a monocular depth map. Since monocular depth estimation typically involves scale and offset uncertainties, this invention does not directly use absolute depth values ​​as equal-weighted supervision, but instead uses the rendered depth... and monocular depth Perform scale, offset, or local normalization alignment.

[0071] In one alternative implementation, an inverse depth form can be used:

[0072]

[0073] in To prevent constants with zero denominators, a local normalization method can be used, subtracting the mean or median from the depth within the local window and then dividing by the standard deviation or mean absolute deviation. This reduces the impact of monocular depth absolute scale error on 3D Gaussian optimization.

[0074] (v) Uncertainty Gating Mechanism

[0075] In step S500, for pixels Or, the three-dimensional Gaussian elements associated with that pixel, constructing uncertainty. Uncertainty arises from binocular reprojection error. Monocular depth gradient Local texture confidence Obscuring marks and Gaussian visibility One or more of these factors are used to determine the local texture confidence. It can be calculated from gray-level gradient, local variance, or feature response; monocular depth gradient It can be obtained from the first-order gradient or edge response of the depth map; Gaussian visibility It can be obtained from the cumulative Gaussian transparency, visible count, or coverage weight statistics of the pixels involved in rendering.

[0076] In one alternative expression, uncertainty can be represented as:

[0077]

[0078] in, to For weight parameters, This represents the normalization function. The corresponding gate weights are:

[0079]

[0080] in, The larger the value, the more reliable the geometric supervision corresponding to that pixel or Gaussian. The smaller the value, the more likely the region is to be affected by occlusion, weak texture, depth abrupt changes, or low visibility, and its geometric supervision strength should be reduced. This is a bias parameter and can take negative values; to These are weight parameters, which can be preset as fixed hyperparameters based on the characteristics of the scenario, or learned through gradient descent during training. In a typical implementation, these parameters are set as fixed hyperparameters, with the following values ​​as references: (Control the overall bias so that the gating default bias is reliable when there are no obvious anomalies). (The weight of binocular reprojection error is set to a large value so that even a small error can significantly increase the uncertainty.) (Weights of monocular depth gradient) (Penalty weight for missing texture confidence). (The weight of the occlusion marker directly increases the uncertainty of the occluded area.) (Gaussian visibility compensation weights, with higher visibility appropriately reducing uncertainty). The above values ​​are applicable to LLFF-type forward-facing scenes and 3-input view configurations. In practice, they can be adjusted appropriately based on the baseline distance and texture richness of the specific dataset.

[0081] In Gaussian gating, a three-dimensional Gaussian primitive can be... Gaussian-level reliability is obtained by weighting and averaging the pixel-level gated weights across multiple projection regions. This reliability is used for subsequent control of Gaussian splitting and pruning, enabling uncertainty information to participate not only in loss function calculation but also in 3D Gaussian lifecycle management.

[0082] (vi) Joint optimization objectives

[0083] In step S600, a joint optimization objective is constructed, consisting of color reconstruction loss, monocular depth consistency loss, binocular consistency loss, and a smoothing or regularization term. The total loss function can be expressed as:

[0084]

[0085] in, The reconstruction loss between the rendered colormap and the input image; The monocular depth consistency loss is adjusted by the gating weights; The binocular reprojection consistency loss is adjusted by gating weights; For depth or inverse depth smoothing constraints; For Gaussian scaling, transparency, or spatial distribution regularization; , , , This refers to the balancing weights for the corresponding loss terms.

[0086] In one alternative implementation:

[0087]

[0088]

[0089] in This is the robust error function (either Huber loss function or Charbonnier loss function can be selected according to actual needs). After weighting, the geometric constraints of reliable regions are preserved or enhanced, while the impact of erroneous depths or mismatches in unreliable regions on parameter updates is weakened. Subsequently, the Gaussian center position, scale, rotation, transparency, and color parameters are updated via backpropagation.

[0090] (vii) Edge-guided Gaussian splitting

[0091] In step S700, the projection area and average gating weight of each 3D Gaussian primitive under multiple viewpoints are statistically analyzed according to a preset iteration interval. Color residual, depth error, image edge response, and depth edge response. A Gaussian primitive is split when it satisfies the following conditions: the number of times it is visible is greater than a threshold, the average gating weight is greater than a threshold, the residual or gradient is greater than a threshold, and it is located near an image edge or depth edge.

[0092] The splitting direction can be determined based on the image edge normal, depth variation direction, projection gradient direction, or the principal axis of the 3D covariance. The split child Gaussian inherits some color and transparency parameters from the parent Gaussian, is scaled down accordingly, and its position is slightly offset along the determined direction. For regions with low texture, occlusion, large binocular errors, or low gating weights, even with large image residuals, splitting can be temporarily delayed or suppressed to reduce the generation of floating or redundant Gaussians in unreliable regions.

[0093] In a typical implementation, edge detection uses the Canny operator, with high and low thresholds set to 100 and 50 respectively, to calculate an edge response map in the image space; "near the edge" is defined as a region whose Euclidean distance from the nearest edge pixel does not exceed 3 pixels. The gating weight threshold is used in the Gaussian splitting trigger condition. The threshold is set to 0.5, the visibility threshold is set to 3 times, and the color residual threshold is set to 0.02. The split interval is executed once every 100 iterations, consistent with the original 3DGS densification strategy.

[0094] (viii) Annealing and pruning

[0095] In step S800, a pruning threshold that varies with training iterations is set. .in, This represents the current iteration number. This represents the total number of iterations. The pruning process can be linear, exponential, or cosine-based, gradually varying from the initial threshold to the target threshold. In the early stages of training, more lenient pruning conditions are used to avoid accidentally deleting effective structures that are not yet fully optimized. In the later stages of training, the intensity of pruning is gradually increased for structures with low transparency, low visibility, high uncertainty, persistently large cross-viewpoint errors, or anomalously Gaussian scales.

[0096] In one alternative implementation, a Gaussian Gaussian cell is deleted when it simultaneously satisfies the following conditions: transparency is below a transparency threshold, number of views is below a visibility threshold, average gating weight is below a reliability threshold, or binocular consistency error consistently exceeds an error threshold over multiple iterations. Through annealing, this invention can balance early-stage structure exploration during training with later-stage redundancy cleanup.

[0097] In one specific embodiment, the total number of training iterations Set the threshold to 10000 for annealing and pruning. Used to control the intensity of Gaussian pruning, and based on an initial threshold in a linear annealing manner. Gradually change to the termination threshold . Specifically:

[0098]

[0099] in This represents the current iteration number. In an LLFF-class forward-facing scene with a 3-input view configuration... 0.003 is acceptable. A value of 0.01 can be used. Therefore, the pruning conditions are relatively lenient in the early stages of training to avoid accidentally deleting effective Gaussians that have not yet been fully optimized; the pruning conditions are gradually tightened in the later stages of training to remove anomalous Gaussians with low transparency, low visibility, high uncertainty, or persistently large cross-view consistency errors.

[0100] In other alternative embodiments, the annealing pruning threshold Alternatively, exponential annealing, cosine annealing, or piecewise annealing can be used, and this invention is not limited to a specific annealing function form.

[0101] (ix) Output and System Structure

[0102] In step S900, after training is complete, an optimized 3D Gaussian scene file is output. When the camera intrinsic and extrinsic parameters of the new target viewpoint are input, the target viewpoint image is rendered using a 3D Gaussian rasterizer. This output can be used for 3D scene browsing, virtual reality display, robot environmental perception, digital twins, or other applications that require recovering the appearance of a scene from a small number of images.

[0103] The present invention also provides a system for performing the above-described method, the overall architecture of which is as follows: Figure 2 As shown, the system includes: a data acquisition module for acquiring sparse viewpoint images and camera parameters; a 3D Gaussian initialization module for generating initial 3D Gaussian primitives; a monocular depth estimation module for outputting monocular depth priors for each viewpoint; a binocular consistency calculation module for calculating reprojection errors between viewpoints based on camera geometry; an uncertainty gating module for outputting pixel-level or Gaussian-level reliability weights; a joint optimization module for updating 3D Gaussian parameters based on a weighted loss function; an edge-guided splitting module for performing Gaussian densification in high-confidence structure edge regions; an annealing pruning module for progressively removing redundant or unreliable Gaussians during the training phase; and a new viewpoint rendering module for outputting rendered images based on target viewpoint parameters.

[0104] In one implementation, the system includes a processor and a memory. The memory stores a computer program for executing the above-described method. When the processor runs the computer program, it calls a data acquisition module, a 3D Gaussian initialization module, a monocular depth estimation module, a binocular consistency calculation module, an uncertainty gating module, a joint optimization module, an edge-guided splitting module, an annealing pruning module, and a new perspective rendering module to complete sparse perspective 3D Gaussian representation optimization and new perspective rendering.

[0105] Compared with the prior art, the advantages of the above solutions of the present invention include:

[0106] This invention alleviates depth ambiguity and geometric degradation under sparse perspectives. Compared to basic 3DGS methods that rely solely on image reconstruction loss, this invention provides additional geometric constraints through binocular consistency and online monocular depth priors, compensating for the lack of multi-view photometric consistency under sparse perspectives. This significantly improves the stability of Gaussian unit spatial location and depth accuracy, reducing background collapse and depth misalignment issues.

[0107] This invention reduces the risk of error propagation from monocular depth priors. Compared to schemes that directly use equal-weight monocular depth supervision, this invention eliminates the absolute scale error of monocular depth through scale alignment, inverse depth transformation, or local normalization. Furthermore, it uses uncertainty gating to suppress depth abrupt changes, weak textures, and occluded regions with weights, preventing erroneous depth information from misleading the Gaussian optimization process and preventing edge structure distortion and local over-smoothing.

[0108] Improve the robustness of binocular consistency constraints. Compared to schemes that apply binocular constraints alone, this invention combines local texture, occlusion state, depth gradient, and Gaussian visibility to determine the reliability of supervision from multiple dimensions. This avoids introducing incorrect matching supervision in areas with low texture, repetitive texture, occlusion, or insufficient viewpoint overlap, and reduces the interference of invalid constraints on the optimization process.

[0109] This invention achieves precise targeting of Gaussian densification, reducing floating artifacts. Compared to ordinary image gradient-driven densification strategies, this invention uses gating reliability, edge response, and residual information together for splitting decisions, prioritizing densification at high-confidence edges of real structures and suppressing blind splitting in low-confidence regions, thereby reducing the generation of floating Gaussians and pseudo-structures from the source.

[0110] Balancing training exploration with later simplification, this invention optimizes the compactness of Gaussian representations. Compared to fixed-threshold pruning schemes, this invention employs an annealing-based dynamic pruning strategy. In the early stages of training, a loose threshold is used to retain potentially effective structures, while in the later stages, the threshold is gradually tightened. By combining transparency, visibility, gating weights, and cross-view consistency error, anomalous Gaussians are filtered and eliminated from multiple dimensions, achieving a balance between structure exploration and redundancy removal.

[0111] A systematic end-to-end optimization scheme is formed. This invention constructs a complete technical process from sparse perspective input, 3D Gaussian initialization, multi-source geometric prior construction, uncertainty adaptive adjustment, dynamic control of Gaussian structure to new perspective rendering, and provides a system architecture with corresponding functional modules, which can realize the complete technical implementation from both methodological and system levels.

[0112] To further illustrate the feasibility of the technical solution of the present invention and the technical effects of each module, a more specific example is given below.

[0113] Implementation configuration: The dataset is selected from the Fern scene in LLFF, and the number of input viewpoints is [number missing]. Training views were selected using equally spaced sampling, while test views were divided according to the standard protocol every 8th image. The initial point cloud was obtained using PDCNet+ (using only three images selected from sparse views). Camera pose was taken from the standard output of colmap, and image resolution was downsampled to half the original resolution (approximately 1008×756). The monocular depth estimation network used Depth-Anything-V2, pre-compiling and caching depth maps online for the training images. In the binocular consistency construction, the virtual slave camera was randomly translated horizontally, with the translation range... Set to 30% of the baseline length of adjacent training views. Total training iterations. The optimizer uses Adam, and the loss weights are set to... , , , The gating parameters are set according to the typical values ​​mentioned above. Edge detection uses the Canny operator (threshold 100 / 50) with an edge radius of 3 pixels. Linear annealing is used for annealing. , .

[0114] Control group A is based on the 3DGS method, control group B is based on control group A with the addition of fixed-weight double prior supervision, and control group C is the complete scheme of this invention.

[0115]

[0116] In summary, this invention addresses the key challenges of optimizing 3D Gaussian representations from a sparse perspective. Based on multi-source geometric prior complementarity and with multi-factor uncertainty gating as the core, it extends reliability assessment from loss weighting to the entire process of Gaussian structure densification and pruning. This solution not only resolves the depth ambiguity and edge blurring issues caused by insufficient geometric constraints in a sparse perspective but also avoids structural distortion caused by the propagation of a single prior error. Simultaneously, it improves the geometric accuracy and structural compactness of 3D Gaussian representations, making it widely applicable in various technical fields such as 3D scene browsing, virtual reality, digital twins, and robot environmental perception.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for optimizing 3D Gaussian representation for synthesizing novel sparse viewpoints, characterized in that, Includes the following steps: Obtain a sparse viewpoint image set of the target scene and the corresponding camera parameters; A 3D Gaussian scene representation is initialized based on the image set and camera parameters, and the 3D Gaussian scene representation contains multiple 3D Gaussian primitives; Construct binocular reprojection geometric constraints between a reference viewpoint and at least one source viewpoint, and obtain the monocular depth prior of the reference viewpoint; At least based on binocular reprojection error, monocular depth gradient, and Gaussian visibility information, pixel-level gating weights are generated to characterize the reliability of geometric supervision. The pixel-level gating weights are used to adaptively weight the binocular consistency loss constructed based on the binocular reprojection geometric constraints and the monocular depth consistency loss constructed based on the monocular depth prior. Combined with the color reconstruction loss, a joint optimization objective is constructed to update the parameters of each 3D Gaussian element. The pixel-level gating weights are aggregated into Gaussian reliability weights corresponding to each three-dimensional Gaussian element, and the Gaussian reliability weights are used to regulate the splitting, densification and pruning process of the three-dimensional Gaussian element. The output is an optimized 3D Gaussian scene representation used for image rendering from a new perspective of the target.

2. The 3D Gaussian representation optimization method for sparse viewpoint synthesis according to claim 1, characterized in that: The specific steps for generating pixel-level gated weights include: At least two of the following factors were selected and weighted: binocular reprojection error, monocular depth gradient, local texture confidence, occlusion markers, and Gaussian visibility. The weighted combination result is mapped to a pixel-level uncertainty value by a normalization function. The higher the uncertainty value, the lower the reliability of the geometric supervision. Pixel-level gate weights are obtained based on the pixel-level uncertainty values, and the pixel-level gate weights of a single 3D Gaussian primitive in the multi-view projection region are weighted and averaged to obtain the Gaussian reliability weights of the corresponding 3D Gaussian primitive.

3. The 3D Gaussian representation optimization method for sparse viewpoint synthesis according to claim 1, characterized in that: When obtaining the monocular depth prior from the reference viewpoint, the monocular depth estimation result is scale-aligned with the rendered depth map. The scale alignment process employs inverse depth transformation and / or local normalization. The inverse depth transformation converts the depth value into an inverse depth form; The local normalization process removes the mean and normalizes the depth values ​​within the local window.

4. The 3D Gaussian representation optimization method for sparse viewpoint synthesis according to claim 1, characterized in that: The specific processes involved in regulating the splitting and densification of three-dimensional Gaussian units include: The Gaussian reliability weights of each 3D Gaussian element, the color residuals of the corresponding regions, the depth errors, and the edge responses are statistically analyzed according to the preset iteration intervals. When a 3D Gaussian primitive satisfies the visibility requirement, the Gaussian reliability weight is greater than the threshold, the residual or gradient is greater than the threshold, and the projection position is located in the image edge or depth edge region, the 3D Gaussian primitive is split. The splitting direction is determined according to the image edge normal, depth change direction, projection gradient direction or 3D covariance principal axis direction. For regions with low texture, occlusion, or Gaussian reliability weights below a threshold, the splitting of 3D Gaussian primitives is temporarily suspended or suppressed.

5. The 3D Gaussian representation optimization method for sparse viewpoint synthesis according to claim 1, characterized in that: The pruning process for regulating three-dimensional Gaussian elements specifically includes: A pruning threshold is set that dynamically changes with the number of training iterations. The pruning threshold is gradually changed from an initial loose threshold to a final strict threshold according to linear annealing, exponential annealing, cosine annealing or piecewise annealing. A loose pruning condition is used in the early stage of training, and the pruning intensity is gradually increased in the middle and late stages of training. When a 3D Gaussian primitive simultaneously satisfies the following conditions: transparency is below a threshold, number of visible objects is below a threshold, Gaussian reliability weight is below a threshold, or cross-view consistency error is continuously higher than a threshold within consecutive iteration cycles, the 3D Gaussian primitive is deleted.

6. The 3D Gaussian representation optimization method for sparse viewpoint synthesis according to claim 1, characterized in that: The specific steps for constructing binocular reprojection geometric constraints include: Based on camera pose, viewpoint overlap ratio and baseline angle, at least one source viewpoint is selected from the sparse viewpoint image set as the reference viewpoint. The reference viewpoint pixels are back-projected into 3D points according to the current rendering depth, and then projected onto the source viewpoint to obtain the matching position; When the matching position is within the effective image range, the binocular reprojection error is calculated; Set occlusion markers for invalid areas, areas with inconsistent front and rear projections, and occluded areas.

7. The 3D Gaussian representation optimization method for sparse viewpoint synthesis according to claim 1, characterized in that: The joint optimization objective also includes at least one of a deep smoothing constraint term and a Gaussian regularization term; The depth smoothing constraint term is used to constrain the local smoothness of the depth or inverse depth; The Gaussian regularization term is used to constrain the scale, transparency, or spatial distribution of three-dimensional Gaussian elements.

8. The 3D Gaussian representation optimization method for sparse viewpoint synthesis according to claim 1, characterized in that: Initialize the 3D Gaussian scene representation using any of the following methods: When a sparse point cloud exists that is obtained by restoring the structure through motion, the point cloud coordinates are used as the initial position of the three-dimensional Gaussian center. When no sparse point cloud is available, pixels are back-projected into 3D space based on the monocular depth map and camera parameters to generate initial Gaussian centers.

9. The 3D Gaussian representation optimization method for sparse viewpoint synthesis according to claim 1, characterized in that: The steps of constructing binocular reprojection geometric constraints, generating gating weights, optimizing parameters, and updating Gaussian structure are executed cyclically until the preset training termination condition is met.

10. A three-dimensional Gaussian representation optimization system for sparse viewpoint synthesis, characterized in that, include: The data acquisition module is used to acquire a sparse viewpoint image set of the target scene and the corresponding camera parameters; A 3D Gaussian initialization module is used to initialize a 3D Gaussian scene representation based on the image set and camera parameters. The 3D Gaussian scene representation contains multiple 3D Gaussian primitives. The binocular consistency calculation module is used to construct the binocular reprojection geometric constraints between the reference view and at least one source view; The monocular depth processing module is used to obtain the monocular depth prior from the reference viewpoint; An uncertainty gating module is used to generate pixel-level gating weights to characterize the reliability of geometric supervision based at least on binocular reprojection error, monocular depth gradient and Gaussian visibility information, and to aggregate the pixel-level gating weights into Gaussian reliability weights corresponding to each three-dimensional Gaussian element. The joint optimization module is used to adaptively weight the binocular consistency loss constructed based on the binocular reprojection geometric constraints and the monocular depth consistency loss constructed based on the monocular depth prior using the pixel-level gating weights, and to construct a joint optimization objective by combining the color reconstruction loss, and update the parameters of each 3D Gaussian element. The Gaussian structure control module is used to regulate the splitting, densification, and pruning process of three-dimensional Gaussian primitives using the Gaussian-level reliability weights. The output rendering module is used to output an optimized 3D Gaussian scene representation and render the image based on the target's new perspective parameters.

Images