A sparse view three-dimensional gaussian reconstruction method, system, device and storage medium
By using multi-view geometric consistency verification and generative model error correction, the problems of missing geometric consistency and smooth texture details in sparse view 3D Gaussian reconstruction were solved, achieving high-quality 3D reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-02
AI Technical Summary
Existing sparse view 3D Gaussian reconstruction methods suffer from problems such as lack of geometric consistency, inability to distinguish the source of rendering errors, and excessive smoothing of texture details, resulting in high noise and blurred details in the reconstruction results.
The confidence scores of pseudo-view images are calculated by multi-view geometric consistency verification and weighted training is performed to identify the geometric consistency inlier rate of high error regions, construct a spatial variation modulation field, and use a generative model for error correction.
It improves the effectiveness of training data and the robustness of model initialization, suppresses floating artifacts, preserves high-frequency details, and improves reconstruction quality.
Smart Images
Figure CN122134936A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and more particularly to a sparse vision method. Figure 3 Vigos reconstruction methods, systems, equipment, and storage media. Background Technology
[0002] 3D Gaussian Splatting (3DGS) is a technique designed to address the challenge of achieving high-fidelity 3D scene reconstruction and novel perspective synthesis using only a very small number of 2D input images. Due to the sparsity of the input perspective, geometric constraints are severely lacking, making traditional methods prone to overfitting or geometric collapse.
[0003] To overcome the above-mentioned shortcomings, existing technologies have mainly focused on the following three dimensions to form mainstream improvement solutions: (1) Data augmentation: Using pre-trained 2D or 3D diffusion models, new pseudo-viewpoint images are generated to augment the dataset, conditioned on sparse reference images.
[0004] (2) Adaptive densification: During 3DGS training, the average gradient of the view space is monitored to determine the growth (splitting or cloning) of Gaussian primitives to increase the density of point clouds.
[0005] (3) Closed-loop self-supervised optimization: relying on manually designed regularization terms (such as depth smoothing) or introducing a pre-trained monocular depth estimation network as prior knowledge to constrain the geometry of unseen viewpoints.
[0006] Existing 3D Gaussian sputtering reconstruction workflows for sparse views typically begin by directly generating pseudo-view images using a pre-trained diffusion model to augment the sparse training set. This is followed by initializing the point cloud using a motion recovery structure and constructing a 3D Gaussian model, then conducting adaptive densification training and regularization parameter optimization based on positional gradients. However, this workflow lacks a geometric consistency verification mechanism during the data augmentation stage. The directly introduced pseudo-view images often contain geometric illusions inconsistent with the real scene, leading to a significant amount of noise being introduced into the initial reconstruction model.
[0007] However, the aforementioned existing technologies have the following objective drawbacks: (1) Lack of geometric consistency and quality screening mechanism: The content of the images generated by the diffusion model may lack strict three-dimensional geometric constraints, producing "illusion" content that is inconsistent with the original input geometry. Existing methods usually use all generated images for training indiscriminately, which introduces a lot of noise and reduces the signal-to-noise ratio of the three-dimensional reconstruction.
[0008] (2) Inability to distinguish the source of error, and rigid and simplistic densification strategy: In sparse view, high rendering error may be caused by "under-reconstruction" (insufficient detail) or "over-reconstruction" (floating artifacts). A single gradient threshold cannot distinguish between these two situations, resulting in excessive densification in sparse areas and the generation of a large number of "floating objects", or blurring in textured areas due to insufficient constraints.
[0009] (3) Excessive smoothing of texture details and lack of "self-correction" ability: Traditional geometric regularization (such as smoothness constraints) tends to smooth out high-frequency details, resulting in blurry reconstruction results. Moreover, existing methods are usually unidirectional constraints. Once the model gets stuck in a local optimum, it is difficult to get out. There is a lack of a mechanism to use the prior ability of generative models to repair the "bad results" of the current rendering and guide the model update in reverse. Summary of the Invention
[0010] This application aims to at least solve the technical problems existing in the prior art and provide a sparse view Figure 3 Vigos reconstruction methods, systems, equipment, and storage media.
[0011] Firstly, the present invention provides a sparse view Figure 3 Vigaussian reconstruction methods include: Obtain a real image dataset for sparse view scenes. The real image dataset includes real sparse images and the camera poses corresponding to the real sparse images. A generative model is used to synthesize pseudo-viewpoint images based on real sparse images and the camera poses corresponding to the real sparse images, and the camera poses corresponding to the pseudo-viewpoint images are determined. The confidence score of the pseudo-view image is calculated by multi-view geometric consistency verification, and the pseudo-view image is weighted according to the confidence score. The pseudo-viewpoint images and their corresponding camera poses are added to the training of the 3D Gaussian model. The rendering loss of the pseudo-viewpoint images is calculated by weighting the confidence scores as weights, and the 3D Gaussian model is trained under supervision. During the training of the 3D Gaussian model, the features of the current training view and the neighboring views are compared to identify the high error region of the current training view and calculate the geometric consistency inlier rate of the high error region; based on the geometric consistency inlier rate, a spatial variation modulation field is constructed to dynamically adjust the compaction gradient threshold of the Gaussian points. In the later stages of training the 3D Gaussian model, a generative model is used to correct the rendering results of the 3D Gaussian model under unseen viewpoints to obtain the reconstructed 3D Gaussian model.
[0012] Optionally, the step of synthesizing a pseudo-viewpoint image based on a real sparse image and the camera pose corresponding to the real sparse image using a generative model, and determining the camera pose corresponding to the pseudo-viewpoint image, includes: Real sparse images are input into a pre-trained view synthesis model to generate pseudo-viewpoint images; The generated pseudo-viewpoint image is registered into the original sparse reconstruction model using the motion reconstruction tool to calculate the camera extrinsic parameters corresponding to the pseudo-viewpoint image, thereby obtaining the camera pose corresponding to the pseudo-viewpoint image.
[0013] Optionally, the steps of identifying high-error regions in the current training view and calculating the geometric consistency inlier rate of the high-error regions include: Calculate the photometric residual between the rendered image and the real image of the current training view, and define the region where the residual exceeds a set threshold as a high error region; Select the nearest neighboring view that is spatially closest to the current training view, and extract the matching points between the current training view and the neighboring view that are located within the high error region to generate a subset of feature matching point pairs; The camera pose of the current training view is used to triangulate and reproject a subset of feature matching point pairs. The reprojection error is calculated, and the proportion of feature matching point pairs with reprojection errors less than a preset threshold in the subset is counted. The proportion of feature matching point pairs with reprojection errors less than the preset threshold in the subset is used as the geometric consistency inlier rate of the high error region.
[0014] Alternatively, the view synthesis model can be a diffusion model, a generative adversarial network, or a video frame interpolation model.
[0015] Optionally, the confidence score is calculated using the following formula: ; in, This represents the confidence score. Represents the coordinates of two-dimensional feature points in a pseudo-viewpoint image. This represents the coordinates of a two-dimensional feature point in the real sparse image that is spatially closest to the pseudo-viewpoint image. This represents the set of feature matching pairs between a pseudo-viewpoint image and the nearest real sparse image in spatial distance from the pseudo-viewpoint image. This indicates the reprojection error. This represents the pixel error threshold.
[0016] Optionally, the formula for calculating the geometrically consistent interior point ratio is: ; in, Represents the geometrically consistent interior point rate. This represents the coordinates of two-dimensional feature points in the current training view. Represents the coordinates of a two-dimensional feature point in the nearest neighboring view in the current training view space; Located in the high error area of the current view A subset of matching point pairs within the range.
[0017] Optionally, the step of using a generative model to correct the rendering results of the 3D Gaussian model under an unseen viewpoint to obtain the reconstructed 3D Gaussian model includes: Select a reference pose from the training view pose set, apply isotropic Gaussian noise to its translation vector, and generate a new pseudo translation vector; Find the two reference poses in the training set that are closest to the new pseudo-translation vector in Euclidean distance, and calculate the interpolation factor based on the distance; Using the spherical linear interpolation algorithm, the rotation quaternions of the two reference poses that are closest to the new pseudo-translation vector in Euclidean distance are interpolated according to the interpolation factor to obtain smooth pseudo-rotation quaternions. The pseudo-translation vector and pseudo-rotation quaternion are combined to form a complete pseudo-camera pose; The initial pseudo-view is obtained by performing differentiable rendering of the current 3D Gaussian model under the pseudo-camera pose. The initial pseudo-view is input into the pre-trained generative image refining model, which outputs a refined pseudo-ground image. The difference loss between the initial pseudo-view and the pseudo-real image is calculated, the gradient of the difference loss is backpropagated, the parameters of the 3D Gaussian model are updated, and the reconstructed 3D Gaussian model is obtained.
[0018] Secondly, the present invention provides a sparse view Figure 3 The Vigaussian reconstruction system includes: The acquisition module is used to acquire the real image dataset in the sparse view scene. The real image dataset includes real sparse images and the camera poses corresponding to the real sparse images. The data augmentation module is used to synthesize pseudo-viewpoint images based on real sparse images and the camera poses corresponding to real sparse images using a generative model, and to determine the camera poses corresponding to the pseudo-viewpoint images. The processing module is used to calculate the confidence score of the pseudo-view image through multi-view geometric consistency verification, and to weight the pseudo-view image according to the confidence score; The training module is used to add pseudo-viewpoint images and their corresponding camera poses to the training of the 3D Gaussian model, and to calculate the rendering loss of the pseudo-viewpoint images by weighting the confidence scores, thereby supervising the training of the 3D Gaussian model. During the training of the 3D Gaussian model, the features of the current training view and the neighboring views are compared to identify the high error region of the current training view and calculate the geometric consistency inlier rate of the high error region; based on the geometric consistency inlier rate, a spatial variation modulation field is constructed to dynamically adjust the compaction gradient threshold of the Gaussian points. In the later stages of training the 3D Gaussian model, a generative model is used to correct the rendering results of the 3D Gaussian model under unseen viewpoints to obtain the reconstructed 3D Gaussian model.
[0019] Thirdly, the present invention provides an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the sparse view described above. Figure 3 Vigaussian reconstruction method.
[0020] Fourthly, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the sparse view described above. Figure 3 Vigaussian reconstruction method.
[0021] In summary, this application includes the following beneficial technical effects: This application first performs data augmentation on a real image dataset to generate pseudo-viewpoint images. After data augmentation, the confidence score of the pseudo-viewpoint images is calculated through multi-view geometric consistency verification and used for weighted training, solving the geometric illusion pollution problem in generative data augmentation and improving the effectiveness of training data and the robustness of model initialization. Second, during training, the geometric consistency inlier rate in high-error regions is calculated and a spatial variation modulation field is constructed to dynamically adjust the densification gradient threshold, breaking through the bottleneck of traditional densification strategies being unable to distinguish between under-reconstruction and over-reconstruction, and achieving effective suppression of floating artifacts and accurate reconstruction of details. Finally, in the later stage of training, a generative model is used to actively correct the rendering results under unseen viewpoints, overcoming the defect of excessively smooth texture caused by passive regularization and improving the data quality of the reconstructed 3D Gaussian model. Attached Figure Description
[0022] Figure 1 A sparse view provided in an embodiment of the present invention Figure 3 A flowchart illustrating the VIGAS reconstruction method; Figure 2 A sparse view provided in an embodiment of the present invention Figure 3 Experimental results of the Dimensional Gaussian reconstruction method on the LLFF dataset; Figure 3 A sparse view provided in one embodiment of the present invention Figure 3 Experimental results of the Dimension Gaussian reconstruction method on the Mip-NeRF 360 dataset; Figure 4 An embodiment of the present invention provides an implementation of the sparse view. Figure 3 A schematic diagram of the structure of an electronic device using the Vigaussian reconstruction method.
[0023] Reference numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface.
[0024] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0025] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0026] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0027] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.
[0028] Reference Figure 1 As shown, this is a sparse view provided in an embodiment of the present invention. Figure 3 A flowchart illustrating the Dimensional Gaussian reconstruction method. This method primarily addresses core issues arising from severely lacking geometric constraints under sparse input conditions, such as scene overfitting, geometric structure collapse (e.g., floating artifacts), and difficulty in recovering high-frequency texture details. In this embodiment, the sparse view... Figure 3 The Vigaussian reconstruction method includes: S1. Obtain the real image dataset in the sparse view scene.
[0029] Specifically, the real image dataset includes real sparse images and the camera poses corresponding to the real sparse images; S2. Using a generative model, a pseudo-viewpoint image is synthesized based on the real sparse image and the camera pose corresponding to the real sparse image, and the camera pose corresponding to the pseudo-viewpoint image is determined.
[0030] Specifically, a generative model is used to synthesize a pseudo-viewpoint image based on the real sparse image and the corresponding camera pose, and the camera pose corresponding to the pseudo-viewpoint image is determined, including: S21. Input the real sparse image into the pre-trained view synthesis model to generate a pseudo-view image; The view synthesis model can be a diffusion model, a generative adversarial network, or a video frame interpolation model.
[0031] S22. Using the structure-of-motion motion tool, the generated pseudo-viewpoint image is registered into the original sparse reconstruction model to calculate the camera extrinsic parameters corresponding to the pseudo-viewpoint image, thereby obtaining the camera pose corresponding to the pseudo-viewpoint image.
[0032] In sparse view scenarios, the input contains only a very small set of real images. and its corresponding camera pose To supplement the missing texture and geometric information from the viewpoint, a pre-trained conditional diffusion model is first used for viewspace augmentation.
[0033] Specifically, a pre-trained view synthesis model is used to synthesize realistic sparse images. As a conditional input, generate a new pseudo-viewpoint image. Since the generative model directly outputs images rather than pose-inclusive data, it is necessary to use structure-of-motion (SfM) tools (such as COLMAP) to process the generated images. The generated image is registered into the original sparse reconstruction model, thereby solving for the camera extrinsic parameters. .
[0034] At this point, we have obtained an expanded candidate dataset containing the generated pseudo-viewpoint images. Camera pose and pseudo-view images .
[0035] S3. Calculate the confidence score of the pseudo-view image through multi-view geometric consistency verification, and weight the pseudo-view image according to the confidence score.
[0036] Existing diffusion models often generate content that does not match the geometry of the real scene (i.e., "geometric illusion"), or the pose estimated by COLMAP may be inaccurate. To remove these "dirty data," this method introduces a geometric consistency check.
[0037] The core idea of this step is that if a generated image is real, then it should be able to form the correct epipolar geometry with neighboring real images in three-dimensional space.
[0038] Feature extraction and matching includes the following steps: 1. First, select and generate the view. The closest real-world reference view in terms of spatial distance Use robust local feature extractors (such as SuperPoint) and matchers (such as LightGlue) to extract a set of feature matching pairs between them. : in, and These are the coordinates of two-dimensional feature points in the two images.
[0039] 2. Triangulation and Reprojection: For sets Each pair of matching points Using its corresponding camera pose and Triangulation is performed to recover the three-dimensional spatial points. To verify And the accuracy of image content, 3D points Reproject back to the generated view From the camera plane, the theoretical projection point is obtained. .
[0040] The mathematical expression is as follows: in, This represents the projection transformation function from a three-dimensional world coordinate system to a two-dimensional image coordinate system.
[0041] 3. Reprojection error calculation: Calculate the observation points With theoretical reprojection point The Euclidean distance between them, i.e., the reprojection error : Note: Here The abstract representation represents the process of finding the intersection of back-projected rays; in actual calculations, it is usually solved directly by solving a set of simultaneous equations. .
[0042] The quality of the generated image is quantified using reprojection error. A strict pixel error threshold is set. (For example (pixels).
[0043] Interior point statistics: statistical matching sets In the middle, the reprojection error is less than the threshold. The number of point pairs, these points are called geometrically consistent inliers.
[0044] Calculate the inlier rate as the value of the generated image. The final confidence score is calculated using the following formula: ; in, This represents the confidence score. Represents the coordinates of two-dimensional feature points in a pseudo-viewpoint image. This represents the coordinates of a two-dimensional feature point in the real sparse image that is spatially closest to the pseudo-viewpoint image. This represents the set of feature matching pairs between a pseudo-viewpoint image and the nearest real sparse image in spatial distance from the pseudo-viewpoint image. This indicates the reprojection error. This represents the pixel error threshold.
[0045] high A value of 1 indicates that the generated image closely matches the real image in terms of geometry and has accurate pose estimation, making it a high-quality sample.
[0046] Low The value implies that the generated image may contain a lot of illusions or that SfM pose estimation has failed, and is therefore considered noise.
[0047] This invention innovatively introduces a multi-view geometric consistency verification mechanism in the conditional view synthesis stage. By backprojecting feature points from the generated view into 3D space and reprojecting them back into the reference view, the reprojection error is calculated to determine the inlier ratio. This inlier ratio is then used as a confidence score for soft-weighting the training loss. This mechanism essentially sets a quality gate for the generated data, automatically eliminating or downweighting geometrically inconsistent low-quality samples, ensuring that the model learns geometric information only from high-confidence pseudo-views. This not only effectively expands the sparse training set and solves the cold-start problem, but also eliminates the contamination of the model by erroneous geometric priors at the source, significantly improving the robustness of the initial reconstruction.
[0048] S4. Add the pseudo-viewpoint image and the corresponding camera pose to the training of the 3D Gaussian model, and use the confidence score as the weight to calculate the rendering loss of the pseudo-viewpoint image, thereby supervising the training of the 3D Gaussian model.
[0049] The sparse dataset is augmented with the powerful prior knowledge of generative models. However, to prevent the generative model from generating illusions or incorrect geometric structures, a strict multi-view geometric consistency verification mechanism is introduced, and the verification results are converted into confidence weights to guide the training of the 3D Gaussian (3DGS) model.
[0050] The validated generated images are added to the training process of 3D Gaussian Splatting. To prevent low-quality images from corrupting the model, a soft-weighting strategy is adopted, that is, the confidence scores are used directly. Weights used as the loss function.
[0051] During training, for generating views Render the current 3DGS model to obtain the rendered image. And calculate its enhancement loss. : in: , For miniaturized rasterizers, These are the parameters of the Gaussian model.
[0052] for Norm (Manhattan distance).
[0053] It serves as an adaptive gating mechanism: when At that time, the image is fully trusted, and the model tries its best to fit the view and fill in the missing perspective information.
[0054] when When the gradient of the image is blocked, the model ignores the erroneous sample, thus avoiding overfitting of spurious geometry.
[0055] During the training of the 3D Gaussian model, the features of the current training view and the neighboring views are compared to identify the high error region of the current training view and calculate the geometric consistency inlier rate of the high error region; based on the geometric consistency inlier rate, a spatial variation modulation field is constructed to dynamically adjust the compaction gradient threshold of the Gaussian points. In the later stages of training the 3D Gaussian model, a generative model is used to correct the rendering results of the 3D Gaussian model under unseen viewpoints to obtain the reconstructed 3D Gaussian model.
[0056] Specifically, the steps of identifying high-error regions in the current training view and calculating the geometric consistency inlier rate of high-error regions include: S401. Calculate the photometric residual between the rendered image and the real image of the current training view, and define the region where the residual exceeds the set threshold as the high error region. S402. Select the nearest neighboring view that is spatially closest to the current training view, and extract the matching points between the current training view and the neighboring view that are located within the high error region to generate a subset of feature matching point pairs; S403. Triangulate and reproject the subset of feature matching point pairs using the camera pose of the current training view, calculate the reprojection error, and count the proportion of feature matching point pairs with reprojection errors less than a preset threshold in the subset. Use the proportion of feature matching point pairs with reprojection errors less than a preset threshold in the subset as the geometric consistency inlier rate of the high error region.
[0057] In standard 3D Gaussian geometry, high errors are always considered to require densification; however, in sparse views, high errors may originate from "under-reconstruction" (requiring densification) or "over-reconstruction / floating artifacts" (requiring suppression). This method utilizes multi-view geometric constraints (SfM principle) to decouple these two cases and constructs a spatial modulation field to dynamically adjust the densification threshold.
[0058] During the training process of this method, the system calculates the current viewpoint. The difference between the rendered image and the real image Or, using the photometric residual, an error map is obtained. The high-error region is defined as... From a sparse perspective, The causes are divided into two types: Scenario A (Underweight): The scene geometry is realistic, but details are blurred due to complex textures or insufficient Gaussian spheres. The solution is to enhance densification (lowering the threshold for splitting / cloning).
[0059] Case B (Over-reconstruction): The 3D Gaussian model generates floating points that do not conform to physical laws in regions lacking observations. These points only appear correct (or have errors) from the current viewpoint, but are completely invalid from other viewpoints. The countermeasure is to suppress densification (raise the threshold for splitting / cloning to prevent the spread of erroneous geometry).
[0060] To distinguish between the two situations mentioned above, this method introduces multi-view geometric consistency as a criterion. If the content of a high-error area can find a geometrically consistent corresponding point in a neighboring view, it indicates that it is a true physical structure error caused by situation A; otherwise, it is an artifact error caused by situation B.
[0061] The error source decoupling method based on geometric consistency is as follows: Step 1: Local Feature Extraction and Filtering - Select the Current Training View The nearest neighbor view in space Use a robust feature extractor (such as SuperPoint + LightGlue) to extract the set of all matching feature point pairs between the two. Filter out the high-error areas located in the current view. The set of matching point pairs within : in Views and The coordinates of two-dimensional feature points on the surface; Step 2: Triangulation and reprojection verification uses epipolar geometry principles to verify the three-dimensional consistency of these matching points.
[0062] Back projection / triangulation: utilizing camera pose and matching point pairs Recover the corresponding three-dimensional space points .
[0063] Reprojection: Reprojecting 3D points Reproject back to reference view From the camera plane, the theoretical projection point is obtained. .
[0064] Error calculation: Calculate the actual observation points With theoretical projection point Reprojection error between : (Note: In the formula, Unproj and Proj represent the geometric transformation process from 2D to 3D and back to 2D, which depends on accurate camera intrinsic and extrinsic parameters.) ) Step 3: Calculate the consistency internal point rate and set a geometric error threshold. (Usually at the pixel level, such as 2px). Statistical set Mean error The number of "interior points" is calculated, and the interior point rate of the high error region is also calculated. : ; in, Represents the geometrically consistent interior point rate. This represents the coordinates of two-dimensional feature points in the current training view. Represents the coordinates of a two-dimensional feature point in the nearest neighboring view in the current training view space; Located in the high error area of the current view A subset of matching point pairs within the range.
[0065] This indicates that areas with high errors have strong geometric support in nearby viewpoints, and are considered under-reconstructed (true details are not restored).
[0066] This indicates that a high-error region cannot be found to have a geometrical correspondence in a nearby viewpoint, and is highly likely to be an over-reconstruction (floating artifact).
[0067] (2) Spatial variation modulation field and adaptive densification Based on the intrinsic rate, a spatially-variant modulation field (SVMF) is constructed to dynamically adjust the compaction gradient threshold of 3DGS.
[0068] Modulation function design: Construct a modulation function about monotonically decreasing function This is used to output the final gradient threshold: in: Interior point ratio ; This represents the upper limit of the threshold, corresponding to a higher penalty / higher threshold (e.g., 1.7). This is the lower bound of the threshold, corresponding to a low penalty / low threshold (e.g., 0.5). The hyperparameter for adjusting the curvature of the curve (e.g., 0.5); This function implements the standard gradient threshold. Dynamic scaling or alternatives: when (Strong consistency / under-reconstruction): At this point, the threshold drops to its lowest point. The system determines that the region is a real, complex texture, and therefore lowers the densification threshold, encouraging Gaussian points to be more easily split or cloned, thereby generating more Gaussian points to fit the details.
[0069] when (Weak consistency / artifacts): At this point, the threshold rises to its highest point. The system determines that the region may be floating noise, and therefore significantly increases the densification threshold, making it extremely difficult for Gaussian points to meet the splitting conditions, thereby suppressing the erroneous geometric growth of the region and even removing it in subsequent pruning operations.
[0070] for The region, the threshold is and The smooth transition between these elements allows for precise control over the densification process.
[0071] This invention proposes an "Adaptive Densification Method Based on Multi-View Geometric Consistency (CAAD)". By constructing a local feature matching and reprojection verification process, the geometric consistency inlier rate of each high-error region is quantified, and a Spatial Variation Modulation Field (SVMF) is designed accordingly. This modulation field establishes a dynamic mapping relationship: the densification threshold is reduced in regions with high inlier rates (strong geometric consistency), and the threshold is increased in regions with low inlier rates (severe geometric conflicts). This allows the 3D Gaussian model to accurately identify and suppress false floating points (over-reconstructed regions) caused by a lack of viewpoints, while simultaneously refining the complex textures of the real structure (under-reconstructed regions). This targeted strategy completely solves the problem of geometric collapse under sparse views, preserving high-frequency details of the scene to the maximum extent while removing artifacts.
[0072] In this embodiment, a reconstructed 3D Gaussian model is obtained by correcting the rendering results of the 3D Gaussian model from an unseen viewpoint using a generative model, including: S410. Select a reference pose from the training view pose set, apply isotropic Gaussian noise to its translation vector, and generate a new pseudo translation vector. S420. Find the two reference poses in the training set that are closest to the new pseudo-translation vector in Euclidean distance, and calculate the interpolation factor based on the distance. S430. Using the spherical linear interpolation algorithm, the rotation quaternions of the two reference poses that are closest to the new pseudo-translation vector in Euclidean distance are interpolated according to the interpolation factor to obtain smooth pseudo-rotation quaternions. S440. Combine the pseudo-translation vector and pseudo-rotation quaternion to form a complete pseudo-camera pose; S450. Use the current 3D Gaussian model to perform differentiable rendering under the pseudo-camera pose to obtain the initial pseudo-view. S460. Input the initial pseudo-view into the pre-trained generative image refining model and output the refined pseudo-real image. S470. Calculate the difference loss between the initial pseudo-view and the pseudo-true image, backpropagate the gradient of the difference loss, update the parameters of the three-dimensional Gaussian model, and obtain the reconstructed three-dimensional Gaussian model.
[0073] In the mid-to-late stages of training, a generative model (such as a diffusion model) is used as a teacher to correct and refine the rendering results of the 3D Gaussian model (3DGS) under unseen viewpoints. The repaired image is then used as a pseudo-ground truth to construct a closed-loop feedback loop that guides the parameter updates of the 3DGS. This solves the problems of local optima and texture blurring caused by the lack of supervision signals in sparse views.
[0074] In one specific implementation of this embodiment, the steps for correcting the rendering result of the 3D Gaussian model under an unseen viewpoint are as follows: (1) Pseudo-camera pose construction; To ensure the model receives supervision even in unseen regions, a series of geometrically plausible and smoothly transitioning new observation perspectives must first be generated. Random poses cannot be generated arbitrarily, as this could lead to uncontrollable output from the generated model. Therefore, this method employs an interpolation perturbation strategy based on the reference pose.
[0075] A complete camera pose From the translation vector and rotation quaternions Definition; from an existing set of training view poses Select a reference pose Within a local neighborhood centered on this point, isotropic Gaussian noise is applied to generate a new pseudo-translation vector. : in: To control the standard deviation hyperparameter of the disturbance range (to ensure that the new perspective does not deviate too far from the scene); It is an identity matrix.
[0076] Rotation Quaternion Interpolation: In order to ensure that the camera is still "looking" at the subject of the scene, the new rotation parameters cannot simply add noise, but need to be interpolated according to the position change.
[0077] Find new locations in the training set. The two reference poses closest to each other in Euclidean distance are denoted as follows: and .
[0078] calculate Distance to these two neighbors and Define the interpolation factor. : ; (Note: This weighting design makes...) Leave The closer, The smaller the value, the more biased the SLERP results become. ) Calculate smooth rotations in quaternion space using the SLERP algorithm for spherical linear interpolation. :
[0079] in, It is the angle between two quaternions.
[0080] Ultimately, Convert to rotation matrix ,and Combined into a complete pseudo camera pose .
[0081] (2) Initial pseudo-view rendering Using the current training phase of the 3D Gaussian model (Including parameters such as the position, covariance, and color of all Gaussian spheres), processed by a differentiable rasterizer. In the generated pseudo pose Rendering will proceed as follows: State analysis: Due to This represents a perspective not covered in the training set, and the model may not have fully converged or may be overfitting at this point. It typically includes defects such as geometric artifacts (e.g., floaters, background collapse) and / or missing textures (blurring or holes due to sparse viewpoints).
[0082] (3) Generative Image Refinement Those with defects Input into a pre-trained generative image refining model (For example, Image-to-Image based Diffusion Models such as DiFix or ControlNet).
[0083] This generative model leverages its prior knowledge of natural images learned from large-scale data to... Denoising and completion are performed to output a high-quality, refined image. : This is considered a potential ground truth image from this perspective. The generative model acts as the teacher here, correcting the non-physical artifacts rendered by the student (3DGS) and filling in reasonable texture details.
[0084] (4) Closed-loop self-supervised update A loss function is constructed to force the rendering result of the 3DGS model to converge with the refined image, thereby forming a closed-loop optimization.
[0085] Loss calculation: Calculate the initial rendered image With refined images The difference loss between them (usually used) loss): (Note: Here) The gradient is truncated, meaning the generative model is not updated; only the 3DGS parameters are updated. ) Parameter update: The gradient is backpropagated to the 3DGS property parameters (position). Rotation Scaling Opacity spherical harmonic coefficient .
[0086] Closed-loop effect: As training progresses, The quality will gradually improve Generative Models Fewer and fewer repairs are needed. and Gradually converging. This process helps 3DGS models escape local optima from a sparse perspective and recover a continuous geometric structure that is more in line with physical laws.
[0087] This invention constructs a closed-loop self-supervised optimization (CLSS-GR) loop based on generative refinement. An initial image is rendered by generating geometrically smooth pseudo-camera poses, and a pre-trained generative inpainting model is used as a teacher to actively repair imperfections in the rendered image. The repaired, high-quality image is then used as a pseudo-ground value to guide 3DGS updates. This mechanism transforms the powerful prior knowledge of the generative model into a supervisory signal for active error correction. It not only fills in texture holes in unobserved areas but also guides the model out of local optima caused by sparse input. Compared to traditional smoothing regularization, this method can recover sharper, more realistic high-frequency texture details, significantly improving the perceptual quality of new perspective synthesis.
[0088] To verify the sparse view of this application Figure 3The effectiveness of the VIGAS reconstruction method was experimentally verified on sparse input settings (such as LLFF 3 views and Mip-NeRF360 24 views) of mainstream datasets such as LLFF and Mip-NeRF360. See Tables 1 and 2 for details. Figure 2 and Figure 3 This application has sparse vision. Figure 3 The Vigaussian reconstruction method significantly outperforms existing state-of-the-art methods (such as FSGS and SparseGS) in key metrics such as PSNR, SSIM, and LPIPS. It is particularly effective in eliminating background floating noise and restoring the fine structure of objects, and has extremely high practical value.
[0089] Table 1. Experimental results of the method in this application on the LLFF dataset.
[0090] Table 2. Experimental results of the method in this application on the Mip-NeRF 360 dataset.
[0091] Based on the same inventive concept, an embodiment of the present invention provides a sparse view Figure 3 Vigos Reconstruction System.
[0092] The sparse view described in this invention Figure 3 The Vigaussian reconstruction system can be installed in electronic devices. Depending on the implemented function, the sparse view... Figure 3 The Vigaussian reconstruction system includes: The acquisition module is used to acquire the real image dataset in the sparse view scene. The real image dataset includes real sparse images and the camera poses corresponding to the real sparse images. The data augmentation module is used to synthesize pseudo-viewpoint images based on real sparse images and the camera poses corresponding to real sparse images using a generative model, and to determine the camera poses corresponding to the pseudo-viewpoint images. The processing module is used to calculate the confidence score of the pseudo-view image through multi-view geometric consistency verification, and to weight the pseudo-view image according to the confidence score; The training module is used to add pseudo-viewpoint images and their corresponding camera poses to the training of the 3D Gaussian model, and to calculate the rendering loss of the pseudo-viewpoint images by weighting the confidence scores, thereby supervising the training of the 3D Gaussian model. During the training of the 3D Gaussian model, the features of the current training view and the neighboring views are compared to identify the high error region of the current training view and calculate the geometric consistency inlier rate of the high error region; based on the geometric consistency inlier rate, a spatial variation modulation field is constructed to dynamically adjust the compaction gradient threshold of the Gaussian points. In the later stages of training the 3D Gaussian model, a generative model is used to correct the rendering results of the 3D Gaussian model under unseen viewpoints to obtain the reconstructed 3D Gaussian model.
[0093] The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.
[0094] The sparse view provided in the above embodiments Figure 3 The various variations and specific examples in the Gaussian reconstruction method are also applicable to the sparse view in this embodiment. Figure 3 The Vigaussian reconstruction system, through the aforementioned sparse view Figure 3 A detailed description of the Gaussian reconstruction method will clearly demonstrate to those skilled in the art the sparse view in this embodiment. Figure 3 The implementation method of the Vigaussian reconstruction system will not be described in detail here for the sake of brevity.
[0095] This application also discloses an electronic device, such as Figure 4 As shown, this is a sparse view provided by an embodiment of the present invention. Figure 3 A schematic diagram of an electronic device for the Vigaussian reconstruction method. The electronic device may include at least one processor 10, a memory 11 communicatively connected to the at least one processor, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as a sparse view... Figure 3 The method and procedure for Vigos reconstruction.
[0096] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It runs or executes programs or modules stored in the memory 11 (e.g., executing sparse view...). Figure 3 Methods such as Vigos reconstruction, and calling data stored in memory 11 to perform various functions of the electronic device and process data.
[0097] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software installed on the electronic device and various types of data, such as sparse visual data... Figure 3 The code for the Vigaussian reconstruction method can also be used to temporarily store data that has already been output or will be output.
[0098] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.
[0099] Communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), and optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.
[0100] Figure 4 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 4The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0101] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to at least one processor 10 via a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be elaborated further here.
[0102] It should be understood that the embodiments are for illustrative purposes only and are not limited to this structure in the scope of the patent application.
[0103] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.
[0104] This application provides a computer-readable storage medium, including, for example, any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM). This computer-readable storage medium stores sparse views that can be loaded by a processor and executed according to the above embodiments. Figure 3 A computer program for the Vigaussian reconstruction method.
[0105] In the description of this specification, the references to terms such as "an embodiment," "some embodiments," "example," "specific example," "a implementation," "a preferred implementation," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0106] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
Claims
1. A method for 3D Gaussian reconstruction of sparse views, characterized in that, The method includes: Obtain a real image dataset for sparse view scenes. The real image dataset includes real sparse images and the camera poses corresponding to the real sparse images. A generative model is used to synthesize pseudo-viewpoint images based on real sparse images and the camera poses corresponding to the real sparse images, and the camera poses corresponding to the pseudo-viewpoint images are determined. The confidence score of the pseudo-view image is calculated by multi-view geometric consistency verification, and the pseudo-view image is weighted according to the confidence score. The pseudo-viewpoint images and their corresponding camera poses are added to the training of the 3D Gaussian model. The rendering loss of the pseudo-viewpoint images is calculated by weighting the confidence scores as weights, and the 3D Gaussian model is trained under supervision. During the training of the 3D Gaussian model, the features of the current training view and the neighboring views are compared to identify the high error region of the current training view and calculate the geometric consistency inlier rate of the high error region; based on the geometric consistency inlier rate, a spatial variation modulation field is constructed to dynamically adjust the compaction gradient threshold of the Gaussian points. In the later stages of training the 3D Gaussian model, a generative model is used to correct the rendering results of the 3D Gaussian model under unseen viewpoints to obtain the reconstructed 3D Gaussian model.
2. The sparse view 3D Gaussian reconstruction method as described in claim 1, characterized in that, The process of synthesizing pseudo-viewpoint images based on real sparse images and their corresponding camera poses using a generative model, and determining the camera poses corresponding to the pseudo-viewpoint images, includes: Real sparse images are input into a pre-trained view synthesis model to generate pseudo-viewpoint images; The generated pseudo-viewpoint image is registered into the original sparse reconstruction model using the motion reconstruction tool to calculate the camera extrinsic parameters corresponding to the pseudo-viewpoint image, thereby obtaining the camera pose corresponding to the pseudo-viewpoint image.
3. The sparse view 3D Gaussian reconstruction method as described in claim 1, characterized in that, The steps for identifying high-error regions in the current training view and calculating the geometric consistency inlier rate for these high-error regions include: Calculate the photometric residual between the rendered image and the real image of the current training view, and define the region where the residual exceeds a set threshold as a high error region; Select the nearest neighboring view that is spatially closest to the current training view, and extract the matching points between the current training view and the neighboring view that are located within the high error region to generate a subset of feature matching point pairs; The camera pose of the current training view is used to triangulate and reproject a subset of feature matching point pairs. The reprojection error is calculated, and the proportion of feature matching point pairs with reprojection errors less than a preset threshold in the subset is counted. The proportion of feature matching point pairs with reprojection errors less than the preset threshold in the subset is used as the geometric consistency inlier rate of the high error region.
4. The sparse view 3D Gaussian reconstruction method as described in claim 3, characterized in that, The view synthesis model can be a diffusion model, a generative adversarial network, or a video frame interpolation model.
5. The sparse view 3D Gaussian reconstruction method as described in claim 1, characterized in that, The formula for calculating the confidence score is: ； in, This represents the confidence score. Represents the coordinates of two-dimensional feature points in a pseudo-viewpoint image. This represents the coordinates of a two-dimensional feature point in the real sparse image that is spatially closest to the pseudo-viewpoint image. This represents the set of feature matching pairs between a pseudo-viewpoint image and the nearest real sparse image in spatial distance from the pseudo-viewpoint image. This indicates the reprojection error. This represents the pixel error threshold.
6. The sparse view 3D Gaussian reconstruction method as described in claim 5, characterized in that, The formula for calculating the geometric consistency interior point ratio is as follows: ； in, Indicates the geometrically consistent interior point rate. This represents the coordinates of two-dimensional feature points in the current training view. Represents the coordinates of a two-dimensional feature point in the nearest neighboring view in the current training view space; Located in the high error area of the current view A subset of matching point pairs within.
7. The sparse view three-dimensional Gaussian reconstruction method according to any one of claims 1 to 6, characterized in that, The process of using a generative model to correct the rendering results of a 3D Gaussian model under an unseen viewpoint to obtain a reconstructed 3D Gaussian model includes: Select a reference pose from the training view pose set, apply isotropic Gaussian noise to its translation vector, and generate a new pseudo translation vector; Find the two reference poses in the training set that are closest to the new pseudo-translation vector in Euclidean distance, and calculate the interpolation factor based on the distance; Using the spherical linear interpolation algorithm, the rotation quaternions of the two reference poses that are closest to the new pseudo-translation vector in Euclidean distance are interpolated according to the interpolation factor to obtain smooth pseudo-rotation quaternions. The pseudo-translation vector and pseudo-rotation quaternion are combined to form a complete pseudo-camera pose; The initial pseudo-view is obtained by performing differentiable rendering of the current 3D Gaussian model under the pseudo-camera pose. The initial pseudo-view is input into the pre-trained generative image refining model, which outputs a refined pseudo-ground image. The difference loss between the initial pseudo-view and the pseudo-real image is calculated, the gradient of the difference loss is backpropagated, the parameters of the 3D Gaussian model are updated, and the reconstructed 3D Gaussian model is obtained.
8. A sparse view 3D Gaussian reconstruction system, used to implement the sparse view 3D Gaussian reconstruction method according to any one of claims 1 to 7, characterized in that, include: The acquisition module is used to acquire the real image dataset in the sparse view scene. The real image dataset includes real sparse images and the camera poses corresponding to the real sparse images. The data augmentation module is used to synthesize pseudo-viewpoint images based on real sparse images and the camera poses corresponding to real sparse images using a generative model, and to determine the camera poses corresponding to the pseudo-viewpoint images. The processing module is used to calculate the confidence score of the pseudo-view image through multi-view geometric consistency verification, and to weight the pseudo-view image according to the confidence score; The training module is used to add pseudo-viewpoint images and their corresponding camera poses to the training of the 3D Gaussian model, and to calculate the rendering loss of the pseudo-viewpoint images by weighting the confidence scores, thereby supervising the training of the 3D Gaussian model. During the training of the 3D Gaussian model, the features of the current training view and the neighboring views are compared to identify the high error region of the current training view and calculate the geometric consistency inlier rate of the high error region; based on the geometric consistency inlier rate, a spatial variation modulation field is constructed to dynamically adjust the compaction gradient threshold of the Gaussian points. In the later stages of training the 3D Gaussian model, a generative model is used to correct the rendering results of the 3D Gaussian model under unseen viewpoints to obtain the reconstructed 3D Gaussian model.
9. An electronic device, characterized in that, The electronic device includes: At least one processor (10); and, A memory (11) communicatively connected to the at least one processor (10); The memory (11) stores a computer program that can be executed by the at least one processor (10) to enable the at least one processor (10) to perform the sparse view three-dimensional Gaussian reconstruction method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program; when the computer program is executed by a processor, it implements the sparse view three-dimensional Gaussian reconstruction method as described in any one of claims 1 to 7.