3D Gaussian splashing method for sparse camera visual angle
By optimizing the 3D Gaussian model through hybrid viewpoint layered depth regularization, scale regularization, and dual structure perception loss, the problems of poor rendering quality and artifacts under sparse viewpoints are solved, generating a new view with more accurate geometry and clearer textures.
Patent Information
- Application Number
- CN202511492653.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-20
- Publication Date
- 2026-02-06
AI Technical Summary
Existing sparse-view 3D Gaussian splashing methods have shortcomings in geometric accuracy and artifact handling. In particular, the depth estimator output is subject to severe noise interference under sparse viewpoints, resulting in poor rendering quality. Existing technologies have failed to effectively solve the artifact problem.
By combining viewpoint-layered depth regularization, scale regularization, and dual structure-aware loss, the 3D Gaussian model is optimized to generate more accurate Gaussian primitives. The rendering process is improved by combining multiple loss functions, including hybrid viewpoint-layered depth loss, scale regularization loss, and dual structure-aware loss.
It achieves more accurate geometric structure capture, reduces artifacts, improves texture structure clarity, and generates high-quality new views.
Smart Images
Figure CN121482233A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and graphics technology, and relates to a new view synthesis technology, specifically a 3D Gaussian splashing method for sparse camera perspectives. Background Technology
[0002] Novel view synthesis (NVS) technology aims to generate rendering results from arbitrary viewpoints from inputs from multiple camera perspectives. While traditional neural radiation field (NeRF)-based methods can achieve high-quality rendering results, their training and inference speeds are relatively slow, making them difficult to meet the needs of real-time applications.
[0003] In recent years, the 3D Gaussian Splash (3DGS) method, as an explicit representation method, has attracted widespread attention due to its real-time rendering speed, high-quality rendering, and fast training speed. However, in real-world scenarios, acquiring images with sufficient viewpoints is costly, so we study 3DGS techniques for sparse viewpoints. However, when 3DGS techniques are designed for sparse viewpoints, overfitting often occurs due to insufficient geometric cues. Furthermore, the rendering quality of 3D Gaussian techniques largely depends on pre-computed point clouds, but in the case of sparse input views, due to the lack of rich supervision information, Structure of Motion (SfM) often struggles to generate reliable point clouds. Point cloud degradation leads to excessively large Gaussian units in low-density areas, especially at image edges, resulting in significant artifacts that severely impact the quality of the new view. Existing 3DGS techniques for sparse viewpoints still suffer from the following problems: poor geometric accuracy, as noise interference in the depth estimator output affects the accuracy of depth information; and the artifact problem remains unresolved. Existing techniques do not employ specific processing methods for artifacts generated during reconstruction, resulting in most artifacts still present in the final reconstruction result. Summary of the Invention
[0004] To address the aforementioned technical problems in the existing technology, this invention proposes a 3D Gaussian splashing method for sparse camera perspectives, the specific technical solution of which is as follows: A 3D Gaussian splashing method for sparse camera viewpoints includes: Step 1: Acquire sparse input images through the camera, and use a 3D Gaussian model for initialization to generate Gaussian primitives. The sparse input images include training viewpoint images. Step 2: Reduce the size of the Gaussian element by minimizing the volume prior, and obtain the optimized Gaussian element by calculating the scale regularization loss; Step 3: Generate a pseudo camera and acquire pseudo viewpoint images. Extract depth information from the training viewpoint images and pseudo viewpoint images, and align the layer depth of the hybrid viewpoints by calculating the hybrid viewpoint layered depth loss. Step 4: Reassemble the training viewpoint image and the pseudo-viewpoint image, and calculate the dual-structure perception loss to align the global and local structures; Step 5: Optimize the 3D Gaussian model by combining the hybrid viewpoint alignment depth loss, dual structure perception loss, and scale regularization loss. Step 6: Use the optimized 3D Gaussian model to render the output result through the Gaussian splashing process.
[0005] Furthermore, step 1 specifically includes: Step 1-1: Take several overlapping scene images from different angles using a camera; Steps 1-2: Using the Colmap tool and based on the SFM structure-of-motion (SOFM) technique, for each sparse input image, find the correspondence between feature points in different images to achieve feature point association, generate a sparse point cloud, and solve the camera's intrinsic and extrinsic parameters. Steps 1-3: Use the 3D coordinates of the sparse point cloud as the mean μ of the Gaussian elements, and initialize the opacity, rotation coefficient, and spherical harmonic coefficient of the Gaussian elements.
[0006] Furthermore, step 2 specifically includes: Step 2-1: Use the KNN algorithm to obtain the scaling factor s of the Gaussian elements, and extract all Gaussian elements in the scene sequentially. The three scaling factors The size of each Gaussian element is obtained by multiplying its three scaling factors together. Step 2-2: Multiply the dimensions of all Gaussian elements in the scene by their three scaling factors, then sum them sequentially. Constrain the dimensions of the Gaussian elements using scale regularization loss, where the scale regularization loss is: .
[0007] Furthermore, step 3 specifically includes: Step 3-1: Select the two closest training viewpoints from different perspectives and calculate the average camera orientation P. t and the average quaternion q t Then, a pseudo-camera is generated by interpolating between the two, thus obtaining the pseudo-viewpoint. Add white noise at the 3-DOF camera position Finally, a pseudo-viewpoint image is rendered. ; Step 3-2: Use a pre-trained monocular depth estimator to extract training viewpoint images. and pseudo-viewpoint images Estimated depth and To supplement geometric cues; the viewpoint image and pseudo-viewpoint image are trained using a blending calculation with an opacity value α, resulting in rendering depths for both. and At the same time, the entire estimated depth map and The image is divided into m blocks of the same size, and the depth map is as follows: , , and ; Step 3-3: Calculate the loss between estimated depth and rendering depth using the Pearson correlation coefficient (PCC).
[0008] Furthermore, in step 3-3, the calculation of the loss between the estimated depth and the rendering depth using the Pearson correlation coefficient (PCC) specifically involves: first, calculating the depth loss of the training viewpoint. Then calculate the hierarchical depth loss of the training viewpoint. Then the total depth loss of the training viewpoint is Similarly, the depth loss and hierarchical depth loss under pseudo-viewpoints are calculated as follows: and The total depth loss under pseudo-viewpoints is , The asymptotic coefficients are denoted by n, which represents the number of training epochs. The final depth loss, obtained by combining the training viewpoint and pseudo-viewpoint depth loss, is the hybrid viewpoint alignment depth loss. .
[0009] Furthermore, step 4 specifically includes: Step 4-1: Use SSIM structural similarity calculation to train viewpoint images and rendering images Local structural similarity between them: ; Step 4-2: Transfer the training viewpoint images Divided into N small image blocks each small image block The pixels within the image are randomly shuffled K times, and these K recombined image blocks are stitched together according to the width dimension to obtain the recombined large image block. There are a total of N; then the image is rendered. Do the same operation to get and The global structural similarity P-SSIM is calculated using the following expression: , ; Step 4-3: Weighted fusion of local structural similarity and global structural similarity.
[0010] Furthermore, in step 5, the 3D Gaussian model incorporates conventional color L1 loss as a basic loss term.
[0011] Furthermore, in step 5, the 3D Gaussian model employs the Adam optimizer to perform iterative optimization operations during the optimization of model parameters.
[0012] Furthermore, step 6 specifically includes: Step 6-1: Calculate the position and shape of the 3D Gaussian elements projected onto the 2D plane using the Jacobian matrix and camera intrinsic and extrinsic parameters; Step 6-2: Divide the image into several tiles to achieve parallel processing, and calculate the Gaussian elements of the depth-sorted projection to calculate their color contribution to the pixel. Use the transparency value α to mix and calculate the final pixel color.
[0013] Compared with the prior art, the present invention has the following beneficial effects: 1) More accurate geometric structure: By using hybrid viewpoint layered depth regularization, the geometric structure of the scene can be captured more accurately, and the noise of the depth estimator can be dealt with to a limited extent, making the spatial position of the Gaussian unit more accurate; 2) Fewer artifacts: By using scale regularization, smaller and more compact Gaussian units are obtained, thereby reducing the impact of artifacts; 3) Clearer texture structure: By using dual structure perception loss to capture the global and local structural relationships, the texture structure of the image is clearer and more reasonable. Attached Figure Description
[0014] Figure 1 This is a flowchart illustrating the overall process of a 3D Gaussian splashing method for sparse camera perspectives according to an embodiment of the present invention. Detailed Implementation
[0015] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0016] like Figure 1 As shown, this embodiment of a 3D Gaussian splashing method for sparse camera viewpoints includes: Step 1: Obtain the sparse input image and perform initialization processing to generate Gaussian primitives; details are as follows: Step 1-1: Prepare sparse input images. Take several overlapping scene images from different perspectives using a camera.
[0017] Steps 1-2: Obtain sparse point cloud. Using the Colmap tool and based on SFM (Structured Motion Recovery) technology, for each sparse input image, the correspondence between feature points in different images is found to achieve feature point association, generate sparse point cloud, and solve for the camera's intrinsic and extrinsic parameters.
[0018] Steps 1-3: Generate initial Gaussian elements. Use the 3D coordinates of the sparse point cloud as the mean μ of the Gaussian elements, and initialize the opacity, rotation coefficient, and spherical harmonic coefficient of the Gaussian elements.
[0019] Step 2: Reduce the size of the Gaussian primitives using the volume minimization prior to obtain the optimized Gaussian primitives; the details are as follows: Step 2-1: Extract Gaussian pixel dimensions. Use the KNN algorithm to obtain the scaling factor s of the Gaussian pixels, and then extract all Gaussian pixels in the scene sequentially. The three scaling factors The size of each Gaussian element is obtained by multiplying its three scaling factors together.
[0020] Step 2-2: Calculate the scaling regularization loss: The dimensions of all Gaussian elements in the scene are summed sequentially after multiplying by their three scaling factors. The size of the Gaussian elements is then constrained by a scale regularization loss to reduce artifacts and produce a more detailed image.
[0021] In step 2 above, scale regularization calculates the product of three scaling factors for each Gaussian cell in the scene and sums these products to obtain the relevant loss. This loss allows the Gaussian cells in the scene to be controlled to be smaller and more compact, thus resolving artifacts and blurring issues caused by large-volume Gaussian cells in edge regions.
[0022] Step 3: Generate a pseudo-camera using the camera training viewpoint and acquire pseudo-viewpoint images. Use the hybrid viewpoint hierarchical depth regularization module to perform hierarchical depth alignment between the training viewpoint and the pseudo-viewpoint, extract the depth information of the training viewpoint and the pseudo-viewpoint, and calculate the hybrid viewpoint hierarchical depth loss. This drives the Gaussian elements to move to the appropriate position, achieving accurate geometric constraints for the Gaussian elements. Specifically: Step 3-1: Synthesize pseudo-viewpoints. Specifically, the synthesized pseudo-viewpoints... Yes: Select the two closest training viewpoints and calculate their average camera orientation P. t and the average quaternion q t A pseudo-camera is generated by interpolating between the two. Additionally, white noise is added to the 3-DOF camera position. To enhance the diversity of camera positions. Finally, pseudo-viewpoint images are rendered based on these generated camera parameters. .
[0023] Step 3-2: Obtain depth information. Use a pre-trained monocular depth estimator (DepthAnythingV2) to extract training viewpoint images. and pseudo-viewpoint images Estimated depth information and To supplement geometric cues and train viewpoint images That is, images taken by the camera from different viewpoints for training are used to mix and calculate the training viewpoint image and the pseudo viewpoint image using the transparency value α, resulting in rendering depths for both. and At the same time, the entire estimated depth map and The image is divided into m blocks of the same size, and the depth map is as follows: , , and .
[0024] Step 3-3: Align the hierarchical depth of the hybrid viewpoint. Calculate the loss between the estimated depth and the rendered depth using the Pearson correlation coefficient (PCC): First, calculate the depth loss for the training viewpoint. Then calculate the hierarchical depth loss of the training viewpoint. Then the total depth loss of the training viewpoint is Similarly, the depth loss under pseudo-viewpoints and the hierarchical depth loss are calculated as follows: and The total depth loss under pseudo-viewpoints is , The asymptotic coefficient is denoted by n, which represents the number of training epochs. The final depth loss, calculated by combining the training viewpoint and the pseudo-viewpoint, yields the hybrid viewpoint alignment depth loss. .
[0025] In step 3 above, the hybrid viewpoint hierarchical depth regularization module extracts depth information from the input viewpoint image and the synthesized pseudo-viewpoint image, and performs hierarchical depth loss calculation on the extracted depth information and the corresponding viewpoint's rendering depth. For the synthesized pseudo-viewpoint, its camera parameters are obtained by averaging the camera parameters of the two input viewpoints and adding white noise, thus obtaining the corresponding pseudo-viewpoint's rendering image. For the input image and pseudo-viewpoint image, a high-performance monocular depth estimator is used to obtain their respective depth maps. Then, the above depth maps and the corresponding viewpoint's rendered depth maps are subjected to Pearson correlation loss calculation for the entire depth map and the block depth maps, respectively, to address the geometric imperfection problem.
[0026] Step 4: Divide the training viewpoint image and the pseudo viewpoint image into blocks and reassemble them. Use dual structure-aware loss to align the global and local structures, balancing the global and local structural information to make the generated image's texture structure clear and reasonable; the details are as follows:
[0027] Step 4-1: Calculate structural similarity. This invention uses SSIM structural similarity to calculate the training viewpoint image. and rendering images Local structural similarity between them: .
[0028] Step 4-2: Reassemble image patches and calculate global structural similarity. This invention uses training viewpoint images... Divided into N small image blocks each small image block The pixels within the image are randomly shuffled K times, and these K recombined image blocks are stitched together according to the width dimension to obtain the recombined large image block. There are a total of N. Then, the image is rendered. Do the same operation to get and .
[0029] After reconstructing the image patches, the global (long-distance) structural similarity P-SSIM is calculated, as shown in the following expression: , .
[0030] Step 4-3: Calculate the dual structure perception loss. This invention weightedly fuses local and global structural similarities to balance global and local structural similarities, resulting in a clearer structure and texture in the final rendered image. .
[0031] In step 4 above, the dual-structure perception loss module calculates the local and global structural similarity between the rendered image and the training image. Traditional structural similarity metrics can only compare the structures of adjacent pixels. To address this, the input image and its corresponding rendered image are divided into several small blocks. The pixels within each block are then randomly shuffled and rearranged, and the blocks are stitched together according to their width to form a larger image block. By calculating the structural similarity metric of this larger image block, comparisons of long-distance structural relationships are achieved. Through weighted fusion of local and global structural similarity, both global and local structural relationships can be captured simultaneously, resulting in a clearer and more reasonable texture structure in the rendered image.
[0032] Step 5: Optimize the 3D Gaussian model using various loss functions; details are as follows: Step 5-1: Define the total loss function of the model. This invention first introduces the conventional color L1 loss as the basic loss term. Based on this, it further incorporates hybrid viewpoint layered depth loss and scale regularization loss to reduce artifacts; finally, to improve the structural consistency of the rendering results, an additional dual structure-aware supervision term is introduced, together forming the complete total loss function: .
[0033] Step 5-2, Optimization Process. During the optimization of model parameters, the Adam optimizer is used to perform iterative optimization operations. The initial learning rate is set to 0.01, and it gradually decreases during training. , , , Set them to 0.8, 0.05, 0.01, and 0.2 respectively.
[0034] Step 6: Render the output result using the Gaussian splashing process. Details are as follows: Step 6-1: Project 3D Gaussian elements onto the camera plane. Calculate the position and shape of the 3D Gaussian projection onto the 2D plane using the Jacobian matrix and camera intrinsic and extrinsic parameters.
[0035] Step 6-2, Rasterization. The image is divided into several tiles for parallel processing, and the Gaussian elements of the projection are sorted according to depth. Their color contribution to the pixels is calculated, and the final pixel color is obtained using alpha mixing.
[0036] In summary, compared to traditional view synthesis methods, the method of this invention, by integrating multiple regularization mechanisms and achieving synergistic effects, exhibits significant advantages in sparse input scenarios, generating new views with higher geometric accuracy, clearer texture rendering, and richer detail. In particular, key components specifically designed for sparse viewpoint scenarios, such as the hybrid viewpoint hierarchical depth regularization module, scale regularization module, and dual-structure perception loss, effectively suppress overfitting and artifact problems during scene reconstruction, thus providing an efficient and feasible technical solution for fields such as digital twins and immersive interaction.
[0037] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A 3D Gaussian splatting method oriented to sparse camera view, characterized in that, The method comprises the following steps: Step 1, acquiring a sparse input image by a camera, and performing initialization processing on the sparse input image by using a 3D Gaussian model to generate a Gaussian cell, wherein the sparse input image comprises a training viewpoint image; Step 2, reducing the size of the Gaussian cell by using a volume minimization prior, and obtaining an optimized Gaussian cell by calculating a scale regularization loss; Step 3, generating a pseudo camera and acquiring a pseudo viewpoint image, extracting depth information of the training viewpoint image and the pseudo viewpoint image, and aligning hierarchical depth of the mixed viewpoint by calculating a mixed viewpoint hierarchical depth loss; Step 4, recombining the training viewpoint image and the pseudo viewpoint image, and calculating a double structure perception loss to align global and local structures; Step 5, optimizing the 3D Gaussian model by combining the mixed viewpoint aligned depth loss, the double structure perception loss and the scale regularization loss; Step 6, using the optimized 3D Gaussian model to render an output result by a Gaussian splatting process.
2. The 3D Gaussian spatter method of claim 1, wherein, The step 1 specifically comprises: Step 1-1, taking a plurality of scene images with an overlapping degree from different angles by using a camera; Step 1-2, using a Colmap tool to find a corresponding relationship of feature points between different images based on a SFM (Structure From Motion) technology, realizing feature point association, generating a sparse point cloud, and solving internal and external parameters of the camera for each sparse input image; Step 1-3, taking 3D coordinates of the sparse point cloud as a mean value μ of the Gaussian cell, and initializing opacity, rotation coefficients and spherical harmonic coefficients of the Gaussian cell.
3. The 3D Gaussian spatter method of claim 1, wherein, The step 2 specifically comprises: Step 2-1, obtain the scaling coefficients s of the Gaussian cell by using KNN algorithm, and sequentially extract all Gaussian cells in the scene three scaling coefficients , multiply the three scaling coefficients of each Gaussian cell as its size; Step 2-2, adding sizes of all Gaussian cells in the scene in sequence after multiplying three scaling coefficients of the Gaussian cells, and constraining the size of the Gaussian cell by a scale regularization loss, wherein the scale regularization loss is: 。 4. The 3D Gaussian spatter method of claim 1, wherein, The step 3 specifically comprises: Step 3-1, select two closest training views from different perspectives, calculate the average camera direction P t and the average quaternion q t , and interpolate between the two to generate a pseudo camera, i.e. get the pseudo view point Add white noise to the 3-DOF camera position , and finally render the pseudo view point image ; Step 3-2, using the pre-trained monocular depth estimator, extract the training view image and the estimated depth of the pseudo view image ; supplement the geometric cues with the estimated depth of the training view image and the pseudo view image and ; mix the calculated training view image and the pseudo view image with the transparency value a to obtain the rendering depth of the two respectively and , and divide the whole estimated depth map and into m small block depth maps of the same size respectively , , and ; Step 3-3, calculating a loss between estimated depth and rendered depth by using a Pearson correlation coefficient (PCC).
5. The 3D Gaussian spatter method of claim 4, wherein, In step 3-3, the loss between the estimated depth and the rendered depth is calculated using the Pearson correlation coefficient PCC, which is specifically: first, the depth loss of the training viewpoint is calculated Then, the hierarchical depth loss of the training viewpoint is calculated The total depth loss of the training viewpoint is ; similarly, the depth loss and the hierarchical depth loss of the pseudo viewpoint are calculated as and The total depth loss of the pseudo viewpoint is , is the progressive coefficient, and n is the training round number; finally, the total depth loss of the mixed training viewpoint and the pseudo viewpoint is obtained by mixing the depth loss of the mixed viewpoint .
6. The 3D Gaussian spatter method of claim 4, wherein, The step 4 specifically comprises: Step 4-1, Compute local structural similarity between training view images using SSIM structural similarity and rendered images : ; Step 4-2, training the view image into N small image blocks , randomly shuffle the pixels in each small image block K times, and splice the K reorganized image blocks according to the width dimension to obtain a reorganized large image block , a total of N; then perform the same operation on the rendered image to obtain and ; calculate the global structural similarity P-SSIM, expressed as follows: ， ； Step 4-3, fusing local structure similarity and global structure similarity by weighting.
7. The 3D Gaussian spatter method of claim 1, wherein, In the step 5, the 3D Gaussian model introduces a conventional color L1 loss as a basic loss term.
8. The 3D Gaussian spatter method of claim 1, wherein, In the step 5, the 3D Gaussian model performs an iterative optimization operation by using an Adam optimizer in the optimization process of model parameters.
9. The 3D Gaussian spatter method of claim 1, wherein, The step 6 specifically comprises: Step 6-1, calculating a position and a shape of the 3D Gaussian cell projected onto a 2D plane by using a Jacobian matrix and internal and external parameters of the camera; Step 6-2, dividing an image into a plurality of tiles to realize parallel processing, and calculating a color contribution of the projected Gaussian cell to a pixel according to a depth order, and calculating a final pixel color by using an opacity value α for mixing.