A sparse uncalibrated view Gaussian splatting three-dimensional reconstruction method based on scene quality perception

Abstract

The present application provides a kind of sparse uncalibrated view Gaussian sputtering three-dimensional reconstruction method based on scene quality perception.First, the input view defects are detected by scene quality perception module, output image level defect label and pixel level defect probability graph, and corresponding optimizer (abnormal illumination, specular reflection, rough / weak texture) is called to carry out targeted enhancement;Second, the optimized view is input into the encoder-decoder architecture based on Transformer, and the three-dimensional Gaussian primitive parameters are directly predicted in the normalized source Gaussian space, avoiding camera pose estimation error;Finally, a two-stage global Gaussian screening scheme is adopted, and the floating artifacts are suppressed by voxel density screening and outlier rejection, and high-fidelity three-dimensional reconstruction results are output.The method effectively solves the problems of inaccurate pose estimation, image quality sensitivity and geometric distortion under sparse view conditions.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D reconstruction technology, specifically relating to a method for 3D scene reconstruction by combining scene quality perception and view enhancement, and using 3D Gaussian sputtering representation, under sparse view conditions where camera calibration parameters are not provided. Background Technology

[0002] Reconstructing 3D scenes with high fidelity from sparse 2D views is a highly challenging frontier topic in computer vision. In recent years, despite significant progress in techniques such as neural radiation fields, they still face considerable challenges when applied to sparse, uncalibrated views. Current methods suffer from the following drawbacks when handling such tasks: 1. They are highly dependent on the accuracy of camera pose estimation of the input view. Under sparse view conditions, insufficient feature matching can easily lead to large cumulative errors in pose estimation, thus affecting reconstruction quality. 2. They are sensitive to the quality of the input image. When there is abnormal lighting (such as overexposure or underexposure), specular reflection on the surface, or rough geometric texture, feature extraction and matching become difficult. Even with advanced pose estimation algorithms, reconstruction errors can be nonlinearly amplified, ultimately resulting in low fidelity of the 3D scene. 3. Some existing methods rely on point clouds as 3D representations, which has inherent limitations in rendering efficiency, fine representation of geometric details, and capture of dynamic scenes. 4. While emerging 3D Gaussian sputtering methods excel in rendering speed and quality, they are prone to producing physically inaccurate "floating" artifacts when geometric constraints are insufficient or optimization deviations occur, affecting the realism of the scene. Therefore, how can we reduce our reliance on precise camera pose? How can we improve robustness to low-quality input images? And how can we generate geometrically accurate, high-fidelity 3D scenes free of visual artifacts? All of these urgently require new ideas and methods.

[0003] To effectively overcome the shortcomings of sparse view-based 3D scene reconstruction, such as inaccurate camera pose estimation, sensitivity to input image quality, geometric distortion and floating artifacts in the reconstruction results, and low fidelity of the final 3D scene, this paper proposes a scene quality-aware method for Gaussian sputtering 3D reconstruction using sparse uncalibrated views. This method helps to efficiently and accurately generate complex 3D scenes with high fidelity from sparse input images of varying quality. Summary of the Invention

[0004] The purpose of this invention is to provide a method for Gaussian sputtering 3D reconstruction based on scene quality awareness using sparse, uncalibrated views. To overcome the shortcomings of 3D scene reconstruction, such as sensitivity to input image quality, susceptibility to floating artifacts, and low fidelity of the final 3D scene, this method can efficiently and accurately generate complex 3D scenes with high fidelity.

[0005] This invention first uses a scene quality awareness module to assess and classify the sparse view input by the user. Then, based on the classification results, it calls the corresponding optimizer to specifically optimize views with quality issues. Next, it employs a Transformer-based encoder-decoder architecture to directly predict the optimized view as a set of 3D Gaussian parameters in a normalized source Gaussian space, thus avoiding errors introduced by camera pose estimation. Finally, it performs a two-stage global optimization on the generated Gaussian primitives to eliminate reconstruction artifacts, ultimately predicting and reconstructing a high-fidelity 3D scene. Specifically, it includes the following steps: Step S1: Scene quality perception and optimization. The scene quality perception module performs defect detection on the input sparse view, outputting image-level defect labels and pixel-level defect probability maps. Based on the defect labels, the corresponding view optimizer is called to enhance the input view, resulting in an optimized view. For example... Figure 2 As shown, step S1 includes: step S11 inputting the view; step S12 extracting multi-scale features based on the ResNet-50 backbone network and the Feature Pyramid Network (FPN); step S13 generating a pixel-level defect probability map through a cascaded channel attention and spatial attention mechanism; step S14 thresholding the defect probability map and determining the defect region based on the defect region area ratio; step S15 outputting image-level defect labels (multiple labels are allowed); step S16 sequentially calling the abnormal lighting optimizer, specular reflection optimizer, and / or coarse / weak texture optimizer according to a preset priority, or weighting and fusing multiple optimization results based on the pixel-level defect probability map, and outputting an optimized view.

[0006] Step S2: 3D Gaussian parameter prediction. The optimized view is input into the encoder-decoder architecture. In a preset, normalized source Gaussian space decoupled from camera pose, the 3D Gaussian meta-parameter set used to characterize the scene geometry and appearance is directly predicted. Preferably, as follows... Figure 6 As shown, step S2 includes: step S21 inputting optimized views (at least two and not exceeding a preset threshold N); step S22 encoding the input views, preferably using Patch Embedding and ViT encoder to extract visual features; step S23 fusing multi-view information through a cross-attention mechanism during the decoding stage; step S24 regressing three-dimensional Gaussian primitive parameters in a normalized source Gaussian space using a Gaussian prediction head, wherein the parameters include at least spatial position, rotation, anisotropic scaling factor, opacity, and spherical harmonic function coefficients; and step S25 outputting a set of three-dimensional Gaussian primitive parameters.

[0007] Step S3, 3D Scene Reconstruction and Optimization: Based on the 3D Gaussian primitive parameter set, initial 3D Gaussian sputtering rendering is performed, and a two-stage global Gaussian filtering scheme is used to perform voxel density filtering and outlier removal on the 3D Gaussian primitives to suppress floating artifacts and obtain the 3D reconstruction result. Preferably, as follows... Figure 7 As shown, step S3 includes: step S31 inputting a set of three-dimensional Gaussian primitive parameters; step S32 performing initial 3D Gaussian sputtering rendering based on a differentiable renderer; step S33 voxelizing the Gaussian primitives and performing a first-stage voxel density screening, using a preset voxel side length, and retaining voxel primitives with a number of primitives not less than the density threshold; step S34 performing a second-stage outlier removal, removing outlier primitives based on a comparison of the average distance between each primitive and its several nearest neighbors with the outlier threshold, preferably removing Gaussian primitives with an average distance greater than the outlier threshold; and step S35 outputting a three-dimensional Gaussian representation for rendering as the three-dimensional reconstruction result. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of the overall process of the present invention (steps S1 to S3).

[0009] Figure 2 This is a schematic diagram of the scene quality perception classification process proposed in this invention (steps S11 to S16).

[0010] Figure 3 This is a schematic diagram of the optimization process for abnormal lighting scenarios proposed in this invention (steps S161 to S165).

[0011] Figure 4 This is a schematic diagram of the reflective scene optimization process proposed in this invention (steps S166 to S170).

[0012] Figure 5 This is a schematic diagram of the rough geometry scene optimization process proposed in this invention (steps S171 to S176).

[0013] Figure 6 This is a schematic diagram of the Gaussian parameter prediction process proposed in this invention (steps S21 to S25).

[0014] Figure 7 This is a schematic diagram of the three-dimensional scene reconstruction and global filtering process proposed in this invention (steps S31 to S35).

[0015] Figure 8 This is an example diagram illustrating the optimization effect of 3D scene reconstruction quality in the embodiment. Detailed Implementation

[0016] Steps S1-S3 constitute the overall process proposed in this invention. (As follows...) Figure 1As shown, the method includes: Step S1 Scene quality perception and optimization: Defect detection is performed on the input sparse uncalibrated RGB view, outputting image-level defect labels and pixel-level defect probability maps, and the view optimizer is called according to the defect labels to obtain an optimized view; Step S2 Gaussian parameter prediction: The optimized view is input into the encoder-decoder architecture, and the three-dimensional Gaussian meta-parameter set is directly predicted in the normalized source Gaussian space decoupled from the camera pose; Step S3 3D scene reconstruction and optimization: 3D Gaussian sputtering rendering is performed based on the three-dimensional Gaussian meta-parameter set, and a two-stage global Gaussian filtering scheme is used for voxel density filtering and outlier removal, outputting the 3D reconstruction result. Optionally, the encoder-decoder network can be set to two parallel decoding branches to predict two sets of Gaussian parameters respectively, and the two sets of parameters are weighted and fused before entering the subsequent global filtering step.

[0017] Step S1 involves scene quality perception and classification. For example... Figure 2 As shown, the process is used to automatically identify quality defects in the input image, specifically including: Step S11 Input view; Step S12 Extract multi-scale features based on ResNet-50 backbone network and Feature Pyramid Network (FPN); Step S13 Input the multi-scale features into a cascaded channel attention and spatial attention mechanism to generate a pixel-level defect probability map; Step S14 Threshold the defect probability map, preferably using a pixel threshold p0 to filter defective pixels, and comparing the defect area ratio with a ratio threshold r0 to determine the defect area; Step S15 Output image-level defect labels, allowing multiple labels to be output; Step S16 Execute defect processing strategy according to the defect labels: sequentially call the corresponding optimizer according to the preset priority of abnormal lighting - specular reflection - rough / weak texture, or perform weighted fusion of multiple optimizer outputs based on each defect probability map to output an optimized view.

[0018] Abnormal lighting optimizer process. For example... Figure 3 As shown, when step S16 determines that there is an abnormal lighting defect, the abnormal lighting optimizer is invoked to enhance the view. Specifically, this includes: step S161 inputting the view containing the lighting defect; step S162 executing the pixel enhancement module in the local branch to achieve pixel-level brightness and exposure correction; step S163 executing the window attention mechanism in the global branch to optimize the global color and white balance parameters; step S164 fusing the outputs of step S162 and step S163 (preferably convolutional fusion); and step S165 outputting the lighting-optimized view.

[0019] Specular reflection optimizer process. (Example) Figure 4As shown, when step S16 determines that a specular reflection defect exists, the specular reflection optimizer is invoked, specifically including: step S166 inputting a view containing specular reflection; step S167 encoding the input view into the latent space through a VAE encoder; step S168 generating or repairing surface normal information and suppressing reflection interference using conditional diffusion denoising U-Net under the guidance of ControlNet geometric constraints; step S169 decoding the repaired latent representation into an optimized image through a VAE decoder; and step S170 outputting the specular reflection optimized view.

[0020] Coarse / weak texture optimizer process. (e.g.) Figure 5 As shown, when step S16 determines that there is a coarse or weak texture defect, the coarse / weak texture optimizer is invoked, specifically including: step S171 inputting a view lacking texture; step S172 performing intrinsic channel decomposition, decomposing the RGB image into a diffuse reflectance channel and an irradiance channel; step S173 performing back diffusion (guided by denoising U-Net) to enhance the texture of the diffuse reflectance channel; step S174 performing intrinsic channel recombination, re-fusing the enhanced diffuse reflectance and irradiance channels; step S175 using improved DDIM sampling to generate the final image; and step S176 outputting a coarse / weak texture optimized view.

[0021] Step S2: 3D Gaussian parameter prediction process. (See attached diagram.) Figure 6 As shown, the process is used to directly predict 3D Gaussian primitive parameters in a normalized source Gaussian space decoupled from camera pose. Specifically, it includes: Step S21: Input an optimized view; Step S22: Divide the input view into non-overlapping image blocks using Patch Embedding and perform linear mapping, then extract visual features using a ViT encoder; Step S23: In the decoding stage, fuse multi-view information through a cross-attention mechanism to obtain fused features; Step S24: Input the fused features into a Gaussian prediction head to regress 3D Gaussian primitive parameters in the normalized source Gaussian space. The parameters include at least spatial position, rotation, anisotropic scaling factor, opacity, and spherical harmonic function coefficients; Step S25: Output a set of 3D Gaussian primitive parameters.

[0022] Step S3 involves 3D scene reconstruction and a two-stage global Gaussian filtering process. (For example...) Figure 7As shown, the process is used to suppress floating artifacts during 3D Gaussian sputtering reconstruction, specifically including: Step S31: Inputting a 3D Gaussian primitive parameter set; Step S32: Performing initial 3D Gaussian sputtering rendering based on a differentiable renderer to obtain an initial rendering result; Step S33: Performing the first stage of voxel density filtering: Voxelizing the 3D space, using a preset voxel side length, counting the number of primitives within each voxel, and retaining voxel primitives whose number of primitives within the voxel is not less than the density threshold; Step S34: Performing the second stage of outlier removal: Calculating several nearest neighbor sets for each retained primitive and calculating the average distance, comparing the average distance with the outlier threshold, and removing outlier primitives whose average distance is greater than the outlier threshold; Step S35: Outputting the filtered 3D Gaussian representation as the 3D reconstruction result.

[0023] Model training and inference process. During the training of the scene quality perception module, data with defect annotations or pseudo-labels generated by threshold rules can be used as supervision. The classification cross-entropy loss and pixel-level segmentation loss (a weighted sum of binary cross-entropy and Dice loss) are used for training. During inference, the defect probability map is thresholded with a pixel threshold p0 to obtain a defect mask, and the defect mask area ratio is calculated to output image-level defect labels. During encoder-decoder and Gaussian prediction head training, the input view can be scaled to a uniform resolution (256*256) and divided into non-overlapping image blocks (16*16) for encoding. The predicted Gaussian primitives are rendered into the target view through a differentiable 3D Gaussian sputtering renderer, and supervised by a weighted sum of pixel reconstruction loss and perceptual loss, where the perceptual loss weight can be 0.02-0.1. During the training phase, only the parameters of the Gaussian prediction head and / or decoder can be updated, while the backbone feature extraction network and each view optimizer remain frozen to improve training stability. To evaluate the effectiveness of the method of this invention, the RealEstate-10K and ACID datasets were selected, and PSNR, SSIM, and LPIPS were used as evaluation metrics. The results are shown in Table 1.

[0024] Depend on Figure 8 As can be seen, the first two rows show scenes with light leakage; the third row shows scenes containing specular reflective surfaces; and the fourth and fifth rows present scenes with blurred / weakly textured surfaces. The first two columns show the original low-quality scene and its 3D reconstruction, respectively, while the third and fourth columns show the scene after view optimization and its 3D reconstruction. It can be observed that, compared to the unoptimized input, after applying the quality perception and view optimization of this invention, floating artifacts and point cloud holes in the reconstruction results are suppressed. Furthermore, in typical low-quality scenes such as abnormal lighting, specular reflection, and weak textures, the geometric continuity and appearance consistency of the reconstruction are improved.

Claims

1. A 3D reconstruction method based on scene quality awareness and using sparse uncalibrated views for 3D Gaussian sputtering, characterized in that, Specifically, the following steps are included: Step 1, Scene Quality Perception and Optimization: The scene quality perception module is used to detect defects in the input view, outputting image-level defect labels and pixel-level defect probability maps. Based on the defect labels, the corresponding view optimizer is called to enhance the input view and obtain an optimized view. When multiple types of defects are detected, the corresponding optimizer executes sequentially according to a preset priority or performs weighted fusion of multiple optimization results based on a pixel-level defect probability map. Step 2: Gaussian parameter prediction: The optimized view is input into the encoder-decoder architecture, and the 3D Gaussian primitive parameter set used to characterize the scene geometry and appearance is directly predicted in a preset, decoupled normalized source Gaussian space. Step 3: 3D scene reconstruction and optimization: Initial 3D Gaussian sputtering rendering is performed based on the 3D Gaussian primitive parameter set, and a two-stage global Gaussian filtering scheme is used to perform voxel density filtering and outlier removal on the 3D Gaussian primitives to suppress floating artifacts and obtain the 3D reconstruction result.

2. The method as described in claim 1, characterized in that: The scene quality perception module extracts multi-scale features based on the ResNet-50 backbone network and feature pyramid network, and generates a pixel-level defect probability map through a cascaded channel attention and spatial attention mechanism. When the proportion of pixel areas with defect probabilities greater than pixel threshold p0 in the pixel-level defect probability map exceeds the proportion threshold r0, the corresponding image-level defect label is output, where p0 is 0.4-0.7 and r0 is 0.5%-5%.

3. The method as described in claim 1, characterized in that: The view optimizer includes one or more of the following: abnormal lighting optimizer, specular reflection optimizer, and coarse / weak texture optimizer; when an image-level defect label indicates the existence of a corresponding defect, the corresponding optimizer is invoked to enhance the input view; when multiple types of defects exist, they are executed sequentially according to the priority of abnormal lighting-specular reflection-coarse / weak texture, or the outputs of multiple optimizers are weighted and fused based on the pixel-level defect probability map.

4. The method as described in claim 1, characterized in that: The encoder-decoder architecture is based on a visual Transformer, where the encoder extracts visual features from the input 2D image, and the decoder fuses multi-view information through a cross-attention mechanism and directly predicts 3D Gaussian meta-parameters. The parameters include at least: spatial position, rotation quaternion, anisotropic scaling factor, opacity, and spherical harmonic coefficients used for view-dependent appearance modeling.

5. The method as described in claim 1, characterized in that: The two-stage global Gaussian screening scheme includes: a first stage of voxel density screening, in which the normalized source Gaussian space is voxelized using a preset voxel side length, the number of Gaussian primitive center points in each voxel is counted, and voxel primitives with a number of primitives not less than the density threshold are retained, wherein the voxel side length is preferably 1 / 256 to 1 / 64; the density threshold can be determined based on the statistics of voxel counts (mean and standard deviation), or based on the total number of primitives and a preset ratio, wherein the preset ratio is preferably 0.5% to 5%; a second stage of outlier removal, in which the average distance of several nearest neighbors of each Gaussian primitive center point retained in the first stage is calculated, and Gaussian primitives with an average distance greater than the outlier threshold are removed, wherein the number of nearest neighbors is preferably 8 to 32, and the outlier threshold can be determined based on the statistics of nearest neighbor distances (mean and standard deviation).

Images