A method for processing image degradation problem of Gaussian rendering based on gradient control

Abstract

The application discloses a method for processing image degradation problems of Gaussian rendering based on gradient control, comprising the following steps: obtaining a degraded image and performing pretreatment, using a pre-trained 2D image recovery model to recover the pretreated degraded image to obtain a preliminary recovery image; using a sparse point cloud generated by a motion recovery structure to initialize a 3D Gaussian model; using the preliminary recovery image to train the 3D Gaussian model, introducing a gradient control strategy in the training process, and iteratively optimizing Gaussian parameters to solve the view inconsistency problem; using the optimized 3D Gaussian model to process, completing the processing of the degraded image and three-dimensional scene reconstruction. The application converts various types of input image degradation problems into view inconsistency, introduces a gradient control strategy to alleviate the inaccuracy of Gaussian densification and optimization caused by view inconsistency, and well adapts to various degradation scenes, while maintaining high-fidelity rendering and real-time rendering capability.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for processing image degradation problems based on gradient-controlled Gaussian rendering, especially image degradation problems in blurry, low-resolution, low-light and underwater scenes. Background Art

[0002] Obtaining accurate 3D scene representations and synthesizing novel views from 2D images has been an ongoing challenge in computer vision. This problem is crucial in both computer vision and graphics, with potential applications including street view navigation, augmented reality / virtual reality, and robotics. Various approaches typically leverage multi-view images and camera poses to synthesize images from arbitrary viewpoints and reconstruct 3D scenes. Neural Radiance Fields (NeRF) significantly improves the ability to recover high-quality 3D scene representations from 2D images by leveraging a multi-layer perceptron (MLP) to learn from 5D coordinate inputs and employing differentiable volume rendering techniques. Much research has focused on improving NeRF's performance, particularly with regard to training and rendering efficiency. Recently, 3D Gaussian Rendering (3DGS) has extended implicit neural rendering to explicit point clouds, significantly improving both training and rendering efficiency and quality. In practical applications of 3D Gaussian Rendering, various forms of image degradation are often introduced, such as blurring due to camera motion or loss of focus, noise from low-light conditions, and degradation caused by scattering in low-resolution or underwater media.

[0003] Due to the limitations of 2D image restoration models, they struggle to restore degraded images to the same degree as under ideal conditions, inevitably leading to inconsistencies in the restored images. These inconsistencies mislead the 3D Gaussian function into generating incorrect gradients, which in turn leads to low-quality new view synthesis.

[0004] Therefore, how to improve the quality of 3D scene reconstruction and new view synthesis remains an urgent problem to be solved. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the existing technology and to propose a method for processing image degradation problems based on gradient-controlled Gaussian rendering.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for processing image degradation problems based on gradient-controlled Gaussian rendering, comprising the following steps:

[0008] Obtain degraded images and perform preprocessing;

[0009] The preprocessed degraded image is recovered using a pre-trained 2D image restoration model, the degradation problem is converted into a view inconsistency problem, and a preliminary recovered image is obtained;

[0010] A 3D Gaussian model is initialized by using a sparse point cloud generated by a motion recovery structure;

[0011] A 3D Gaussian model is trained using the preliminary recovered image, a gradient control strategy is iteratively applied in the training process to solve the view inconsistency problem, and the Gaussian parameters are iteratively optimized; the gradient control strategy includes a fuzzy Gaussian selection strategy and a gradient weighted optimization;

[0012] A three-dimensional scene reconstruction is completed using the optimized 3D Gaussian model.

[0013] Further, the 2D image restoration model includes a Uformer model, a SwinIR model and a Retinexformer model.

[0014] Further, the gradient control strategy includes a fuzzy Gaussian selection strategy and a gradient weighted optimization, wherein:

[0015] The fuzzy Gaussian selection strategy traverses all views during the 3D Gaussian densification process, calculates and updates the view space position gradient of each Gaussian, and distinguishes between clear Gaussians and fuzzy Gaussians through a Gaussian densification threshold; the clear Gaussian is a Gaussian whose updated view space position gradient in all views is consistent with the Gaussian densification threshold, and is allowed to perform cloning or splitting operations; the fuzzy Gaussian is a Gaussian whose updated view space position gradient in at least one view is inconsistent with the Gaussian densification threshold, and is prohibited from performing densification operations;

[0016] The gradient weighted optimization obtains a fuzzy ratio by calculating the ratio of the number of fuzzy views to the total number of views, and weights the gradient of the fuzzy Gaussian in the fuzzy view based on the fuzzy ratio to reduce the influence of the wrong gradient.

[0017] Further, the determination condition of the clear Gaussian is that the updated view space position gradient in all views is greater than the Gaussian densification threshold, or is less than or equal to the Gaussian densification threshold; the determination condition of the fuzzy Gaussian is that the updated average view space position gradient in at least one view is greater than the Gaussian densification threshold, and the updated average view space position gradient in all other views is less than or equal to the Gaussian densification threshold; or the updated average view space position gradient in at least one view is less than or equal to the Gaussian densification threshold, and the updated average view space position gradient in all other views is greater than the Gaussian densification threshold.

[0018] Further, the splitting operation condition of the explicit Gauss is that the updated average visual spatial position gradient is greater than a Gauss density threshold, and the sum of three dimensions is greater than a Gauss scale threshold.

[0019] The splitting operation condition of the explicit Gauss is that the updated average visual spatial position gradient is greater than a Gauss density threshold, and the sum of three dimensions is greater than a Gauss scale threshold.

[0020] Further, the gradient weighting optimization includes the following steps:

[0021] The ratio of the number of ambiguous visual angles to the total number of visual angles is calculated as a weighting coefficient;

[0022] The original gradient of the ambiguous Gauss at the ambiguous visual angle is multiplied by the weighting coefficient to obtain a weighted gradient;

[0023] The Gauss parameters are updated using the weighted gradient.

[0024] In a second aspect, the present application provides an image degradation problem processing system using the above method, comprising the following modules:

[0025] An image acquisition and preprocessing module is configured to acquire a degraded image and perform preprocessing;

[0026] A preliminary recovery module is configured to select a corresponding 2D image recovery model according to the degradation type, and output a preliminary recovery image;

[0027] A degraded image processing module is configured to process the preliminary recovery image using the optimized 3D Gauss model, complete the processing of the degraded image, and reconstruct a three-dimensional scene.

[0028] In a third aspect, the present application provides a computer readable storage medium storing instructions which, when executed by a computer, perform the above method.

[0029] In a fourth aspect, the present application provides an electronic device comprising a processor and a memory, wherein the memory stores instructions which, when executed by the processor, perform the above method.

[0030] Compared with the prior art, the present application has the following advantages:

[0031] The application converts various types of input image degradation problems into view inconsistency, and introduces a gradient control strategy to alleviate the inaccuracy of Gaussian densification and optimization caused by view inconsistency, prevent false gradients from affecting the densification process, ensure high-quality scene reconstruction, and avoid new view synthesis performance degradation caused by false Gaussian densification. Through the gradient control strategy (including the fuzzy Gaussian selection strategy and the gradient weighted optimization), the influence of false gradients is effectively reduced, the quality of the final new view synthesis is improved, and various degradation scenes are well adapted, while maintaining high-fidelity rendering and real-time rendering capability. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A complete flowchart of the Gaussian rendering processing of various image degradation problems based on gradient control is implemented for the application. DETAILED DESCRIPTION

[0033] The application will be further described below.

[0034] As shown in the Figure 1 The application provides a method for processing image degradation problems based on gradient control of Gaussian rendering, comprising the following steps:

[0035] Step S1, obtaining the degraded image and pre-processing.

[0036] Step S2, using a pre-trained 2D image restoration model to restore the pre-processed degraded image, converting the degradation problem into a view inconsistency problem, and obtaining a preliminary restored image. Specifically as follows:

[0037] Due to the diversity of the causes of various degraded images, it is challenging to directly find a general 3D Gaussian splatting rendering method to process all types of degraded images. On the other hand, various two-dimensional image restoration models have shown excellent performance. Therefore, a pre-trained two-dimensional image restoration model is used to convert various input image degradation problems, such as blur, low resolution, weak light and underwater, into view inconsistency.

[0038] Given K degraded images, a set of restored images is obtained using a pre-trained 2D image restoration model. Specifically, the Uformer model is used for deblurring for blur degradation, the SwinIR model is used for super-resolution reconstruction for low-resolution degradation, and the Retinexformer model is used for enhancement for low-light degradation.

[0039] The embodiments of the application are performed on four representative data sets, covering various degradation scenarios:

[0040] LLFF dataset: contains forward-facing images of real scenes, degradation includes 4 times down-sampling (low resolution) and Gaussian blur (motion blur).

[0041] Reblur24 dataset: synthetic dataset containing 24 real camera motion blur for quantitative evaluation of deblurring effects.

[0042] UIEB dataset: underwater image enhancement benchmark dataset containing color distortion and light scattering caused by seawater.

[0043] SeaThru-NeRF dataset: real underwater scene reconstructed by NeRF with severe scattering blur and color cast.

[0044] All images are preprocessed to 512x512 resolution for training and testing.

[0045] Step S3, initialize the 3D Gaussian model with the sparse point cloud generated by motion recovery structure.

[0046] Step S4, introduce a gradient control strategy to solve the view inconsistency problem in the 3D Gaussian model training process, and iteratively optimize the Gaussian parameters; the gradient control strategy includes a blur Gaussian selection strategy and a gradient weighted optimization.

[0047] The specific method is as follows:

[0048] The 3D Gaussian distribution is initialized by the free sparse point cloud generated by SfM (Structure from Motion). Then, adaptive density control is applied to reconstruct empty areas, including two types: cloning or splitting. The cloning operation aims to clone more Gaussian distributions to ensure full coverage of unreconstructed areas. The splitting operation is designed to split large Gaussian distributions representing small-scale areas into two parts, which corresponds to the case of over-reconstruction.

[0049] ①Blur Gaussian selection strategy

[0050] In the densification process of 3D Gaussian rendering, K viewpoints are traversed and the corresponding view space position gradient is obtained As follows:

[0051] Where, is the Gaussian pixel space projection point under the kth view, is the Gaussian loss under the kth view.

[0052] Then, by accumulating or subtracting the gradient of each view, the updated average view space position gradient As follows:

[0053] where M is the total number of views participating in the computation. In this way, those Gaussians with smaller or larger gradients can be distinguished. For those Gaussians far away from the Gaussian densification threshold, their final densification results remain consistent even if they contain erroneous gradients. Therefore, when the updated view-space position gradients of all views are consistent:

[0054]

[0055] where is the Gaussian densification threshold, which is defined as a clear Gaussian and allows it to be cloned or split.

[0056] For ambiguous Gaussians, that is, those updated average view-space position gradients satisfy:

[0057]

[0058]

[0059] Gaussians that are not consistent with the Gaussian densification threshold are defined by the present application as ambiguous Gaussians and are not allowed to perform densification operations.

[0060] In this way, the present application can prevent erroneous gradients from affecting the densification process and ensure high-quality scene reconstruction. To prevent the performance of new view synthesis from being degraded by erroneous Gaussians densification, only when the following conditions are met, clear Gaussian functions are allowed to be densified:

[0061]

[0062]

[0063] where is the Gaussian densification threshold, is the Gaussian scale threshold. In addition, according to the majority rule, those minority views will be defined as ambiguous views.

[0064] ② Gradient-weighted optimization

[0065] Even if erroneous Gaussians are identified by the ambiguous Gaussian selection strategy (AGS) and prevented from densification, erroneous gradients will still affect those ambiguous Gaussians, ultimately leading to a decrease in the quality of new view synthesis. Therefore, the gradient control strategy introduces gradient-weighted optimization (GWO) to further reduce the impact of these erroneous gradients.

[0066] ​In the gradient weighting optimization process, the present application assigns different weights to the gradients under different viewing angles according to the blur degree of the fuzzy Gaussian. The blur degree is obtained by calculating the ratio of the number of fuzzy viewing angles k to the total number of viewing angles K, and is defined as:

[0067]

[0068] wherein a represents the blur ratio, k represents the number of fuzzy viewing angles, and K represents the total number of all viewing angles.

[0069] Next, the gradient corresponding to the fuzzy viewing angle of the fuzzy Gaussian is weighted according to the proportion a. Specifically, the gradient g ka of the fuzzy Gaussian under the fuzzy viewing angle is weighted into g kw , as follows:

[0070]

[0071] The weighted gradient gkw is updated in the densification process. The purpose of this is to reduce the influence of fuzzy viewing angles, so that inconsistent viewing angles have less influence on the final scene synthesis. The gradient weighting optimization is iteratively updated together with the densification until the training is completed. Through this weighting method, the present application can effectively reduce the influence of inconsistent viewing angles, thereby improving the quality of the final new viewing angle synthesis.

[0072] After the training is completed, an ablation study is performed to compare the use of the fuzzy Gaussian selection strategy with the use of the strategy. As shown in Model 1 and Model 4 in Table 1, the use of the fuzzy Gaussian selection strategy improves the PSNR by 2.5 dB, and is superior to other methods in all indicators. The use of the fuzzy Gaussian selection strategy can obtain better image details.

[0073] Gradient weighting optimization (GWO) is discussed. First, the present application compares the results of not using gradient weighting optimization and using gradient weighting optimization. As shown in Model 4 and Model 6 in Table 1, the use of gradient weighting optimization improves the PSNR by 0.1 dB, and performs well in all indicators. The slight improvement of GWO is due to the fact that AGS has already alleviated most of the inconsistent influences. As shown in Model 1 and Model 3 in Table 1, the use of GWO alone can improve the PSNR performance by 2.0 dB, which makes the present application consider GWO as an important part of the method.

[0074] Table 1

[0075]

[0076] The above description of the embodiments is only for the purpose of helping to understand the method of the present application and its core idea. It should be noted that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application. Various modifications to the embodiments are apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Accordingly, the present application will not be limited to the embodiments shown in the present application, but will be consistent with the widest scope consistent with the principles and novel features disclosed in the present application.

Claims

1. A method for processing image degradation problems based on gradient-controlled Gaussian rendering, characterized in that: The method comprises the following steps: Obtain degraded images and perform preprocessing; Use the pre-trained 2D image restoration model to restore the pre-processed degraded image, transform the degradation problem into a view inconsistency problem, and obtain a preliminary restored image; Initialize the 3D Gaussian model using the sparse point cloud generated by motion recovery structure; A 3D Gaussian model is trained using a preliminary restored image, and a gradient control strategy is iteratively applied in the training process to solve the view inconsistency problem and iteratively optimize the Gaussian parameters; the gradient control strategy includes a fuzzy Gaussian selection strategy and a gradient weighted optimization; the fuzzy Gaussian selection strategy traverses all view angles during the 3D Gaussian densification process, calculates and updates the visual space position gradient of each Gaussian, and distinguishes between clear Gaussians and fuzzy Gaussians through the Gaussian densification threshold; the clear Gaussian is a Gaussian whose updated visual space position gradient is consistent with the Gaussian densification threshold for all view angles, and cloning or splitting operations are allowed; the fuzzy Gaussian is a Gaussian whose updated visual space position gradient is inconsistent with the Gaussian densification threshold for at least one view angle, and densification operations are prohibited; wherein, the judgment condition for the clear Gaussian is: the updated visual space position of all view angles is consistent with the Gaussian densification threshold. The gradients are all greater than the Gaussian density threshold, or are all less than or equal to the Gaussian density threshold; the judgment condition of the fuzzy Gaussian is: there is at least one perspective whose updated average visual space position gradient is greater than the Gaussian density threshold, and the updated average visual space position gradients of all other perspectives are less than or equal to the Gaussian density threshold; or there is at least one perspective whose updated average visual space position gradient is less than or equal to the Gaussian density threshold, and the updated average visual space position gradients of all other perspectives are greater than the Gaussian density threshold; the cloning operation condition of the clear Gaussian is: the updated average visual space position gradient is greater than the Gaussian density threshold, and the sum of the three dimensions is less than or equal to the Gaussian scale threshold; the splitting operation condition of the clear Gaussian is: the updated average visual space position gradient is greater than the Gaussian density threshold, and the sum of the three dimensions is greater than the Gaussian scale threshold; The optimized 3D Gaussian model is used to complete the 3D scene reconstruction.

2. The method for processing image degradation problems based on gradient-controlled Gaussian rendering according to claim 1, characterized in that: The 2D image restoration model includes a Uformer model, a SwinIR model and a Retinexformer model.

3. The method for processing image degradation problems based on gradient-controlled Gaussian rendering according to claim 1, characterized in that: The gradient weighted optimization calculates the ratio of the number of blurred view angles to the total number of view angles to obtain a blur ratio, and weights the gradient of the blurred Gaussian under the blurred view angle based on the blur ratio to reduce the influence of erroneous gradients.

4. The method for processing image degradation problems based on gradient-controlled Gaussian rendering according to claim 1, characterized in that: The gradient weighted optimization comprises the following steps: Calculate the ratio of the number of blurred perspectives to the total number of perspectives as a weighting coefficient; Multiply the original gradient of the blurred Gaussian at the blurred perspective by the weighting coefficient to obtain the weighted gradient; Update the Gaussian parameters using weighted gradients.

5. A system for processing image degradation problems using the method according to any one of claims 1 to 4, characterized in that: Includes the following modules: Image acquisition and preprocessing module, used to acquire degraded images and perform preprocessing; A preliminary restoration module is used to select a corresponding 2D image restoration model according to the degradation type and output a preliminary restored image; The degraded image processing module uses the optimized 3D Gaussian model to process the preliminary restored image, completing the processing of the degraded image and the reconstruction of the three-dimensional scene.

6. A computer-readable storage medium, characterized in that Instructions are stored thereon, and when executed by a computer, the method according to any one of claims 1 to 4 is performed.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores instructions, and when executed by the processor, performs the method according to any one of claims 1 to 4.

Images