A method for synthesizing new views using noisy images
By leveraging 3D Gaussian ellipsoids and deep learning networks in noise images, the problem of noise impact in low-light conditions is solved, and new view synthesis with high quality and consistency is achieved.
Patent Information
- Application Number
- CN202411508326.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2044-10-28
AI Technical Summary
Traditional new view synthesis technology is susceptible to noise under low light conditions, resulting in a decline in image quality, and there are differences in details for denoising different views, resulting in inconsistencies between views.
Using a method, using colmap to initialize 3D Gaussian ellipsoids from noise images, and extract and simulate noise distributions through pre-trained blind image denoising networks and U-Net-based noise extraction networks, combining 3D Gaussian Splatting and residuals Network, learn pixel-by-pixel masks, fuse noise distributions to render high-quality views.
It can effectively remove noise under low light conditions, ensure consistency between views, improve the quality and efficiency of new view synthesis, which is significantly better than the existing technology.
Smart Images
Figure CN119048369B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of three-dimensional reconstruction and relates to a new view synthesis method. Background Art
[0002] 3D reconstruction is an important research direction in the field of computer vision and graphics. Its purpose is to reconstruct the 3D model of an object or scene by processing and analyzing 2D image data. At present, 3D reconstruction has been widely used in many fields such as medical imaging, cultural relics protection, autonomous driving and smart cities. Traditional reconstruction algorithms mainly rely on explicit expressions, such as voxels, grids, point clouds, etc. However, these methods often have "holes" and artifacts in the reconstruction process, and it is difficult to continuously reconstruct the details and textures in the scene. In contrast, the reconstruction technology based on neural radiance field (NeRF) significantly improves the details and realism of reconstruction through the implicit expression of neural networks, and has the ability to generate high-quality images from novel perspectives. However, due to the extensive use of multi-layer perceptrons (MLPs) and unnecessary sampling of blank areas, the efficiency of scene reconstruction is significantly reduced. 3D Gaussian Splatting (3D GS) uses anisotropic 3D Gaussian and tile-based rasterization technology to not only ensure the synthesis quality of the view, but also accelerate the training and rendering time.
[0003] Although recent new view synthesis techniques have shown great potential and impressive results under idealized data, they still face challenges in practical applications. For example, in low-light conditions, due to insufficient ambient light, cameras often need to use high ISO (sensitivity) settings and extend exposure time to capture sufficient image information. Although these measures can alleviate the problem of insufficient light to a certain extent, they will inevitably lead to increased image noise. These noises not only reduce the clarity of the image, but also pose a serious obstacle to subsequent scene reconstruction and view synthesis, such as Figure 1 shown.
[0004] Current new view synthesis techniques mainly rely on high-quality image input and accurate initialization point clouds, but noisy input images are difficult to meet the ideal conditions required for reconstruction and are prone to reconstruction failure. To solve this problem, an intuitive solution is to first denoise the noisy image, then use the processed image to reconstruct the scene and synthesize a new view. Although this method can improve the quality of reconstruction to a certain extent, independent denoising of different views will result in differences in detail processing, leading to inconsistent correspondence between views, such as Figure 2 shown.
[0005] NAN is based on the IBRNet method, trained in multiple scenes, and uses NeRF's inherent ability to integrate information between multiple views, demonstrating the advantages of NeRF technology in burst denoising. However, this multi-frame denoising method is only effective when the difference between adjacent frames is small. Once the pose difference of the input view is large, it will cause view synthesis to fail. Summary of the invention
[0006] In order to solve the problem that traditional new view synthesis methods are difficult to synthesize new views under noisy input conditions and ensure the geometric consistency of the viewing angle, the present invention provides a method for synthesizing new views using noisy images. The method can reconstruct a clear scene using a series of noisy images and synthesize a clear view at any viewing angle.
[0007] The objective of the present invention is achieved through the following technical solutions:
[0008] A method for synthesizing a new view using a noisy image comprises the following steps:
[0009] Step (1) using the colmap method to obtain the initial pose from the multi-view noise image and initialize the 3D Gaussian ellipsoid;
[0010] Step (2) uses the pre-trained blind image denoising network to obtain a high-quality image from the input noisy image and extract the initial noise distribution. The specific steps are as follows:
[0011] The noisy image Input to a pre-trained blind image denoising network De-noising is performed in the image to obtain a high-quality image after denoising. , and then subtract the original noise image from it to get the initial noise distribution :
[0012]
[0013] Step (3) Using the U-Net-based noise extraction network Further extract fine noise distribution from the original noisy image :
[0014]
[0015] Step (4) Render the 3D Gaussian ellipsoid generated in step (1) from the scene to obtain the predicted image ,design The network combines image, viewpoint and position information to learn the noise mask for each pixel. The extracted noise distribution is weighted using the noise mask to simulate real noise. The specific steps are as follows:
[0016] Step (41) 3D Gaussian distribution of each Gaussian ellipsoid in space It is expressed by the following formula:
[0017]
[0018] Among them, Σ is the covariance matrix, which represents the shape and direction of the Gaussian distribution, and μ is the mean, which represents the center position of the Gaussian distribution;
[0019] Step (42) projects the 3D Gaussian distribution representing the ellipsoid onto a 2D Gaussian distribution , the projected covariance matrix Σ′ is given by:
[0020]
[0021] in, is the view transformation matrix, J is the Jacobian matrix of the affine transformation; the position of the Gaussian ellipsoid in 2D space can be obtained by projecting the pose initialized in step (1);
[0022] Step (43) During the rendering process, the image is rendered using a fast GPU sorting algorithm and tile-based rasterization technology. The specific method is as follows:
[0023] Each tile in the two-dimensional image space is regarded as a unit, which is sorted based on depth and rendered using alpha blending. After rendering, an image at any perspective is obtained. The color of each pixel in :
[0024] (4)
[0025] in, represents the set of all Gaussian distributions overlapping the pixel, is the color of a 2D Gaussian distribution, is the density;
[0026] Step (44) Design a residual network combined with a multi-layer perceptron The network controls the noise distribution extracted by step (2) and step (3) by learning a pixel-by-pixel mask The weights of are used to adapt to the heterogeneity of noise between pixels in different views and within views. The specific steps are as follows:
[0027] First, using convolutional neural networks Extracting RGB features from the image rendered in step (43);
[0028] Then, the extracted RGB features are embedded with learnable viewpoints and position embedding As feature input Network, where is the view index, is the pixel coordinate
[0029] Finally, the learned mask features are transformed into Mapped to between 0 and 1 to get the final mask , the formula is as follows:
[0030]
[0031]
[0032] Step (5) Using the simulated noise distribution and the image rendered by the 3D Gaussian ellipsoid Combined, predicting noisy images , the specific steps are as follows:
[0033] Step (51) uses the mask learned in step (4) to fuse the noise extracted in steps (2) and (3):
[0034]
[0035] Step (52) converts the fused noise Image obtained with 3D Gaussian rendering Add together to get the predicted noisy image :
[0036]
[0037] Step (6) converts the predicted noise image Create a color loss with the original noisy input image , joint structural similarity loss , Mask loss The entire reconstruction process is constrained to optimize the properties of the feature extraction network and the 3D Gaussian ellipsoid.
[0038] Compared with the prior art, the present invention has the following advantages:
[0039] The present invention generates a clear scene model from a noisy image through training and renders a high-quality view from a new perspective, effectively solving the problem that traditional new view synthesis relies on clear image input, but in low-light environments, the image is easily affected by noise, resulting in quality degradation. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1This is an image synthesized from a noisy image using 3D GS at a new perspective. In the figure: the top image is the input noisy image, the middle image is the image synthesized from a new perspective using 3D GS, and the bottom image is the reference clear image.
[0041] Figure 2 The results of reconstruction using a single image denoising image. In the figure: (a) input noisy image, (b) result after denoising using a single image, (c) new view synthesis result of 3D GS reconstruction using the denoised image, and (d) reference clear image.
[0042] Figure 3 This is the reconstruction process of the algorithm.
[0043] Figure 4 This is the network structure diagram for noise extraction based on U-Net.
[0044] Figure 5 for Network structure diagram.
[0045] Figure 6 : The comparison between the method of the present invention and the NeRF method in terms of the quality and speed of synthesizing new views when the noise gain is 16.
[0046] Figure 7 This is a comparison between the “horns” scene and other methods when the noise gain is 4. In the figure: (a) input noisy image, (b) view synthesis result of NAN at a new perspective, (c) view synthesis result of NeRF_per at a new perspective, (d) view synthesis result of 3D GS_pre at a new perspective, (e) view synthesis result of the method of the present invention at a new perspective, and (f) reference view.
[0047] Figure 8 : This is a comparison between the “trex” scene and other methods when the noise gain is 4. In the figure: (a) input noisy image, (b) view synthesis result of NAN at a new perspective, (c) view synthesis result of NeRF_per at a new perspective, (d) view synthesis result of 3D GS_pre at a new perspective, (e) view synthesis result of the method of the present invention at a new perspective, and (f) reference view.
[0048] Fig. 9: This is a comparison with other methods in the "benchflower" scene when the noise gain is 4. In the figure: (a) input noisy image, (b) view synthesis result of NAN at a new perspective, (c) view synthesis result of NeRF_per at a new perspective, (d) view synthesis result of 3D GS_pre at a new perspective, (e) view synthesis result of the method of the present invention at a new perspective, and (f) reference view.
[0049] Fig.10 : This is a comparison with other methods in the "livingroom" scene when the noise gain is 4. In the figure: (a) input noisy image, (b) view synthesis result of NAN at a new perspective, (c) view synthesis result of NeRF_per at a new perspective, (d) view synthesis result of 3D GS_pre at a new perspective, (e) view synthesis result of the method of the present invention at a new perspective, and (f) reference view.
[0050] Fig.11 For the comparison between the method of the present invention and other methods in the trash can scene captured by a camera, the figure shows: (a) input noisy image, (b) the result of the single view denoising method, (c) the view synthesis result of NAN at a new perspective, (d) the view synthesis result of NeRF_per at a new perspective, (e) the view synthesis result of 3D GS_pre at a new perspective, and (f) the view synthesis result of the method of the present invention at a new perspective.
[0051] Fig.12 For the comparison between the method of the present invention and other methods in the electric fan scene captured by a camera, the figure shows: (a) input noisy image, (b) the result of the single view denoising method, (c) the view synthesis result of NAN at a new perspective, (d) the view synthesis result of NeRF_per at a new perspective, (e) the view synthesis result of 3D GS_pre at a new perspective, and (f) the view synthesis result of the method of the present invention at a new perspective. DETAILED DESCRIPTION
[0052] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.
[0053] The present invention provides a method for realizing new view synthesis using a noisy image, which mainly consists of four parts: 3D Gaussian Splatting, a noise extraction module, a noise simulation module, and a loss function optimization:
[0054] 1. 3D Gaussian Splatting
[0055] 3D Gaussian Splatting is a method for real-time radiation field rendering that combines 3D Gaussian distribution and efficient differentiable renderer. The scene representation starts with a sparse point cloud generated using the colmap method. These point clouds are initialized as 3D Gaussian distributions, each defined by a mean, covariance matrix, and properties such as opacity and color. A compact representation of the scene is achieved by optimizing the properties of these Gaussian distributions. 3D Gaussian distribution for each Gaussian ellipsoid in space It is expressed by the following formula:
[0056] (1)
[0057] Among them, Σ is the covariance matrix, which represents the shape and direction of the Gaussian distribution, and μ is the mean, which represents the center position of the Gaussian distribution.
[0058] In order to render the Gaussian ellipsoid into the 2D image space, the 3D Gaussian distribution representing the ellipsoid needs to be projected into a 2D Gaussian distribution. The projected covariance matrix Σ′ is given by:
[0059] (2)
[0060] in, is the view transformation matrix, and J is the Jacobian matrix of the affine transformation. For efficient optimization by gradient descent, the covariance matrix Σ can be represented by the scaling matrix S and the rotation matrix R:
[0061] (3)
[0062] The position of the Gaussian ellipsoid in 2D space can be obtained by projecting the pose initialized in step (1). During the rendering process, the image is rendered using a fast GPU sorting algorithm and tile-based rasterization technology. Each tile in the two-dimensional image space is regarded as a unit, which is sorted based on depth and rendered using α-blending. This process can be understood as aggregating 2D Gaussian distributions according to the depth order of pixels or tiles, and after rendering, images from any perspective can be obtained. The color of each pixel in :
[0063] (4)
[0064] in, Represents the set of all Gaussian distributions overlapping the pixel; is the color of a 2D Gaussian distribution, represented by spherical harmonic coefficients; density The 2D Gaussian distribution after projection Calculated by multiplying the learned opacity per point.
[0065] 2. Noise Extraction Module
[0066] The noise extraction module is composed of a blind image denoising network and a noise extraction network. Train a 3D Gaussian model in the image to render clear views from any perspective The core is to simulate the noise distribution characteristics in the input image and fuse these simulated noises with the image rendered based on 3D Gaussian distribution. By constructing the constraint relationship between the fused noise image and the original input image, the parameter configuration of the 3D Gaussian ellipsoid is further optimized, and finally the noise-free high-quality image can be generated at any angle. Before simulating noise, it is necessary to extract the noise distribution from the input image. It is divided into the following two steps:
[0067] First, the noisy image Input to a pre-trained blind image denoising network De-noising is performed in the image to obtain a high-quality image after denoising. . Then subtract the original noise image from it to get the initial noise distribution:
[0068] (5)
[0069] Thanks to the pre-trained blind image denoising network The denoising result of It is inappropriate to simulate the input noise image, so it is necessary to further correct (supplement) this noise distribution.
[0070] Therefore, the present invention also designs a noise extraction network based on the U-Net architecture , the network structure is as follows Figure 4 As shown. By using a symmetrical structure and a skip connection mechanism, multi-level features are effectively fused to further extract noise distribution. The multi-scale feature fusion and spatial information retention capabilities in the network enable the network to accurately capture global and local detail noise features in noise extraction tasks, improve the accuracy and efficiency of noise extraction, and provide high-quality noise distribution information for subsequent noise simulation:
[0071] (6)
[0072] Through the above two steps, the noise distribution in the input image can be extracted more comprehensively. Specifically, the pre-trained blind image denoising network Extract an initial noise estimate from , while the noise extraction network Further extracting finer features to correct the information loss caused by blind image denoising and obtain a more accurate noise distribution .
[0073] 3. Noise simulation module
[0074] The noise simulation module consists of The network is composed of a convolutional neural network, which is used to extract features from images. The network is used to aggregate the extracted image features and viewpoint and position embeddings.
[0075] The main task of this part is to fuse the extracted noise with the image obtained by 3D Gaussian rendering to simulate the original noisy input image. The main method is to combine the residual network and the multi-layer perceptron (MLP) The network learns a pixel-wise mask and uses this mask to control the weights of two extracted noises. The network structure is as follows Figure 5 shown.
[0076] Since the noise in the input image usually varies between different viewing angles and different pixels of the same image, the present invention is The network provides a learnable perspective embedding and position embedding , where n is the view index and x is the pixel coordinate. In addition, the present invention also combines the RGB features extracted from the rendered image. The feature is generated by a convolutional neural network Finally, the learned mask features are extracted from the rendered image through the sigmoid function. Mapped to between 0 and 1 to get the final mask , the formula is as follows:
[0077] (7)
[0078] (8)
[0079] in, is the view index, Represents pixel coordinates. Using the learned weights, the noise extracted in the noise extraction module is aggregated to simulate the final noise distribution :
[0080] (9)
[0081] in, is the noise distribution obtained by the pre-trained network, is the corrected noise distribution extracted from the original noisy image. Finally, the fused noise is combined with the image obtained by 3D Gaussian rendering. Add together to get the noisy image predicted by the model :
[0082] (10)
[0083] 4. Loss Function Optimization
[0084] In order to optimize the entire model, the present invention designs a comprehensive loss function to constrain and train the parameters of the model.
[0085] First, use The loss is used to minimize the difference between the predicted noisy image and the actual noisy image, and the color loss is established, which is defined as follows:
[0086] (11)
[0087] This part of the loss aims to ensure that the noise image predicted by the model is as close as possible to the real noise image of the input.
[0088] Secondly, the high-quality images obtained by using the pre-trained denoising network and the estimated noise , define a pre-trained reconstruction loss , further increasing the stability of model training:
[0089] (12)
[0090] Considering that the loss function designed in this way is easy to make The value of is close to 1, that is The effect of A sparsity loss is added to constrain the mask, defined as follows:
[0091] (13)
[0092] The present invention also adds structural similarity loss , which is used to measure the structural similarity between the rendered image and the real image. This part of the loss can capture the structural differences in the image and further improve the quality of the rendered image. Finally, the comprehensive loss function of the present invention is defined as follows:
[0093] (14)
[0094] in, and is a hyperparameter. Through the design of this loss function, each part of the model can be effectively constrained to improve the quality and consistency of the generated images.
[0095] like Figure 3 As shown in Figure 1, the specific steps of reconstructing a 3D scene from a noisy input image and synthesizing a new perspective are as follows:
[0096] (1) The colmap method is used to obtain the initial pose from the multi-view noisy image and initialize the 3D Gaussian ellipsoid.
[0097] (2) Use a pre-trained blind image denoising network to obtain high-quality images and extract a rough noise distribution (Pre_noise).
[0098] (3) A noise extraction network based on U-Net is used to further extract fine noise distribution from the original noisy image (Re_noise).
[0099] (4) The 3D Gaussian ellipsoid is rendered from the scene to obtain an image. The noise simulation module combines the image, viewpoint and position information to learn the noise mask for each pixel. The noise mask is weighted with the extracted noise distribution to simulate the real noise.
[0100] (5) The simulated noise distribution is combined with the image rendered by the 3D Gaussian ellipsoid to predict the noise image.
[0101] (6) A color loss is established between the predicted noise image and the input noise image, and the structure loss and mask loss are combined to constrain the entire reconstruction process and optimize the properties of the feature extraction network and the 3D Gaussian ellipsoid.
[0102] 1. Evaluation indicators
[0103] The quality of the synthesized new perspective image is evaluated using peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). PSNR quantifies the difference between images by calculating the mean squared error (MSE) between the image synthesized by the model and the original image. The higher the PSNR value, the higher the similarity between the two images. SSIM takes into account aspects such as the brightness, contrast, and structure of the image. It ranges from 0 to 1, and the larger the value, the more similar the images are.
[0104] 2. Quantitative comparison
[0105] In order to verify the superiority of the method of the present invention, it is compared with other SOTA methods: (1) NAN, (2) NeRF_pre, and (3) 3D GS_pre. NAN is a classic method for burst denoising. It synthesizes images of new perspectives by learning adjacent perspective information. Although it can achieve good view synthesis effects, it needs to be trained in advance and the training time is long (it takes about one week to train with a single A100). Before comparing with it, it is pre-trained on the NeRF and IBRNet datasets to obtain better comparison results. In addition, since there are few new view synthesis methods for noisy input, the present invention selects the classic new view synthesis methods NeRF and 3D GS to compare with the method of the present invention. In order to reduce the impact of noise, the single image denoising method Diffbir is first used to denoise the noisy image, and then the reconstruction algorithm of NeRF and 3D GS is executed to synthesize the new view. These two processes are defined as NeRF_per and 3D GS_per.
[0106] Tables 1 and 2 show the quantitative comparison results of the method of the present invention and other sota methods on the NeRF dataset and IBRNet dataset, respectively. The performance scores and average scores of each method under different noise gains are listed in the table, with the best value highlighted in bold and the suboptimal value highlighted in underline. It can be seen that the method of the present invention can significantly improve the existing methods while removing the noise effect in view synthesis.
[0107] Table 1 Quantitative comparison results of the proposed method and other SOTA methods under different noise gains in the NeRF dataset.
[0108] Table 2 Quantitative comparison results of the proposed method and other SOTA methods under different noise gains on the IBRNet dataset.
[0109] 3. Qualitative comparison
[0110] like Figure 6 As shown in the figure, thanks to the flexible 3D Gaussian distribution representation and efficient rendering mechanism, the method of the present invention can render at a faster speed while ensuring the quality of view synthesis. The method of the present invention is significantly better than NeRF_pre trained for 12 hours (200k iterations) to synthesize new perspective views, such as books on a piano and display boards in a museum, with only 10 minutes of training (3K iterations).
[0111] Figure 7~Figure 10The experimental results of the method of the present invention and other SOTA methods are shown in the synthetic data set when the noise gain is 4. Through qualitative comparison, it can be seen that the quality of the new views synthesized by the method of the present invention is better than the results of the SOTA method under the synthetic noise data set, such as the distant railings in the horns scene, the roof spotlights in the trex scene, the handrails of the stairs in the benchflower scene, and the chairs in the livingroom scene. The method of the present invention can effectively remove noise and synthesize new perspective views with high quality.
[0112] Fig.11 and Fig.12 is the experimental result under the real data set. Fig.11 (c) It can be seen that when the inter-frame motion is large, the multi-frame fusion effect of NAN is not good, and it is difficult to overlap the views. Fig.11 In (b), when a single view is denoised using a single image denoising method, the noise in the view can be significantly reduced, but unnecessary error information will be introduced. When synthesizing new views, this error information will lead to inconsistent correspondence between views, which significantly affects the synthesis of new views of NeRF_per and 3D GS_per, as shown in Figure 2. Fig.11 (d) Fig.11 (e). However, thanks to the excellent noise extraction and fusion modules, the method of the present invention can remove noise while retaining the real details of the scene and synthesize high-quality new perspective views. Fig.12 As can be seen in (b), the single view denoising algorithm can effectively remove the noise in the image without generating any erroneous features. However, when using these denoised images to synthesize new views, artifacts still appear, such as Fig.12 (d) Fig.12 (e). This is due to the processing differences between adjacent views, which leads to geometric mismatch when synthesizing new views. Fig.12 (f), the method of the present invention effectively solves this problem.
[0113] From the qualitative analysis results of the synthetic dataset and the real dataset, it can be seen that the method of the present invention can restore more details while removing noise, and solves the problem of inconsistent view correspondence caused by single view denoising.
Claims
1. A method for synthesizing a new view using a noisy image, characterized in that The method comprises the following steps: Step (1) using the colmap method to obtain the initial pose from the multi-view noise image and initialize the 3D Gaussian ellipsoid; Step (2) uses the pre-trained blind image denoising network to obtain a high-quality image from the input noisy image and extract the initial noise distribution. The specific steps are as follows: The noisy image Input to a pre-trained blind image denoising network De-noising is performed in the image to obtain a high-quality image after denoising. , and then subtract the original noise image from it to get the initial noise distribution : Step (3) Using the U-Net-based noise extraction network Further extract fine noise distribution from the original noisy image : Step (4) Render the 3D Gaussian ellipsoid generated in step (1) from the scene to obtain the predicted image ,design The network combines image, viewpoint and position information to learn a noise mask for each pixel, and uses the noise mask to weight the extracted noise distribution to simulate real noise. Step (5) Using the simulated noise distribution and the image rendered by the 3D Gaussian ellipsoid Combined, predicting noisy images ; Step (6) The predicted noise image Create a color loss with the original noisy input image , joint structural similarity loss , Mask loss The entire reconstruction process is constrained to optimize the properties of the feature extraction network and the 3D Gaussian ellipsoid.
2. The method for realizing new view synthesis using a noisy image according to claim 1, characterized in that The specific steps of step (4) are as follows: Step (41) 3D Gaussian distribution of each Gaussian ellipsoid in space It is expressed by the following formula: Among them, Σ is the covariance matrix, which represents the shape and direction of the Gaussian distribution, and μ is the mean, which represents the center position of the Gaussian distribution; Step (42) projects the 3D Gaussian distribution representing the ellipsoid onto a 2D Gaussian distribution , the projected covariance matrix Σ′ is given by: in, is the view transformation matrix, J is the Jacobian matrix of the affine transformation; the position of the Gaussian ellipsoid in 2D space is obtained by projecting the pose initialized in step (1); Step (43) during the rendering process, the image is rendered using a fast GPU sorting algorithm and tile-based rasterization technology; Step (44) Design a residual network combined with a multi-layer perceptron The network controls the noise distribution extracted by step (2) and step (3) by learning a pixel-by-pixel mask The weights of are used to adapt to the heterogeneity of noise between pixels in different views and within views.
3. The method for realizing new view synthesis using a noisy image according to claim 2, characterized in that The specific method of step (43) is as follows: Each tile in the two-dimensional image space is regarded as a unit, which is sorted based on depth and rendered using alpha blending. After rendering, an image at any perspective is obtained. The color of each pixel in : in, represents the set of all Gaussian distributions overlapping the pixel, is the color of a 2D Gaussian distribution, is the density.
4. The method for realizing new view synthesis using a noisy image according to claim 2, characterized in that The specific steps of step (44) are as follows: First, using convolutional neural networks Extracting RGB features from the image rendered in step (43); Then, the extracted RGB features are embedded with learnable viewpoints and position embedding As feature input Network, where is the view index, is the pixel coordinate Finally, the learned mask features are transformed into Mapped to between 0 and 1 to get the final mask , the formula is as follows: 。 5. The method for realizing new view synthesis using a noisy image according to claim 1, characterized in that The specific steps of step (5) are as follows: Step (51) uses the mask learned in step (4) to fuse the noise extracted in steps (2) and (3): Step (52) converts the fused noise Image obtained with 3D Gaussian rendering Add together to get the predicted noisy image : 。 6. The method for realizing new view synthesis using a noisy image according to claim 1, characterized in that The color loss The definition is as follows: Pre-trained reconstruction loss The definition is as follows: Mask loss The definition is as follows: Comprehensive loss function The definition is as follows: in, and is a hyperparameter.