Denoising diffusion model and local linear embedding regularization three-dimensional Gaussian sputtering method
Through the denoising diffusion model and the regularization of local linear embedding, the problem of insufficient synthesis capability of new perspective synthesis technology when the input view is sparse is solved, the accuracy and rendering speed of new perspective images are improved, and pseudo-reducing , achieving higher quality image synthesis.
Patent Information
- Application Number
- CN202410918055.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-10
AI Technical Summary
The existing new perspective synthesis technology is difficult to effectively synthesize new perspective images when the input views are sparse, especially compared with neural radiation fields, which lacks the ability to synthesize high-fidelity perspective angles.
Using the denoising diffusion model and local linear embedding regularization of three-dimensional Gaussian sputtering method, the Gaussian representation is optimized to improve the new perspective image through sparse reconstruction, neighborhood Gaussian densification, local linear embedding regularization and denoising diffusion model regularization strategies. accuracy and quality.
Improves the accuracy and quality of new perspective images for sparse image synthesis, reduces artifacts, achieves faster rendering speed and higher synthesis performance, and enhances the smoothness and density of Gaussian representations.
Smart Images

Figure CN118941457A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to immersive media, namely, the field of new perspective synthesis technology, and in particular, to a denoising diffusion model and a three-dimensional Gaussian sputtering method with local linear embedding regularization. Background Art
[0002] Novel View Synthesis (NVS) has long been a fundamental problem in computer vision research. It represents the task of rendering unseen views based on a series of input views of a scene. With the continuous advancement of technology, Novel View Synthesis (NVS) is opening up new areas of visual media interaction and presentation, providing broad possibilities for future digital content creation and consumption.
[0003] In recent years, novel view synthesis (NVS) has attracted increasing attention due to the success of Neural Radiance Fields (NeRFs). NeRFs encodes 3D scenes by mapping 3D point positions to colors and volume densities using a multi-layer perceptron (MLP) and synthesizing images through volume rendering. It exhibits exceptionally high-fidelity view synthesis capabilities when the input views are dense. However, its ability to synthesize new views when the input views are sparse requires further exploration. Summary of the invention
[0004] In view of the defects in the prior art, the purpose of the present disclosure is to provide a 3D Gaussian sputtering method with a denoised diffusion model and local linear embedding regularization.
[0005] According to one aspect of the present disclosure, a 3D Gaussian sputtering method with a denoised diffusion model and local linear embedding regularization is provided, comprising:
[0006] Performing sparse reconstruction processing on the sparse image to determine an initial Gaussian distribution of the sparse image;
[0007] Performing Gaussian densification processing on the initial Gaussian distribution of the sparse image by using a neighborhood Gaussian densification method, and determining a Gaussian representation of the sparse image after the Gaussian densification processing;
[0008] Performing a local linear embedding regularization process on the Gaussian representation of the sparse image using a local linear embedding regularization strategy, and determining the Gaussian representation processed by the local linear embedding regularization process;
[0009] A denoising diffusion model regularization strategy is adopted to perform image restoration processing on the image corresponding to the Gaussian representation processed by the local linear embedding regularization to determine a synthesized new viewing angle image.
[0010] Optionally, the method further comprises:
[0011] The Gaussian representation is optimized using a preset texture loss function and a preset relative depth loss function to determine the synthesized new perspective image.
[0012] Optionally, performing sparse reconstruction processing on the sparse image to determine an initial Gaussian distribution of the sparse image includes:
[0013] Performing feature point cloud extraction processing on the sparse image to determine a sparse point cloud of the sparse image;
[0014] The sparse point cloud of the sparse image is fused to determine an initial Gaussian distribution of the sparse image.
[0015] Optionally, the performing Gaussian densification processing on the initial Gaussian distribution of the sparse image by using a neighborhood Gaussian densification method, and determining the Gaussian representation of the sparse image after the Gaussian densification processing comprises:
[0016] According to the initial Gaussian distribution of the sparse image, determining the distance between each Gaussian component and each Gaussian component;
[0017] According to the distance between each Gaussian component and each Gaussian component and a preset distance threshold, a new Gaussian component is inserted into the initial Gaussian distribution of the sparse image to determine the Gaussian representation of the sparse image after the Gaussian densification process.
[0018] Optionally, inserting a new Gaussian component into the initial Gaussian distribution of the sparse image according to the distance between each Gaussian component and each Gaussian component and a preset distance threshold, and determining the Gaussian representation of the sparse image after Gaussian densification processing includes:
[0019] If the distance between the Gaussian component and another Gaussian component is greater than the preset distance threshold, insert a new Gaussian component between the Gaussian component and another Gaussian component to determine the Gaussian representation of the sparse image after Gaussian densification;
[0020] If the distance between the Gaussian component and another Gaussian component is not greater than the preset distance threshold, no new Gaussian component is inserted between the Gaussian component and another Gaussian component, and the Gaussian representation of the sparse image after Gaussian densification is determined.
[0021] Optionally, the adopting a local linear embedding regularization strategy to perform local linear embedding regularization processing on the Gaussian representation of the sparse image, and determining the Gaussian representation processed by the local linear embedding regularization processing, comprises:
[0022] Determine spatial position information of a preset number of neighboring points of the Gaussian function of each sparse image according to the spatial position information of the Gaussian representation of the sparse image;
[0023] Determining a reconstruction error of a neighborhood of the Gaussian function of the sparse image according to spatial position information of a preset number of neighboring points of the Gaussian function of the sparse image;
[0024] The reconstruction error of the neighborhood of the Gaussian function is minimized to determine the Gaussian representation that has undergone the local linear embedding regularization process.
[0025] Optionally, the adopting a denoising diffusion model regularization strategy to perform image restoration processing on the image corresponding to the Gaussian representation processed by the local linear embedding regularization to determine a synthesized new perspective image includes:
[0026] Using a denoising diffusion model to perform forward diffusion processing on the image corresponding to the Gaussian representation processed by the local linear embedding regularization, adding Gaussian noise to the image during the forward diffusion process, and determining image samples with added Gaussian noise;
[0027] Using the denoising diffusion model to perform noise prediction processing on random RGB-D patches, and determining the predicted value of the score function;
[0028] According to the predicted value of the score function, the denoising diffusion model is used as a score function predictor to perform back diffusion processing on the image samples with added Gaussian noise, and the Gaussian noise of the image samples with added Gaussian noise is removed during the back diffusion process to determine the synthesized new perspective image.
[0029] According to a second aspect of the present disclosure, a three-dimensional Gaussian sputtering system based on a denoising diffusion model and local linear embedding regularization is provided, comprising:
[0030] A sparse reconstruction module, used for performing sparse reconstruction processing on the sparse image and determining an initial Gaussian distribution of the sparse image;
[0031] A Gaussian densification processing module, used to perform Gaussian densification processing on the initial Gaussian distribution of the sparse image by using a neighborhood Gaussian densification method, and determine the Gaussian representation of the sparse image after the Gaussian densification processing;
[0032] A local linear embedding regularization processing module, used to perform local linear embedding regularization processing on the Gaussian representation of the sparse image using a local linear embedding regularization strategy, and determine the Gaussian representation after the local linear embedding regularization processing;
[0033] The new perspective synthesis module is used to perform image restoration processing on the image corresponding to the Gaussian representation processed by the local linear embedding regularization process by adopting a denoising diffusion model regularization strategy to determine a synthesized new perspective image.
[0034] According to a third aspect of the present disclosure, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method provided in the first aspect of the present disclosure are implemented.
[0035] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:
[0036] a memory having a computer program stored thereon;
[0037] A processor is used to execute the computer program in the memory to implement the steps of the method provided in the first aspect of the present disclosure.
[0038] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:
[0039] Through the above technical scheme, the initial Gaussian distribution of the sparse image is densified by using a neighborhood Gaussian densification method, which can supplement the Gaussian representation of the sparse image and improve the accuracy of synthesizing new perspective images using the sparse image. The Gaussian representation of the sparse image is locally linearly embedded and regularized by using a local linear embedding regularization strategy, which can provide a more compact and smoother Gaussian distribution, improve the smoothness of the local distribution of the Gaussian representation, and improve the quality of synthesized new perspective images. The denoising diffusion model regularization strategy is used to prevent overfitting and reduce artifacts, reduce the interference of human factors, and achieve faster rendering speed and higher performance of synthesizing new perspective images. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:
[0041] Figure 1 It is a flowchart of a denoising diffusion model and a three-dimensional Gaussian sputtering method with local linear embedding regularization according to an exemplary embodiment.
[0042] Figure 2 It is a process schematic diagram of a three-dimensional Gaussian sputtering method based on a denoising diffusion model and local linear embedding regularization according to an exemplary embodiment.
[0043] Figure 3 It is a schematic diagram of process decomposition of a neighborhood Gaussian densification method according to an exemplary embodiment.
[0044] Figure 4It is a process decomposition diagram of a local linear embedding regularization strategy according to an exemplary embodiment.
[0045] Figure 5 The figure is a schematic diagram of a process decomposition of a denoising diffusion model regularization strategy according to an exemplary embodiment.
[0046] Figure 6 is a schematic diagram showing a performance comparison on a MipNeRF 360 dataset according to an exemplary embodiment.
[0047] Figure 7 The block diagram of a three-dimensional Gaussian sputtering system based on a denoising diffusion model and local linear embedding regularization is shown according to an exemplary embodiment. DETAILED DESCRIPTION
[0048] The present disclosure is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but are not intended to limit the present disclosure in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present disclosure. These all fall within the scope of protection of the present disclosure.
[0049] Figure 1 It is a flowchart of a denoising diffusion model and a three-dimensional Gaussian sputtering method with local linear embedding regularization according to an exemplary embodiment. Figure 2 It is a schematic diagram of another process of three-dimensional Gaussian sputtering based on a denoising diffusion model and local linear embedding regularization according to an exemplary embodiment.
[0050] like Figure 1 , Figure 2 As shown, the present disclosure provides a three-dimensional Gaussian sputtering method with a denoising diffusion model and local linear embedding regularization, including S11 to S14.
[0051] S11, performing sparse reconstruction processing on the sparse image to determine an initial Gaussian distribution of the sparse image.
[0052] Among them, the sparse reconstruction process uses Structure-from-Motion (SfM) to initialize the three-dimensional Gaussian distribution of the sparse image.
[0053] In a possible embodiment, S11 may include S21 to S22.
[0054] S21, performing feature point cloud extraction processing on the sparse image to determine a sparse point cloud of the sparse image.
[0055] S22, performing fusion processing on the sparse point cloud of the sparse image to determine an initial Gaussian distribution of the sparse image.
[0056] For example, when rendering a 2D image, the Gaussian distributions that affect the same pixel are organized and fused to determine the color of the pixel, thereby determining the initial Gaussian distribution of the sparse image. The contribution of each Gaussian distribution to the pixel color is calculated using the color value and transparency of the Gaussian distribution:
[0057]
[0058] Among them, c represents the pixel color after fusion, c i Denotes the color derived from the SH function of the i-th Gaussian.
[0059] α i and α j To use the covariance Σ′∈R 2×2 Evaluated by multiplication with a 2D Gaussian function of opacity.
[0060] Among them, covariance Σ′=JWΣ -1 W T JT is the projection of the three-dimensional covariance Σ onto the camera coordinates, where J represents the Jacobian matrix of the affine approximation of the projective transformation and W represents the view transformation matrix.
[0061] S12, performing Gaussian densification processing on the initial Gaussian distribution of the sparse image by using a neighborhood Gaussian densification method, and determining a Gaussian representation of the sparse image after the Gaussian densification processing.
[0062] Among them, a neighborhood Gaussian densification method is used to insert new Gaussian components into the initial Gaussian distribution of the sparse image to supplement the Gaussian components of the sparse image.
[0063] Figure 3 It is a schematic diagram of process decomposition of a neighborhood Gaussian densification method according to an exemplary embodiment.
[0064] In a possible embodiment, S12 may include S31 to S32.
[0065] S31, determining the distance between each Gaussian component according to the initial Gaussian distribution of the sparse image.
[0066] like Figure 3 As shown, the graph-based method constructs the connection relationship between Gaussian components, wherein the center point of each colored ring graphic represents a Gaussian component, and the Euclidean distance between the Gaussian components represents the interaction between the Gaussian components.
[0067] S32, inserting new Gaussian components into the initial Gaussian distribution of the sparse image according to the distance between each Gaussian component and a preset distance threshold, and determining the Gaussian representation of the sparse image after Gaussian densification processing.
[0068] In the initial stage of model training, Gaussian pairs exceeding a preset distance threshold can be identified by determining the Euclidean distance between each Gaussian component and each Gaussian component.
[0069] like Figure 3 As shown, as an example, if the Euclidean distance between a Gaussian component and another Gaussian component is greater than a preset distance threshold, a new Gaussian component is inserted between the Gaussian component and the other Gaussian component to determine the Gaussian representation of the sparse image after Gaussian densification.
[0070] A new Gaussian component is inserted between Gaussian pairs whose distance is greater than a preset threshold value. The position of the new Gaussian component is located at the geometric center position between the Gaussian pairs, and the initial parameters of the new Gaussian component are set to the average value of the Gaussian parameters of the Gaussian pairs.
[0071] Among them, the initial parameters of the new Gaussian component include mean and covariance.
[0072] In some possible embodiments, the parameters of the new Gaussian component are dynamically adjusted, and based on the difference between the Gaussian representation of the sparse image after Gaussian densification and the original scene data, the parameters of the newly inserted Gaussian component are iteratively optimized to minimize the difference between the Gaussian representation of the sparse image after Gaussian densification and the original scene data, thereby improving the scene expression ability and rendering quality of the DLGS model.
[0073] As another example, if the distance between a Gaussian component and another Gaussian component is not greater than a preset distance threshold, no new Gaussian component is inserted between the Gaussian component and the other Gaussian component, and the Gaussian representation of the sparse image after Gaussian densification is determined.
[0074] Wherein, no new Gaussian component is inserted between Gaussian pairs whose distance is not greater than a preset threshold.
[0075] In the middle and late stages of model training, when the scene representation obtained by inserting new Gaussian components is dense enough, the neighborhood Gaussian densification method is stopped and no new Gaussian components are inserted.
[0076] In the present disclosure, an adaptive distance threshold adjustment mechanism can be used to dynamically adjust the preset distance threshold according to the training process of the model, so as to quickly increase the Gaussian density in the early stage of model training and refine the Gaussian distribution in the middle and late stages of model training, thereby improving the accurate representation of the scene.
[0077] According to the requirements of specific scenarios, machine learning algorithms can also be used to automatically optimize Gaussian parameters, such as the position, size, and shape of Gaussian components, as well as the parameters used in the Gaussian densification process, such as the preset distance threshold and the number of Gaussians connected to each component, to improve the applicability and scenario expression capabilities of the DLGS model.
[0078] S13, using a local linear embedding regularization strategy to perform local linear embedding regularization processing on the Gaussian representation of the sparse image, and determining the Gaussian representation processed by the local linear embedding regularization processing.
[0079] Figure 4 It is a process decomposition diagram of a local linear embedding regularization strategy according to an exemplary embodiment.
[0080] Among them, S13 may include S41 to S43.
[0081] S41, determining spatial position information of a preset number of neighboring points of the Gaussian function of each sparse image according to the spatial position information of the Gaussian representation of the sparse image.
[0082] like Figure 4 As shown, suppose that the Gaussian representation of the sparse image includes n Gaussian functions. In the three-dimensional space, the spatial position information of each Gaussian function is denoted as μ i , determine the spatial position information of the preset number k neighboring points of each Gaussian function, where the spatial position information of the jth neighboring point is recorded as μ ij .
[0083] S42, determining a reconstruction error of a neighborhood of the Gaussian function of the sparse image according to spatial position information of a preset number of neighboring points of the Gaussian function of the sparse image.
[0084] Continuing with the above example, define the weight matrix W∈R n×k The reconstruction error of the neighborhood of the Gaussian function of the sparse image, that is, the optimization target, is:
[0085]
[0086] Among them, W ij represents the reconstruction weight of the jth neighbor of the i-th Gaussian function.
[0087] Detailed derivation of the optimization objectives:
[0088]
[0089] in,
[0090] W i =[W i1 ,Wi2 ,…,W ik ]∈R 1×k ..
[0091] Among them, W i The second partial derivative of is expressed as a semi-positive definite matrix, then the optimization objective is a convex optimization problem.
[0092] S43, minimizing the reconstruction error of the neighborhood of the Gaussian function, and determining a Gaussian representation that has been subjected to local linear embedding regularization.
[0093] Continuing with the above example, the Lagrange multiplication method is used to calculate the reconstruction error of the neighborhood of the Gaussian function of the sparse image, that is, to solve the optimization target.
[0094] The optimization objective function is expressed using Lagrange multipliers as:
[0095]
[0096] Where L(W,λ) represents the solution function and λ represents the Lagrange multiplier.
[0097] The definition of e is as follows:
[0098]
[0099] e=[1,1,…,1] T ∈=R k×1
[0100] For the objective function W i Find the partial derivative and set it to zero, and get the following:
[0101]
[0102] For the objective function λ i Find the partial derivative and set it to zero, and get the following:
[0103]
[0104] W i e=1,
[0105]
[0106] Then the only solution to the minimum value of the optimization objective is:
[0107]
[0108] Among them, λ satisfies:
[0109]
[0110] Once we have the matrix, we can determine the reconstruction error of each Gaussian function’s neighborhood by calculation, and thus determine the loss function of the Local Linear Embedding (LLE) regularization strategy used in training as:
[0111]
[0112] Among them, L LLE Represents the loss function of the locally linear embedding regularization strategy.
[0113] By minimizing the reconstruction error of the neighborhood of the Gaussian function and using the loss function of the local linear embedding regularization strategy for constraint, the Gaussian representation processed by the local linear embedding regularization is determined, the linear combination of the Gaussian and its nearest neighbors is constrained to be as close as possible, and the smooth local distribution of the Gaussian function is improved, thereby improving the local detail performance of the scene in the synthesized perspective image.
[0114] S14, using a denoising diffusion model regularization strategy to perform image restoration processing on the image corresponding to the Gaussian representation processed by local linear embedding regularization, to determine a synthesized new perspective image.
[0115] Figure 5 The figure is a schematic diagram of a process decomposition of a denoising diffusion model regularization strategy according to an exemplary embodiment.
[0116] In a possible embodiment, S14 may include S51 to S52.
[0117] S51, using a denoising diffusion model to perform forward diffusion processing on the image corresponding to the Gaussian representation processed by local linear embedding regularization, adding Gaussian noise to the image during the forward diffusion process, and determining image samples with the added Gaussian noise.
[0118] like Figure 5 As shown, in the process of forward diffusion processing using the denoising diffusion model (DDM), Gaussian noise is gradually added to the images x0~q(x) corresponding to the Gaussian representation processed by local linear embedding regularization to generate noisy image samples.
[0119]
[0120] ∈ τ-1 ~N(0,I)
[0121] α τ =1-β τ
[0122] Among them, x τ represents the noise sample, that is, the image sample with Gaussian noise added, α τ represents the difference from the variance, xτ-1 represents the previous noise sample, β τ represents variance, `o τ-1 Represents Gaussian noise
[0123] By variance Controls the amount of Gaussian noise.
[0124] The noise function is a Gaussian function, defined as:
[0125]
[0126] in, represents the noise function, α s Represents a parameter in a procedure.
[0127] In order to effectively generate arbitrary noise samples, the noise sample processing is reparameterized:
[0128]
[0129] When τ→∞, the noise sample x τ The distribution of converges to an isotropic unit Gaussian distribution.
[0130] S52, using a denoising diffusion model to perform noise prediction processing on the random RGB-D patch to determine a predicted value of the score function.
[0131] The score function represents the prior probability score of scene geometry and color (σ,c).
[0132] S53, according to the predicted value of the score function, a denoising diffusion model is used as a score function predictor to perform back diffusion processing on the image samples with added Gaussian noise, and the Gaussian noise of the image samples with added Gaussian noise is removed during the back diffusion process to determine a synthesized new perspective image.
[0133] The score function predictor determines the posterior probability scores of the density field and the color field (σ, c) given the training views.
[0134] Continuing with the above example, the score function is expressed as the gradient of the log-likelihood value and the gradient of the logarithmic prior
[0135] For the input training image I, the posterior probability of the density field and color field is expressed as:
[0136]
[0137] Among them, I represents the input training image, σ represents the density field, and c represents the color field
[0138] Removing the normalization constant that depends only on the training image I, the logarithm of the posterior probability of the density field and the color field is expressed as:
[0139]
[0140] During the training of the denoising diffusion model (DDM), by maximizing All you need to do is calculate the score function.
[0141] Continuing with the above example, in the process of back diffusion processing using the denoising diffusion model (DDM), it is expressed as:
[0142]
[0143] Among them, x τ can be used as the average value μ(x τ ,τ) input, using a pre-trained neural network ∈ θ (x τ ,τ) predicts the noise ∈ based on the noise input and the noise level τ-1 , then determine the average value μ(x τ ,τ):
[0144]
[0145] The loss function when training the denoising diffusion model (DDM) is:
[0146]
[0147] in,
[0148] From the above embodiment, it can be seen that based on the relationship between the noise predictor of the noise diffusion model and the score function, the noise predictor of the noise diffusion model is proportional to the score function:
[0149]
[0150] Among them, ∈ θ (x τ ,τ) represents the noise predictor based on the noise diffusion model, Represents a score function.
[0151] Through the above technical solution, the present disclosure can use a noise diffusion model (DDM) as a score function predictor to regularize three-dimensional Gaussian sputtering (3D-GS).
[0152] For example, we establish a scoring function ∈ based on the RGBD patch distribution θ({C(r), D(r)|r∈P}), and establish a (σ,c). Where P represents a set of rays originating from any camera and passing through a random 48×48 pixel block.
[0153] In order to effectively manage the size of the gradient, the output ∈ is further standardized in the regularization process θ ({C(r),D(r)|r∈P}).
[0154] The gradient of the log-posteriori curve can be expressed as:
[0155]
[0156] By combining the equation Using the noise diffusion model as (σ, c), we can obtain the noise diffusion model regularization loss:
[0157]
[0158] Through the above technical scheme, a denoising diffusion model regularization strategy is introduced in Gaussian sputtering, which uses the denoising diffusion model as the prior of the scene density field and color field, and minimizes the photometric difference between the training image and the scene differentiable rendering, guiding the fitting of the Gaussian distribution, which can prevent overfitting and reduce noise and artifacts.
[0159] In one possible embodiment, a denoising diffusion model (DDM) can be pre-trained on RGB-D patches on the Hypanem dataset to predict the logarithmic gradient of the joint probability distribution of color and depth patches.
[0160] During Gaussian sputtering training, random RGB-D patches are presented and the estimated log-likelihood gradients are back-propagated to the density and color fields. Gaussian sputtering is trained using stochastic gradient descent, and the logarithmic gradient is sufficient. The estimated log-likelihood gradient is back-propagated to the NeRF network during training as a regularizer. Thus, the denoised diffusion model (DDM) gradient encourages Gaussian sputtering of the density and color fields, reducing noise and artifacts.
[0161] Through the present disclosure, a neighborhood Gaussian densification method is used to perform Gaussian densification processing on the initial Gaussian distribution of a sparse image, which can supplement the Gaussian representation of the sparse image and improve the accuracy of synthesizing new perspective images using the sparse image. A local linear embedding regularization strategy is used to perform local linear embedding regularization processing on the Gaussian representation of the sparse image, which can provide a more compact and smoother Gaussian distribution, improve the smoothness of the local distribution of the Gaussian representation, and improve the quality of synthesized new perspective images. A denoising diffusion model regularization strategy is used to prevent overfitting and reduce artifacts, reduce the interference of human factors, and achieve faster rendering speed and higher performance of synthesizing new perspective images.
[0162] In some possible embodiments, a denoising diffusion model and a three-dimensional Gaussian sputtering method with local linear embedding regularization may also include S15.
[0163] S15, optimizing the Gaussian representation using a preset texture loss function and a preset relative depth loss function to determine a synthesized new perspective image.
[0164] The preset texture loss function is expressed as the difference between the rendered view of the synthetic new perspective of the input image and the original rendered view of the input image. -Photometric reconstruction loss based on the difference in norm.
[0165]
[0166] Among them, L RGB represents the preset texture loss function, C represents the original rendered view of the input image, that is, the real image, A rendered view representing a synthesized new perspective of the input image, i.e., the predicted image.
[0167] The preset relative depth loss function is a relaxed relative depth loss based on Pearson correlation, which is expressed as:
[0168]
[0169] Among them, L D represents the preset relative depth loss function, D represents the predicted depth, represents the monocular depth map generated by the pre-trained dense prediction transformer (DPT) at different viewpoints, Cov() represents the covariance, Corr() represents the correlation coefficient, and Var() represents the variance.
[0170] By adopting a preset relative depth loss function, the depth structure is aligned with loose constraints without being affected by the difference in absolute depth values, and the proportional difference between the real scene and the estimated depth can be resolved.
[0171] In a possible embodiment, a peak signal-to-noise ratio (PSNR), a structural similarity index (SSIM), and a learning-based perceptual image block similarity (LPIPS) may be used to evaluate the quality of the synthesized new-view image, wherein the peak signal-to-noise ratio (PSNR, in dB) is used to evaluate the quality of RGB reconstruction, and the higher the peak signal-to-noise ratio, the higher the quality of the RGB reconstruction of the synthesized new-view image; the structural similarity index (SSIM, in %) is used to evaluate the quality of the reconstructed image, and the higher the structural similarity index, the higher the image quality of the synthesized new-view image; the learning-based perceptual image block similarity (LPIPS) is used to evaluate the similarity between image blocks, and the lower the learning-based perceptual image block similarity, the higher the similarity between the image blocks.
[0172] A denoising diffusion model and a three-dimensional Gaussian sputtering method with local linear embedding regularization provided in the present invention are compared with seven other methods (including RegNeRF, DSNeRF, DDP-NeRF, InfoNeRF, TensoRF, DietNeRF and ViP-NeRF). The above methods can be experimented on the public datasets LLFF dataset, RealEstate10K dataset and MipNeRF360 dataset.
[0173] Among them, the LLFF dataset contains 8 forward scenes, each scene contains a different number of frames, and the spatial resolution is 4032×3024; the RealEstate10K dataset contains 10 million frames of camera poses from approximately 80,000 video clips, and the first 5 scenes are selected, each containing 50 frames with a spatial resolution of 1024×576; the Mip-NeRF 360 dataset contains 9 scenes, each of which is characterized by a complex central object or area and a detailed background. Seven scenes are used for comparison, each with more than 100 views and a resolution of more than 3115×2078; the Mip-NeRF 360 dataset uses 8 frames for testing, and 24 frames are evenly selected from the remaining frames for training.
[0174] The comparison results on the LLFF dataset are shown in Table 1:
[0175]
[0176] Table 1
[0177] As shown in Table 1, in the case of 2 input viewing angles, 3 input viewing angles, and 4 input viewing angles, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the method provided by the present disclosure are higher than those of other methods, and the learning-based perceptual image block similarity (LPIPS) is lower than that of other methods, indicating that the method provided by the present disclosure has a high ability in replicating and preserving the original image structure, and the image generated by the method provided by the present disclosure is consistent with human visual perception, surpassing other methods. From an objective perspective, the quantitative indicators of the LLFF dataset reinforce the superior performance of the present method.
[0178] The comparison results on the RealEstate10K dataset are shown in Table 2:
[0179]
[0180]
[0181] Table 2
[0182] As can be seen from Table 2, in the cases of 2 input perspectives, 3 input perspectives, and 4 input perspectives, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the method provided by the present invention are higher than those of other methods, and the learning-based perceptual image block similarity (LPIPS) is lower than that of other methods, indicating that the method provided by the present invention has excellent image structure and texture reproduction capabilities, and has high-quality image reconstruction capabilities, and the synthesized new perspective images are in line with human visual perception.
[0183] Figure 6 is a schematic diagram showing a performance comparison on a MipNeRF 360 dataset according to an exemplary embodiment.
[0184] like Figure 6 As shown in the figure, the method provided by the present disclosure shows excellent subjective effects when qualitatively compared with 24 input views on the MipNeRF 360 dataset. The method provided by the present disclosure enhances local details and contrast levels, and the synthesized image is consistent with the real ground with high fidelity. For example, the texture of natural elements, such as trees and wood, presents excellent clarity and depth, adding a vivid and realistic feeling to the overall image.
[0185] In contrast, other methods perform poorly in reconstructing local details, presenting images with reduced contrast and unclear edges, resulting in a flatter and less appealing visual experience, and other methods introduce artifacts and noise that impair the visual quality of the image.
[0186] The method disclosed herein avoids common artifacts and noise, achieves a cleaner appearance, reduces visual interference, and thus achieves a more consistent and natural immersive viewing experience. The accuracy of the scene reconstruction method provided by the present disclosure produces visually pleasing images that are not only closer to reality, but also more aesthetically superior.
[0187] Figure 7 The block diagram of a three-dimensional Gaussian sputtering system based on a denoising diffusion model and local linear embedding regularization is shown according to an exemplary embodiment.
[0188] Based on the same concept, the present disclosure also provides an image restoration system based on wavelet transform and multi-scale residuals, such as Figure 7 As shown, it includes: a sparse reconstruction module 110, a Gaussian densification processing module 120, a local linear embedding regularization processing module 130, and a new perspective synthesis module 140.
[0189] The sparse reconstruction module 110 is used to perform sparse reconstruction processing on the sparse image and determine the initial Gaussian distribution of the sparse image;
[0190] A Gaussian densification processing module 120 is used to perform Gaussian densification processing on the initial Gaussian distribution of the sparse image by using a neighborhood Gaussian densification method, and determine a Gaussian representation of the sparse image after the Gaussian densification processing;
[0191] A local linear embedding regularization processing module 130 is used to perform local linear embedding regularization processing on the Gaussian representation of the sparse image using a local linear embedding regularization strategy, and determine the Gaussian representation after the local linear embedding regularization processing;
[0192] The new perspective synthesis module 140 is used to perform image restoration processing on the image corresponding to the Gaussian representation processed by the local linear embedding regularization process by adopting a denoising diffusion model regularization strategy to determine a synthesized new perspective image.
[0193] Through the above technical scheme, the initial Gaussian distribution of the sparse image is densified by using a neighborhood Gaussian densification method, which can supplement the Gaussian representation of the sparse image and improve the accuracy of synthesizing new perspective images using the sparse image. The Gaussian representation of the sparse image is locally linearly embedded and regularized by using a local linear embedding regularization strategy, which can provide a more compact and smoother Gaussian distribution, improve the smoothness of the local distribution of the Gaussian representation, and improve the quality of synthesized new perspective images. The denoising diffusion model regularization strategy is used to prevent overfitting and reduce artifacts, reduce the interference of human factors, and achieve faster rendering speed and higher performance of synthesizing new perspective images.
[0194] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0195] Based on the same concept as above, in another embodiment of the present disclosure, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor is used to execute a denoising diffusion model and a three-dimensional Gaussian sputtering method with local linear embedding regularization when executing the program.
[0196] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.
[0197] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.
[0198] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.
[0199] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.
[0200] In an embodiment of the present disclosure, a non-temporary computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a denoising diffusion model and a three-dimensional Gaussian sputtering method with local linear embedding regularization in any of the above embodiments are implemented.
[0201] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0202] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0203] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0204] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0205] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.
[0206] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.
Claims
1. A 3D Gaussian sputtering method with a denoised diffusion model and local linear embedding regularization, It is characterized in that include: Performing sparse reconstruction processing on the sparse image to determine an initial Gaussian distribution of the sparse image; Performing Gaussian densification processing on the initial Gaussian distribution of the sparse image by using a neighborhood Gaussian densification method, and determining a Gaussian representation of the sparse image after the Gaussian densification processing; Performing a local linear embedding regularization process on the Gaussian representation of the sparse image using a local linear embedding regularization strategy, and determining the Gaussian representation processed by the local linear embedding regularization process; A denoising diffusion model regularization strategy is adopted to perform image restoration processing on the image corresponding to the Gaussian representation processed by the local linear embedding regularization to determine a synthesized new viewing angle image.
2. The method according to claim 1, It is characterized in that The method further comprises: The image corresponding to the Gaussian representation subjected to the local linear embedding regularization processing is optimized by using a preset texture loss function and a preset relative depth loss function to determine the synthesized new perspective image.
3. The method according to claim 1, It is characterized in that The performing sparse reconstruction processing on the sparse image to determine the initial Gaussian distribution of the sparse image includes: Performing feature point cloud extraction processing on the sparse image to determine a sparse point cloud of the sparse image; The sparse point cloud of the sparse image is fused to determine an initial Gaussian distribution of the sparse image.
4. The method according to claim 1, It is characterized in that The method of performing Gaussian densification processing on the initial Gaussian distribution of the sparse image by using a neighborhood Gaussian densification method, and determining the Gaussian representation of the sparse image after the Gaussian densification processing, comprises: According to the initial Gaussian distribution of the sparse image, determining the distance between each Gaussian component and each Gaussian component; According to the distance between each Gaussian component and each Gaussian component and a preset distance threshold, a new Gaussian component is inserted into the initial Gaussian distribution of the sparse image to determine the Gaussian representation of the sparse image after the Gaussian densification process.
5. The method according to claim 4, It is characterized in that The inserting of a new Gaussian component into the initial Gaussian distribution of the sparse image according to the distance between each Gaussian component and each Gaussian component and a preset distance threshold, and determining the Gaussian representation of the sparse image after Gaussian densification processing, comprises: If the distance between the Gaussian component and another Gaussian component is greater than the preset distance threshold, insert a new Gaussian component between the Gaussian component and another Gaussian component to determine the Gaussian representation of the sparse image after Gaussian densification; If the distance between the Gaussian component and another Gaussian component is not greater than the preset distance threshold, no new Gaussian component is inserted between the Gaussian component and another Gaussian component, and the Gaussian representation of the sparse image after Gaussian densification is determined.
6. The method according to claim 1, It is characterized in that The adopting the local linear embedding regularization strategy to perform local linear embedding regularization processing on the Gaussian representation of the sparse image, and determining the Gaussian representation after the local linear embedding regularization processing, comprises: Determine spatial position information of a preset number of neighboring points of the Gaussian function of each sparse image according to the spatial position information of the Gaussian representation of the sparse image; Determining a reconstruction error of a neighborhood of the Gaussian function of the sparse image according to spatial position information of a preset number of neighboring points of the Gaussian function of the sparse image; The reconstruction error of the neighborhood of the Gaussian function is minimized to determine the Gaussian representation that has undergone the local linear embedding regularization process.
7. The method according to claim 1, It is characterized in that The method of using a denoising diffusion model regularization strategy to perform image restoration processing on the image corresponding to the Gaussian representation processed by the local linear embedding regularization to determine a synthesized new perspective image includes: Using a denoising diffusion model to perform forward diffusion processing on the image corresponding to the Gaussian representation processed by the local linear embedding regularization, adding Gaussian noise to the image during the forward diffusion process, and determining image samples with added Gaussian noise; Using the denoising diffusion model to perform noise prediction processing on random RGB-D patches, and determining the predicted value of the score function; According to the predicted value of the score function, the denoising diffusion model is used as a score function predictor to perform back diffusion processing on the image samples with added Gaussian noise, and the Gaussian noise of the image samples with added Gaussian noise is removed during the back diffusion process to determine the synthesized new perspective image.
8. A three-dimensional Gaussian sputtering system based on a denoised diffusion model and local linear embedding regularization, It is characterized in that include: A sparse reconstruction module, used for performing sparse reconstruction processing on the sparse image and determining an initial Gaussian distribution of the sparse image; A Gaussian densification processing module, used to perform Gaussian densification processing on the initial Gaussian distribution of the sparse image by using a neighborhood Gaussian densification method, and determine the Gaussian representation of the sparse image after the Gaussian densification processing; A local linear embedding regularization processing module, used to perform local linear embedding regularization processing on the Gaussian representation of the sparse image using a local linear embedding regularization strategy, and determine the Gaussian representation after the local linear embedding regularization processing; The new perspective synthesis module is used to perform image restoration processing on the image corresponding to the Gaussian representation processed by the local linear embedding regularization process by adopting a denoising diffusion model regularization strategy to determine a synthesized new perspective image.
9. A non-transitory computer-readable storage medium having stored thereon a computer program, It is characterized in that When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.
10. An electronic device, It is characterized in that include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Real image denoising method based on multi-scale fusion and edge enhancement
CN112233038A
Sparse view three-dimensional reconstruction method based on generative scene image
CN118262050A
Cited By
CT image reconstruction method based on three-dimensional diffusion model and related device
CN119516114A
Three-dimensional reconstruction method and device based on structured three-dimensional Gaussian representation
CN119625169A
Generalized efficient Gaussian representation learning method and device, equipment and storage medium
CN119741423A
Gaussian image rendering method and device
CN119850805A
Dynamic Gaussian scene reconstruction method based on depth regularization
CN119991974A
