Three-dimensional reconstruction method based on fractional gradient optimization and three-dimensional Gaussian sputtering fusion

Abstract

The invention discloses a three-dimensional reconstruction method based on fractional gradient optimization and three-dimensional Gaussian sputtering fusion. A sparse point cloud is generated through a motion structure recovery technology and input picture data; gaussian primitives with the point cloud as the center are generated in the space; performing rasterization rendering on the scene represented by the Gaussian primitive; carrying out joint optimization function calculation on the rendered picture and the real picture; the fractional calculus is pruned to define and design a fractional order optimizer; optimizing the Gaussian primitive parameters by adopting a fractional order optimizer; and finishing the pruning of the Gaussian ellipsoid according to a self-adaptive pruning strategy. A fractional order optimizer is used for guiding Gaussian primitives to carry out gradient descent, the problem that three-dimensional Gaussian sputtering falls into a local minimum value in three-dimensional reconstruction is effectively relieved through fractional order gradient guidance, and joint loss optimization is carried out on rendered images and real images. The edge information of the image can be better concerned, and a reconstruction scene with richer details can be obtained.

Description

Technical Field

[0001] The invention relates to fractional-order calculus and scene three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method based on fractional-order gradient optimization and three-dimensional Gaussian sputtering fusion. Background Art

[0002] 3D Gaussian sputtering has achieved a series of groundbreaking results in the field of 3D reconstruction. It can provide display scene representation and novel view synthesis without relying on neural networks such as Neural Radiance Field (NeRF), and has made great breakthroughs in the quality and real-time performance of reconstruction. Unlike traditional methods, 3D Gaussian sputtering uses Gaussian distribution at the position of each point to represent the influence area of ​​the point, which effectively reduces the complexity of the data while improving the efficiency of calculation and rendering. However, there may be problems with optimization falling into local minima. On the one hand, since Gaussian primitives have local support, which means that if the distance to the correct position exceeds a few standard deviations, the gradient will disappear; on the other hand, since there will be occlusion features in the process of the Gaussian ellipsoid moving in space, the loss cannot be stably optimized.

[0003] In order to solve the problem of local minima that is prone to occur during the 3D reconstruction of 3D Gaussian sputtering, previous works either performed differentiable operations on Gaussian pruning and replication, or predicted the information of Gaussian balls based on generative networks, which reduced the real-time performance of 3D Gaussian sputtering reconstruction. Secondly, the prediction of Gaussian parameters with prior information is unstable and requires additional training costs. Summary of the invention

[0004] In view of the above problems, the present invention provides a three-dimensional reconstruction method based on the fusion of fractional gradient optimization and three-dimensional Gaussian sputtering, constructs a fractional gradient optimizer and designs a specific loss optimization function to utilize stronger gradient vectors. Without increasing a large amount of computational overhead, it guides the optimization process of three-dimensional reconstruction, thereby alleviating the problem of three-dimensional Gaussian sputtering falling into local minima in three-dimensional reconstruction.

[0005] The present invention adopts the following technical solutions to solve the technical problems. A three-dimensional reconstruction method based on fractional gradient optimization and three-dimensional Gaussian sputtering fusion, the steps of which are as follows:

[0006] 1) Generate sparse point cloud through motion structure recovery technology and input image data;

[0007] 2) Generate Gaussian primitives centered on the point cloud in space;

[0008] 3) Rasterize and render the scene represented by the Gaussian primitive;

[0009] 4) Perform joint optimization function calculation on the rendered image and the real image:

[0010]

[0011] Among them: λ1, λ2, λ3 represent weighting coefficients, L1 represents the mean absolute error loss between the rendered image and the real image, L D-SSIM represents the structural similarity between the rendered image and the real image; L e Represents the edge loss function based on the sobel operator; Represents the mean absolute error loss function based on fractional order correction;

[0012] 5) Prune the definition of fractional calculus to design a fractional optimizer; by analyzing the applicable scenarios of several mainstream fractional-order definitions, prune the Caputo fractional-order calculus definition, eliminate its non-locality, and use it to optimize the fractional gradient of the information contained in the Gaussian primitive as follows:

[0013]

[0014] Among them: α represents the order of the fractional order, δ represents a very small number used to prevent division by zero;

[0015] 6) A fractional order optimizer is used to optimize the Gaussian primitive parameters; the definition of the Gaussian primitive is expressed as:

[0016] {g k =(μ,Σ,σ,s)};

[0017] Where: μ represents the center coordinate of the Gaussian primitive, Σ represents the shape confidence of the Gaussian primitive, σ represents opacity and S represents color information;

[0018] By combining the pruned fractional-order definition with the Adam optimizer, a fractional-order optimizer is designed to achieve fractional-order optimization of the attributes contained in the above Gaussian primitives; combining the non-local characteristics of fractional calculus, an adaptive learning rate adjustment strategy is designed, and the non-local characteristics are realized through the exponential moving weighted average strategy, which is expressed as:

[0019] EMA t =α×x t +(1-α)×EMA t-1 ;

[0020] Among them, α is used as the weight decay coefficient to calculate the memory depth; the learning rate is dynamically adjusted according to the mean and variance changes of historical data;

[0021] 7) According to the adaptive pruning strategy, the pruning of the Gaussian ellipsoid is completed.

[0022] Compared with the prior art, the present invention has the following advantages and positive effects: the present invention integrates fractional calculus with three-dimensional Gaussian sputtering, uses fractional optimizers to guide Gaussian primitives to perform gradient descent, and uses fractional gradient guidance to effectively alleviate the problem of three-dimensional Gaussian sputtering falling into local minima in three-dimensional reconstruction. When calculating the loss, by performing joint loss optimization on the rendered image and the real image, compared with the original function, better attention can be paid to the edge information of the image, and a more detailed reconstructed scene can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is an overall framework diagram of an embodiment of the present invention;

[0024] Figure 2 It is a pruning flow chart of fractional order definition in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below in conjunction with the accompanying drawings and implementation methods. It should be understood that the implementation method is only used to illustrate the present invention and is not used to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the claims attached to the present application.

[0026] The present invention relates to a 3D reconstruction method based on the fusion of fractional gradient optimization and 3D Gaussian sputtering. Based on the 3D Gaussian sputtering algorithm, the fractional calculus and the gradient descent algorithm are fused, and a new fractional optimizer is proposed to guide the optimization of various attributes in the Gaussian primitive. At the same time, in view of the possible instability in the fractional optimization, a learning rate adjustment algorithm based on the exponential moving weighted average strategy is proposed to stabilize the training process to obtain better training results (such as Figure 1 ), including the following steps:

[0027] 1) Generate sparse point clouds through motion structure recovery technology and input image data; Use motion recovery structure technology to restore sparse point clouds in three-dimensional space by combining scene images from multiple perspectives and corresponding camera poses.

[0028] 2) Generate a Gaussian primitive centered on the point cloud in space; after obtaining the sparse point cloud, initialize the sparse point cloud in space as a Gaussian primitive with color, opacity, shape information and spherical harmonic function to represent the color, and transform the Gaussian primitive in space according to the projection matrix to obtain the Gaussian distribution in the pixel coordinate system.

[0029] 3) The scene represented by the Gaussian primitive is rasterized and rendered; the Gaussian primitive in space is transformed into pixels in a pixel coordinate system through a rasterization pipeline in three-dimensional Gaussian sputtering, thereby obtaining a rendered image.

[0030] 4) Calculate the joint optimization function of the rendered image and the real image: Calculate the loss of the rendered view and the real scene view by using the above joint loss function, in addition to calculating the photometric loss of the rendered image and the real scene image. By combining the application of fractional calculus in edge detection, a loss function based on the sobel operator for calculating the edge loss between the rendered image and the real image is designed to guide the fractional gradient optimizer to optimize the edge loss of the image, which is defined as:

[0031] Where N represents the number of edge pixels, render_edge i and gt_edge i They represent the i-th pixel value of the rendered image and the real image after being detected by the Sobel operator.

[0032] In addition, a fractional-order corrected mean absolute error loss function is designed to constrain the fractional-order optimizer, which is defined as:

[0033]

[0034] Combined with the fractional order optimizer designed in step 6), the attributes contained in the Gaussian primitive are optimized.

[0035] In this embodiment, the loss function of the scene 3D reconstruction model can be calculated in the following way: using the scene image under multiple viewpoints and the corresponding camera poses to perform pseudo view synthesis to obtain a rendered image, constructing a photometric consistency loss based on the rendered image and the real scene image; constructing an edge loss based on the edge information of the rendered image and the real scene image; constructing a mean absolute error loss after fractional order correction based on the rendered image and the real scene image; using the loss function jointly constructed using photometric consistency loss, edge loss, and mean absolute error loss based on fractional order correction to calculate the loss of the scene 3D reconstruction model. Its joint optimization loss function can be expressed as:

[0036]

[0037] Among them: λ1, λ2, λ3 represent weighting coefficients, L1 represents the mean absolute error loss, L D-SSIM represents the structural similarity error; L e Represents the edge loss function based on the sobel operator; Represents the mean absolute error loss function based on fractional-order correction; the edge loss in the formula provides explicit supervision for fractional-order optimization by combining the edge information of the rendered view with the mean absolute error loss of fractional-order correction, so that the fractional-order optimization can guide the model to be closer to the optimal solution than the integer-order optimization.

[0038] 5) Prune the fractional calculus definition to design the fractional optimizer; the pruning process is as follows Figure 2 As shown in the figure, the three classic fractional-order definitions (i.e., Grünwald-Letnikov definition, Riemann–Liouville definition, and Caputo definition) are compared. By analyzing different fractional-order definitions and their application scenarios, fractional-order definitions are selected. Finally, the Caputo fractional-order definition is selected to reduce the gradient accumulation of the fractional-order gradient optimization process by eliminating non-local features. In order to reduce its computational complexity, the calculation form of the Caputo fractional-order definition is simplified, and the stability of the gradient calculation is ensured by setting a very small number. An optimizer based on fractional-order calculus is designed to realize fractional-order optimization of various attributes of the Gaussian primitive, thereby providing a stronger gradient guidance for the Gaussian primitive, so that it can be closer to the target position than the integer-order optimizer. The fractional-order derivative calculation can be expressed as follows after simplification:

[0039]

[0040] In the above formula, α represents the order of the fractional order. By using the modified fractional calculus based on Caputo definition instead of integer calculus to calculate the biased first-order moment estimate and second-order moment estimate of the gradient, the preliminary combination of fractional calculus and adaptive moment estimation algorithm (Adam) is completed.

[0041] 6) Use fractional-order optimizer to optimize Gaussian primitive parameters: On the one hand, the non-local features will be simplified in the optimization process by pruning fractional-order calculus to reduce the complexity of the operation and reduce the impact of error information on gradient optimization; on the other hand, to ensure that the non-locality of the fractional order is reflected in the algorithm, the learning rate is dynamically adjusted by calculating the average change amplitude of the first-order moment and the second-order moment of the iterative steps within the memory depth through the moving exponential average strategy. Through the above two aspects, it is ensured that the optimization of various attributes of the Gaussian primitive using the fractional-order optimizer maintains a downward trend at the macro level.

[0042] 7) According to the adaptive pruning strategy, the pruning of Gaussian primitives is completed; the optimization efficiency of Gaussian primitives using fractional-order optimizers is worse than that of integer-order optimizers.

[0043] The present invention combines a sliding window-based Gaussian filter with adaptive density control to prune and replicate Gaussian primitives, thereby improving rendering speed and ensuring density geometry details while reducing redundant representations in the scene.

Claims

1. A 3D reconstruction method based on fractional gradient optimization and 3D Gaussian sputtering fusion, the steps of which are as follows: 1) Generate sparse point cloud through motion structure recovery technology and input image data; 2) Generate Gaussian primitives centered on the point cloud in space; 3) Rasterize and render the scene represented by the Gaussian primitive; 4) Perform joint optimization function calculation on the rendered image and the real image: L=(1-λ1)L1+λ1L D-SSIM +λ2L e +λ3L FC-L1 ; in: λ1, λ2, and λ3 represent weighting coefficients, L1 represents the mean absolute error loss between the rendered image and the real image, and L D-SSIM represents the structural similarity between the rendered image and the real image; L e represents the edge loss function based on the sobel operator; L FC-L1 Represents the mean absolute error loss function based on fractional order correction; 5) Prune the definition of fractional calculus to design a fractional optimizer; by analyzing the applicable scenarios of several mainstream fractional-order definitions, prune the Caputo fractional-order calculus definition, eliminate its non-locality, and use it to optimize the fractional gradient of the information contained in the Gaussian primitive as follows: Among them: α represents the order of the fractional order, δ represents a very small number used to prevent division by zero; 6) A fractional order optimizer is used to optimize the Gaussian primitive parameters; the definition of the Gaussian primitive is expressed as: {g} k =(μ,∑,σ,s)}; Where: μ represents the center coordinate of the Gaussian primitive, ∑ represents the shape confidence of the Gaussian primitive, σ represents opacity and S represents color information; By combining the pruned fractional-order definition with the Adam optimizer, a fractional-order optimizer is designed to achieve fractional-order optimization of the attributes contained in the above Gaussian primitives; combining the non-local characteristics of fractional calculus, an adaptive learning rate adjustment strategy is designed, and the non-local characteristics are realized through the exponential moving weighted average strategy, which is expressed as: EMA t =α×x t +(1-a)×EMA t-1 ; Among them, α is used as the weight decay coefficient to calculate the memory depth; the learning rate is dynamically adjusted according to the mean and variance changes of historical data; 7) According to the adaptive pruning strategy, the pruning of the Gaussian ellipsoid is completed.

Images