3D Reconstruction Method Based on the Fusion of Fractional Gradient Optimization and 3D Gaussian Sputtering

Abstract

The present invention discloses a three-dimensional reconstruction method based on the fusion of fractional gradient optimization and three-dimensional Gaussian sputtering. A sparse point cloud is generated through the structure from motion technology and the input picture data; Gaussian primitives centered on the point cloud are generated in space; rasterization rendering is performed on the scene represented by the Gaussian primitives; a joint optimization function is calculated for the rendered picture and the real picture; a fractional-order optimizer is designed by trimming the definition of fractional calculus; the fractional-order optimizer is used to optimize the parameters of the Gaussian primitives; according to the adaptive trimming strategy, the trimming of the Gaussian ellipsoid is completed. By using the fractional-order optimizer to guide the Gaussian primitives to perform gradient descent, the problem that the three-dimensional Gaussian sputtering falls into local minima in three-dimensional reconstruction is effectively alleviated by using the guidance of the fractional gradient. By performing joint loss optimization on the rendered image and the real image compared with the original function, the edge information of the image can be better focused on, and a reconstructed scene with richer details can be obtained.

Description

Technical Field

[0001] The present invention relates to fractional calculus and three-dimensional scene reconstruction technology, and particularly to a three-dimensional reconstruction method based on fractional gradient optimization and three-dimensional Gaussian splatting fusion. Background Art

[0002] Three-dimensional Gaussian splatting has achieved a series of groundbreaking results in the field of three-dimensional reconstruction. It can provide a display scene representation and novel view synthesis without relying on neural networks such as Neural Radiance Fields (NeRF), and has made significant breakthroughs in the quality and real-time performance of reconstruction. Different from traditional methods, three-dimensional Gaussian splatting represents the influence area of a point by using a Gaussian distribution at the position of each point, thus effectively reducing the complexity of data while improving the computational and rendering efficiency. However, the problem of optimization falling into local minima may occur. On the one hand, due to the local support of Gaussian primitives, which means that if the distance to the correct position exceeds several standard deviations, the gradient will disappear; on the other hand, since there will be cases of occluding features during the movement of Gaussian ellipsoids in space, the loss cannot be stably optimized.

[0003] Regarding the problem of local minima that easily occur during the three-dimensional reconstruction process using three-dimensional Gaussian splatting, previous work was either to perform differentiable operations on Gaussian pruning and replication, or to predict the information of Gaussian spheres based on generative networks, which led to a reduction in the real-time performance of three-dimensional Gaussian splatting for reconstruction. Secondly, predicting Gaussian parameters with the help of prior information is unstable and requires additional training costs. Summary of the Invention

[0004] To solve the above problems, the present invention provides a three-dimensional reconstruction method based on fractional gradient optimization and three-dimensional Gaussian splatting fusion. A fractional gradient optimizer is constructed and a specific loss optimization function is designed to utilize a stronger gradient vector to guide the optimization process of three-dimensional reconstruction without increasing a large amount of computational overhead, so as to alleviate the problem of three-dimensional Gaussian splatting falling into local minima in three-dimensional reconstruction.

[0005] The present invention adopts the following technical solutions to solve its technical problems. A three-dimensional reconstruction method based on fractional gradient optimization and three-dimensional Gaussian splatting fusion, the steps are as follows:

[0006] 1) Generate sparse point clouds through structure from motion technology and input image data;

[0007] 2) Generate Gaussian primitives centered on the point clouds in space;

[0008] 3) Rasterize and render the scene represented by the Gaussian primitives;

[0009] 4) Calculate the joint optimization function for the rendered image and the real image:

[0010]

[0011] Wherein: λ1, λ2, and λ3 represent weighting coefficients, L1 represents the mean absolute error loss between the rendered image and the ground truth image, and L D-SSIM represents the structural similarity between the rendered image and the ground truth image; L e represents the edge loss function based on the sobel operator; represents the mean absolute error loss function after fractional order correction;

[0012] 5) Prune the fractional calculus definition to design a fractional order optimizer; by analyzing the applicable scenarios of several current mainstream fractional order definitions, prune the Caputo fractional calculus definition, and after eliminating its non-locality, use it to perform fractional order gradient optimization on the information contained in the Gaussian primitive as follows:

[0013]

[0014] Wherein: α represents the order of the fractional order, and δ represents an extremely small number used to prevent division by zero;

[0015] 6) Use the fractional order optimizer to optimize the Gaussian primitive parameters; the definition of the Gaussian primitive is expressed as:

[0016] {g k =(μ,∑,σ,s)};

[0017] Wherein: μ represents the center coordinates of the Gaussian primitive, ∑ represents the shape confidence of the Gaussian primitive, σ represents the opacity, and S represents the color information;

[0018] By combining the pruned fractional order definition with the Adam optimizer, design a fractional order optimizer to achieve fractional order optimization of the attributes contained in the above Gaussian primitive; combining the non-local characteristics of fractional calculus, design an adaptive learning rate adjustment strategy, and implement non-local features through the exponential moving weighted average strategy, and the exponential moving weighted average strategy is expressed as:

[0019] EMA t =α×x t +(1-α)×EMA t-1 ;

[0020] Wherein, α is used as the weight decay coefficient for calculating the memory depth; dynamically adjust the learning rate according to the comparison of the mean and variance changes of historical data;

[0021] 7) Complete the pruning of the Gaussian ellipsoid according to the adaptive pruning strategy.

[0022] Compared with the prior art, the present invention has the following advantages and positive effects: The present invention combines fractional calculus with three-dimensional Gaussian sputtering. By using a fractional optimizer to guide the Gaussian primitive to perform gradient descent, the effective use of fractional gradients can alleviate the problem that three-dimensional Gaussian sputtering falls into local minima in three-dimensional reconstruction. When calculating the loss, by jointly optimizing the loss of the rendered image and the real image compared with the original function, it is possible to better focus on the edge information of the image and obtain a reconstructed scene with richer details. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is the overall framework diagram of the embodiment in the present invention;

[0024] Figure 2 is the pruning flowchart of the fractional order definition in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] The following further elaborates on the specific implementation of the present invention in conjunction with the drawings and embodiments. It should be understood that this embodiment is only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this patent application.

[0026] The present invention relates to a three-dimensional reconstruction method based on the fusion of fractional gradient optimization and three-dimensional Gaussian sputtering. Based on the three-dimensional Gaussian sputtering algorithm, the present invention combines fractional calculus and the gradient descent algorithm, and proposes a new fractional optimizer to guide the optimization of each attribute in the Gaussian primitive; at the same time, aiming at the possible instability in fractional optimization, a learning rate adjustment algorithm based on the exponential moving weighted average strategy is proposed to stabilize the training process to obtain better training results (as Figure 1 shown), including the following steps:

[0027] 1) Generate sparse point clouds through structure from motion technology and input picture data; adopt the structure from motion technology to restore the sparse point clouds in the three-dimensional space by combining the scene pictures from multiple perspectives and the corresponding camera poses.

[0028] 2) Generate Gaussian primitives centered on the point clouds in space; after obtaining the sparse point clouds, initialize the sparse point clouds in space as Gaussian primitives with color, opacity, shape information, and spherical harmonic function representing color, and perform coordinate transformation on the Gaussian primitives in space according to the projection matrix to obtain the Gaussian distribution in the pixel coordinate system.

[0029] 3) Rasterize and render the scene represented by Gaussian primitives; map the Gaussian primitives in space to pixels in the pixel coordinate system through the rasterization pipeline in 3D Gaussian sputtering after spatial coordinate transformation to obtain the rendered image.

[0030] 4) Calculate the joint optimization function for the rendered image and the real image: Calculate the loss between the rendered view and the real scene view by using the above joint loss function. In addition to calculating the photometric loss between the rendered image and the real scene image, a loss function based on the sobel operator for calculating the edge loss between the rendered image and the real image is designed by combining the application of fractional calculus in edge detection, so as to guide the fractional gradient optimizer to optimize the edge loss of the image, defined as:

[0031] where N represents the number of edge pixels, render_edge i and gt_edge i represent the i-th pixel values of the rendered image and the real image after being detected by the Sobel operator respectively.

[0032] In addition, a mean absolute error loss function based on fractional order correction is designed to constrain the fractional order optimizer, defined as:

[0033]

[0034] Combine the fractional order optimizer designed in step 6) to optimize the attributes contained in the Gaussian primitives.

[0035] In this embodiment, the loss function of the scene three-dimensional reconstruction model can be calculated in the following way: Use the scene images and corresponding camera poses under multiple perspectives to perform pseudo-view synthesis to obtain the rendered image, and construct the photometric consistency loss according to the rendered image and the real scene image; construct the edge loss according to the edge information of the rendered image and the real scene image; construct the mean absolute error loss after fractional order correction according to the rendered image and the real scene image; adopt the photometric consistency loss, the edge loss, and the mean absolute error loss based on fractional order correction to jointly construct the loss function to calculate the loss of the scene three-dimensional reconstruction model. The joint optimization loss function can be expressed as:

[0036]

[0037] where: λ1, λ2, λ3 represent the weighting coefficients, L1 represents the mean absolute error loss, L D-SSIM represents the structural similarity error; L e represents the edge loss function based on the sobel operator; Denote the mean absolute error loss function after fractional-order correction; the edge loss in the formula provides explicit supervision for fractional-order optimization by combining the edge information of the rendered view with the fractional-order corrected mean absolute error, enabling fractional-order optimization to guide the model closer to the optimal solution compared to integer-order optimization.

[0038] 5) Prune the fractional calculus definition to design a fractional-order optimizer; the pruning process is as Figure 2 shown. The three classical fractional-order definitions (i.e., Grünwald-Letnikov definition, Riemann–Liouville definition, and Caputo definition) are compared. By analyzing different fractional-order definitions and their application scenarios, the fractional-order definition is selected. Finally, the Caputo fractional-order definition is chosen to reduce the gradient accumulation in the fractional-order gradient optimization process by eliminating non-local features. To reduce its computational complexity, the calculation form of the Caputo fractional-order definition is simplified, and the stability during gradient calculation is ensured by setting an extremely small number. Design an optimizer based on fractional calculus to achieve fractional-order optimization of the various attributes of the Gaussian primitive, thereby providing stronger gradient guidance for the Gaussian primitive and enabling it to be closer to the target position compared to the integer-order optimizer. After simplification, the fractional-order derivative calculation can be expressed as:

[0039]

[0040] In the above formula, α represents the order of the fractional order. By using the modified fractional calculus based on the Caputo definition instead of the integer-order calculus, the biased first-order moment estimate and second-order moment estimate for calculating the gradient are completed, and the initial combination of fractional calculus and the Adaptive Moment Estimation (Adam) algorithm is achieved.

[0041] 6) Use the fractional-order optimizer to optimize the parameters of the Gaussian primitive: On the one hand, simplify the non-local features in the optimization process by pruning the form of the fractional calculus to reduce the computational complexity and at the same time reduce the impact of error information on gradient optimization; on the other hand, to ensure that the non-locality of the fractional order is reflected in the algorithm, adjust the learning rate dynamically by moving the exponential moving average strategy and calculating the average change amplitude of the first-order moment and second-order moment of the number of iterations within the memory depth. Through the above two aspects, ensure that the optimization of the various attributes of the Gaussian primitive using the fractional-order optimizer shows a downward trend at the macroscopic level.

[0042] 7) Complete the pruning of the Gaussian primitive according to the adaptive pruning strategy; the optimization efficiency of the Gaussian primitive using the fractional-order optimizer is worse than that of the integer-order optimizer.

[0043] The present invention combines a Gaussian filter based on a sliding window with adaptive density control to prune and duplicate Gaussian primitives, reducing redundant representations in the scene while enhancing rendering speed and ensuring density geometric details.

Claims

1. A 3D reconstruction method based on fractional gradient optimization and 3D Gaussian sputtering fusion, the steps of which are as follows: 1) Generate sparse point cloud through motion structure recovery technology and input image data; 2) Generate Gaussian primitives centered on the point cloud in space; 3) Rasterize and render the scene represented by the Gaussian primitive; 4) Perform joint optimization function calculation on the rendered image and the real image: in: λ1, λ2, and λ3 represent weighting coefficients, L1 represents the mean absolute error loss between the rendered image and the real image, and L D-SSIM represents the structural similarity between the rendered image and the real image; L e Represents the edge loss function based on the sobel operator; Represents the mean absolute error loss function based on fractional order correction; 5) Prune the definition of fractional calculus to design a fractional optimizer; by analyzing the applicable scenarios of several mainstream fractional-order definitions, prune the Caputo fractional-order calculus definition, eliminate its non-locality, and use it to optimize the fractional gradient of the information contained in the Gaussian primitive as follows: Among them: α represents the order of the fractional order, δ represents a very small number used to prevent division by zero; 6) A fractional order optimizer is used to optimize the Gaussian primitive parameters; the definition of the Gaussian primitive is expressed as: {g} k =(μ,∑,σ,s)}; Where: μ represents the center coordinate of the Gaussian primitive, ∑ represents the shape confidence of the Gaussian primitive, σ represents opacity and S represents color information; By combining the pruned fractional-order definition with the Adam optimizer, a fractional-order optimizer is designed to achieve fractional-order optimization of the attributes contained in the above Gaussian primitives; combining the non-local characteristics of fractional calculus, an adaptive learning rate adjustment strategy is designed, and the non-local characteristics are realized through the exponential moving weighted average strategy, which is expressed as: EMA t =α×x t +(1-a)×EMA t-1 ; Among them, α is used as the weight decay coefficient to calculate the memory depth; the learning rate is dynamically adjusted according to the mean and variance changes of historical data; 7) According to the adaptive pruning strategy, the pruning of the Gaussian ellipsoid is completed.

Images