Gaussian spatter inverse problem optimization method and device for body-aware three-dimensional environment reconstruction

Abstract

The application discloses a kind of Gauss splashing three-dimensional reconstruction method and device based on inverse problem optimization.The method first obtains the multi-view image of target scene, completes camera calibration, generates sparse point cloud and initializes three-dimensional Gaussian ellipsoid set;High-frequency sub-band is extracted using multi-layer frequency domain detail decomposition, and pixel-level contribution weight mapping is constructed;Three-dimensional Gauss splashing reconstruction is modeled as inverse problem optimization, the mapping relationship between ellipsoid parameters and two-dimensional observation is established by differentiable rendering, and frequency prior constraint is introduced;In the rendering alpha synthesis stage, add spatial level contribution modulation, dynamically adjust pixel weight according to Gaussian ellipsoid depth;Adaptive constraint mixed loss function in frequency domain is used to iteratively optimize ellipsoid parameters, and the three-dimensional reconstruction result is output.The application can improve the high-frequency texture reconstruction accuracy, reduce artifacts and geometric distortion, improve the reconstruction stability and rendering quality, and is suitable for body intelligent perception, three-dimensional reconstruction and new view synthesis.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, computer graphics and artificial intelligence, and in particular to a Gaussian splashing inverse problem optimization method and apparatus for embodied intelligent 3D environment reconstruction. Background Technology

[0002] In recent years, embodied intelligent systems have placed extremely high demands on high-fidelity, real-time 3D reconstruction of complex and unknown environments during their interaction with the physical world. 3D Gaussian Splatting (3DGS), as an emerging scene representation method, has shown great potential in the field of 3D scene reconstruction due to its efficient rendering speed and realistic visual effects. Reconstructing a Gaussian representation of a 3D scene from multi-view 2D images can essentially be viewed as a typical inverse problem solution process.

[0003] However, in complex real-world embodied intelligence applications, this inverse problem often exhibits a high degree of ill-posedness. Influenced by complex factors such as drastic changes in viewing perspective, dense high-frequency structures in the scene (e.g., fine textures, complex edges), and depth discontinuities caused by object occlusion, traditional 3DGS optimization processes are prone to getting stuck in local optima or experiencing ill-conditioned convergence. This directly leads to problems such as blurred details, rendering artifacts, and a significant decrease in geometric consistency across multiple viewpoints in the reconstructed 3D environment, severely limiting the ability of embodied intelligence systems to accurately perceive the environment and engage in intelligent interaction.

[0004] Currently, mainstream improvement methods primarily attempt to alleviate the aforementioned problems by optimizing basic parameters of 3DGS or introducing conventional regularization terms. However, these existing techniques fail to deeply explore and fully integrate the frequency distribution features of images with spatial depth perception mechanisms. When faced with environments containing complex high-frequency details and abrupt depth changes, existing models exhibit weak perception of high-frequency structures, making it difficult to achieve a good balance between noise suppression and scene structure preservation. Therefore, existing methods still suffer from significant deficiencies in reconstruction stability and generalization ability in complex scenes. Summary of the Invention

[0005] This invention provides a Gaussian splash 3D reconstruction method and system based on inverse problem optimization. The 3DGS scene reconstruction process is modeled as a nonlinear inverse problem driven by differentiable rendering. By introducing multi-scale frequency priors and depth-aware constraints, the integrated optimization of high-frequency structures and spatial geometry in complex scenes is achieved, thereby improving the quality of 3D scene reconstruction and rendering stability.

[0006] This invention provides a Gaussian splash 3D reconstruction method based on inverse problem optimization, comprising: Step 1: Use a camera to take surround shots of the target scene, with sufficient overlap between adjacent viewpoints, to obtain multi-view images of the target scene. Use the structure-of-motion method to calibrate the camera pose and generate a sparse point cloud of the scene. Initialize a three-dimensional Gaussian ellipsoid set based on the sparse point cloud. Step 2: Perform multi-level frequency domain detail decomposition on the multi-view image, extract the horizontal, vertical and diagonal high-frequency sub-bands and calculate the high-frequency energy, and construct a pixel-level pixel contribution weight mapping. Step 3: The 3D Gaussian splash scene is reconstructed into an inverse problem optimization model. The mapping relationship between the 3D Gaussian ellipsoid parameters and the 2D observation image is established through a differentiable rendering operator. The pixel contribution weight mapping is introduced as a frequency prior constraint into the Gaussian ellipsoid parameter optimization process. Step 4: In the α-compositing stage of rendering, a spatial hierarchy contribution modulation mechanism is introduced to dynamically modulate the contribution weight of each Gaussian ellipsoid to the pixel color based on the depth value between the Gaussian ellipsoid and the camera. Step 5: Construct a hybrid loss function with frequency domain structure adaptive constraints, and iteratively update the position parameters, covariance parameters, color parameters, and transparency parameters of the Gaussian ellipsoid to finally output the 3D reconstruction result of the target scene.

[0007] Furthermore, step 2 specifically includes: Step 21: Perform multi-level frequency domain detail decomposition on the input image. Multi-scale decomposition yields a set of coefficients. ,in, For the first Low-frequency subband coefficients after layer decomposition , and They represent the first High-frequency subband coefficients in the horizontal, vertical, and diagonal directions of layer decomposition; Step 22, retain in each level of decomposition , and , and will Setting to zero yields The coefficient set is obtained. ; Step 23, perform inverse discrete wavelet transform on... Reconstruction is performed to obtain the high-frequency response map of the input image. ; Step 24: Based on the high-frequency subband coefficients at each scale and in each direction, calculate the position of each pixel using equation (1). High-frequency energy at the location : (1) In the formula, , , They represent the results obtained by discrete wavelet transform decomposition. High-frequency sub-bands in the horizontal, vertical, and diagonal directions at that location; Step 25, for high-frequency energy Normalization is performed to map the energy values ​​to the [0,1] interval, resulting in pixel-level pixel contribution weight mapping.

[0008] Furthermore, the inverse problem optimization model in step 3 is expressed as equation (2): (2) In the formula, These are the optimal scene parameters obtained after optimization. The set of 3D scene parameters to be optimized. For data fidelity items, The regularization coefficient is . For regularization terms, To render the image.

[0009] Furthermore, the spatial hierarchy contribution modulation mechanism in step 4 is expressed as equation (3): (3) In the formula, Contribute weights to the pixels of the i-th Gaussian ellipsoid. Let be the center depth of the i-th Gaussian ellipsoid, and j be the index variable used to iterate through all Gaussian ellipsoids in the scene. This is a parameter used to control the decay rate of the depth with respect to the weights.

[0010] Furthermore, the hybrid loss function in step 5 is expressed as equation (4): (4) in, To reconstruct errors, For structural consistency terms, For the frequency constraint term, it is described by Equation (5), where α is the weight of the reconstruction error and structural consistency term, and α is used to balance pixel reconstruction accuracy and structural similarity. β is the frequency constraint weight, which is used to control the strength of the frequency constraint. (5) In the formula, Indicates pixel frequency response. This is the frequency enhancement factor. N Total number of pixels p For pixel index, and Ip The rendered image and the real image are respectively in pixels. pThe color value at that location, This represents the square of the L2 norm.

[0011] Furthermore, in step 5, the gradient backpropagation algorithm described by equation (6) is used to iteratively update the Gaussian ellipsoid parameters: (6) In the formula, θ represents the Gaussian ellipsoid parameters, including position parameters, covariance parameters, color parameters, and opacity parameters. The learning rate controls the step size for each parameter update. This is the current iteration step number. Let θ be the gradient of the total loss function with respect to the parameter θ.

[0012] The present invention also provides a Gaussian splash 3D reconstruction device based on inverse problem optimization, comprising: The image acquisition and initialization unit is used to acquire multi-view images of the target scene, use the structure of motion reconstruction method to perform camera pose calibration and generate sparse point cloud of the scene, and initialize a three-dimensional Gaussian ellipsoid set based on the sparse point cloud. The frequency analysis unit is used to perform multi-level frequency domain detail decomposition on multi-view images, extract horizontal, vertical, and diagonal high-frequency sub-bands and calculate high-frequency energy, and construct pixel-level pixel contribution weight mapping. The inverse problem optimization unit is used to reconstruct the three-dimensional Gaussian splash scene into an inverse problem optimization model. It establishes the mapping relationship between the three-dimensional Gaussian ellipsoid parameters and the two-dimensional observation image through a differentiable rendering operator, and introduces the pixel contribution weight mapping as a frequency prior constraint into the Gaussian ellipsoid parameter optimization process. The depth-aware rendering unit is used to introduce a spatial hierarchy contribution modulation mechanism in the alpha compositing stage of rendering, which dynamically modulates the contribution weight of each Gaussian ellipsoid to the pixel color based on the depth value between the Gaussian ellipsoid and the camera. The scene output unit is used to construct a hybrid loss function that integrates the pixel contribution weights, iteratively update the position parameters, covariance parameters, color parameters, and transparency parameters of the Gaussian ellipsoid, and output the 3D reconstruction result of the target scene.

[0013] Furthermore, the frequency analysis unit is specifically configured to perform frequency analysis on the input image using a multi-level frequency domain detail decomposition method. Multi-scale decomposition yields a set of coefficients. ,in, For the first Low-frequency subband coefficients after layer decomposition , and They represent the first High-frequency subband coefficients in the horizontal, vertical, and diagonal directions are decomposed into layers; preservation is achieved in each layer decomposition. , and , and will Setting to zero yields The coefficient set is obtained. ; By using inverse discrete wavelet transform Reconstruction is performed to obtain the high-frequency response map of the input image. Based on the high-frequency subband coefficients at each scale and in each direction, the position of each pixel is calculated using equation (1). High-frequency energy at the location For high-frequency energy Normalization is performed to map the energy values ​​to the [0,1] interval, resulting in pixel-level pixel contribution weight mapping; (1) In the formula, , , They represent the results obtained by discrete wavelet transform decomposition. High-frequency subbands in the horizontal, vertical and diagonal directions.

[0014] Furthermore, the depth-aware rendering unit adopts a spatial hierarchy contribution modulation mechanism, expressed as Equation (3): (3) In the formula, For the first i The pixel contribution weight of each Gaussian ellipsoid For the first i The central depth of a Gaussian ellipsoid j An index variable used to iterate through all Gaussian ellipsoids in the scene. This is a parameter used to control the decay rate of the depth with respect to the weights.

[0015] Furthermore, the inverse problem optimization unit adopts an inverse problem optimization model: (2) In the formula, These are the optimal scene parameters obtained after optimization. The set of 3D scene parameters to be optimized. For data fidelity items, The regularization coefficient is . For regularization terms, To render the image; The hybrid loss function in the scene output unit is expressed as equation (4): (4) In the formula, To reconstruct errors, For structural consistency terms, The frequency constraint term is described in equation (5). The weights for the reconstruction error and structural consistency term are used to balance pixel reconstruction accuracy and structural similarity. These are the frequency constraint weights, used to control the strength of the frequency constraints. (5) In the formula, Indicates pixel frequency response. This is the frequency enhancement factor. N Total number of pixels p For pixel index, and I The rendered image and the real image are respectively in pixels. p The color value at that location, This represents the square of the L2 norm.

[0016] Compared with existing technologies, this invention provides a unified description of the 3DGS reconstruction process from the perspective of inverse problem optimization, improving theoretical integrity and physical interpretability; it utilizes multi-scale wavelet frequency priors to enhance the modeling capability of high-frequency regions and improve the texture detail recovery effect; it introduces a depth-aware rendering mechanism to reduce occlusion artifacts in regions with abrupt depth changes; it improves structural stability and rendering quality in complex scenes through frequency-weighted hybrid loss; and it can achieve more stable and high-fidelity 3D scene reconstruction in complex indoor and outdoor scenes. Attached Figure Description

[0017] Figure 1 This is a flowchart of the Gaussian splashing inverse problem optimization method for embodied intelligent 3D environment reconstruction according to an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described in detail below with reference to the embodiments of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] This invention provides a Gaussian splash 3D reconstruction method based on inverse problem optimization. Addressing the issues of instability in high-frequency structure recovery and insufficient geometric consistency across multiple views in 3DGS, this method, from the perspective of inverse problem optimization, unifies the optimization of the 3DGS model into an inverse problem solution process under finite observation conditions. By introducing frequency priors, depth-aware constraints, and a frequency-weighted loss function, it effectively reduces the instability of solutions in the high-frequency subspace, thereby improving the accuracy and visual quality of 3D reconstruction in complex scenes.

[0020] The Gaussian splash 3D reconstruction method based on inverse problem optimization in this embodiment includes: Step 1: Use a camera to take surround shots of the target scene, with sufficient overlap between adjacent viewpoints to meet the feature matching conditions required for multi-view geometric reconstruction, in order to obtain multi-view images of the target scene.

[0021] The structure-of-motion method is used for feature extraction, feature matching, and camera pose estimation to complete camera pose calibration and generate sparse point clouds of the scene.

[0022] The sparse point cloud is used as the initial location of the Gaussian ellipsoid center. A 3D Gaussian ellipsoid set is initialized based on the sparse point cloud, thus completing the initialization of the 3DGS model. Specifically, the sparse point cloud is used to initialize the 3D Gaussian ellipsoid set: Where: K is the number of Gaussian ellipsoids; Let the k-th Gaussian ellipsoid be represented as: ,in: Indicates the location of the center of Gaussian; Represent the covariance matrix; Indicates color parameters; This represents the opacity parameter.

[0023] The three-dimensional Gaussian distribution function is expressed as equation (6): (6) In the formula, superscript The transpose operation represents the spatial offset of a vector, which is originally a column vector. Converting to a row vector to meet the requirement of aligning with the multiplication dimension of the inverse of the covariance matrix, yields a scalar value. This operation itself does not change the geometric meaning of the vector. The superscript {-1} denotes the covariance matrix. The inverse matrix of is physically significant because it performs directional normalization on spatial offsets.

[0024] Furthermore, to ensure the positive definiteness of the covariance matrix, the covariance matrix is ​​parameterized as follows: ,in: It is a rotation matrix; is the scale matrix.

[0025] Step 2 addresses the lack of scene frequency structure guidance in controlling the number and scale of Gaussian ellipsoids in traditional 3DGS methods. This invention introduces a multi-level frequency domain decomposition method to perform multi-level frequency domain decomposition on the input multi-view images, extracting high-frequency structural information such as horizontal, vertical, and diagonal high-frequency sub-bands as priors to guide the density distribution and scale configuration of the Gaussian ellipsoids in 3D space. High-frequency energy is calculated, and a pixel-level pixel contribution weight mapping is constructed. Here, the high-frequency sub-band is the region where the high-frequency energy value exceeds a preset threshold. The high-frequency energy is calculated from the sum of the squares of the coefficients of the high-frequency sub-bands in the horizontal, vertical, and diagonal directions at each scale. The threshold is typically set using a percentage method, directly retaining the top p% of pixels by energy, usually between 10% and 30%.

[0026] In one embodiment, step 2 specifically includes: Step 21: Apply multi-level frequency domain detail decomposition methods, such as two-dimensional discrete wavelet transform, to the input image. conduct Multi-scale decomposition yields a set of coefficients. This is to capture local structural information of the image, such as edges and textures, at that scale and in that direction. For the first Low-frequency subband coefficients after layer decomposition , and They represent the first High-frequency subband coefficients in the horizontal, vertical and diagonal directions of layer decomposition.

[0027] Step 22: To highlight high-frequency structural information in the image, retain [the relevant information] in each decomposition layer. , and , and will Setting to zero yields The coefficient set is obtained. .

[0028] Step 23, perform inverse discrete wavelet transform on... Reconstruction is performed to obtain the high-frequency response map of the input image. This high-frequency response map can effectively highlight the edges and textured areas in the image, while suppressing the influence of large-scale smooth structures.

[0029] High frequency response diagram Threshold segmentation is performed to locate high-frequency salient regions. : ,in, The threshold is adaptively determined based on the statistical characteristics of the high-frequency response. (Set) The pixels in the image correspond to regions with significant structural changes, such as object edges, densely textured areas, and fine geometric structures. The center point of each 3D Gaussian ellipsoid is projected onto the 2D plane of each view using a view projection matrix, obtaining the frequency characteristics corresponding to the projection location. Based on the frequency attributes of the region where the Gaussian ellipsoid is located, differentiated structural adjustment operations are performed. For regions located in high-frequency significant areas... The Gaussian ellipsoid within is split to increase the local spatial sampling density. Assume the Gaussian ellipsoid to be split has a mean... Covariance matrix Opacity And color c, after splitting, a new Gaussian ellipsoid is generated, and set as follows: , , , , ,in, Represents the covariance matrix The main axis direction, This is a scaling function used to adjust the size of the split Gaussian ellipsoid.

[0030] For Gaussian ellipsoids located in the low-frequency smooth region, pruning is performed first to remove redundant Gaussian units and reduce the overall complexity of the model. At the same time, the scale of the Gaussian ellipsoids in the low-frequency region is appropriately enlarged to maintain the continuous representation effect of the smooth region; the scale of the Gaussian ellipsoids in the high-frequency region is appropriately reduced to enhance the ability to express detailed structures.

[0031] By using the above-mentioned frequency response-based structural control method, under the premise of limited computing resources, more Gaussian ellipsoids can be reasonably allocated to regions with higher scene structural complexity, thereby improving the overall representation efficiency of the model for 3D scenes and the reconstruction accuracy of high-frequency details.

[0032] Step 24: Based on the high-frequency subband coefficients at each scale and in each direction, calculate the position of each pixel using equation (1). High-frequency energy at the location : (1) in, , , They represent the results obtained by discrete wavelet transform decomposition. High-frequency subbands in the horizontal, vertical and diagonal directions.

[0033] This high-frequency energy It has a clear physical meaning: in the edges and textured regions of an image, pixel values ​​change rapidly in space, and the corresponding wavelet coefficients have larger amplitudes, therefore... The value is relatively large; in the smooth area, the pixel value changes slowly and the high-frequency coefficient approaches zero. Therefore, the value approaches zero. This correspondence makes it possible to intuitively reflect the structural complexity within the neighborhood of each pixel in the image, which can be used as a natural measure to evaluate the importance of local details.

[0034] Step 25, normalize the high-frequency energy map and map the energy values to the unified [0, 1] interval to obtain the pixel-level pixel contribution weight map.

[0035] Step 3, construct the three-dimensional Gaussian splash scene reconstruction into an inverse problem optimization model, and establish the mapping relationship between the three-dimensional Gaussian ellipsoid parameters and the two-dimensional observed image through the original differentiable rendering operator in the 3DGS method, so as to realize the joint optimization of the scene structure and the rendering result.

[0036] First, project the three-dimensional Gaussian ellipsoid onto the two-dimensional image plane through the camera imaging model. For the i-th Gaussian ellipsoid, the projection form of its center point is expressed as: , where: K is the camera intrinsic matrix; (R, t) are the camera rotation matrix and translation vector respectively; is the coordinate of the three-dimensional Gaussian center in the world coordinate system; is the two-dimensional projection coordinate.

[0037] Furthermore, to maintain the spatial shape and orientation information of the Gaussian ellipsoid in the two-dimensional image plane, perform a two-dimensional projection transformation on the three-dimensional covariance matrix. The two-dimensional covariance matrix is expressed as:

[0038] where: W is the transformation matrix from the world coordinate to the camera coordinate; J is the Jacobian matrix of the projection function, is the covariance matrix of the transformed two-dimensional Gaussian ellipse.

[0039] After obtaining the two-dimensional Gaussian ellipse, all Gaussian ellipsoids perform transparency accumulation according to the depth order, and the final pixel color is expressed as Equation (7): (7) In the formula, represents the own color of the i-th Gaussian ellipsoid, represents the opacity of the i-th Gaussian ellipsoid at the pixel position x, represents the cumulative transmittance of the i-th Gaussian ellipsoid, that is, the product of the probabilities that all the previous closer Gaussian kernels j < i do not block this point.

[0040] Furthermore, represent the generation process from the three-dimensional scene to the two-dimensional image as: , where: For observation images; Here, is the differentiable rendering operator; n is the noise term. Based on the above forward imaging model, the 3D scene reconstruction process is further modeled as an inverse optimization problem. In this way, the spatial projection of the 3D Gaussian ellipsoid, differentiable rendering, and inverse optimization process are unified into the same mathematical framework, and the pixel contribution weight mapping is introduced as a frequency prior constraint into the Gaussian ellipsoid parameter optimization process.

[0041] In one embodiment, the inverse problem optimization model in step 3 is expressed as equation (2): (2) in, These are the optimal scene parameters obtained after optimization. The set of 3D scene parameters to be optimized. The data fidelity term (loss function) is used to measure the rendered image. The difference between the observed image Y and the actual image Y. is the regularization coefficient, a non-negative scalar weight, used to balance the relative importance of data fidelity terms and regularization terms. As a regularization term, prior constraints are imposed to guide the optimization process, alleviate ill-qualities, and avoid overfitting. To render the image.

[0042] It should be noted that this invention uses a regularized inverse problem optimization model for solution, but it can also be implemented equivalently using constraint optimization or probabilistic inference.

[0043] Step 4 introduces a spatial hierarchy contribution modulation mechanism in the α-compositing stage of rendering, dynamically modulating the contribution weight of each Gaussian ellipsoid to pixel color based on the depth value between the Gaussian ellipsoid and the camera.

[0044] This step is for traditional 3DGS... The α-blending model relies solely on depth sorting without explicitly utilizing depth values ​​for contribution weight modulation. This invention introduces a spatial hierarchy contribution modulation mechanism during the rendering stage. By incorporating the absolute depth information of the Gaussian ellipsoid into the contribution weight calculation process, it enhances the model's ability to distinguish the spatial geometric layers of the scene. Specifically, the operation is as follows: In the forward rendering stage of 3DGS, for the first... A three-dimensional Gaussian ellipsoid is used to calculate pixel contribution weights through a spatial hierarchy contribution modulation mechanism described by equation (3): (3) In the formula, Contribute weights to the pixels of the i-th Gaussian ellipsoid. is the central depth of the i-th Gaussian ellipsoid, and j is an index variable used to traverse all Gaussian ellipsoids in the scene. is a parameter used to control the attenuation rate of the weight with depth. When controlling, when it increases, the weight decreases faster with depth; when it decreases, the change of the weight is more gentle.

[0045] This weight function ensures that the weight value decreases monotonically as the distance between the Gaussian ellipsoid and the camera increases: Gaussian ellipsoids closer to the camera obtain weights approaching 1, and Gaussian ellipsoids farther away obtain weights approaching 0. This design is based on a fundamental visual perception principle: in most natural scenes, the observed color of a pixel is mainly determined by its nearest visible surface, and farther surfaces are either completely occluded with a contribution of zero or have only a weak influence through indirect lighting and their contributions should be reasonably suppressed.

[0046] After introducing the pixel contribution weight, the rendering equation of 3DGS is modified to a depth-weighted form. For a certain pixel, its color is calculated by the expression in Equation (8): (8) In the formula, represents the self-color of the i -th Gaussian ellipsoid, represents the opacity of the i -th Gaussian ellipsoid at the pixel position x, represents the cumulative transmittance of the i-th Gaussian ellipsoid, that is, the product of the probabilities that all previous closer Gaussian kernels j < i do not occlude this point. It is further equivalently expressed as Equation (9): (9) In the formula, is the cumulative transmittance of the first Gaussian ellipsoids to the -th Gaussian ellipsoid.

[0047] It can be seen from Equation (8) and Equation (9) that the pixel contribution weight The weighting directly affects the contribution of each Gaussian ellipsoid to the final pixel color, ensuring a reasonable distribution of influence across Gaussian ellipsoids at different depth levels during optimization. During backpropagation, the closer Gaussian ellipsoids, with their higher weights, exhibit larger gradients corresponding to their color parameters, leading the optimization process to prioritize fitting foreground structures and primary visible surfaces. The gradients of the farther Gaussian ellipsoids are moderately suppressed, effectively reducing their interference with the colors of complex occluded areas. It's important to emphasize that this pixel contribution weighting only affects the color calculation and does not alter the cumulative opacity or the rendering process determined by depth ranking. This design ensures the physical validity of the depth blending mechanism itself, while the weights primarily influence the optimization process by adjusting the gradient magnitude, which is particularly crucial for handling thin structural edges and regions with discontinuous depth.

[0048] Of course, pixel contribution weights can be implemented using exponential functions, reciprocal functions, sigmoid functions, or other monotonically decreasing functions.

[0049] Step 5: Construct a hybrid loss function with adaptive constraints in the frequency domain structure. Use a stochastic gradient descent algorithm or Adam optimization algorithm based on gradient backpropagation to iteratively update the position parameters, covariance parameters, color parameters, and transparency parameters of the Gaussian ellipsoid, and finally output the 3D reconstruction result of the target scene.

[0050] This step addresses the problem that traditional pixel-domain loss functions assign the same weight to all pixel positions, causing gradient signals in high-frequency detail regions to be overwhelmed by large areas of low-frequency regions. This invention utilizes the energy spectrum of wavelet transform to construct pixel-level pixel contribution weights and designs a frequency-weighted hybrid loss function as the optimization objective. Specifically, the operations are as follows: For the real image under the current training perspective to conduct L Layered discrete wavelet decomposition yields high-frequency detail subband coefficients at various scales.

[0051] By comprehensively characterizing high-frequency information at different scales and orientations in the image, the position of each pixel... High-frequency energy at the location The high-frequency energy map is obtained through equation (1), and the energy values ​​are normalized to a unified range. The pixel-level pixel contribution weights are then obtained through equation (3). After normalization, the weights Mapped to the [0,1] interval: the region with the highest high-frequency energy has a weight close to 1, and the region with the lowest high-frequency energy has a weight close to 0. The pixel-level pixel contribution weights constructed above are then... Introducing a loss function, a hybrid loss function for frequency modulation is constructed, expressed as equation (4): (4) In the formula, To reconstruct errors, For structural consistency terms, The frequency constraint term is described in equation (5), where α is the weight of the reconstruction error and structural consistency term, and β is the frequency constraint weight. α is used to balance pixel reconstruction accuracy and structural similarity, and its value ranges from 0 to 1. β is used to control the strength of the frequency constraint. For example, if α = 0.2 and β = 0.1, then 80% of the focus is on pixel error, 20% on structural similarity, and the high-frequency constraint accounts for 10% of the regularization strength.

[0052] (5) In the formula, Indicates pixel frequency response. This is the frequency enhancement factor. N Total number of pixels p For pixel index, and Ip The rendered image and the real image are respectively in pixels. p The color value at that location, This represents the square of the L2 norm.

[0053] It can be clearly seen from the formula that the pixel contribution weight Error terms applied to each pixel: in high-frequency textures and edge regions, When the value approaches 1, the error of the corresponding pixel is amplified to twice the original weight, thus obtaining a stronger gradient signal; in smooth regions, When the value approaches 0, the error remains at its original magnitude, and the optimization process is not excessively affected. Through this differentiated weighting mechanism, the model pays more attention to edge and texture regions during training, thereby achieving a better balance between overall reconstruction quality and local detail fidelity.

[0054] Through the above training process, the resulting 3DGS model significantly improves the ability to express high-frequency details and multi-view geometric consistency while maintaining global structural stability. By applying the trained model to new viewpoint synthesis tasks, high-fidelity scene images from any new viewpoint can be rendered.

[0055] In one embodiment, step 5 uses the gradient backpropagation algorithm described by equation (6) to iteratively update the Gaussian ellipsoid parameters: (6) In the formula, θ represents the Gaussian ellipsoid parameters, including position parameters, covariance parameters, color parameters, and opacity parameters; η is the learning rate, used to control the step size of each parameter update; and t is the current iteration step (i.e., the number of optimizations). The gradient of the total loss function with respect to the parameter θ is calculated by backpropagation using the chain rule through a differentiable rendering process. and These represent the parameter values ​​after iterations at step t and t+1, respectively. Through continuous iterative optimization, the model gradually converges to the optimal parameters, ultimately outputting a new perspective reconstruction result of the target 3D scene. The final 3D scene reconstruction result and the new perspective rendered image are output after iterative optimization.

[0056] This invention also provides a Gaussian splash 3D reconstruction device based on inverse problem optimization, comprising: The image acquisition and initialization unit is used to acquire multi-view images of the target scene, use the structure of motion reconstruction method to perform camera pose calibration and generate sparse point cloud of the scene, and initialize a three-dimensional Gaussian ellipsoid set based on the sparse point cloud. The frequency analysis unit is used to perform multi-level frequency domain detail decomposition on multi-view images, extract horizontal, vertical, and diagonal high-frequency sub-bands and calculate high-frequency energy, and construct pixel-level pixel contribution weight mapping. The inverse problem optimization unit is used to reconstruct the three-dimensional Gaussian splash scene into an inverse problem optimization model. It establishes the mapping relationship between the three-dimensional Gaussian ellipsoid parameters and the two-dimensional observation image through a differentiable rendering operator, and introduces the pixel contribution weight mapping as a frequency prior constraint into the Gaussian ellipsoid parameter optimization process. The depth-aware rendering unit is used to introduce a spatial hierarchy contribution modulation mechanism in the alpha compositing stage of rendering, which dynamically modulates the contribution weight of each Gaussian ellipsoid to the pixel color based on the depth value between the Gaussian ellipsoid and the camera. The scene output unit is used to construct a hybrid loss function that integrates the pixel contribution weights, iteratively update the position parameters, covariance parameters, color parameters, and transparency parameters of the Gaussian ellipsoid, and output the 3D reconstruction result of the target scene.

[0057] In one embodiment, the frequency analysis unit is specifically configured to: perform analysis on the input image. Multi-scale decomposition is used to obtain low-frequency subband coefficients and multi-directional high-frequency subband coefficients; high-frequency subband coefficients are retained and low-frequency subband coefficients are set to zero; high-frequency response map is obtained by inverse discrete wavelet transform; pixel-level high-frequency energy map is calculated based on high-frequency subband coefficients; pixel-level pixel contribution weight mapping in the [0,1] interval is obtained by normalizing the high-frequency energy map.

[0058] In one embodiment, the depth-aware rendering unit adopts a spatial hierarchy contribution modulation mechanism, as expressed in equation (3).

[0059] In one embodiment, the inverse problem optimization unit adopts the inverse problem optimization model, as shown in Equation (2), and the hybrid loss function in the scene output unit is expressed as Equation (4).

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Gaussian splash 3D reconstruction method based on inverse problem optimization, characterized in that, include: Step 1: Use a camera to take surround shots of the target scene, with sufficient overlap between adjacent viewpoints, to obtain multi-view images of the target scene. Use the structure-of-motion method to calibrate the camera pose and generate a sparse point cloud of the scene. Initialize a three-dimensional Gaussian ellipsoid set based on the sparse point cloud. Step 2: Perform multi-level frequency domain detail decomposition on the multi-view image, extract the horizontal, vertical and diagonal high-frequency sub-bands and calculate the high-frequency energy, and construct a pixel-level pixel contribution weight mapping. Step 3: The 3D Gaussian splash scene is reconstructed into an inverse problem optimization model. The mapping relationship between the 3D Gaussian ellipsoid parameters and the 2D observation image is established through a differentiable rendering operator. The pixel contribution weight mapping is introduced as a frequency prior constraint into the Gaussian ellipsoid parameter optimization process. Step 4: In the α-compositing stage of rendering, a spatial hierarchy contribution modulation mechanism is introduced to dynamically modulate the contribution weight of each Gaussian ellipsoid to the pixel color based on the depth value between the Gaussian ellipsoid and the camera. Step 5: Construct a hybrid loss function with frequency domain structure adaptive constraints, and iteratively update the position parameters, covariance parameters, color parameters, and transparency parameters of the Gaussian ellipsoid to finally output the 3D reconstruction result of the target scene.

2. The Gaussian splash 3D reconstruction method based on inverse problem optimization as described in claim 1, characterized in that, Step 2 specifically includes: Step 21: Perform multi-level frequency domain detail decomposition on the input image. Multi-scale decomposition yields a set of coefficients. ,in, For the first Low-frequency subband coefficients after layer decomposition , and They represent the first High-frequency subband coefficients in the horizontal, vertical, and diagonal directions of layer decomposition; Step 22, retain in each level of decomposition , and , and will Setting to zero yields The coefficient set is obtained. ; Step 23, perform inverse discrete wavelet transform on... Reconstruction is performed to obtain the high-frequency response map of the input image. ; Step 24: Based on the high-frequency subband coefficients at each scale and in each direction, calculate the position of each pixel using equation (1). High-frequency energy at the location : （1） In the formula, , , They represent the results obtained by discrete wavelet transform decomposition. High-frequency sub-bands in the horizontal, vertical, and diagonal directions at that location; Step 25, for high-frequency energy Normalization is performed to map the energy values ​​to the [0,1] interval, resulting in pixel-level pixel contribution weight mapping.

3. The Gaussian splash 3D reconstruction method based on inverse problem optimization as described in claim 1, characterized in that, The inverse problem optimization model in step 3 is expressed as equation (2): （2） In the formula, These are the optimal scene parameters obtained after optimization. The set of 3D scene parameters to be optimized. For data fidelity items, The regularization coefficient is . For regularization terms, To render the image.

4. The Gaussian splash 3D reconstruction method based on inverse problem optimization as described in any one of claims 1-3, characterized in that, The spatial hierarchy contribution modulation mechanism in step 4 is expressed as equation (3): （3） In the formula, Contribute weights to the pixels of the i-th Gaussian ellipsoid. Let be the center depth of the i-th Gaussian ellipsoid, and j be the index variable used to iterate through all Gaussian ellipsoids in the scene. This is a parameter used to control the decay rate of the depth with respect to the weights.

5. The Gaussian splash 3D reconstruction method based on inverse problem optimization as described in claim 4, characterized in that, The mixed loss function in step 5 is expressed as equation (4): （4） in, To reconstruct errors, For structural consistency terms, For the frequency constraint term, it is described by Equation (5), where α is the weight of the reconstruction error and structural consistency term, and α is used to balance pixel reconstruction accuracy and structural similarity. β is the frequency constraint weight, which is used to control the strength of the frequency constraint. （5） In the formula, Indicates pixel frequency response. This is the frequency enhancement factor. N Total number of pixels p For pixel index, and Ip The rendered image and the real image are respectively in pixels. p The color value at that location, This represents the square of the L2 norm.

6. The Gaussian splash 3D reconstruction method based on inverse problem optimization as described in claim 5, characterized in that, In step 5, the gradient backpropagation algorithm described by equation (6) is used to iteratively update the Gaussian ellipsoid parameters: （6） In the formula, θ represents the Gaussian ellipsoid parameters, including position parameters, covariance parameters, color parameters, and opacity parameters. The learning rate controls the step size for each parameter update. This is the current iteration step number. Let θ be the gradient of the total loss function with respect to the parameter θ.

7. A Gaussian splash 3D reconstruction device based on inverse problem optimization, characterized in that, include: The image acquisition and initialization unit is used to acquire multi-view images of the target scene, use the structure of motion reconstruction method to perform camera pose calibration and generate sparse point cloud of the scene, and initialize a three-dimensional Gaussian ellipsoid set based on the sparse point cloud. The frequency analysis unit is used to perform multi-level frequency domain detail decomposition on multi-view images, extract horizontal, vertical, and diagonal high-frequency sub-bands and calculate high-frequency energy, and construct pixel-level pixel contribution weight mapping. The inverse problem optimization unit is used to reconstruct the 3D Gaussian splash scene into an inverse problem optimization model. It establishes the mapping relationship between the 3D Gaussian ellipsoid parameters and the 2D observation image through a differentiable rendering operator, and introduces the pixel contribution weight mapping as a frequency prior constraint into the Gaussian ellipsoid parameter optimization process. The depth-aware rendering unit is used to introduce a spatial hierarchy contribution modulation mechanism in the alpha compositing stage of rendering, which dynamically modulates the contribution weight of each Gaussian ellipsoid to the pixel color based on the depth value between the Gaussian ellipsoid and the camera. The scene output unit is used to construct a hybrid loss function that integrates the pixel contribution weights, iteratively update the position parameters, covariance parameters, color parameters, and transparency parameters of the Gaussian ellipsoid, and output the 3D reconstruction result of the target scene.

8. The Gaussian splash 3D reconstruction device based on inverse problem optimization as described in claim 7, characterized in that, The frequency analysis unit is specifically configured to perform frequency analysis on the input image using a multi-level frequency domain detail decomposition method. Multi-scale decomposition yields a set of coefficients. ,in, For the first Low-frequency subband coefficients after layer decomposition , and They represent the first High-frequency subband coefficients in the horizontal, vertical, and diagonal directions are decomposed into layers; preservation is achieved in each layer decomposition. , and , and will Setting to zero yields The coefficient set is obtained. ; By using inverse discrete wavelet transform Reconstruction is performed to obtain the high-frequency response map of the input image. Based on the high-frequency subband coefficients at each scale and in each direction, the position of each pixel is calculated using equation (1). High-frequency energy at the location For high-frequency energy Normalization is performed to map the energy values ​​to the [0,1] interval, resulting in pixel-level pixel contribution weight mapping; （1） In the formula, , , They represent the results obtained by discrete wavelet transform decomposition. High-frequency subbands in the horizontal, vertical and diagonal directions.

9. The Gaussian splash 3D reconstruction device based on inverse problem optimization as described in claim 7 or 8, characterized in that, The depth-sensing rendering unit adopts a spatial hierarchical contribution modulation mechanism, expressed as Equation (3): （3） In the formula, For the first i The pixel contribution weight of each Gaussian ellipsoid For the first i The central depth of a Gaussian ellipsoid j An index variable used to iterate through all Gaussian ellipsoids in the scene. This is a parameter used to control the decay rate of the depth with respect to the weights.

10. The Gaussian splash 3D reconstruction device based on inverse problem optimization as described in claim 9, characterized in that, The inverse problem optimization unit uses an inverse problem optimization model: （2） In the formula, These are the optimal scene parameters obtained after optimization. The set of 3D scene parameters to be optimized. For data fidelity items, The regularization coefficient is . For regularization terms, To render the image; The hybrid loss function in the scene output unit is expressed as equation (4): （4） In the formula, To reconstruct errors, For structural consistency terms, The frequency constraint term is described in equation (5). The weights for the reconstruction error and structural consistency term are used to balance pixel reconstruction accuracy and structural similarity. These are the frequency constraint weights, used to control the strength of the frequency constraints. （5） In the formula, Indicates pixel frequency response. This is the frequency enhancement factor. N Total number of pixels p For pixel index, and I The rendered image and the real image are respectively in pixels. p The color value at that location, This represents the square of the L2 norm.

Images