A three-dimensional reconstruction method based on structured gaussian and neural kernel field
By constructing a collaborative optimization method of structured Gaussian representation and neural kernel function field, the problem of focusing on rendering and reconstruction tasks in existing technologies is solved, and the unity of efficient rendering and accurate 3D reconstruction is achieved, which is applicable to the generation of new urban perspective images and urban 3D modeling.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN UNIV
- Filing Date
- 2025-09-03
- Publication Date
- 2026-05-26
AI Technical Summary
Existing technologies struggle to achieve high-quality rendering and robust surface reconstruction of large-scale scenes simultaneously within a unified framework. This is especially true when dealing with complex scenes, where there is a focus on one aspect over the other in rendering and reconstruction tasks, making it difficult to balance efficiency and accuracy.
By constructing a structured Gaussian representation as the basic spatial structure of the neural kernel field, and combining the structured Gaussian representation and the neural kernel field for synergistic optimization, we can achieve efficient rendering of images from new perspectives and generate accurate 3D models.
Outputting high-quality rendered images and robust 3D surface models simultaneously within a unified framework improves scalability and overall efficiency in handling large-scale complex scenes, while avoiding the complexity and inconsistencies caused by independent pipelines.
Smart Images
Figure CN121190658B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of surveying and mapping science and technology, and relates to a three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields, which is mainly applied to the generation of new perspective images of cities and the three-dimensional modeling of cities. Background Technology
[0002] In recent years, significant progress has been made in the digitization and reconstruction of 3D scenes. Among these advancements, image-based neural rendering methods, such as neural radiation fields and their variants, have demonstrated exceptional performance in synthesizing high-quality images with novel perspectives. Recently developed 3D Gaussian sputtering methods have achieved new heights in both rendering speed and quality by optimizing a set of 3D Gaussians. However, these Gaussian sputtering-based methods primarily focus on efficient and realistic image rendering; their direct output is not a standardized geometric surface model or precise distance information readily usable by downstream tasks. Extracting high-quality geometric information from their rendered representation remains a challenge.
[0003] Meanwhile, significant progress has been made in 3D surface reconstruction techniques, particularly those based on implicit representations. Neural implicit surface reconstruction methods can learn implicit functions of a scene from input data such as point clouds and obtain high-quality surface meshes by extracting isosurfaces. These methods exhibit good robustness and scalability in handling large-scale, incomplete, or noisy data. However, since the core objective of these methods is geometric reconstruction, they typically require additional steps or different networks to achieve high-quality image rendering, and their rendering efficiency and view adaptability may not be as good as dedicated neural rendering methods.
[0004] Existing methods often prioritize either rendering or reconstruction, making it difficult to achieve high performance and scalability for both within a unified framework, especially when dealing with large-scale and complex 3D scenes. A significant challenge in this field is how to effectively combine the advantages of both approaches to build a unified solution that delivers high-quality, scalable rendering while providing accurate and robust 3D geometric representations. Summary of the Invention
[0005] The purpose of this invention is to provide a 3D scene processing method and system that combines structured Gaussian representation and neural kernel function fields, aiming to solve the problem in existing technologies that it is difficult to simultaneously achieve high-quality rendering of large-scale scenes and robust surface reconstruction within a unified framework. This invention achieves complementary advantages and synergistic operation of the two representation forms by using the constructed structured Gaussian representation as the basic spatial structure of the neural kernel function field. Specifically, the method of this invention constructs a structured Gaussian representation based on initial scene data, and then constructs a neural kernel function field based on this representation. Through joint optimization, the structured Gaussian representation can efficiently render images from new perspectives, while accurately representing the implicit surfaces of the scene and generating 3D models.
[0006] The technical solution of this invention is: a three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields, comprising the following steps:
[0007] Step 1, data preparation, includes: 1) multi-view images; 2) camera intrinsic and extrinsic parameters of the images; 3) point cloud for initialization;
[0008] Step 2: Based on the input data in Step 1, construct a structured Gaussian representation of the 3D scene;
[0009] Step 3: Based on the structured Gaussian representation, construct the neural kernel function field, using sparse voxel grids and anchor points as the basic spatial structure of the neural kernel function field;
[0010] Step 4: Perform co-optimization training on the structured Gaussian representation and the neural kernel field, which updates the parameters of the structured Gaussian representation and the neural kernel field based on multi-view images and using reconstruction loss;
[0011] Step 5: Using the results of the co-optimization of structured Gaussian representation and neural kernel function field, the surface of the 3D scene is reconstructed to obtain a 3D surface model and a new perspective image as output.
[0012] Furthermore, the specific implementation method of step 2 is as follows:
[0013] Step 2.1, anchor point initialization, which is to obtain a sparse anchor point mesh by voxelizing the initial point cloud;
[0014] Step 2.2, Dynamic Neural Gaussian Generation: Based on anchor point features and camera intrinsic and extrinsic parameters, the properties of the neural Gaussian are dynamically predicted using a multilayer perceptron, as shown below:
[0015]
[0016] In the formula, The identifier comprises a set of neural Gaussian properties including color, opacity, and covariance. It is a multilayer perceptron. For anchor point features, and This refers to the relative distance and direction between the anchor point and the camera.
[0017] Furthermore, step 2 also includes: anchor point optimization and adjustment, employing a gradient-driven growth strategy and importance pruning: calculating gradients through backpropagation of differentiable rendering loss. ,when When the opacity exceeds the first threshold, an anchor point is added. Anchor points whose cumulative opacity is less than the second threshold after several consecutive iterations are removed.
[0018] Furthermore, the neural kernel function field is constructed in step 3 as follows:
[0019] Step 3.1, Hierarchical Voxel Structure Construction: A sparse convolutional network is used to process the input point cloud, generating a multi-layer voxel mesh. The input point cloud originates from the anchor points of the structured Gaussian representation. The center of each voxel layer is denoted as... The predicted features are denoted as And feature fields are constructed using Bézier interpolation. ;
[0020] Step 3.2, define the kernel function, the kernel function for each layer. Bézier kernel supported by the inner product of the characteristic fields and compactness composition:
[0021]
[0022] In the formula, For query point, The center point of the voxel;
[0023] Step 3.3: Minimize the difference between the gradient of the implicit field at the voxel center and the normal predicted by the neural nucleus field, as well as the value at the input point:
[0024]
[0025] in, , It is the normal vector of the voxel center point cloud. This is to input the point cloud location. It is the spatial gradient operator. It is a level First scalar coefficients of individual elements. It is the optimal solution for scalar coefficients. It is the first Hierarchical voxel center point The predicted normal vector at that location, It is the total number of layers in the layered voxel mesh. It is the first The number of voxels in a hierarchical voxel grid. This is the total number of points in the input point cloud.
[0026] Furthermore, the total loss used in collaborative optimization training Combining rendering loss and geometric regularization loss, the calculation formula is as follows:
[0027]
[0028] in, It is a hyperparameter that balances rendering loss and geometric loss. It is the geometric regularization loss, calculated using the following formula: Rendering loss , It is a constant. L1 is the structural similarity loss, and L2 is the absolute error loss.
[0029] Furthermore, the L1 loss calculates the mean absolute error between the pixels of the rendered image and the real image, using the following formula:
[0030]
[0031] in Indicates pixel position, This represents the total number of pixels. It's rendering an image. It is a real image.
[0032] Furthermore, the formula for calculating the structural similarity loss is as follows:
[0033]
[0034] in, For the number of windows, via sliding window Compare the structure, brightness, and contrast of local areas:
[0035]
[0036] in The mean within the window. It's rendering an image. It is a real image. Variance / covariance , It is a constant.
[0037] Furthermore, in step 5, the methods for generating the new perspective image and the 3D surface model are as follows:
[0038] Step 5.1: For a given new viewpoint, based on the anchor point position and associated Gaussian attributes in the structured Gaussian representation, project each Gaussian from the three-dimensional space onto the two-dimensional image plane corresponding to the new viewpoint. Sort the two-dimensional Gaussians according to the depth information in the camera parameters. Superimpose the sorted two-dimensional Gaussians through Alpha blending. Calculate the color value of each pixel to obtain the image corresponding to the new viewpoint.
[0039] Step 5.2: Output a depth map through the structured Gaussian representation differentiable rendering pipeline, and fuse the depth map using a truncated sign function fusion algorithm to obtain a 3D surface model.
[0040] The present invention also provides a three-dimensional reconstruction system based on structured Gaussian and neural nucleus fields, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a three-dimensional reconstruction method based on structured Gaussian and neural nucleus fields as described in the above technical solution.
[0041] The present invention also provides a computer storage medium storing a computer-readable program, characterized in that: when the computer-readable program is invoked, it executes a three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields as described in the above technical solution.
[0042] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0043] 1) This invention achieves the unification of scene rendering and surface reconstruction, and outputs high-quality rendered images and robust 3D surface models in one framework, avoiding the complexity and potential inconsistencies caused by using separate pipelines.
[0044] 2) By leveraging the compactness and efficiency of structured Gaussian representation in handling large-scale scenes, and the scalability and robustness of neural kernel field in implicit surface learning and to sparse data, the method demonstrates superior scalability and overall efficiency when handling large-scale complex scenes. Attached Figure Description
[0045] Figure 1 This is a flowchart of the present invention; Detailed Implementation
[0046] The present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.
[0047] This invention provides a three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields, such as... Figure 1 As shown, it includes the following steps:
[0048] Step 1, Data Preparation. The input data required for this method includes: 1) multi-view images; 2) camera intrinsic and extrinsic parameters of the images; 3) point cloud for initialization;
[0049] Step 2: Based on the input data in Step 1, construct a structured Gaussian representation of the 3D scene;
[0050] Step 3: Based on the structured Gaussian representation, construct the neural kernel function field, where the sparse voxel grid and anchor points are used as the basic spatial structure of the neural kernel function field;
[0051] Step 4: Perform co-optimization training on the structured Gaussian representation and the neural kernel field, which is based on multi-view images and uses reconstruction loss to update the parameters of the structured Gaussian representation and the neural kernel field;
[0052] Step 5: Using the results of the co-optimization of structured Gaussian representation and neural kernel function field, the surface of the 3D scene is reconstructed to obtain a 3D surface model and a new perspective image as output.
[0053] In the aforementioned three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields, step 2, specifically the method for constructing the structured Gaussian representation, is as follows:
[0054] Step 2.1, anchor point initialization, which is to obtain a sparse anchor point mesh by voxelizing the initial point cloud;
[0055] Step 2.2, Dynamic Neural Gaussian Generation: Based on anchor point features and camera intrinsic and extrinsic parameters, the properties of the neural Gaussian are dynamically predicted using a multilayer perceptron.
[0056]
[0057] In the formula, The identifier comprises a set of neural Gaussian properties including color, opacity, and covariance. It is a multilayer perceptron. For anchor point features, and This refers to the relative distance and direction between the anchor point and the camera.
[0058] Step 2.3, Anchor point optimization and adjustment, adopting a gradient-driven growth strategy and importance pruning: the gradient is calculated through backpropagation of differentiable rendering loss. ,when Anchor points are added periodically, and anchor points whose cumulative opacity is less than 0.005 after 10 consecutive iterations are removed.
[0059] In the aforementioned three-dimensional reconstruction method based on structured Gaussian and neural kernel fields, step 3 involves the construction and solution method of the neural kernel function field as follows:
[0060] Step 3.1: Hierarchical voxel structure construction. A sparse convolutional network is used to process the input point cloud, which originates from the anchor points of a structured Gaussian representation, generating a multi-layer voxel mesh. The center of each voxel layer... Predictive features And feature fields are constructed using Bézier interpolation. ;
[0061] Step 3.2, define the kernel function, the kernel function for each layer. Bézier kernel supported by the inner product of the characteristic fields and compactness composition:
[0062]
[0063] In the formula, For query point, The center point of the voxel;
[0064] Step 3.3: Minimize the difference between the gradient of the implicit field at the voxel center and the normal predicted by the neural nucleus field, as well as the value at the input point:
[0065]
[0066] in, , It is the normal vector of the voxel center point cloud. This is to input the point cloud location. It is the spatial gradient operator. It is a level First scalar coefficients of individual elements. It is the optimal solution for scalar coefficients. It is the first Hierarchical voxel center point The predicted normal vector at that location (generated by the neural kernel function field). It is the total number of layers in the layered voxel mesh. It is the first The number of voxels in a hierarchical voxel grid. This is the total number of points in the input point cloud.
[0067] In the aforementioned 3D reconstruction method based on structured Gaussian and neural nuclear fields, step 4 involves using multi-view image data, calculating the loss through differentiable rendering, and updating parameters via backpropagation.
[0068] The total loss combines rendering loss and geometric regularization:
[0069]
[0070] in, It is a hyperparameter that balances rendering loss and geometric loss. It is the geometric regularization loss, calculated using the following formula: Rendering loss , L1 loss calculates the mean absolute error between the pixels of the rendered image and the real image:
[0071]
[0072] in Indicates pixel position, This represents the total number of pixels. It's rendering an image. It is a real image.
[0073] It is a structural similarity loss. , For the number of windows, via sliding window Compare the structure, brightness, and contrast of local areas:
[0074]
[0075] in, It's rendering an image. It is a real image. The mean within the window. Variance / covariance , It is a constant.
[0076] In the aforementioned three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields, step 5 involves the following method for generating the new perspective image and the three-dimensional surface model:
[0077] Step 5.1: For a given new viewpoint, based on the anchor point position and associated Gaussian attributes in the structured Gaussian representation, project each Gaussian from the three-dimensional space onto the two-dimensional image plane corresponding to the new viewpoint. Sort the two-dimensional Gaussians according to the depth information in the camera parameters. Superimpose the sorted two-dimensional Gaussians through Alpha blending. Calculate the color value of each pixel to obtain the image corresponding to the new viewpoint.
[0078] Step 5.2: Output a depth map through the structured Gaussian representation differentiable rendering pipeline, and fuse the depth map using a truncated sign function fusion algorithm to obtain a 3D surface model.
[0079] We conducted comparative experiments on reconstruction accuracy on two publicly available datasets, Tanks and Temples and GauUScene, and the results are shown in Tables 1 and 2. For both datasets, we used the F1 score to measure the effectiveness of our method. The F1 score evaluates reconstruction accuracy by calculating the comparison between the real point cloud and the reconstructed surface. Specifically, the F1 score measures how accurately the model reconstructs the geometry of the real point cloud when reconstructing a 3D surface, and how well it balances correctly reconstructed regions (true positives) with incorrectly reconstructed regions (false positives and false negatives).
[0080] Table 1. Accuracy comparison on the Tanks and Temples datasets (measured by F1 score)
[0081]
[0082] Table 2. Accuracy and efficiency comparison on the GauUScene dataset
[0083]
[0084] The experimental results show that the present invention has a strong advantage on both datasets. On the Tanks and Temples dataset, the F1 score is better than other comparative methods, and on the GauUScene dataset, it also shows excellent performance, maintaining a high F1 score while still having high efficiency. This fully demonstrates the superiority of the present invention in the 3D surface reconstruction task. It can not only accurately reconstruct the geometric structure in complex scenes, but also optimize the computational efficiency, and has a stronger practical application potential than other methods.
[0085] The NeuS method listed is derived from the literature Wang P, Liu L, Liu Y, et al. NeuS: learning neural implicit surfaces by volume rendering for multi-view reconstruction[C]. Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021: 27171-27183;
[0086] Geo-NeuS is sourced from the literature Fu Q, Xu Q, Ong Y S, et al. Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction[J]. Advances in Neural Information Processing Systems, 2022, 35: 3403-3416;
[0087] 2DGS is sourced from the literature Huang B, Yu Z, Chen A, et al. 2d gaussian splatting for geometrically accurate radiance fields[C]. ACM SIGGRAPH 2024 conference papers. 2024: 1-11;
[0088] GVKF is sourced from the literature Song G, Cheng C, Wang H. GVKF: Gaussian voxel kernel functions for highly efficient surface reconstruction in open scenes[J]. Advances in Neural Information Processing Systems, 2024, 37: 104792-104815;
[0089] GOF is sourced from the literature Yu Z, Sattler T, Geiger A. Gaussian opacity fields: Efficient adaptive surface reconstruction in unbounded scenes[J]. ACM Transactions on Graphics (TOG), 2024, 43(6): 1-13;
[0090] PGSR comes from the literature Chen D, Li H, Ye W, et al. Pgsr: Planar-based gaussiansplating for efficient and high-fidelity surface reconstruction[J]. IEEETransactions on Visualization and Computer Graphics, 2024;
[0091] CityGaussianV2 is derived from the paper Liu Y, Luo C, Mao Z, et al. CityGaussianV2: Efficient and Geometrically Accurate Reconstruction for Large-Scale Scenes[J]. arXiv preprint arXiv:2411.00771, 2024.
[0092] The above description of the embodiments is merely an illustration of the basic technical solution of the present invention and is not limited to the above embodiments. It should be noted that those skilled in the art can make any simple modifications and refinements to the specific embodiments described, but without departing from the basic spirit of the present invention or exceeding the scope defined by the claims.
Claims
1. A three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields, characterized in that, Includes the following steps: Step 1, Input data preparation, including: 1) multi-view images; 2) camera intrinsic and extrinsic parameters of the images; 3) point cloud for initialization; Step 2: Based on the input data in Step 1, construct a structured Gaussian representation of the 3D scene; Step 3: Based on structured Gaussian representation, construct a neural kernel function field, using a sparse voxel grid and anchor points as the basic spatial structure of the neural kernel function field; wherein, the construction of the neural kernel function field includes: Step 3.1, Hierarchical Voxel Structure Construction: A sparse convolutional network is used to process the input point cloud, generating a multi-layer voxel mesh. The input point cloud originates from the anchor points of the structured Gaussian representation. The center of each voxel layer is denoted as... The predicted features are denoted as And feature fields are constructed using Bézier interpolation. ; Step 3.2, define the kernel function, the kernel function for each layer. Bézier kernel supported by the inner product of the characteristic fields and compactness composition: In the formula, For query point, The center point of the voxel; Step 3.3: Minimize the difference between the gradient of the implicit field at the voxel center and the normal predicted by the neural nucleus field, as well as the value at the input point: in, , It is the normal vector of the voxel center point cloud. This is to input the point cloud location. It is the spatial gradient operator. It is a level First scalar coefficients of individual elements. It is the optimal solution for scalar coefficients. It is the first Hierarchical voxel center point The predicted normal vector at that location, It is the total number of layers in the layered voxel mesh. It is the first The number of voxels in a hierarchical voxel grid. This is the total number of points in the input point cloud; Step 4 involves co-optimizing the structured Gaussian representation and the neural kernel field, using multi-view images and reconstruction loss to update the parameters of both. The co-optimization training employs a total loss... Combining rendering loss and geometric regularization loss, the calculation formula is as follows: in, It is a hyperparameter that balances rendering loss and geometric loss. It is the geometric regularization loss, calculated using the following formula: Rendering loss , It is a constant. L1 is the structural similarity loss, and L2 is the absolute error loss. Step 5: Using the results of the co-optimization of structured Gaussian representation and neural kernel function field, the surface of the 3D scene is reconstructed to obtain a 3D surface model and a new perspective image as output.
2. The three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields as described in claim 1, characterized in that: The specific implementation method of step 2 is as follows: Step 2.1, anchor point initialization, which is to obtain a sparse anchor point mesh by voxelizing the initial point cloud; Step 2.2, Dynamic Neural Gaussian Generation: Based on anchor point features and camera intrinsic and extrinsic parameters, the properties of the neural Gaussian are dynamically predicted using a multilayer perceptron, as shown below: In the formula, The identifier comprises a set of neural Gaussian properties including color, opacity, and covariance. It is a multilayer perceptron. For anchor point features, and This refers to the relative distance and direction between the anchor point and the camera.
3. The three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields as described in claim 1, characterized in that: Step 2 also includes: anchor point optimization and adjustment, employing a gradient-driven growth strategy and importance pruning: calculating gradients through backpropagation of differentiable rendering loss. ,when When the opacity exceeds the first threshold, an anchor point is added. Anchor points whose cumulative opacity is less than the second threshold after several consecutive iterations are removed.
4. The three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields as described in claim 1, characterized in that: L1 loss is calculated as the mean absolute error between the pixels of the rendered image and the real image. The calculation formula is as follows: in Indicates pixel position, This represents the total number of pixels. It's rendering an image. It is a real image.
5. The three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields as described in claim 1, characterized in that: The formula for calculating structural similarity loss is as follows: in, For the number of windows, via sliding window Compare the structure, brightness, and contrast of local areas: in The mean within the window. It's rendering an image. It is a real image. Variance / covariance , It is a constant.
6. The three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields as described in claim 1, characterized in that: In step 5, the methods for generating the new perspective image and the 3D surface model are as follows: Step 5.1: For a given new viewpoint, based on the anchor point position and associated Gaussian attributes in the structured Gaussian representation, project each Gaussian from the three-dimensional space onto the two-dimensional image plane corresponding to the new viewpoint. Sort the two-dimensional Gaussians according to the depth information in the camera parameters. Superimpose the sorted two-dimensional Gaussians through Alpha blending. Calculate the color value of each pixel to obtain the image corresponding to the new viewpoint. Step 5.2: Output a depth map through the structured Gaussian representation differentiable rendering pipeline, and fuse the depth map using a truncated sign function fusion algorithm to obtain a 3D surface model.
7. A three-dimensional reconstruction system based on structured Gaussian and neural nuclear fields, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields as described in any one of claims 1 to 6.
8. A computer storage medium storing a computer-readable program, characterized in that: When the computer-readable program is invoked, it executes a three-dimensional reconstruction method based on structured Gaussian and neural nuclear fields as described in any one of claims 1 to 6.