3D scene reconstruction method based on Gaussian splattering

Abstract

The present invention discloses a three-dimensional scene reconstruction method based on Gaussian splattering, which belongs to the field of three-dimensional scene processing technology and includes the following steps: S1, obtaining an initial three-dimensional scene and generating a three-dimensional Gaussian splattering sphere based on several feature points of the initial three-dimensional scene; S2, using the three-dimensional Gaussian splattering sphere to screen the area to be reconstructed in the initial three-dimensional scene; S3, smoothing the area to be reconstructed to complete the three-dimensional scene reconstruction. Based on the geometric relationship between the Gaussian sphere and the area to be reconstructed, the present invention designs adaptive smoothing weights. The weights assign higher smoothing strengths in flat areas to suppress noise, and reduce smoothing strengths in feature areas to retain details. This achieves efficient three-dimensional scene reconstruction and is particularly suitable for refined modeling of complex dynamic scenes and low-texture areas, providing an innovative technical solution for the field of three-dimensional vision.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional scene processing, and in particular relates to a three-dimensional scene reconstruction method based on Gaussian splattering. Background Art

[0002] Three-dimensional scene reconstruction is a core task in computer vision and graphics, aiming to recover the scene's geometric structure and surface properties from multi-source sensor data. With the rise of applications such as the metaverse, robotic navigation, and augmented reality, the requirements for reconstruction efficiency, detail preservation, and adaptability to dynamic scenes are increasing. However, traditional methods still face the following challenges: 1. The conflict between computational efficiency and geometric accuracy: Mesh representations (such as TSDF and Marching Cubes) rely on dense sampling, resulting in memory and computational overhead that grows cubically with resolution, making them difficult to process large-scale scenes in real time. 2. Implicit representations (such as Neural Radiance Fields (NeRF)) can generate high-precision details, but suffer from slow inference speed and limited ability to model dynamic scenes. 3. Reconstruction challenges in dynamic scenes and sparsely textured areas: Areas with missing texture (such as white walls) or dynamic objects are prone to holes or noise. Consequently, existing methods lack awareness of scene geometry, leading to inappropriate smoothing weight allocation. Gaussian splatting, a sparse scene representation method in which each primitive contains position and color attributes, can be applied to 3D scene reconstruction. Summary of the Invention

[0003] In order to solve the above problems, the present invention proposes a three-dimensional scene reconstruction method based on Gaussian splashing.

[0004] The technical solution of the present invention is: a three-dimensional scene reconstruction method based on Gaussian splattering includes the following steps:

[0005] S1. Obtain an initial three-dimensional scene and generate a three-dimensional Gaussian splash sphere based on several feature points of the initial three-dimensional scene;

[0006] S2, based on the RGB value of each pixel point in the initial three-dimensional scene, using a three-dimensional Gaussian splatter sphere to screen the area to be reconstructed in the initial three-dimensional scene;

[0007] S3. Determine the smoothing weight of each pixel point in the area to be reconstructed, perform smoothing processing on the area to be reconstructed, and complete the three-dimensional scene reconstruction.

[0008] Furthermore, S1 includes the following sub-steps:

[0009] S11, obtaining an initial three-dimensional scene, and extracting several SIFT feature points of the initial three-dimensional scene;

[0010] S12, generating a two-dimensional splash area based on a number of SIFT feature points of the initial three-dimensional scene;

[0011] S13. Rotate the two-dimensional splash area around the long axis to generate a three-dimensional Gaussian splash sphere.

[0012] The beneficial effect of the above further scheme is: in the present invention, SIFT feature points are invariant to scale, rotation and brightness changes, can stably capture geometric key points in three-dimensional scenes, and provide reliable anchor points for subsequent Gaussian sphere generation. The spatial distribution of SIFT feature points is used to adaptively generate a two-dimensional splash area, rather than uniformly covering the scene, so that reconstruction resources are concentrated on high-information areas (such as object boundaries and texture overlaps). The long axis and short axis determined based on the Euclidean distance of the feature points are used to align the shape of the two-dimensional area with the main direction of the scene, reduce redundant coverage, and provide geometric rationality for subsequent three-dimensional sphere generation. Combining SIFT feature points with Gaussian splashing, end-to-end optimization from feature extraction to three-dimensional representation is achieved, avoiding excessive smoothing or loss of details in traditional methods.

[0013] Furthermore, S12 includes the following sub-steps:

[0014] S121, calculating the spatial distribution value of the SIFT feature point according to the descriptor of each SIFT feature point;

[0015] S122, taking the SITF feature point with the largest spatial distribution value as the first vertex, and taking the SITF feature point with the farthest Euclidean distance from the first vertex in the initial three-dimensional scene as the second vertex;

[0016] S123 , connecting the first vertex and the second vertex as the major axis of the two-dimensional splash area, and using half of the Euclidean distance between the first vertex and the second vertex as the minor axis distance of the two-dimensional splash area to generate a two-dimensional splash area.

[0017] The beneficial effects of this further solution are as follows: In the present invention, the feature point with the maximum spatial distribution value is used as the first vertex, and the farthest feature point is used as the second vertex, ensuring that the major axis is along the main extension direction of the scene, so that the two-dimensional region covers the core structure of the scene and reduces interference from irrelevant areas. The minor axis distance is half the major axis (which can also be adjusted dynamically), and the region scale is dynamically adjusted based on the distribution of scene feature points to avoid excessive coverage or information loss caused by a fixed ratio, and to balance reconstruction accuracy and efficiency. By rotating the two-dimensional splatter region around the major axis to generate an ellipsoid, the Gaussian distribution primitives are aligned with the main direction of the scene, improving the geometric fit of the 3D reconstruction and supporting real-time reconstruction of large-scale scenes compared to voxel or grid representations.

[0018] Furthermore, in S121, the spatial distribution value of the SIFT feature point The calculation formula is:

[0019] ;

[0020] Where, represents the dot product operation, Represents the descriptor of SIFT feature points, represents the transpose operation, Represents the dimension of the descriptor of SIFT feature points, Represents the descriptor of the SIFT feature point with the closest Euclidean distance to the SIFT feature point, Indicates the number of SIFT feature points, Represents the calculation of the maximum eigenvalue of a matrix.

[0021] The beneficial effect of the above further solution is that in the present invention, the SIFT descriptor is a vector calculated based on the location and scale of the local extremum in the scale space and the gradient within the neighborhood, generally a 128-dimensional column vector. By comparing the descriptor differences between a feature point and its nearest neighbor, low-quality or duplicate features are filtered out, improving the geometric consistency of the reconstruction result.

[0022] Furthermore, S2 includes the following sub-steps:

[0023] S21, taking two pixel points adjacent to each other along the horizontal axis, two pixel points adjacent to each other along the vertical axis, and two pixel points adjacent to each other along the vertical axis as a six-neighborhood set of the pixel point;

[0024] S22. Determine the weight of each neighboring pixel point in the six neighborhood sets corresponding to each pixel point based on the three-dimensional Gaussian splatter sphere;

[0025] S23, generating a fuzziness for the pixel according to the RGB value of the pixel and the weight of each neighboring pixel;

[0026] S24: Pixels whose blurriness is smaller than their own RGB values ​​are used as the to-be-reconstructed area of ​​the initial three-dimensional scene.

[0027] The beneficial effect of the above-mentioned further solution is as follows: In the present invention, local structure perception capabilities in three-dimensional space are constructed through neighboring pixels in three directions: horizontal, vertical, and vertical. Compared with two-dimensional neighborhoods (such as four or eight neighborhoods), a six-neighborhood set more accurately captures the neighboring pixels in a three-dimensional scene, providing spatial continuity constraints for subsequent reconstruction. The six-neighborhood structure avoids over-smoothing of the three-dimensional scene, especially when processing anisotropic textures (such as fibers and stripes), which can preserve more directional details. The present invention only optimizes areas with low blur, which is more adaptable to real-time reconstruction requirements than global processing. The present invention combines the geometric constraints of a three-dimensional Gaussian sphere with color blur screening to achieve joint optimization from spatial structure to surface properties, solving the geometric distortion and color bleeding problems encountered in traditional methods. Through dynamic adjustment of weights and blur, it can adapt to different scenes and is robust to sensor noise, lighting changes, and dynamic object residue.

[0028] Furthermore, in S22, the weights of the neighboring pixels in the six neighborhood sets corresponding to the pixel point are The calculation formula is:

[0029] ;

[0030] Where, represents the neighborhood pixel point, Represents the set of pixels on the outer surface of a three-dimensional Gaussian splash sphere, Represents the RGB value of the pixel, Indicates the RGB value of the neighboring pixel in the six-neighborhood set corresponding to the pixel.

[0031] Furthermore, in S23, the fuzziness The calculation formula is:

[0032] ;

[0033] Where, represents the hyperparameter, Represents the RGB value of the pixel, Indicates that the pixel corresponds to the first The weight of the neighboring pixels, Indicates that the pixel corresponds to the first RGB values ​​of neighboring pixels.

[0034] The beneficial effect of this further solution is that, in this invention, the blurriness combines the current pixel color value with a weighted average of its neighborhood, balancing local color consistency and spatial smoothness through hyperparameters. The blurriness quantifies the compatibility of the current pixel with its neighborhood, providing a quantitative indicator for subsequent screening and avoiding indiscriminate processing of the entire scene.

[0035] Furthermore, S3 includes the following sub-steps:

[0036] S31, connecting two vertices of the three-dimensional Gaussian splatter sphere to the pixel positions of the area to be reconstructed, to form a smooth angle;

[0037] S32, calculating the smoothing weight of each pixel point in the area to be reconstructed according to the smoothing angle;

[0038] S33. Based on the smoothing weight of each pixel point, Laplace smoothing is performed on each pixel point in the area to be reconstructed to complete the three-dimensional scene reconstruction.

[0039] Furthermore, in S32, the smoothing weight of the pixel The calculation formula is:

[0040] ;

[0041] Where, Indicates the angle between pixel positions. Indicates taking a random number.

[0042] The beneficial effect of the above further scheme is: in the present invention, by connecting the two vertices (long axis endpoints) of the Gaussian splatter sphere with the pixel points of the area to be reconstructed, an angle aligned with the main geometric direction of the scene is formed, so that the smoothing operation is performed along the natural extension direction of the scene, avoiding excessive smoothing of the edge or texture overlapping area. The angle implicitly encodes the relative position relationship between the pixel point and the main structure of the scene, providing geometric constraints for subsequent weight calculations, and reducing the damage to key features (such as object contours). Controllable randomness is introduced to avoid "water ripple" artifacts caused by excessive regularization of weights, and enhance the robustness of the algorithm to noise or outliers. At the same time, Laplace smoothing controls the adjustment amplitude of each pixel through weights, effectively removing noise in flat areas (high weights) and avoiding blurred edges or texture details in feature areas (low weights). Prevent the optimization process from falling into local optimality.

[0043] The beneficial effects of the present invention are:

[0044] (1) The present invention extracts 3D-SIFT feature points of the initial scene and constructs an adaptive three-dimensional Gaussian splatter sphere. The position, scale, and covariance matrix of the sphere are dynamically determined by the distribution of feature points, focusing on the key structures of the scene and improving the accuracy of geometric reconstruction.

[0045] (2) The present invention utilizes the spatial distribution relationship between pixel RGB values ​​and Gaussian spheres to dynamically identify the target to be reconstructed, avoiding indiscriminate processing of the entire scene and significantly reducing computational overhead;

[0046] (3) Based on the geometric relationship between the Gaussian sphere and the area to be reconstructed, the present invention designs adaptive smoothing weights. The weights allocate higher smoothing strength in flat areas to suppress noise, and reduce the smoothing strength in feature areas to retain details, thereby achieving efficient three-dimensional scene reconstruction. It is particularly suitable for refined modeling of complex dynamic scenes and low-texture areas, providing an innovative technical solution for the field of three-dimensional vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Flowchart of the 3D scene reconstruction method based on Gaussian splashing. DETAILED DESCRIPTION

[0048] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0049] like Figure 1 As shown, the present invention provides a three-dimensional scene reconstruction method based on Gaussian splattering, comprising the following steps:

[0050] S1. Obtain an initial three-dimensional scene and generate a three-dimensional Gaussian splash sphere based on several feature points of the initial three-dimensional scene;

[0051] S2, based on the RGB value of each pixel point in the initial three-dimensional scene, using a three-dimensional Gaussian splatter sphere to screen the area to be reconstructed in the initial three-dimensional scene;

[0052] S3. Determine the smoothing weight of each pixel point in the area to be reconstructed, perform smoothing processing on the area to be reconstructed, and complete the three-dimensional scene reconstruction.

[0053] In this embodiment of the present invention, S1 includes the following sub-steps:

[0054] S11, obtaining an initial three-dimensional scene, and extracting several SIFT feature points of the initial three-dimensional scene;

[0055] S12, generating a two-dimensional splash area based on a number of SIFT feature points of the initial three-dimensional scene;

[0056] S13. Rotate the two-dimensional splash area around the long axis to generate a three-dimensional Gaussian splash sphere.

[0057] In the present invention, SIFT feature points are invariant to changes in scale, rotation, and brightness, and can stably capture geometric key points in three-dimensional scenes, providing reliable anchor points for subsequent Gaussian sphere generation. Two-dimensional splatter regions are adaptively generated through the spatial distribution of SIFT feature points, rather than uniformly covering the scene, so that reconstruction resources are concentrated in high-information areas (such as object boundaries and texture overlaps). The long and short axes determined based on the Euclidean distance of the feature points align the shape of the two-dimensional region with the main direction of the scene, reducing redundant coverage and providing geometric rationality for subsequent three-dimensional sphere generation. Combining SIFT feature points with Gaussian splattering enables end-to-end optimization from feature extraction to three-dimensional representation, avoiding over-smoothing or detail loss in traditional methods.

[0058] In this embodiment of the present invention, S12 includes the following sub-steps:

[0059] S121, calculating the spatial distribution value of the SIFT feature point according to the descriptor of each SIFT feature point;

[0060] S122, taking the SITF feature point with the largest spatial distribution value as the first vertex, and taking the SITF feature point with the farthest Euclidean distance from the first vertex in the initial three-dimensional scene as the second vertex;

[0061] S123 , connecting the first vertex and the second vertex as the major axis of the two-dimensional splash area, and using half of the Euclidean distance between the first vertex and the second vertex as the minor axis distance of the two-dimensional splash area to generate a two-dimensional splash area.

[0062] In this method, the feature point with the maximum spatial distribution value is used as the first vertex, and the farthest feature point is used as the second vertex. The major axis is aligned along the main extension direction of the scene, ensuring that the two-dimensional region covers the core structure of the scene and reducing interference from irrelevant areas. The minor axis distance is half the major axis (which can also be adjusted dynamically). The region scale is dynamically adjusted based on the distribution of scene feature points to avoid excessive coverage or information loss caused by a fixed ratio, balancing reconstruction accuracy and efficiency. By rotating the two-dimensional splatter region around its major axis to generate an ellipsoid, the Gaussian distribution primitives are aligned with the main direction of the scene, improving the geometric fit of the 3D reconstruction and supporting real-time reconstruction of large-scale scenes compared to voxel or grid representations.

[0063] In the embodiment of the present invention, in S121, the spatial distribution value of the SIFT feature point The calculation formula is:

[0064] ;

[0065] Where, represents the dot product operation, Represents the descriptor of SIFT feature points, represents the transpose operation, Represents the dimension of the descriptor of SIFT feature points, Represents the descriptor of the SIFT feature point with the closest Euclidean distance to the SIFT feature point, Indicates the number of SIFT feature points, Represents the calculation of the maximum eigenvalue of a matrix.

[0066] In this paper, the SIFT descriptor is a vector (typically a 128-dimensional column vector) calculated based on the location and scale of the local extrema in scale space, as well as the gradient within its neighborhood. By comparing the descriptor differences between a feature point and its nearest neighbor, low-quality or duplicate features are filtered out, improving the geometric consistency of the reconstruction results.

[0067] In this embodiment of the present invention, S2 includes the following sub-steps:

[0068] S21, taking two pixel points adjacent to each other along the horizontal axis, two pixel points adjacent to each other along the vertical axis, and two pixel points adjacent to each other along the vertical axis as a six-neighborhood set of the pixel point;

[0069] S22. Determine the weight of each neighboring pixel point in the six neighborhood sets corresponding to each pixel point based on the three-dimensional Gaussian splatter sphere;

[0070] S23, generating a fuzziness for the pixel according to the RGB value of the pixel and the weight of each neighboring pixel;

[0071] S24: Pixels whose blurriness is smaller than their own RGB values ​​are used as the to-be-reconstructed area of ​​the initial three-dimensional scene.

[0072] In the present invention, the local structure perception capability in three-dimensional space is constructed through the neighborhood pixels in the horizontal, vertical and vertical directions. Compared with the two-dimensional neighborhood (such as four neighborhoods or eight neighborhoods), the six-neighborhood set more accurately captures the neighboring pixels in the three-dimensional scene, providing spatial continuity constraints for subsequent reconstruction. The six-neighborhood structure avoids excessive smoothing of the three-dimensional scene, especially when processing anisotropic textures (such as fibers and stripes), it can retain more directional details. The present invention only optimizes areas with low blur, which is more adaptable to real-time reconstruction needs than global processing. The present invention combines the geometric constraints of the three-dimensional Gaussian sphere with color blur screening to achieve joint optimization from spatial structure to surface properties, solving the problems of geometric distortion and color bleeding in traditional methods. Through dynamic adjustment of weights and blur, it can adapt to different scenes and is robust to sensor noise, illumination changes and dynamic object residues.

[0073] In the embodiment of the present invention, in S22, the weights of the neighboring pixels in the six neighborhood sets corresponding to the pixel point are: The calculation formula is:

[0074] ;

[0075] Where, represents the neighborhood pixel point, Represents the set of pixels on the outer surface of a three-dimensional Gaussian splash sphere, Represents the RGB value of the pixel, Indicates the RGB value of the neighboring pixel in the six-neighborhood set corresponding to the pixel.

[0076] In the embodiment of the present invention, in S23, the fuzziness The calculation formula is:

[0077] ;

[0078] Where, represents the hyperparameter, Represents the RGB value of the pixel, Indicates that the pixel corresponds to the first The weight of the neighboring pixels, Indicates that the pixel corresponds to the first RGB values ​​of neighboring pixels.

[0079] In this paper, fuzziness combines the current pixel's color value with a weighted average of its neighborhood, balancing local color consistency and spatial smoothness through hyperparameters. The fuzziness quantifies the compatibility of the current pixel with its neighbors, providing a quantitative indicator for subsequent screening and avoiding indiscriminate processing across the entire scene.

[0080] In this embodiment of the present invention, S3 includes the following sub-steps:

[0081] S31, connecting two vertices of the three-dimensional Gaussian splatter sphere to the pixel positions of the area to be reconstructed, to form a smooth angle;

[0082] S32, calculating the smoothing weight of each pixel point in the area to be reconstructed according to the smoothing angle;

[0083] S33. Based on the smoothing weight of each pixel point, Laplace smoothing is performed on each pixel point in the area to be reconstructed to complete the three-dimensional scene reconstruction.

[0084] In the embodiment of the present invention, in S32, the smoothing weight of the pixel point The calculation formula is:

[0085] ;

[0086] Where, Indicates the angle between pixel positions. Indicates taking a random number.

[0087] In the present invention, by connecting the two vertices (end points of the long axis) of the Gaussian splatter sphere with the pixel points of the area to be reconstructed, an angle aligned with the main geometric direction of the scene is formed, so that the smoothing operation is performed along the natural extension direction of the scene, avoiding excessive smoothing of the edge or texture overlapping area. The angle implicitly encodes the relative position relationship between the pixel point and the main structure of the scene, providing geometric constraints for subsequent weight calculations and reducing damage to key features (such as object contours). Controllable randomness is introduced to avoid "water ripple" artifacts caused by excessive regularization of weights, and enhance the robustness of the algorithm to noise or outliers. At the same time, Laplace smoothing controls the adjustment amplitude of each pixel through weights, effectively removing noise in flat areas (high weights) and avoiding blurred edges or texture details in feature areas (low weights). Prevent the optimization process from falling into local optimality.

[0088] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A three-dimensional scene reconstruction method based on Gaussian splashing, characterized in that: The following steps are involved: S1. Obtain an initial three-dimensional scene and generate a three-dimensional Gaussian splash sphere based on several feature points of the initial three-dimensional scene; S2, based on the RGB value of each pixel point in the initial three-dimensional scene, using a three-dimensional Gaussian splatter sphere to screen the area to be reconstructed in the initial three-dimensional scene; S3, determining the smoothing weight of each pixel point in the area to be reconstructed, smoothing the area to be reconstructed, and completing the 3D scene reconstruction; The S1 includes the following sub-steps: S11, obtaining an initial three-dimensional scene, and extracting several SIFT feature points of the initial three-dimensional scene; S12, generating a two-dimensional splash area based on a number of SIFT feature points of the initial three-dimensional scene; S13, rotating the two-dimensional splash area around the long axis to generate a three-dimensional Gaussian splash sphere; The S12 includes the following sub-steps: S121, calculating the spatial distribution value of the SIFT feature point according to the descriptor of each SIFT feature point; S122, taking the SITF feature point with the largest spatial distribution value as the first vertex, and taking the SITF feature point with the farthest Euclidean distance from the first vertex in the initial three-dimensional scene as the second vertex; S123, connecting the first vertex and the second vertex as the major axis of the two-dimensional splash area, and using half of the Euclidean distance between the first vertex and the second vertex as the minor axis distance of the two-dimensional splash area, to generate a two-dimensional splash area; In the above S121, the spatial distribution value of the SIFT feature point The calculation formula is: ； Where, represents the dot product operation, Represents the descriptor of SIFT feature points, represents the transpose operation, Represents the dimension of the descriptor of SIFT feature points, Represents the descriptor of the SIFT feature point with the closest Euclidean distance to the SIFT feature point, Indicates the number of SIFT feature points, Represents the calculation of the maximum eigenvalue of a matrix.

2. The 3D scene reconstruction method based on Gaussian splattering according to claim 1, characterized in that: The S2 includes the following sub-steps: S21, taking two pixel points adjacent to each other along the horizontal axis, two pixel points adjacent to each other along the vertical axis, and two pixel points adjacent to each other along the vertical axis as a six-neighborhood set of the pixel point; S22. Determine the weight of each neighboring pixel point in the six neighborhood sets corresponding to each pixel point based on the three-dimensional Gaussian splatter sphere; S23, generating a fuzziness for the pixel according to the RGB value of the pixel and the weight of each neighboring pixel; S24: Pixels whose blurriness is smaller than their own RGB values ​​are used as the to-be-reconstructed area of ​​the initial three-dimensional scene.

3. The 3D scene reconstruction method based on Gaussian splattering according to claim 2, characterized in that: In said S22, the pixel point corresponds to the weight of the neighboring pixel points in the six neighborhood sets. The calculation formula is: ； Where, represents the neighborhood pixel point, Represents the set of pixels on the outer surface of a three-dimensional Gaussian splash sphere, Represents the RGB value of the pixel, Indicates the RGB value of the neighboring pixel in the six-neighborhood set corresponding to the pixel.

4. The 3D scene reconstruction method based on Gaussian splattering according to claim 2, characterized in that: In S23, the fuzziness The calculation formula is: ； Where, represents the hyperparameter, Represents the RGB value of the pixel, Indicates that the pixel corresponds to the first The weight of the neighboring pixels, Indicates that the pixel corresponds to the first RGB values ​​of neighboring pixels.

5. The 3D scene reconstruction method based on Gaussian splattering according to claim 1, characterized in that: The S3 includes the following sub-steps: S31, connecting two vertices of the three-dimensional Gaussian splatter sphere to the pixel positions of the area to be reconstructed, to form a smooth angle; S32, calculating the smoothing weight of each pixel point in the area to be reconstructed according to the smoothing angle; S33. Based on the smoothing weight of each pixel point, Laplace smoothing is performed on each pixel point in the area to be reconstructed to complete the three-dimensional scene reconstruction.

6. The 3D scene reconstruction method based on Gaussian splattering according to claim 5, characterized in that: In the step S32, the smoothing weight of the pixel point The calculation formula is: ； Where, Indicates the angle between pixel positions. Indicates taking a random number.

Images