Three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology and related equipment

Abstract

The invention discloses a three-dimensional reconstruction method based on a three-dimensional Gaussian sputtering technology and related equipment, and belongs to the technical field of three-dimensional reconstruction. Firstly, cyclic progressive Gaussian initialization is adopted, and the density of Gaussian point clouds is increased by utilizing pseudo views in the training process, so that more comprehensive geometric coverage is realized; secondly, depth alignment regularization is implemented, depth information is aligned through a sliding window mechanism in combination with precise depth and monocular depth estimation obtained by initializing the 3D point cloud, and effective geometric constraints are provided; finally, a sparse friendly sampling strategy is introduced, Gaussian ellipsoids which are excessively expanded due to sparse input data are processed in a targeted mode, and the geometric and rendering quality of details is improved by segmenting the ellipsoids.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and specifically relates to a three-dimensional reconstruction method and related equipment based on three-dimensional Gaussian sputtering technology. Background Art

[0002] In the field of 3D reconstruction and new view synthesis, 3D Gaussian Splatting (3DGS) is a cutting-edge technology that represents 3D scenes by using a set of anisotropic 3D Gaussian primitives and achieves efficient rendering through differentiable alpha blending rendering techniques. However, 3DGS has some obvious shortcomings when dealing with sparse input views. First, due to the limited number of input views, the initial Gaussian point cloud generated by 3DGS may be too sparse to provide enough geometric information to accurately reconstruct the details of the scene. Second, sparse views lead to insufficient supervision signals during training, making it difficult for the reconstructed model to capture the complex appearance and geometric structure of the scene. In addition, some Gaussian ellipsoids may become abnormally large during the optimization process, which can cause blurring and overfitting of rendering results under new viewpoints, especially at the edges of the scene and in areas with less texture.

[0003] To address these problems, some research works have proposed improvement methods. For example, FreeNeRF improves NeRF training under sparse input through a frequency regularization strategy, while SparseNeRF uses a pre-trained depth estimation model to generate pseudo ground truth depth maps for local depth ranking loss. DNGaussian uses monocular depth and global-local depth normalization to optimize Gaussian parameters. Although these methods improve the reconstruction quality of sparse input scenes to some extent, they usually rely on monocular depth maps that are inconsistent across view scales, which limits their effectiveness in precise geometric constraints.

[0004] In the field of 3D reconstruction, the reconstruction task of sparse input views has always been a challenge, because limited perspectives can hardly provide enough information to capture the complex details of the scene. In this case, traditional 3D Gaussian reconstruction (3DGS) methods often encounter problems such as too sparse initialization point clouds, lack of sufficient supervision signals during training, and uncontrolled Gaussian ellipsoid scale, which leads to a significant decrease in the rendering quality of the reconstruction results under new perspectives, resulting in blurring and distortion. Summary of the invention

[0005] The present invention provides a three-dimensional reconstruction method and related equipment based on three-dimensional Gaussian sputtering technology, which solves the problem of inaccurate reconstruction of existing sparse input views.

[0006] To achieve the above object, the present invention provides the following technical solutions: A three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology, comprising: S1: Get a sparse image dataset; S2: Initialize the sparse image data set to obtain the initial 3D point cloud; S3: Perform cyclic progressive Gaussian initialization on the initial 3D point cloud to obtain Gaussian point cloud data; S4: performing depth alignment on the Gaussian point cloud data to obtain Gaussian point cloud data after depth alignment; S5: Perform sparse friendly sampling on the Gaussian point cloud data after depth alignment; S6: Repeat S3, S4 and S5 1000 times to obtain a three-dimensional reconstructed image.

[0007] Preferably, the steps of initializing the sparse image data set are specifically: The initial 3D point cloud is recovered from the input sparse image dataset in the dataset using structure from motion techniques.

[0008] Preferably, the initial 3D point cloud is subjected to cyclic progressive Gaussian initialization to obtain Gaussian point cloud data as follows: Using the initial 3D point cloud, a set of 3D Gaussian primitives are initialized, and then a pseudo view is rendered using the 3D Gaussian point cloud and used as a new sparse view. The new sparse view is iteratively used to increase the density of the Gaussian point cloud, wherein a new pseudo view is generated around the sparse view in the dataset, and the position of the pseudo view is obtained by adding Gaussian noise around the camera parameters of the training view.

[0009] Preferably, the formula for obtaining the pseudo view position is:

[0010] in and are the positions of the pseudo view and the training view, respectively. is Gaussian noise, Representative training views, Indicates A pseudo view, is a constant used to control the quality of the build.

[0012] Preferably, the operation steps for performing depth alignment on Gaussian point cloud data are specifically as follows: First, a matching error filtering operation is performed on the initial point cloud data to exclude 3D points whose errors are greater than the set threshold. Then, 3D points generated only from pseudo views are excluded, and 3D points visible in a specific view are selected. Finally, a fixed-size window is slid on the rendered depth map and the monocular depth estimation map. The window moves on the image at a set step size to cover the entire image. At each window position, the difference between the rendered depth and the monocular depth is calculated, and a loss function is used to evaluate the consistency of the two depth maps within the window to obtain the depth loss. The parameters of the 3D Gaussian model are adjusted according to the depth loss.

[0013] Preferably, the calculation formula of the depth loss is:

[0014] is the depth loss, where , and Represents rendering depth, SfM extracted depth and monocular depth respectively. is the depth loss, represents the mean absolute error loss, is the Pearson distribution loss, and is the weight coefficient, Representative w windows, W is the total number of windows.

[0015] Preferably, the steps of performing sparse friendly sampling on the Gaussian point cloud data after depth alignment are specifically as follows: During the rendering process, by comparing the rendered image with the real image, calculating the error of each pixel, identifying the pixel with the highest error value, and then analyzing how the color of each pixel is obtained by α-blending multiple Gaussian ellipsoids, determining the Gaussian ellipsoid that contributes the most to the pixel color, the Gaussian ellipsoid will be segmented into multiple Gaussian ellipsoids.

[0016] A three-dimensional reconstruction system based on three-dimensional Gaussian sputtering technology, comprising: Data acquisition module: used to obtain sparse image data sets; Initialization processing module: used to initialize the sparse image data set to obtain the initial 3D point cloud; Density increase module: used to increase the density of Gaussian point cloud using the initial 3D point cloud to obtain Gaussian point cloud data; Depth alignment module: used to perform depth alignment on Gaussian point cloud data to obtain Gaussian point cloud data after depth alignment; Sampling module: used for sparse friendly sampling of Gaussian point cloud data after depth alignment; Reconstruction module: used to repeat S3, S4 and S5 1000 times to obtain a three-dimensional reconstructed image.

[0017] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology are implemented.

[0018] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology.

[0019] Compared with the prior art, the present invention has the following beneficial effects: the present invention proposes a 3D reconstruction method based on 3D Gaussian sputtering technology. Firstly, a cyclic progressive Gaussian initialization is adopted, and pseudo views are used in the training process to increase the density of Gaussian point clouds, so as to achieve more comprehensive geometric coverage; secondly, depth alignment regularization is implemented, and the precise depth obtained by initializing the 3D point cloud and the monocular depth estimation are combined, and the depth information is aligned through a sliding window mechanism to provide effective geometric constraints; finally, a sparse-friendly sampling strategy is introduced to specifically process Gaussian ellipsoids that are over-inflated due to sparse input data, and the geometry and rendering quality of the details are improved by segmenting these ellipsoids. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology of the present invention.

[0021] Figure 2 It is a detailed flow chart of a three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology of the present invention.

[0022] Figure 3 It is a block diagram of a three-dimensional reconstruction system based on three-dimensional Gaussian sputtering technology of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0026] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0027] like Figure 1 As shown, the present invention provides a three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology, comprising: S1: Get a sparse image dataset; S2: Initialize the sparse image data set to obtain the initial 3D point cloud; S3: Perform cyclic progressive Gaussian initialization on the initial 3D point cloud to obtain Gaussian point cloud data; S4: performing depth alignment on the Gaussian point cloud data to obtain Gaussian point cloud data after depth alignment; S5: Perform sparse friendly sampling on the Gaussian point cloud data after depth alignment; S6: Repeat S3, S4 and S5 1000 times to obtain a three-dimensional reconstructed image.

[0028] Detailed steps are as follows Figure 2 As shown, specifically: S1: Prepare 3D reconstruction dataset: First, determine the 3D scene to be reconstructed, then prepare the required photographic equipment, place cameras roughly evenly around the scene, and capture sparse images of the scene from different angles as training data. Make sure the images have enough overlapping areas to facilitate subsequent feature matching and 3D reconstruction training.

[0029] S2: Gaussian point cloud initialization: The initial 3D point cloud of the scene is recovered from the sparse views in the dataset using the Structure from Motion (SfM) technique.

[0030] S3: Cyclic Progressive Gaussian Initialization: First, a set of 3D Gaussian primitives are initialized using the initial 3D point cloud. Then, the pseudo view is rendered using the 3D Gaussian point cloud and used as a new sparse view, and the density of the Gaussian point cloud is increased using the new sparse view. Among them, new pseudo views are generated around the sparse views in the dataset. The positions of these pseudo views are determined by adding Gaussian noise around the camera parameters of the training view. The positions of the pseudo views can be expressed by the following mathematical formula:

[0031] in and are the positions of the pseudo view and the training view, respectively. is Gaussian noise, Representative training views, Indicates A pseudo view, is a constant used to control the quality of the build.

[0032] Step 4: Depth Alignment Regularization: In order to remove inaccurate depth values ​​in the Gaussian point cloud, the first step is to implement matching error filtering. This step identifies those 3D points with large matching errors by analyzing the SfM key point matching report and excludes the depth information of these points. By setting a threshold, for example, points with an average pixel matching error greater than 2 pixels are identified as coarse points, and their depth information will not be used to generate the depth map.

[0033]

[0034] in is the average pixel matching error, is the depth value.

[0035] Next, to ensure the reliability of the depth information, those 3D points generated only from the pseudo view are also removed from the depth map since they may contain errors caused by accidental matching.

[0036]

[0037] in is the matching 3D point, They are training view and pseudo view, Indicates that this point is calculated from these views.

[0038] In addition, using the visibility information provided by the RGB image, we select those 3D points that are visible in a specific view to ensure that the depth values ​​in the depth map reflect the true geometry of the scene. For example, when a foreground object occludes a background object, only the points of the foreground object will be used for depth map generation.

[0039]

[0040] in Representative Points In the i Depth from perspective, Indicates i Cameras, Indicate point To Camera distance.

[0041] Finally, in order to accurately integrate and align the depth information from different sources so that the depth estimation of the entire scene is more accurate and consistent, we designed a sliding window-based alignment technique. This is achieved by sliding a fixed-size window on the rendered depth map and the monocular depth estimation map. The window moves on the image at a set step size to cover the entire image. At each window position, the difference between the rendered depth and the monocular depth is calculated, and a loss function is used to evaluate the consistency of the two depth maps within the window to obtain the depth loss. The parameters of the 3D Gaussian model are adjusted according to the depth loss.

[0042] The calculation formula of depth loss is:

[0043] is the depth loss, where , and Represents rendering depth, SfM extracted depth and monocular depth respectively. is the depth loss, represents the mean absolute error loss, is the Pearson distribution loss, and is the weight coefficient, Representative w windows, W is the total number of windows. By minimizing these losses, the parameters of the 3D Gaussian model are adjusted using the depth alignment regularization strategy to reduce the inconsistency between the rendered depth and the monocular depth.

[0044] Step 5: Sparse-friendly sampling: After depth alignment, in sparse view conditions, due to the limited number of views available for training, some Gaussian ellipsoids may become abnormally large, resulting in blurring and distortion in the rendering results. During the rendering process, we compare the rendered image with the real image, calculate the error of each pixel, and identify those pixels with the highest error values, which indicate that the current Gaussian model fails to accurately capture the geometric details of the scene. Then analyze how the color of each pixel is obtained by alpha-blending multiple Gaussian ellipsoids, and then determine the Gaussian ellipsoid that contributes the most to the pixel color. We will perform segmentation operations on these Gaussian ellipsoids to subdivide them into smaller ellipsoids.

[0045] Step 6: Loop optimization mechanism: After completing step 1, loop step 2 4 times, and then loop and iterate step 3, step 4, and step 5 10,000 times in total, which can achieve the accuracy and completeness of 3D reconstruction, making the reconstruction result real and accurate under the new perspective.

[0046] This application uses three key technologies to enhance rendering quality and reconstruction accuracy to solve the problem of inaccurate reconstruction of sparse input views. Specifically, first, a cyclic progressive Gaussian initialization is used, and pseudo views are used during the training process to increase the density of the Gaussian point cloud to achieve more comprehensive geometric coverage; second, deep alignment regularization is implemented, combining the precise depth obtained by SfM and monocular depth estimation, and the depth information is aligned through a sliding window mechanism to provide effective geometric constraints; finally, a sparse-friendly sampling strategy is introduced to specifically deal with Gaussian ellipsoids that are over-inflated due to sparse input data, and these ellipsoids are segmented to improve the geometry and rendering quality of details.

[0047] like Figure 3 As shown, the present invention provides a three-dimensional reconstruction system based on three-dimensional Gaussian sputtering technology, comprising: Data acquisition module: used to obtain sparse image data sets; Initialization processing module: used to initialize the sparse image data set to obtain the initial 3D point cloud; Density increase module: used to increase the density of Gaussian point cloud using the initial 3D point cloud to obtain Gaussian point cloud data; Depth alignment module: used to perform depth alignment on Gaussian point cloud data to obtain Gaussian point cloud data after depth alignment; Sampling module: used for sparse friendly sampling of Gaussian point cloud data after depth alignment; Reconstruction module: used to repeat S3, S4 and S5 1000 times to obtain a three-dimensional reconstructed image.

[0048] A terminal device is provided in one embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0049] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0050] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0051] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0052] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0053] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0054] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields, and the above-mentioned specific embodiments are only illustrative and instructive, rather than restrictive. Under the guidance of the specification, a person skilled in the art can make many forms without departing from the scope of protection of the claims of the present invention, all of which belong to the scope of protection of the present invention.

Claims

1. A three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology, characterized in that: include: S1: Get a sparse image dataset; S2: Initialize the sparse image data set to obtain the initial 3D point cloud; S3: Perform cyclic progressive Gaussian initialization on the initial 3D point cloud to obtain Gaussian point cloud data; S4: performing depth alignment on the Gaussian point cloud data to obtain Gaussian point cloud data after depth alignment; S5: Perform sparse friendly sampling on the Gaussian point cloud data after depth alignment; S6: Repeat S3, S4 and S5 1000 times to obtain a three-dimensional reconstructed image.

2. The three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: The specific steps for initializing the sparse image data set are: The initial 3D point cloud is recovered from the input sparse image dataset in the dataset using structure from motion techniques.

3. The three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: The initial 3D point cloud is initialized with cyclic progressive Gaussian to obtain Gaussian point cloud data as follows: Using the initial 3D point cloud, a set of 3D Gaussian primitives are initialized, and then a pseudo view is rendered using the 3D Gaussian point cloud and used as a new sparse view. The new sparse view is iteratively used to increase the density of the Gaussian point cloud, wherein a new pseudo view is generated around the sparse view in the dataset, and the position of the pseudo view is obtained by adding Gaussian noise around the camera parameters of the training view.

4. The three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 3, characterized in that: The formula for obtaining the pseudo view position is: in and are the positions of the pseudo view and the training view, respectively. is Gaussian noise, Representative training views, Indicates A pseudo view, is a constant used to control the quality of the build.

5. The three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: The specific steps for deep alignment of Gaussian point cloud data are as follows: First, a matching error filtering operation is performed on the initial point cloud data to exclude 3D points whose errors are greater than the set threshold. Then, 3D points generated only from pseudo views are excluded, and 3D points visible in a specific view are selected. Finally, a fixed-size window is slid on the rendered depth map and the monocular depth estimation map. The window moves on the image at a set step size to cover the entire image. At each window position, the difference between the rendered depth and the monocular depth is calculated, and a loss function is used to evaluate the consistency of the two depth maps within the window to obtain the depth loss. The parameters of the 3D Gaussian model are adjusted according to the depth loss.

6. The three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 5, characterized in that: The calculation formula of depth loss is: is the depth loss, where , and Represents rendering depth, SfM extracted depth and monocular depth respectively. is the depth loss, represents the mean absolute error loss, is the Pearson distribution loss, and is the weight coefficient, Representative w windows, W is the total number of windows.

7. The three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: The specific steps for sparse friendly sampling of Gaussian point cloud data after depth alignment are as follows: During the rendering process, by comparing the rendered image with the real image, calculating the error of each pixel, identifying the pixel with the highest error value, and then analyzing how the color of each pixel is obtained by α-blending multiple Gaussian ellipsoids, determining the Gaussian ellipsoid that contributes the most to the pixel color, the Gaussian ellipsoid will be segmented into multiple Gaussian ellipsoids.

8. A three-dimensional reconstruction system based on three-dimensional Gaussian sputtering technology, characterized in that: include: Data acquisition module: used to obtain sparse image data sets; Initialization processing module: used to initialize the sparse image data set to obtain the initial 3D point cloud; Density increase module: used to increase the density of Gaussian point cloud using the initial 3D point cloud to obtain Gaussian point cloud data; Depth alignment module: used to perform depth alignment on Gaussian point cloud data to obtain Gaussian point cloud data after depth alignment; Sampling module: used for sparse friendly sampling of Gaussian point cloud data after depth alignment; Reconstruction module: used to repeat S3, S4 and S5 1000 times to obtain a three-dimensional reconstructed image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a three-dimensional reconstruction method based on three-dimensional Gaussian sputtering technology as described in any one of claims 1 to 7 are implemented.

Images