A 3D Gaussian-based three-dimensional reconstruction method and computer device

Abstract

The present invention relates to the field of three-dimensional reconstruction, and more specifically to a 3D Gaussian-based three-dimensional reconstruction method and computer device, which improves the accuracy of three-dimensional reconstruction. The method includes: collecting and preprocessing data to obtain a predicted depth map and normal map; rendering using 3D Gaussian Splatting to obtain a color image, normal map, and depth map; subtracting the rendered color image, normal map, and depth map from the training true value, predicted normal map, and predicted depth map, respectively, to obtain a loss function; and minimizing the loss function through stochastic gradient descent to iteratively optimize the distribution, shape, and color of the trained Gaussian. The present invention is suitable for three-dimensional reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction, and in particular to a three-dimensional reconstruction method and a computer device based on 3D Gaussian. Background Art

[0002] Current 3D reconstruction methods, such as a real-time 3D reconstruction method based on a depth map disclosed in CN108961390A, include: obtaining a depth map and an RGB color map of a captured scene using a depth camera; processing each frame of the depth map as follows: completing the depth information, converting its pixels into first-class 3D coordinate points, and calculating the normal vector at each pixel; converting the first-class 3D coordinate points corresponding to the depth map into second-class 3D coordinate points; assigning a voxel block to each second-class 3D coordinate point corresponding to the depth map, and indexing the voxel block using a hash table; updating the SDF value of each voxel in the voxel block by weighted fusion of the voxel blocks, and then extracting the scene surface; obtaining texture information of the scene surface, and calculating the surface normal vector at each voxel on the scene surface.

[0003] The above solution only relies on the depth map for real-time reconstruction, which can effectively improve the reconstruction speed, but the reconstruction accuracy is not high. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a 3D Gaussian-based 3D reconstruction method to improve the accuracy of 3D reconstruction.

[0005] The present invention adopts the following technical solutions to achieve the above-mentioned objectives. In a first aspect, the present invention provides a 3D Gaussian-based 3D reconstruction method, comprising:

[0006] S1, collect data and preprocess the data;

[0007] A set of two-dimensional images is acquired from different viewpoints by a camera, each image containing overlapping areas of the scene;

[0008] Use the ORB feature point extraction algorithm to detect and extract local feature points in each image, then match the feature points in adjacent images, use the fast nearest neighbor search matching algorithm to find the corresponding feature points in multiple images, and use a distance ratio-based strategy to filter out false matches;

[0009] Use the random sampling consensus algorithm to estimate the fundamental matrix between the two images, which is used to describe the relative position and posture between the cameras;

[0010] Decompose the camera pose from the fundamental matrix, including position and translation;

[0011] Using the feature points and camera poses in multiple images, the 3D points in the scene are reconstructed through triangulation. The position of each 3D point is calculated by the intersection of rays from different images in space.

[0012] Initialize the 3D point cloud as a flat Gaussian and use a pre-trained neural network to predict the scene geometry, including the depth map and normal map, based on the input color image. Align the predicted depth map and normal map with the initial scene point cloud to unify their scale and world coordinate system.

[0013] S2, obtain color images, normal maps and depth maps through 3D Gaussian Splatting rendering;

[0014] S3. Subtract the rendered color image, normal map, and depth map from the training true value, predicted normal map, and predicted depth map to obtain the loss function:

[0015]

[0016] Among them, γ, δ, α, β represent, T is the training iteration round, Threshold is the training strategy threshold round, L c is the color loss function, L D , L N , L d , L n are regularity constraints respectively;

[0017]

[0018] Represents the rendered normal map, N mono is the predicted normal map;

[0019]

[0020] Represents the rendered depth map, D mono is the predicted depth map;

[0021]

[0022] ω i represents the superposition weight of the i-th Gaussian, d i is the corresponding depth value, ω j represents the superposition weight of the j-th Gaussian, d j The corresponding depth value;

[0023]

[0024] N represents the normal map generated by the depth map gradient;

[0025] The loss function is minimized by stochastic gradient descent, and the distribution, shape, and color of the training Gaussian are iteratively optimized.

[0026] Furthermore, the iterative optimization training strategy specifically includes:

[0027] If the number of training iterations is less than or equal to the training strategy threshold number, the low-frequency structural information of the scene is optimized by the calculated loss function; otherwise, the high-frequency detail information of the scene is optimized by the calculated loss function.

[0028] Furthermore, the color image obtained by 3D Gaussian Splatting rendering specifically includes:

[0029] Lighting calculation: The fragment shader is responsible for calculating the color of each pixel according to the lighting model. Lighting calculation usually uses vertex normals, light source position, and material attribute information;

[0030] Texture mapping: Mapping a texture image to the surface of a 3D object to provide color information. The fragment shader samples the texture based on the texture coordinates of the object surface to determine the final color of each pixel.

[0031] Color output: The fragment shader outputs the calculated color value to the frame buffer and saves it as a color image.

[0032] Furthermore, the normal map obtained by 3D Gaussian Splatting rendering specifically includes:

[0033] Normal interpolation: During the rasterization process, the vertex normals of the triangle are interpolated to each pixel, and the interpolated normal vector is directly used as the output of the fragment shader;

[0034] Color encoding: The normal is a three-dimensional vector, and the normal vector is mapped to RGB value;

[0035] Normal map output: The fragment shader outputs the encoded normals as color values ​​to generate a normal map.

[0036] Furthermore, the depth map obtained by 3D Gaussian Splatting rendering specifically includes:

[0037] Depth value calculation: Each fragment will calculate its depth value during the rasterization process. This depth value is the depth value after perspective transformation, which indicates the distance of the pixel from the camera.

[0038] Depth normalization: Normalize the depth value to a range so that it can be stored as a grayscale image;

[0039] Depth map output: The fragment shader outputs the normalized depth value as grayscale to generate a depth map.

[0040] Furthermore, step S1 further includes: optimizing camera parameters and positions of 3D points by using a bundle adjustment method.

[0041] In a second aspect, the present invention provides a computer device, comprising a memory, wherein the memory stores program instructions, and when the program instructions are executed, the above-mentioned 3D Gaussian-based three-dimensional reconstruction method is executed.

[0042] The beneficial effects of the present invention are:

[0043] The present invention uses the motion recovery structure method to restore the initial scene point cloud for a series of pictures of the same scene, and initializes the point cloud to a flat Gaussian. At the same time, based on the input color picture, a pre-trained neural network is used to predict the scene's geometric information, depth map, and normal map. The color picture, normal map, and depth map are then rendered using the 3D Gaussian Splatting technology. These pictures are then subtracted from the trained true value, predicted normal map, and predicted depth map to obtain a loss function. The loss function is minimized through stochastic gradient descent, and the distribution, shape, and color of the trained Gaussian are iteratively optimized. This greatly improves the accuracy of 3D reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of a 3D Gaussian-based 3D reconstruction method provided by an embodiment of the present invention;

[0045] Figure 2 This is a flowchart for restoring the initial scene point cloud from a picture of the same scene provided by an embodiment of the present invention;

[0046] Figure 3 It is an iterative optimization flow chart provided by an embodiment of the present invention;

[0047] Figure 4 This is the final rendering accuracy comparison chart provided by the embodiment of the present invention;

[0048] Figure 5 This is a scene reconstruction rendering provided by an embodiment of the present invention;

[0049] Figure 6 This is a quantitative comparison chart of reconstruction indicators provided by an embodiment of the present invention;

[0050] Figure 7 This is a quantitative comparison chart of rendering indicators provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0052] The present invention provides a 3D Gaussian-based 3D reconstruction method, such as Figure 1 、 Figure 2 As shown, specifically including:

[0053] (1) Data preprocessing

[0054] 1. Image Acquisition

[0055] Acquire a set of 2D images from different viewpoints. These images can be taken by a single mobile camera or by multiple cameras simultaneously. Each image must contain overlapping areas of the scene to ensure that feature points can be identified and matched across multiple images.

[0056] 2. Feature point extraction and matching

[0057] Local feature points are detected and extracted in each image. The ORB (Oriented FAST and Rotated BRIEF) feature point extraction algorithm is used to ensure that these feature points remain stable under different scales and rotations. Feature points in adjacent images are then matched. A fast nearest neighbor search matching algorithm is used to find corresponding feature points across multiple images. To improve matching accuracy, a distance ratio-based strategy is used to filter out false matches.

[0058] 3. Fundamental matrix or homography matrix estimation

[0059] The random sampling consensus algorithm is used to estimate the fundamental matrix or homography between two images. The fundamental matrix describes the relative position and pose between cameras, while the homography is used for planar transformations. The random sampling consensus algorithm can effectively filter out false feature matches.

[0060] 4. Camera pose estimation

[0061] Decompose the camera pose (position and translation) from the fundamental matrix. This step usually involves assuming or pre-calibrating the intrinsic parameter matrix (intrinsic parameters of the camera). If the intrinsic parameters of the camera are unknown, they can be estimated through self-calibration.

[0062] 5. Triangulation to reconstruct 3D points

[0063] Using feature points and camera poses from multiple images, the 3D points in the scene are reconstructed through triangulation. The position of each 3D point is calculated by the intersection of rays from different images in space.

[0064] 6. Sparse point cloud optimization

[0065] The initial 3D point cloud is often sparse and may contain noise. To improve the accuracy of 3D reconstruction, optimization of the sparse point cloud is necessary. Bundle adjustment is a commonly used optimization method that can simultaneously optimize camera parameters and the positions of 3D points to minimize projection errors.

[0066] The point cloud is initialized as a flat Gaussian. Based on the input color image, a pre-trained neural network is used to predict the scene's geometric information (depth map, normal map). The predicted depth map and normal map are aligned with the initial scene point cloud to unify their scale and world coordinate system.

[0067] (2) Rendering

[0068] The color image (Render RGB), normal map (Render Normal), and depth map (Render Depth) are rendered using 3D Gaussian Splatting technology.

[0069] The specific steps for rendering color images are:

[0070] 1. Lighting calculation: The fragment shader is responsible for calculating the color of each pixel according to the lighting model. Lighting calculation usually uses information such as vertex normals, light source position, and material properties.

[0071] 2. Texture mapping: Maps a texture image onto the surface of a 3D object to provide color information. The fragment shader samples the texture based on the texture coordinates on the object surface to determine the final color of each pixel.

[0072] 3. Color output: The fragment shader outputs the calculated color value (usually in RGB format) to the frame buffer and saves it as a color image (Render RGB).

[0073] The specific steps for rendering the normal map are:

[0074] 1. Normal interpolation: During the rasterization process, the vertex normals of the triangle are interpolated to each pixel. The interpolated normal vector can be directly used as the output of the fragment shader.

[0075] 2. Color coding: Normal is usually a three-dimensional vector (nx, ny, nz). In order to store it as an image, the normal vector can be mapped to RGB values. For example, the normal component of (-1, 1) is mapped to the range of (0, 1).

[0076] 3. Normal map output: The fragment shader outputs the encoded normal as a color value to generate a normal map (RenderNormal).

[0077] The specific steps for rendering the depth map are:

[0078] 1. Depth value calculation: Each fragment will calculate its depth value (z value) during the rasterization process. This value is usually the depth value after perspective transformation, indicating the distance of the pixel from the camera.

[0079] 2. Depth normalization: Depth values ​​are usually normalized to a range (e.g. [0, 1]) so that they can be stored as grayscale images. Objects that are close have smaller depth values, while objects that are far away have larger depth values.

[0080] 3. Depth map output: The fragment shader outputs the normalized depth value as grayscale to generate a depth map (Render Depth).

[0081] Finally, these images are subtracted from the training true value (GT RGB), the predicted normal map (Predicted Normal), and the predicted depth map (Predicted Depth) to obtain the loss function:

[0082]

[0083] Where γ = 0.1, δ = 0.01, α = 100, β = 5, T is the number of training iterations, Threshold is the training strategy threshold of 13000 rounds, L c is the color loss function, L D ,L N ,L d ,L n They are the regular constraints mentioned below respectively.

[0084]

[0085] in is the rendered normal map, N mono is the predicted normal map.

[0086]

[0087] in is the rendered depth map, D mono is the predicted depth map.

[0088]

[0089] ω i represents the superposition weight of the i-th Gaussian, d i is the corresponding depth value, ω j represents the superposition weight of the j-th Gaussian, d j The corresponding depth value.

[0090]

[0091] Where i represents the number of coincidences with the Gaussian along the ray, and N represents the normal map generated by the depth map gradient.

[0092] According to the above loss function improvement, the loss function is minimized by stochastic gradient descent, and the distribution, shape, and color of the Gaussian training are iteratively optimized. The improvement effect is as follows Figure 3 Shown on the right.

[0093] (3) Training strategy

[0094] According to the scene geometry prior of 3D Gaussian learning, the training phase is divided into two parts. The first part uses formulas (1)(2)(3) to calculate the loss function and optimize the low-frequency structural information of the scene. The second part uses formulas (1)(4)(5) to optimize the high-frequency detail information of the scene. The final rendering accuracy is improved as follows Figure 4 As shown, at the same time, the geometric reconstruction accuracy is as Figure 5 shown.

[0095] This paper employs a new and efficient training strategy to enable Gaussians to effectively represent real surfaces. In the first part of training, predicted depth and normal maps are used to replace the original supervision. This scene-level geometric prior supervision enables the Gaussian to quickly distribute to more accurate poses during densification. In the second part of training, a small amount of Gaussian densification is used to accurately represent high-frequency details. Through an effective loss function, the Gaussian approximates the real surface, addressing the shortcomings of previous methods by preserving the true details of the object.

[0096] The present invention achieved the best results in testing the geometric accuracy of 3D reconstruction on the public dataset DTU. Figure 6 And it also achieved excellent results in reconstructing the new perspective rendering dataset Mip-NeRF360, as shown in the following figure. Figure 7 shown.

[0097] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A 3D Gaussian-based 3D reconstruction method, characterized in that: include: S1. Collect data and preprocess the data to obtain the predicted depth map and normal map; A set of two-dimensional images is acquired from different viewpoints by a camera, each image containing overlapping areas of the scene; Use the ORB feature point extraction algorithm to detect and extract local feature points in each image, then match the feature points in adjacent images, use the fast nearest neighbor search matching algorithm to find the corresponding feature points in multiple images, and use a distance ratio-based strategy to filter out false matches; Use the random sampling consensus algorithm to estimate the fundamental matrix between the two images, which is used to describe the relative position and posture between the cameras; Decompose the camera pose from the fundamental matrix, including position and translation; Using the feature points and camera poses in multiple images, the 3D points in the scene are reconstructed through triangulation. The position of each 3D point is calculated by the intersection of rays from different images in space. Initialize the 3D point cloud as a flat Gaussian and use a pre-trained neural network to predict the scene geometry, including the depth map and normal map, based on the input color image. Align the predicted depth map and normal map with the initial scene point cloud to unify their scale and world coordinate system. S2, obtain color images, normal maps and depth maps through 3D Gaussian Splatting rendering; S3. Subtract the rendered color image, normal map, and depth map from the training true value, predicted normal map, and predicted depth map to obtain the loss function L: Where γ, δ, α, and β represent weights, T is the number of training iterations, Threshold is the threshold number of training strategies, and L c is the color loss function, L D , L N , L d , L n are regularity constraints respectively; Represents the rendered normal map, N mono is the predicted normal map; Represents the rendered depth map, D mono is the predicted depth map; ω i represents the superposition weight of the i-th Gaussian, d i is the corresponding depth value, ω j represents the superposition weight of the j-th Gaussian, d j The corresponding depth value; N represents the normal map generated by the depth map gradient; The loss function is minimized by stochastic gradient descent, and the distribution, shape, and color of the training Gaussian are iteratively optimized.

2. The 3D Gaussian-based 3D reconstruction method according to claim 1, characterized in that: The specific strategies for iterative optimization training include: If the number of training iterations is less than or equal to the training strategy threshold number, the low-frequency structural information of the scene is optimized by the calculated loss function; otherwise, the high-frequency detail information of the scene is optimized by the calculated loss function.

3. The 3D Gaussian-based 3D reconstruction method according to claim 1, wherein: The color images obtained by 3D Gaussian Splatting rendering include: Lighting calculation: The fragment shader is responsible for calculating the color of each pixel according to the lighting model. Lighting calculation usually uses vertex normals, light source position, and material attribute information; Texture mapping: Mapping a texture image to the surface of a 3D object to provide color information. The fragment shader samples the texture based on the texture coordinates of the object surface to determine the final color of each pixel. Color output: The fragment shader outputs the calculated color value to the frame buffer and saves it as a color image.

4. The 3D Gaussian-based 3D reconstruction method according to claim 1, wherein: The normal map obtained by 3D Gaussian Splatting rendering specifically includes: Normal interpolation: During the rasterization process, the vertex normals of the triangle are interpolated to each pixel, and the interpolated normal vector is directly used as the output of the fragment shader; Color encoding: The normal is a three-dimensional vector, and the normal vector is mapped to RGB value; Normal map output: The fragment shader outputs the encoded normals as color values ​​to generate a normal map.

5. The 3D Gaussian-based 3D reconstruction method according to claim 1, wherein: The depth map obtained by 3D Gaussian Splatting rendering specifically includes: Depth value calculation: Each fragment will calculate its depth value during the rasterization process. This depth value is the depth value after perspective transformation, which indicates the distance of the pixel from the camera. Depth normalization: Normalize the depth value to a range so that it can be stored as a grayscale image; Depth map output: The fragment shader outputs the normalized depth value as grayscale to generate a depth map.

6. The 3D Gaussian-based 3D reconstruction method according to claim 1, characterized in that: Step S1 further includes: optimizing camera parameters and positions of 3D points by using a bundle adjustment method.

7. A computer device comprising a memory storing program instructions, characterized in that: When the program instructions are executed, the 3D Gaussian-based 3D reconstruction method according to any one of claims 1 to 6 is executed.

Images