Three-dimensional reconstruction method based on 3D Gaussian and computer device

Abstract

The invention relates to the field of three-dimensional reconstruction, in particular to a three-dimensional reconstruction method based on 3D Gaussian and a computer device, and improves the precision of three-dimensional reconstruction. The method comprises the following steps: collecting data, and preprocessing the data to obtain a predicted depth map and a normal map; a color image, a normal image and a depth image are obtained through 3D Gaussian Splitting rendering; respectively subtracting the color picture, the normal graph and the depth graph obtained by rendering from the training truth value, the predicted normal graph and the predicted depth graph to obtain a loss function; the loss function is minimized through stochastic gradient descent, and Gaussian distribution, shape and color are iteratively optimized and trained. The method is suitable for three-dimensional reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction, and in particular to a three-dimensional reconstruction method and a computer device based on 3D Gaussian. Background Art

[0002] Current 3D reconstruction methods, such as a real-time 3D reconstruction method based on a depth map disclosed in CN108961390A, include: using a depth camera to obtain a depth map and an RGB color map of a captured scene; processing each frame of the depth map as follows: completing the depth information, then converting its pixels into first-class 3D coordinate points, and calculating the normal vector at each pixel; converting the first-class 3D coordinate points corresponding to the depth map into second-class 3D coordinate points; allocating a voxel block to each second-class 3D coordinate point corresponding to the depth map, and indexing the voxel block using a hash table; updating the SDF value of each voxel in the voxel block by weighted fusion of the voxel blocks, and then extracting the scene surface; obtaining the texture information of the scene surface, and calculating the surface normal vector at each voxel on the scene surface.

[0003] The above solution only relies on the depth map for real-time reconstruction, which can effectively improve the reconstruction speed, but the reconstruction accuracy is not high. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a 3D Gaussian-based three-dimensional reconstruction method to improve the accuracy of three-dimensional reconstruction.

[0005] The present invention adopts the following technical solutions to achieve the above-mentioned purpose. In a first aspect, the present invention provides a 3D Gaussian-based 3D reconstruction method, comprising:

[0006] S1, collect data and pre-process the data;

[0007] A set of two-dimensional images are acquired from different perspectives by a camera, each image containing overlapping areas of the scene;

[0008] Use the ORB feature point extraction algorithm to detect and extract local feature points in each image, then match the feature points in adjacent images, use the fast nearest neighbor search matching algorithm to find the corresponding feature points in multiple images, and use a distance ratio-based strategy to filter out false matches;

[0009] Use a random sampling consensus algorithm to estimate the fundamental matrix between the two images, which is used to describe the relative position and posture between the cameras;

[0010] Decompose the camera pose from the fundamental matrix, including position and translation;

[0011] Using the feature points and camera poses in multiple images, the 3D points in the scene are reconstructed through triangulation. The position of each 3D point is calculated by the intersection of rays from different images in space.

[0012] Initialize the 3D point cloud as a flat Gaussian, and use a pre-trained neural network to predict the scene's geometric information based on the input color image. The geometric information includes a depth map and a normal map, and align the predicted depth map and normal map based on the initial scene point cloud to unify their scales and world coordinate systems.

[0013] S2, obtain color images, normal maps and depth maps through 3D Gaussian Splatting rendering;

[0014] S3. Subtract the rendered color image, normal map and depth map from the training true value, predicted normal map and predicted depth map to obtain the loss function:

[0015]

[0016] Among them, γ, δ, α, β represent, T is the training iteration round, Threshold is the training strategy threshold round, Lc is the color loss function, LD, LN, Ld, Ln are regular constraints respectively;

[0017]

[0018] represents the rendered normal map, Nmono is the predicted normal map;

[0019]

[0020] represents the rendered depth map, and Dmono is the predicted depth map;

[0021]

[0022] ω i represents the superposition weight of the i-th Gaussian, d i is the corresponding depth value, ω j represents the superposition weight of the jth Gaussian, d j The corresponding depth value;

[0023]

[0024] N represents the normal map generated by the depth map gradient;

[0025] The loss function is minimized by stochastic gradient descent, and the distribution, shape, and color of the training Gaussian are iteratively optimized.

[0026] Furthermore, the strategies for iterative optimization training specifically include:

[0027] If the training iteration round is less than or equal to the training strategy threshold round, the low-frequency structural information of the scene is optimized by the calculated loss function, otherwise the high-frequency detail information of the scene is optimized by the calculated loss function.

[0028] Furthermore, the color images obtained by 3D Gaussian Splatting rendering include:

[0029] Lighting calculation: The fragment shader is responsible for calculating the color of each pixel according to the lighting model. Lighting calculation usually uses vertex normals, light source positions, and material attribute information;

[0030] Texture mapping: Mapping a texture image to the surface of a 3D object to provide color information. The fragment shader samples the texture according to the texture coordinates on the surface of the object to determine the final color of each pixel.

[0031] Color output: The fragment shader outputs the calculated color value to the frame buffer and saves it as a color image.

[0032] Furthermore, the normal map obtained by 3D Gaussian Splatting rendering specifically includes:

[0033] Normal interpolation: During the rasterization process, the vertex normals of the triangle are interpolated to each pixel, and the interpolated normal vector is directly used as the output of the fragment shader;

[0034] Color encoding: The normal is a three-dimensional vector, and the normal vector is mapped to RGB value;

[0035] Normal map output: The fragment shader outputs the encoded normals as color values ​​to generate a normal map.

[0036] Furthermore, the depth map obtained by 3D Gaussian Splatting rendering specifically includes:

[0037] Depth value calculation: Each fragment will calculate its depth value during the rasterization process. This depth value is the depth value after perspective transformation, which indicates the distance of the pixel from the camera.

[0038] Depth normalization: Normalize the depth value to a range so that it can be stored as a grayscale image;

[0039] Depth map output: The fragment shader outputs the normalized depth value as grayscale to generate a depth map.

[0040] Furthermore, step S1 further includes: optimizing the camera parameters and the positions of the 3D points by a bundle adjustment method.

[0041] In a second aspect, the present invention provides a computer device, including a memory, wherein the memory stores program instructions, and when the program instructions are run, the above-mentioned 3D Gaussian-based three-dimensional reconstruction method is executed.

[0042] The beneficial effects of the present invention are:

[0043] The present invention recovers the initial scene point cloud from a series of pictures of the same scene through the motion recovery structure method, and initializes the point cloud to a flat Gaussian. At the same time, based on the input color picture, a pre-trained neural network is used to predict the geometric information, depth map and normal map of the scene. Then, the color picture, normal map and depth map are rendered by 3D Gaussian Splatting technology, and these pictures are subtracted from the training true value, the predicted normal map and the predicted depth map to obtain the loss function. The loss function is minimized by random gradient descent, and the distribution, shape and color of the training Gaussian are iteratively optimized. The accuracy of three-dimensional reconstruction is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a 3D Gaussian-based 3D reconstruction method provided by an embodiment of the present invention;

[0045] Figure 2 This is a flow chart of restoring the initial scene point cloud from a picture of the same scene provided by an embodiment of the present invention;

[0046] Figure 3 is an iterative optimization flow chart provided by an embodiment of the present invention;

[0047] Figure 4 is a final rendering accuracy comparison chart provided by an embodiment of the present invention;

[0048] Figure 5 is a scene reconstruction effect diagram provided by an embodiment of the present invention;

[0049] Figure 6 is a quantitative comparison chart of reconstruction indicators provided by an embodiment of the present invention;

[0050] Figure 7 It is a quantitative comparison chart of rendering indicators provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0052] The present invention provides a 3D Gaussian-based 3D reconstruction method, such as Figure 1 , Figure 2 As shown, specifically including:

[0053] (1) Data preprocessing

[0054] 1. Image Acquisition

[0055] Acquire a set of 2D images from different viewpoints. These images can be taken by a single mobile camera or by multiple cameras simultaneously. Each image must contain overlapping areas of the scene to ensure that feature points can be identified and matched in multiple images.

[0056] 2. Feature point extraction and matching

[0057] Detect and extract local feature points in each image. The present invention uses the ORB (Oriented FAST and Rotated BRIEF) feature point extraction algorithm to ensure that these feature points remain stable under different scales and rotation conditions. Subsequently, feature points in adjacent images are matched. A fast nearest neighbor search matching algorithm is used to find corresponding feature points in multiple images. And in order to improve the accuracy of matching, a distance ratio-based strategy is used to filter false matches.

[0058] 3. Fundamental matrix or homography matrix estimation

[0059] The random sampling consensus algorithm is used to estimate the fundamental matrix or homography matrix between two images. The fundamental matrix is ​​used to describe the relative position and attitude between cameras, while the homography matrix is ​​used in the case of planar transformation. The random sampling consensus algorithm can effectively filter out false feature matches.

[0060] 4. Camera pose estimation

[0061] Decompose the camera pose (position and translation) from the fundamental matrix. This step usually involves assumptions or pre-calibration of the intrinsic parameter matrix (intrinsic parameters of the camera). If the intrinsic parameters of the camera are unknown, they can be estimated through self-calibration.

[0062] 5. Triangulate and reconstruct 3D points

[0063] Using the feature points and camera poses in multiple images, the 3D points in the scene are reconstructed through triangulation. The position of each 3D point is calculated by the intersection of rays from different images in space.

[0064] 6. Sparse point cloud optimization

[0065] The initial 3D point cloud is usually sparse and may contain noise. In order to improve the accuracy of 3D reconstruction, it is necessary to optimize the sparse point cloud. Bundle adjustment is a commonly used optimization method that can simultaneously optimize the camera parameters and the position of the 3D points to minimize the projection error.

[0066] The point cloud is initialized as a flat Gaussian. At the same time, based on the input color image, a pre-trained neural network is used to predict the scene's geometric information (depth map, normal map). And based on the initial scene point cloud, the predicted depth map and normal map are aligned to unify their scale and world coordinate system.

[0067] (2) Rendering

[0068] The color image (Render RGB), normal map (Render Normal), and depth map (Render Depth) are rendered using 3D Gaussian Splatting technology.

[0069] The specific steps for rendering color images are:

[0070] 1. Lighting calculation: The fragment shader is responsible for calculating the color of each pixel according to the lighting model. Lighting calculation usually uses information such as vertex normals, light source positions, and material properties.

[0071] 2. Texture mapping: Map the texture image to the surface of the 3D object to provide color information. The fragment shader samples the texture according to the texture coordinates on the surface of the object to determine the final color of each pixel.

[0072] 3. Color output: The fragment shader outputs the calculated color value (usually in RGB format) to the frame buffer and saves it as a color image (Render RGB).

[0073] The specific steps for rendering the normal map are:

[0074] 1. Normal interpolation: During the rasterization process, the vertex normals of the triangle will be interpolated to each pixel. The interpolated normal vector can be directly used as the output of the fragment shader.

[0075] 2. Color coding: The normal is usually a three-dimensional vector (nx, ny, nz). In order to store it as an image, the normal vector can be mapped to RGB values. For example, the normal component of (-1, 1) is mapped to the range of (0, 1).

[0076] 3. Normal map output: The fragment shader outputs the encoded normal as a color value to generate a normal map (RenderNormal).

[0077] The specific steps for rendering the depth map are:

[0078] 1. Depth value calculation: Each fragment will calculate its depth value (z value) during the rasterization process. This value is usually the depth value after perspective transformation, indicating the distance of the pixel from the camera.

[0079] 2. Depth normalization: Depth values ​​are usually normalized to a range (e.g. [0,1]) so that they can be stored as grayscale images. Objects that are close have smaller depth values, while objects that are far away have larger depth values.

[0080] 3. Depth map output: The fragment shader outputs the normalized depth value as grayscale to generate a depth map (Render Depth).

[0081] Finally, these images are subtracted from the training true value (GT RGB), the predicted normal map (Predicted Normal), and the predicted depth map (Predicted Depth) to obtain the loss function:

[0082]

[0083] Where γ = 0.1, δ = 0.01, α = 100, β = 5, T is the training iteration round, Threshold is the training strategy threshold of 13000 rounds, Lc is the color loss function, LD, LN, Ld, Ln are the regularization constraints mentioned below respectively.

[0084]

[0085] in is the rendered normal map and Nmono is the predicted normal map.

[0086]

[0087] in is the rendered depth map and Dmono is the predicted depth map.

[0088]

[0089] ω i represents the superposition weight of the i-th Gaussian, d i is the corresponding depth value, ω j represents the superposition weight of the jth Gaussian, d j The corresponding depth value.

[0090]

[0091] Where i represents the number of coincidences with the Gaussian along the ray, and N represents the normal map generated by the depth map gradient.

[0092] According to the above loss function improvement, the loss function is minimized by stochastic gradient descent, and the distribution, shape, and color of the training Gaussian are iteratively optimized. The improvement effects are as follows: Figure 3 Shown on the right.

[0093] (3) Training strategy

[0094] According to the scene geometry prior of 3D Gaussian learning, the training phase is divided into two parts. The first part uses formulas (1), (2), and (3) to calculate the loss function and optimize the low-frequency structural information of the scene. The second part uses formulas (1), (4), and (5) to optimize the high-frequency detail information of the scene. This improves the final rendering accuracy. Figure 4 As shown, at the same time, the geometric reconstruction accuracy is Figure 5 shown.

[0095] The present invention adopts a new and effective training strategy to make Gaussians effectively represent the actual surface. In the first part of the training, the predicted depth map and normal map are used to replace the original supervision, because the scene-level geometric prior supervision enables the Gaussians to quickly distribute to more accurate poses when densified. In the second part of the training, a small amount of Gaussian densification is used to accurately represent high-frequency details. Through an effective loss function, the Gaussians are close to the actual surface, which solves the shortcomings of the previous methods by retaining the real details of the object.

[0096] The present invention achieved the best results in testing the geometric accuracy of 3D reconstruction on the public dataset DTU. Figure 6 And it also achieved excellent results in reconstructing the new perspective rendering dataset Mip-NeRF360, as shown in the following figure. Figure 7 shown.

[0097] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A 3D Gaussian-based 3D reconstruction method, characterized in that: include: S1, collect data, pre-process the data, and obtain the predicted depth map and normal map; A set of two-dimensional images are acquired from different perspectives by a camera, each image containing overlapping areas of the scene; Use the ORB feature point extraction algorithm to detect and extract local feature points in each image, then match the feature points in adjacent images, use the fast nearest neighbor search matching algorithm to find the corresponding feature points in multiple images, and use a distance ratio-based strategy to filter out false matches; Use a random sampling consensus algorithm to estimate the fundamental matrix between the two images, which is used to describe the relative position and posture between the cameras; Decompose the camera pose from the fundamental matrix, including position and translation; Using the feature points and camera poses in multiple images, the 3D points in the scene are reconstructed through triangulation. The position of each 3D point is calculated by the intersection of rays from different images in space. Initialize the 3D point cloud as a flat Gaussian, and use a pre-trained neural network to predict the scene's geometric information based on the input color image. The geometric information includes a depth map and a normal map, and align the predicted depth map and normal map based on the initial scene point cloud to unify their scales and world coordinate systems. S2, obtain color images, normal maps and depth maps through 3D Gaussian Splatting rendering; S3. Subtract the rendered color image, normal map and depth map from the training true value, predicted normal map and predicted depth map to obtain the loss function L: Among them, γ, δ, α, β represent weights, T is the training iteration round, Threshold is the training strategy threshold round, Lc is the color loss function, LD, LN, Ld, Ln are regular constraints respectively; represents the rendered normal map, Nmono is the predicted normal map; represents the rendered depth map, and Dmono is the predicted depth map; ω i represents the superposition weight of the i-th Gaussian, d i is the corresponding depth value, ω j represents the superposition weight of the jth Gaussian, d j The corresponding depth value; N represents the normal map generated by the depth map gradient; The loss function is minimized by stochastic gradient descent, and the distribution, shape, and color of the training Gaussian are iteratively optimized.

2. The 3D Gaussian-based 3D reconstruction method according to claim 1, characterized in that: The strategies for iterative optimization training include: If the training iteration round is less than or equal to the training strategy threshold round, the low-frequency structural information of the scene is optimized by the calculated loss function, otherwise the high-frequency detail information of the scene is optimized by the calculated loss function.

3. The 3D Gaussian-based 3D reconstruction method according to claim 1, characterized in that: The color images obtained by 3D Gaussian Splatting rendering include: Lighting calculation: The fragment shader is responsible for calculating the color of each pixel according to the lighting model. Lighting calculation usually uses vertex normals, light source positions, and material attribute information; Texture mapping: Mapping a texture image to the surface of a 3D object to provide color information. The fragment shader samples the texture according to the texture coordinates on the surface of the object to determine the final color of each pixel. Color output: The fragment shader outputs the calculated color value to the frame buffer and saves it as a color image.

4. The 3D Gaussian-based 3D reconstruction method according to claim 1, characterized in that: The normal map obtained by 3D Gaussian Splatting rendering specifically includes: Normal interpolation: During the rasterization process, the vertex normals of the triangle are interpolated to each pixel, and the interpolated normal vector is directly used as the output of the fragment shader; Color encoding: The normal is a three-dimensional vector, and the normal vector is mapped to RGB value; Normal map output: The fragment shader outputs the encoded normals as color values ​​to generate a normal map.

5. The 3D Gaussian-based 3D reconstruction method according to claim 1, characterized in that: The depth map obtained by 3D Gaussian Splatting rendering specifically includes: Depth value calculation: Each fragment will calculate its depth value during the rasterization process. This depth value is the depth value after perspective transformation, which indicates the distance of the pixel from the camera. Depth normalization: Normalize the depth value to a range so that it can be stored as a grayscale image; Depth map output: The fragment shader outputs the normalized depth value as grayscale to generate a depth map.

6. The 3D Gaussian-based 3D reconstruction method according to claim 1, characterized in that: Step S1 also includes: optimizing the camera parameters and the positions of the 3D points by using a bundle adjustment method.

7. A computer device comprising a memory storing program instructions, characterized in that: When the program instructions are executed, the 3D Gaussian-based three-dimensional reconstruction method according to any one of claims 1 to 6 is executed.

Images