Three-dimensional model reconstruction method based on 3D Gaussian Splitting

Abstract

The invention relates to a three-dimensional model reconstruction method based on 3D Gaussian Splitting. The method comprises the following steps: S1, collecting images of a scene at multiple angles; s2, processing the collected image, and generating sparse point cloud of a scene and a camera pose; s3, performing three-dimensional model reconstruction on the scene by adopting an improved 3D Gaussian Splitting model, taking the sparse point cloud as the input of the model, training the model, storing a training result, and obtaining the vivid rendering of the scene; and S4, performing visual display on the scene by adopting visual software so that a user can observe the scene at any angle. According to the method, three-dimensional model reconstruction can be accurately and efficiently carried out on the scene.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional reconstruction, and in particular relates to a three-dimensional model reconstruction method based on 3D Gaussians Splatting. Background Art

[0002] 3D reconstruction is a technology in the field of computer vision, which refers to the mathematical process and computer technology that uses 2D images or projections to restore the 3D information (such as shape) of an object. This technology has a wide range of applications in fields such as digital twins, augmented reality, and virtual reality. Traditional 3D reconstruction models rely on discrete explicit representations such as meshes, voxels, and point clouds, which approximate the geometry and appearance of the scene by decomposing the continuous 3D space into small, countable units. The models of the above methods have certain limitations. For example, the mesh model may become very complex when dealing with complex topological structures, the voxel model may lose details due to resolution limitations, and the point cloud model may lack accuracy in data sparse areas.

[0003] In recent years, deep learning has been applied to the field of 3D reconstruction, and Neural Radiance Field (NeRF) is a representative work. It is a continuous implicit expression of 3D scenes. It learns the continuous volume representation of scenes through multi-layer perceptrons (MLP), so as to render realistic and high-quality real scenes. However, NeRF training requires 100 to 300k iterations and includes a large number of neural network operations, which leads to slow rendering speed and limits its application in real-time tasks and high-resolution scenes.

[0004] Recently, 3D Gaussian splatting (3D-GS) has attracted attention as an attractive alternative. 3D Gaussian splatting is a volume rendering technique that represents 3D scenes by using Gaussian distributed ellipsoids, and the rendering effect is improved mainly through anti-aliasing and increased illumination adaptability. Among them, the more representative work is Mip-Splatting, which introduces two new filters, namely 3D smoothing filter and 2DMip filter, to achieve alias-free rendering at any scale. In terms of application field expansion, Deformable 3D Gaussians extends 3D-GS to dynamic scenes. The model uses 3D Gaussian distribution to reconstruct scenes and learns them in a canonical space with deformation fields to model monocular dynamic scenes. Scaffold-GS uses anchor points to control the distribution of 3D Gaussians and uses MLP to predict the properties of 3D Gaussians, which effectively reduces the redundancy of Gaussian points and improves the application ability of 3D-GS models in large-scale scenes. At this stage, 3D-GS and its improved models still have certain shortcomings. First, in complex lighting scenes, if there are not enough photos taken from different viewing angles, 3D-GS and its improved models cannot well represent the changes in the natural gloss of the object surface under different viewing angles. Second, in areas without obvious textures or with fine details, 3D-GS and its improved models cannot accurately capture the geometric structure of the scene, and the reconstructed image has a certain gap compared to the real image. Summary of the invention

[0005] The object of the present invention is to provide a three-dimensional model reconstruction method based on 3D Gaussians Splatting, which is conducive to accurately and efficiently reconstructing the three-dimensional model of a scene.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a three-dimensional model reconstruction method based on 3D Gaussians Splatting, comprising the following steps:

[0007] S1, collect images of the scene from multiple angles;

[0008] S2, processing the collected images to generate a sparse point cloud of the scene and camera pose;

[0009] S3, using the improved 3D Gaussian Splatting model to reconstruct the three-dimensional model of the scene, taking the sparse point cloud as the input of the model, training the model, saving the training results, and obtaining a realistic rendering of the scene;

[0010] S4. Use visualization software to visualize the scene so that users can observe the scene from any angle.

[0011] Furthermore, in step S2, the collected images are processed, including: removing blurred images, naming the images according to a set format; and then using COLMAP software to generate a sparse point cloud and camera pose of the scene.

[0012] Furthermore, in step S3, the improved 3D Gaussian Splatting model is a 3D Gaussian Splatting model based on Scffold-GS; perceptual loss is added during the model training process, and the saved training results include anchor point information and MLP training weights.

[0013] Furthermore, in step S4, the visualization software adopts the open source Scaffold-GS-Viewer.

[0014] Furthermore, the improved 3D Gaussian Splatting model is used to reconstruct the three-dimensional model of the scene, which specifically includes the following steps:

[0015] (1) Scene voxelization: First, the scene is voxelized according to the input sparse point cloud, and the scene is divided into voxel units. The voxelization process is as follows:

[0016]

[0017] in, Represents the voxel center set, there are N voxel centers in the voxel center set; represents a sparse point cloud, where the number of points in the sparse point cloud is M; ∈ is the voxel size; Represents a rounding operation; the sparse point cloud is divided by the voxel size and then rounded, and then the overlapping parts between different voxels are deleted;

[0018] (2) Generate 3D Gaussian. First, consider the center v∈V of each voxel as an anchor point. Each anchor point has learnable content features. Scaling Factor and k learnable content biases For each anchor point, each 3D Gaussian it controls has the following properties: Position Opacity Rotation matrix related to covariance Scaling matrix related to covariance and color

[0019] Secondly, the anchor points are pre-filtered to remove invisible anchor points. The properties of the k 3D Gaussians controlled by the anchor points are generated through their properties. According to the positions of the anchor points, the positions of the k 3D Gaussians controlled by them are calculated as follows:

[0020]

[0021] in, represents the learnable content bias of k 3D Gaussians controlled by anchor points, l v Represents the scaling factor of the anchor point; Next, calculate the observation distance and direction between the anchor point and the camera;

[0022]

[0023] Among them, x v Indicates the position of the anchor point, x c Indicates the position of the camera;

[0024] Then, the observation distance δ vc , Observation direction and learnable content features As the input of MLP, multiple independent MLPs are used to generate k 3D Gaussian attributes, specifically: through opacity MLP, denoted as F α , generating k 3D Gaussian opacities; through color MLP, represented as F c , generate k 3D Gaussian colors; rotation matrix Scaling Matrix are all related to the covariance matrix, so by rotating and scaling the MLP, it is represented as F q , and generate the rotation and scaling of k 3D Gaussians at the same time; the calculation process is as follows:

[0025]

[0026] in, represents the opacity of k 3D Gaussians, represents the colors of k 3D Gaussians, It contains two parts, where the first 4 dimensions represent the rotation of k 3D Gaussians The last three dimensions represent the scaling of k 3D Gaussians To make rasterization more efficient, only opacity values ​​greater than a predefined threshold τ are retained. α Neural Gaussian distribution of

[0027] Represent 3D Gaussian by mean μ and covariance matrix Σ;

[0028]

[0029] Where x is an arbitrary position in the three-dimensional scene, Σ is the covariance matrix of the three-dimensional Gaussian distribution; Σ is calculated using the scale matrix S and the rotation matrix R:

[0030] Σ=RSS TR T (8)

[0031] Among them, S is the scale matrix and R is the rotation matrix;

[0032] (3) Project the 3D Gaussian onto the 2D plane to transform it into a 2D Gaussian. According to the given observation transformation W, the covariance matrix Σ' in the camera coordinates is calculated. The 2D Gaussian distribution on the 2D plane is represented as G' i (x'):

[0033]

[0034] Among them, the Jacobian matrix J b is an approximation of the projective transformation defined by the center of a three-dimensional Gaussian; by skipping the third row and column of Σ', we obtain the two-dimensional covariance matrix in ray space G' i (x') represents the 2D Gaussian distribution of the corresponding 2D pixel, x' is an arbitrary position on the 2D pixel plane, and u' is the result of projecting the mean from the 3D coordinates to the 2D pixel coordinates;

[0035] (4) Rendering the color of the pixel using a method based on fast rasterization; first, according to the method of step (3), the three-dimensional Gaussian is converted into a two-dimensional Gaussian G' on the pixel plane. i (x'); then build a fast rasterizer to sort the 2D Gaussian distribution and then calculate the color as follows:

[0036]

[0037] Among them, C(x') represents the color of the pixel, and the alpha blending technology is used according to the two-dimensional Gaussian G' i (x') and opacity α i Calculate the color of each pixel;

[0038] (5) Adopting the adaptive optimization strategy to optimize the scene; for each voxel, calculate the average gradient of the neural Gaussian distribution it contains after N training iterations, denoted as Then, with Voxels with τ are considered under-rendered. g is a predefined threshold; if there is no anchor point established in the adjacent area of ​​the voxel to which the neural Gaussian belongs, a new anchor point is created around the voxel.

[0039] Furthermore, the scene representation is optimized by a scene representation optimization method based on position coding. The scene representation optimization method based on position coding is based on the sine coding in NeRF, combined with the probability estimation of the three-dimensional Gaussian density in the observation direction based on the vMF distribution, to optimize the appearance representation of the high-density area of ​​the three-dimensional Gaussian points, while reducing the redundancy of the anchor points; the implementation method is as follows: first, the observation direction and distance are encoded by a sine function respectively, and the encoded results are used as the input of the MLP, and two independent MLPs, namely, F α 、F q The opacity, scaling matrix and rotation matrix are calculated. Then, a new MLP is added to predict the Gaussian point density value of the voxel to which the anchor point belongs, and the vMF distribution is calculated through the viewing direction and density value encoded by the spherical harmonic function. Using the vMF distribution, the distribution of the viewing direction is concentrated in the area with large density values ​​to better render the complex texture details of the scene and obtain better reconstruction effects.

[0040] Furthermore, the observation angle and distance are position-encoded respectively, and the calculation method is as follows:

[0041] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp)) (12)

[0042] Among them, the function γ(p) is applied to the observation direction respectively. The three components and the anchor point position coordinates x v ;

[0043] The three-dimensional Gaussian point density probability estimation in the observation direction based on vMF distribution is implemented as follows:

[0044] First, a new MLP, F k Predict the density value of the three-dimensional Gaussian point of the voxel where the anchor point is located; use the anchor point position coordinate x v and learnable content features f v As F k The input of , predicts the density of three-dimensional Gaussian points, the density value represents the concentration parameter k; secondly, the normalized observation direction Perform spherical harmonic position encoding and map it to the unit sphere to obtain the spherical harmonic function Finally, according to the concentration parameters k and Calculate the vMF distribution. The vMF distribution concentrates the distribution of the observation direction more in the area with large density value. The specific process is as follows:

[0045] 1) Use MLP to predict concentration parameters;

[0046] k=F q (f v ,δ vc ) (13)

[0047] Where k represents the three-dimensional Gaussian point density;

[0048] 2) Spherical harmonic position encoding of the observation direction;

[0049] Compute the spherical harmonics of the viewing direction and project them onto the unit sphere:

[0050]

[0051] in, represents spherical harmonics, represents each term of the spherical harmonics, is the order, m is less than or equal to A positive integer of ;

[0052] 3) Calculate vMF distribution;

[0053] The vMF distribution is calculated based on the concentration parameter and the result of spherical harmonic position encoding:

[0054]

[0055] in, represents the vMF distribution, which is distributed according to the observation direction on the unit sphere The expectation is calculated with the concentration parameter κ;

[0056] Calculate the properties of a three-dimensional Gaussian. The implementation method is:

[0057] 1) Perform color prediction;

[0058] First, the 3D Gaussian point density is predicted, and then the observation direction and learnable content features f are encoded with it. v and vMF distribution probability IDE as the input of MLP, through F c Get the color of the 3D Gaussian point at this observation angle;

[0059]

[0060] in, represents the colors of k three-dimensional Gaussians, Indicates viewing direction The probability of vMF distribution under the concentration parameter κ;

[0061] 2) Predict opacity, rotation matrix and scaling matrix;

[0062]

[0063] in, represents the opacity of k three-dimensional Gaussians, represents the rotation of k three-dimensional Gaussians Zoom

[0064] Furthermore, the perceptual loss is introduced into the loss function of 3D reconstruction;

[0065] First, the perceptual loss is improved. The improved perceptual loss is as follows:

[0066]

[0067] Among them, G i represents a real image, R represents a rendered image, F represents the first block of the pre-trained VGG-19 model, and C, W, and H represent the dimensions of the output features;

[0068] Then, the improved perceptual loss is added to the loss function of Scaffold-GS, and the total loss function is as follows:

[0069]

[0070] Among them, k represents the current training round, and iteration represents the total training rounds.

[0071] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a three-dimensional model reconstruction method based on 3D Gaussians Splatting, which reconstructs the three-dimensional model of the scene through an improved 3D Gaussian Splatting model, significantly improves the gloss transformation of the 3D Gaussians Splatting model under complex lighting conditions, more accurately reconstructs the underlying details of the object, and can achieve shorter training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a schematic diagram of a method implementation principle of an embodiment of the present invention;

[0073] Figure 2 is a schematic diagram of a voxelized scene according to an embodiment of the present invention;

[0074] Figure 3 is a schematic diagram of calculating vMF distribution according to an embodiment of the present invention;

[0075] Figure 4 4 is a comparison diagram of reconstruction effects of an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0077] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0078] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0079] like Figure 1 As shown, this embodiment provides a 3D model reconstruction method based on 3D Gaussians Splatting, comprising the following steps:

[0080] S1. Image acquisition: collect high-definition images of the scene from multiple angles.

[0081] First, use mobile phones, cameras and other video equipment to shoot the scene on the spot, and obtain several photos of the scene from various angles. The overlap rate between the photos should be low, and the shooting distance from the center of the scene should be the same. The more comprehensive the shooting angle, the better; or use video screenshot software to capture the video data and obtain high-definition pictures of the appropriate scene from various angles.

[0082] S2, 3D point cloud initialization; process the collected images to generate sparse point clouds and camera poses of the scene.

[0083] First, the collected images are processed to remove blurred images and named in the format of 00000 to 99999, ensuring that each image name is a four-digit number and filled with 0s to the required number of digits. Second, the images are used as input to generate the sparse point cloud and camera pose of the scene using the SFM (motion pose estimation) based software COLMAP.

[0084] S3. Use the improved 3D Gaussian Splatting model to reconstruct the three-dimensional model of the scene, use the sparse point cloud as the input of the model, train the model, save the training results, and obtain a realistic rendering of the scene.

[0085] The improved 3D Gaussian Splatting model is based on Scffold-GS (the latest improvement of the 3D-GS model); perceptual loss is added during model training, and the saved training results include anchor point information and MLP training weights.

[0086] S4. Use visualization software to display the scene in 3D so that users can observe the scene from any angle.

[0087] In this embodiment, the visualization software uses the open source Scaffold-GS-Viewer. The visualization software is developed based on the SIBR framework. The SIBR framework was developed by the GRAPHDECO group for the field of novel views. The framework can not only view the final rendering effect, but also view the rendering process of the scene in real time during training.

[0088] Use the saved point cloud information (ply file) and MLP weights (pt file) as input, and use the open source Scaffold-GS-Viewer to view the reconstruction effect of the scene.

[0089] This method uses the generated sparse point cloud and adopts the improved 3D Gaussians Splatting model to perform 3D reconstruction of the scene. After training 30K rounds, the 3D reconstruction model of the scene is obtained. Figure 1 As shown, the basic process of 3D reconstruction is as follows:

[0090] (1) Stage 1: scene voxelization.

[0091] First, the scene is voxelized based on the input sparse point cloud, and the scene is divided into voxel units. Each voxel unit can contain information about its position, density or other properties. Figure 2 As shown in Figure 2, the voxelization process is as follows:

[0092]

[0093] in, Represents the voxel center set, there are N voxel centers in the voxel center set; represents a sparse point cloud, where the number of points in the sparse point cloud is M; ∈ is the voxel size; Denotes a rounding operation. The sparse point cloud is divided by the voxel size and then rounded. If the positions of the sparse point clouds are very close, many overlapping voxels will be generated. Therefore, the overlapping parts between different voxels are deleted and represented by {·} to reduce the redundancy and irregularity in P.

[0094] (2) The second stage: generating three-dimensional Gaussians.

[0095] First, the center v∈V of each voxel is regarded as an anchor point, each of which has learnable content features. Scaling Factor and k learnable content biases For each anchor point, each 3D Gaussian it controls has the following properties: Position Opacity Rotation matrix related to covariance Scaling matrix related to covariance and color

[0096] Secondly, the anchor points are pre-filtered to remove the invisible anchor points. The properties of the k 3D Gaussians controlled by the anchor points are generated through the properties of the above anchor points. Specifically, according to the position of the anchor point, the positions of the k 3D Gaussians controlled by it are calculated as follows:

[0097]

[0098] in, represents the learnable content bias of k 3D Gaussians controlled by anchor points, l v Represents the scaling factor of the anchor point. Next, the observed distance and direction between the anchor point and the camera are calculated.

[0099]

[0100] Among them, x v Indicates the position of the anchor point, x c Indicates the position of the camera.

[0101] Then, the observation distance δ vc , Observation direction and learnable content features As the input of MLP, multiple independent MLPs are used to generate k 3D Gaussian attributes, specifically: through opacity MLP, denoted as F α , generating k 3D Gaussian opacities; through color MLP, represented as F c , generate k 3D Gaussian colors; rotation matrix Scaling Matrix are all related to the covariance matrix, so by rotating and scaling the MLP, it is represented as F q , and generate the rotation and scaling of k 3D Gaussians at the same time. The calculation process is as follows:

[0102]

[0103] in, represents the opacity of k 3D Gaussians, represents the colors of k 3D Gaussians, It contains two parts, where the first 4 dimensions represent the rotation of k 3D Gaussians The last three dimensions represent the scaling of k 3D Gaussians To make rasterization more efficient, only opacity values ​​greater than a predefined threshold τ are retained. α This greatly reduces the computational load and helps the method maintain the same high rendering speed as the original 3D-GS.

[0104] Secondly, a 3D Gaussian is represented by the mean μ and the covariance matrix Σ.

[0105]

[0106] Where x is an arbitrary position in the 3D scene, and Σ is the covariance matrix of the 3D Gaussian distribution. Σ is calculated using the scale matrix S and the rotation matrix R:

[0107] Σ=RSS T R T (8)

[0108] Among them, S is the scale matrix, R is the rotation matrix, and Σ is the covariance matrix of the three-dimensional Gaussian distribution.

[0109] (3) The third stage: project the 3D Gaussian onto the 2D plane to transform it into a 2D Gaussian.

[0110] According to the given observation transformation W, the covariance matrix Σ' in the camera coordinates is calculated, and the two-dimensional Gaussian distribution on the 2D plane is represented as G' i (x'):

[0111]

[0112] Among them, the Jacobian matrix J b is an approximation of the projective transformation defined by the center of a 3D Gaussian. By skipping the third row and column of Σ', we obtain the 2D covariance matrix in ray space G' i (x') represents the two-dimensional Gaussian distribution of the corresponding two-dimensional pixel, x' is an arbitrary position on the two-dimensional pixel plane, and u' is the result of projecting the mean from the three-dimensional coordinates to the two-dimensional pixel coordinates.

[0113] (4) The fourth stage: Rendering pixel colors using a tile-based rasterizer method.

[0114] 3D-GS efficiently renders the scene based on fast rasterization. First, the three-dimensional Gaussian is converted into a two-dimensional Gaussian G' on the pixel plane according to the projection process described in the third stage.i (x'), represented by the covariance matrix Σ'. A fast rasterizer is then constructed to efficiently sort the 2D Gaussian distribution and then calculate the color, which is calculated as follows:

[0115]

[0116] Among them, C(x') represents the color of the pixel, and the alpha blending technology is used according to the two-dimensional Gaussian G' i (x') and opacity α i Calculate the color of each pixel.

[0117] (5) The fifth stage: Adopting adaptive optimization strategy to optimize the scene.

[0118] COLMAP generates fewer points in texture-less and less observed areas, while Scaffold-GS's anchors are calculated from these points. This poses a challenge to the positional diversity of anchors, especially in texture-less and less observed areas. This method proposes an adaptive optimization strategy to discover significant new anchors for the Neural Gaussian. Specifically, for each voxel, the average gradient of the Neural Gaussian distribution it contains over N training iterations is calculated, denoted as Then, with Voxels with τ are considered under-rendered. g is a predefined threshold; if there is no anchor point established in the adjacent area of ​​the voxel to which the neural Gaussian belongs, a new anchor point will be newly created around the voxel.

[0119] In this invention, a scene representation optimization method based on position coding is further proposed to optimize the scene representation. This method is based on sinusoidal coding in Neural Radiance Field (NeRF) and combined with the probability estimation of 3D Gaussian density in the viewing direction based on von Mises-Fisher (vMF) distribution to optimize the appearance representation of areas with high density of 3D Gaussian points and reduce the redundancy of anchor points. In addition, a perceptual loss function is further introduced to better render the high-frequency details of the scene.

[0120] The process of the point optimization method based on position encoding and the improved loss function are as follows:

[0121] 1) Optimization of scene representation of position encoding

[0122] First, a sine function is used to encode the viewing direction and distance respectively, and the encoded results are used as the input of the MLP. α 、F q) calculates the opacity, scaling matrix, and rotation matrix. Then, a new MLP is added to predict the Gaussian point density value of the voxel to which the anchor point belongs, and the vMF distribution is calculated by the observation direction and density value encoded by the spherical harmonic function. If the Gaussian point density within the anchor box is high, it means that the area where the voxel to which the anchor box belongs is under-rendered or has complex textures. Using the vMF distribution, the distribution of the observation direction is concentrated in the area with large density values ​​to better render the complex texture details of the scene and obtain better reconstruction effects.

[0123] [1] Perform sinusoidal position encoding

[0124] Position encoding is a technique that converts the original low-dimensional spatial position into a higher-dimensional vector representation. This method uses a sine function for position encoding to avoid the over-smoothing problem of spatially adjacent sampling points in the MLP representation. This method is introduced into 3D-GS, and the observation angle and distance are position encoded separately, specifically:

[0125] γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 πp)) (12)

[0126] Among them, this function γ(p) is applied to the observation direction The three components (normalized to a value range of [-1,1]) and the anchor point position coordinate x v (After normalization, the value range is [-1, 1]). In this embodiment, set L = 3, γ(x v )’s L=1.

[0127] [2] Probability estimation of three-dimensional Gaussian density in the viewing direction based on von Mises-Fisher (vMF) distribution

[0128] The von Mises-Fisher (vMF) distribution is a continuous probability distribution defined on a unit hypersphere that describes randomness on directional data or unit vectors. The probability density function of the VMF distribution is usually expressed as the concentration of a point on the unit hypersphere relative to the mean direction (or "center" direction), and this concentration is controlled by a concentration parameter (usually denoted κ). As κ increases, the distribution becomes more concentrated in the mean direction; when κ approaches 0, the distribution approaches a uniform distribution on the unit hypersphere.

[0129] The following specifically describes how to optimize the appearance representation of areas with high 3D Gaussian point density based on the probability estimation of the von Mises-Fisher (vMF) distribution in the observation direction.

[0130] First, a new MLP (called F k ) predicts the density value of the three-dimensional Gaussian point of the voxel where the anchor point is located. Because the density is independent of the viewing direction, the anchor point position coordinate x is used v and learnable content features f v As F k The input of , predicts the density of three-dimensional Gaussian points, and the density value represents the concentration parameter k. Secondly, for the normalized observation direction Perform spherical harmonic position encoding and map it to the unit sphere to obtain the spherical harmonic function Finally, according to the concentration parameters k and Calculate the vMF distribution. The vMF distribution will concentrate the distribution of the viewing direction more in the area with large density value (there is under-rendering or complex texture). The specific process is as follows:

[0131] ①Use MLP to predict concentration parameters

[0132] k=F q (f v ,δ vc ) (13)

[0133] Wherein, k represents the three-dimensional Gaussian point density. In this embodiment, F q It is an MLP with 2 layers, where the hidden layer dimension is 32 and uses the Softamx activation function as the output.

[0134] ②Spherical harmonic position encoding of the observation direction

[0135] like Figure 3 As shown, the spherical harmonics of the viewing direction are calculated and mapped onto the unit sphere:

[0136]

[0137] in, represents spherical harmonics, represents each term of the spherical harmonics, is the order, m is less than or equal to A positive integer.

[0138] ③Calculate vMF distribution

[0139] The vMF distribution is calculated based on the concentration parameter and the result of spherical harmonic position encoding:

[0140]

[0141] in, represents the vMF distribution, which is based on the observation direction distribution on the unit sphere The expectation is calculated with the concentration parameter κ.

[0142] [3] The above improvements are used as input to MLP to calculate the properties of the three-dimensional Gaussian

[0143] ①Color prediction;

[0144] First, the three-dimensional Gaussian point density is predicted by sampling formula (12), and then the observation direction and learnable content features f are encoded by it. v and vMF distribution probability IDE as the input of MLP, through F c Get the color of the 3D Gaussian point at this observation angle.

[0145]

[0146] in, represents the colors of k three-dimensional Gaussians, Indicates viewing direction The probability of vMF distribution under the concentration parameter κ. In addition, the expected value of any spherical harmonic under the vMF distribution has the following simple convergence expression:

[0147]

[0148] Among them, the exponential decay function This can be well approximated by a simple exponential function:

[0149]

[0150] ② Opacity, rotation matrix and scaling matrix prediction;

[0151]

[0152] in, represents the opacity of k three-dimensional Gaussians, represents the rotation of k three-dimensional Gaussians Zoom Different from Scaffold-GS, this method adopts Alternative As the input feature of MLP, it avoids the over-smoothing problem of adjacent observation angles in the MLP representation and better renders the details of the scene.

[0153] 2) Perceptual loss function

[0154] In this invention, perceptual loss is introduced into the loss function of 3D reconstruction, and a new loss function is designed. Generally, L1 loss and SSIM are usually used to supervise the training process of 3D reconstruction, and the loss function of Scaffold-GS introduces volume regularization on this basis. However, L1 loss will cause blurring problems when rendering fine details. Therefore, this method better renders the high-frequency details of objects by adding perceptual loss.

[0155] Perceptual Loss is a commonly used loss function in deep learning-based image processing methods. It is mainly used to measure and optimize the quality of images at the perceptual level, rather than simply based on pixel-level differences. Generally, a pre-trained neural network (such as VGG-19) is used to extract features from the input image and the target image, and a certain distance metric is calculated between them to measure the difference between the input image and the target image at the perceptual level.

[0156] Directly using the common perceptual loss function will lead to distortion of the training image and increase the training time. Therefore, instead of using the original perceptual loss function, only the output of the first block before the maximum pooling is used as the loss. At the same time, since the training image of VGG-19 is 224*224, the recognizable pixels are relatively low, so as the number of training rounds increases, its effect will gradually weaken and lead to redundancy of Gaussian points. Therefore, we set a weight for the perceptual loss, and its weight decays linearly with the number of training rounds. The improved perceptual loss is as follows:

[0157]

[0158] Among them, G i represents the real image, R represents the rendered image, F represents the first block of the pre-trained VGG-19 model, and C, W, and H represent the dimensions of the output features.

[0159] Then, the improved perceptual loss is added to the loss function of Scaffold-GS, and the total loss function is as follows:

[0160]

[0161] Among them, k represents the current training round, and iteration represents the total training rounds.

[0162] The technical solution of the present invention is further described below in conjunction with a specific embodiment.

[0163] This embodiment selects the Mip-NeRF360 dataset as a benchmark. This dataset is designed to simulate unbounded scenes in the real world and is currently the most challenging dataset in the field of three-dimensional reconstruction. Similar to 3D-GS and its improved models, this embodiment was tested in the seven scenes it captured, including bicycles, gardens, shrubs, rooms, counters, kitchens, and bonsai. This embodiment uses the high-quality RGB images provided by the dataset, generates a 3D point cloud of the scene through COLMAP initialization, and then uses the improved 3D Gaussians Splatting model to train 30K rounds of iterations to obtain a high-quality three-dimensional model of the scene, evaluate the effect of the model on the test picture, and finally use 3D scene visualization software to visualize the scene.

[0164] The implementation process of this embodiment is as follows:

[0165] 1. Image acquisition

[0166] The images provided by the Mip-NeRF360 dataset are directly used. The images in this dataset are high-definition pictures of scenes taken by cameras at multiple angles.

[0167] 2. 3D point cloud initialization

[0168] Execute the test script, which traverses the dataset, takes all images of a scene as input each time, and uses COLMAP to generate the sparse point cloud and camera pose of the scene.

[0169] 3. Reconstruction using the improved 3D Gaussian Splatting model

[0170] The sparse point cloud and camera pose are used as input, and the improved 3D Gaussians Splatting model is used to train for 30K rounds and save the training results. The test set provided by Mip-NeRF360 is used for testing to evaluate the three indicators of peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and learning-perceptual image similarity (LPIPS).

[0171] 4. 3D scene visualization

[0172] The point cloud information (ply file) and MLP weights (pt file) saved during training are used as input, and the reconstructed scene is viewed using the open source visualization software Scaffold-GS-Viewer.

[0173] The algorithm efficiency and reconstruction effect analysis of this embodiment are as follows:

[0174] First, we compared the performance of this method model with the Scaffold-GS model on the Mip-NeRF360 dataset, as shown in Table 1 below.

[0175] Table 1

[0176]

[0177]

[0178] From the data in Table 1, we can see that compared with the Scaffold-GS model, the PSNR value of this method is improved by about 0.44, and the SSIM value is improved by 0.008. At the same time, we reduced the LPIPS value by 12%, indicating that the image predicted by this method has a higher perceptual similarity with the real image. Figure 4 As shown in the figure, in the bonsai and room scenes with significant lighting changes and no texture, the rendering effect of this method is excellent.

[0179] Furthermore, to demonstrate the effectiveness of this method, this method is compared with the most advanced models, including 3D-GS, Scaffold-GS, Mip-NeRF360, iNGP, and Plenxels, on real datasets. In the experiment, this method and the comparison method used the same training parameters and iterated 30K rounds, which better demonstrated the effectiveness of our method. The results are shown in Table 2 below.

[0180] Table 2

[0181]

[0182] From the data in Table 2, we can see that compared with the most advanced methods, this method has achieved the best results in various indicators.

[0183] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0184] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0185] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0187] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A three-dimensional model reconstruction method based on 3D Gaussians Splatting, characterized in that: The following steps are involved: S1, collect images of the scene from multiple angles; S2, processing the collected images to generate a sparse point cloud of the scene and camera pose; S3, using the improved 3D Gaussian Splatting model to reconstruct the three-dimensional model of the scene, taking the sparse point cloud as the input of the model, training the model, saving the training results, and obtaining a realistic rendering of the scene; S4. Use visualization software to visualize the scene so that users can observe the scene from any angle.

2. The three-dimensional model reconstruction method based on 3D Gaussians Splatting according to claim 1, characterized in that: In step S2, the collected images are processed, including: removing blurred images and naming the images according to a set format; and then using COLMAP software to generate a sparse point cloud and camera pose of the scene.

3. The three-dimensional model reconstruction method based on 3D Gaussians Splatting according to claim 1, characterized in that: In step S3, the improved 3D Gaussian Splatting model is a 3D Gaussian Splatting model based on Scffold-GS; perceptual loss is added during the model training process, and the saved training results include anchor point information and MLP training weights.

4. The three-dimensional model reconstruction method based on 3D Gaussians Splatting according to claim 1, characterized in that: In step S4, the visualization software uses the open source Scaffold-GS-Viewer.

5. The three-dimensional model reconstruction method based on 3D Gaussians Splatting according to claim 1, characterized in that: The improved 3D Gaussian Splatting model is used to reconstruct the three-dimensional model of the scene, which specifically includes the following steps: (1) Scene voxelization: First, the scene is voxelized according to the input sparse point cloud, and the scene is divided into voxel units. The voxelization process is as follows: in, Represents the voxel center set, there are N voxel centers in the voxel center set; represents a sparse point cloud, where the number of points in the sparse point cloud is M; ∈ is the voxel size; Represents a rounding operation; the sparse point cloud is divided by the voxel size and then rounded, and then the overlapping parts between different voxels are deleted; (2) Generate 3D Gaussian. First, consider the center v∈V of each voxel as an anchor point. Each anchor point has learnable content features. Scaling Factor and k learnable content biases For each anchor point, each 3D Gaussian it controls has the following properties: Position Opacity Rotation matrix related to covariance Scaling matrix related to covariance and color Secondly, the anchor points are pre-filtered to remove invisible anchor points. The properties of the k 3D Gaussians controlled by the anchor points are generated through their properties. According to the positions of the anchor points, the positions of the k 3D Gaussians controlled by them are calculated as follows: in, represents the learnable content bias of k 3D Gaussians controlled by the anchor point, and lv represents the scaling factor of the anchor point; then, the observation distance and direction between the anchor point and the camera are calculated; Among them, xv represents the position of the anchor point, and xc represents the position of the camera; Then, the observation distance δvc, the observation direction and learnable content features As the input of MLP, multiple independent MLPs are used to generate the properties of k 3D Gaussians, specifically: the opacity of k 3D Gaussians is generated through the opacity MLP, denoted as Fα; the color of k 3D Gaussians is generated through the color MLP, denoted as Fc; the rotation matrix Scaling Matrix All are related to the covariance matrix, so by rotating and scaling the MLP, denoted as Fq, k 3D Gaussian rotations and scalings are generated at the same time; the calculation process is as follows: in, represents the opacity of k 3D Gaussians, represents the colors of k 3D Gaussians, It contains two parts, where the first 4 dimensions represent the rotation of k 3D Gaussians The last three dimensions represent the scaling of k 3D Gaussians To make rasterization more efficient, only neural Gaussian distributions with opacity values ​​greater than a predefined threshold τα are retained; Represent 3D Gaussian by mean μ and covariance matrix Σ; Where x is an arbitrary position in the three-dimensional scene, Σ is the covariance matrix of the three-dimensional Gaussian distribution; Σ is calculated using the scale matrix S and the rotation matrix R: Σ=RSSTRT (8) Among them, S is the scale matrix and R is the rotation matrix; (3) Project the 3D Gaussian onto the 2D plane to transform it into a 2D Gaussian. According to the given observation transformation W, the covariance matrix Σ' in the camera coordinates is calculated. The 2D Gaussian distribution on the 2D plane is represented as G' i (x'): where the Jacobian matrix Jb is an approximation of the projective transformation defined by the center of the three-dimensional Gaussian; by skipping the third row and column of Σ', we get the two-dimensional covariance matrix in ray space G' i (x') represents the 2D Gaussian distribution of the corresponding 2D pixel, x' is an arbitrary position on the 2D pixel plane, and u' is the result of projecting the mean from the 3D coordinates to the 2D pixel coordinates; (4) Rendering the color of the pixel using a method based on fast rasterization; first, according to the method of step (3), the three-dimensional Gaussian is converted into a two-dimensional Gaussian G' on the pixel plane. i (x'); then build a fast rasterizer to sort the 2D Gaussian distribution and then calculate the color as follows: Among them, C(x') represents the color of the pixel, and the alpha blending technology is used according to the two-dimensional Gaussian G' i (x') and opacity αi to calculate the color of each pixel; (5) Adopting the adaptive optimization strategy to optimize the scene; for each voxel, calculate the average gradient of the neural Gaussian distribution it contains after N training iterations, denoted as Then, with The voxel with τg is considered to be under-rendered, where τg is a predefined threshold; if there is no anchor point established in the adjacent area of ​​the voxel to which the neural Gaussian belongs, a new anchor point is created around the voxel.

6. The three-dimensional model reconstruction method based on 3D Gaussians Splatting according to claim 5, characterized in that: The scene representation is optimized by a scene representation optimization method based on position coding. The scene representation optimization method based on position coding is based on sinusoidal coding in NeRF, combined with the probability estimation of three-dimensional Gaussian density in the observation direction based on vMF distribution, to optimize the appearance representation of the high-density area of ​​three-dimensional Gaussian points, while reducing the redundancy of anchor points. The implementation method is as follows: first, the observation direction and distance are encoded respectively by a sinusoidal function, and the encoded results are used as the input of MLP. The opacity, scaling matrix and rotation matrix are calculated by two independent MLPs, namely Fα and Fq; then, a new MLP is added to predict the Gaussian point density value of the voxel to which the anchor point belongs, and the vMF distribution is calculated by the observation direction and density value encoded by spherical harmonics; using the vMF distribution, the distribution of the observation direction is concentrated in the area with large density value, so as to better render the complex texture details of the scene and obtain a better reconstruction effect.

7. The three-dimensional model reconstruction method based on 3D Gaussians Splatting according to claim 6, characterized in that: The observation angle and distance are respectively encoded in position, and the calculation method is: γ(p)=(sin(2 0 πp),cos(2 0 πp),…,sin(2 L-1 πp),cos(2 L-1 (12) Among them, the function γ(p) is applied to the observation direction The three components of and the anchor point position coordinates xv; The three-dimensional Gaussian point density probability estimation in the observation direction based on vMF distribution is implemented as follows: First, a new MLP, Fk, is used to predict the density value of the 3D Gaussian point of the voxel where the anchor point is located. The anchor point position coordinates xv and the learnable content features fv are used as the input of Fk to predict the density of the 3D Gaussian point, which represents the concentration parameter k. Secondly, the normalized observation direction Perform spherical harmonic position encoding and map it to the unit sphere to obtain the spherical harmonic function Finally, according to the concentration parameters k and Calculate the vMF distribution. The vMF distribution concentrates the distribution of the observation direction more in the area with large density value. The specific process is as follows: 1) Use MLP to predict concentration parameters; k=Fq(fv,δvc) (13) Where k represents the three-dimensional Gaussian point density; 2) Spherical harmonic position encoding of the observation direction; Compute the spherical harmonics of the viewing direction and project them onto the unit sphere: in, represents spherical harmonics, represents each term of the spherical harmonic function, l is the order, and m is a positive integer less than or equal to l; 3) Calculate vMF distribution; The vMF distribution is calculated based on the concentration parameter and the result of spherical harmonic position encoding: in, represents the vMF distribution, which is distributed according to the observation direction on the unit sphere The expectation is calculated with the concentration parameter κ; Calculate the properties of a three-dimensional Gaussian. The implementation method is: 1) Perform color prediction; First, the 3D Gaussian point density is predicted, and then the observation direction after position encoding, the learnable content feature fv and the vMF distribution probability IDE are used as the input of MLP. c Get the color of the 3D Gaussian point at the observation angle; in, represents the colors of k three-dimensional Gaussians, Indicates viewing direction The probability of vMF distribution under the concentration parameter κ; 2) Predict opacity, rotation matrix and scaling matrix; in, represents the opacity of k three-dimensional Gaussians, represents the rotation of k three-dimensional Gaussians Zoom 8. The three-dimensional model reconstruction method based on 3D Gaussians Splatting according to claim 5, characterized in that: Introducing perceptual loss into the loss function of 3D reconstruction; First, the perceptual loss is improved. The improved perceptual loss is as follows: Among them, Gi represents the real image, R represents the rendered image, F represents the first block of the pre-trained VGG-19 model, and C, W, and H represent the dimensions of the output features; Then, the improved perceptual loss is added to the loss function of Scaffold-GS, and the total loss function is as follows: Among them, k represents the current training round, and iteration represents the total training rounds.

Images