A Gaussian sputtering surface reconstruction method based on anisotropic opacity

Abstract

The present invention discloses a Gaussian sputtering surface reconstruction method based on anisotropic opacity. First, the opacity of the Gaussian volume is represented by spherical harmonic functions to make the opacity related to the viewing direction. Second, when performing sputtering rendering, the maximum opacity of the Gaussian volume in each viewing direction is used to calculate the depth transmittance, and the depth of the last Gaussian volume that satisfies the condition that the depth transmittance is greater than a certain value is selected as the surface depth. Finally, the depths from different viewpoints are fused to obtain the reconstructed result of the scene surface. The present invention is applicable to the surface reconstruction of Gaussian sputtering and can significantly improve the reconstruction accuracy of the specular surface while maintaining the reconstruction accuracy of the diffuse surface.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a Gaussian sputtering surface reconstruction method based on anisotropic opacity. Background Art

[0002] In the field of three-dimensional reconstruction, how to reconstruct the light field and surface of a scene based on multi-view images of the scene and their corresponding camera poses is a research topic with great academic and application value. Three-dimensional Gaussian sputtering ( 3D Gaussian Splatting for Real-Time Radiance Field Rendering , ACM Trans. Graph. 42(4)) has been widely used in the industry for its high-quality light field reconstruction and efficient training and inference. However, the reconstruction accuracy of the three-dimensional Gaussian sputtering for the scene surface is relatively low. To address this problem, SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (CVPR2024), 2D Gaussian Splatting for Geometrically Accurate Radiance Fields (SIGGRAPH 2024), High-quality Surface Reconstruction using Gaussian Surfels (SIGGRAPH 2024) have proposed different solutions and achieved good results.

[0003] However, although these methods can achieve good reconstruction results on the diffuse surface of objects, serious distortion still occurs in the reconstruction of the specular surface. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the purpose of the present invention is to provide a Gaussian sputtering surface reconstruction method based on anisotropic opacity for the distortion phenomenon that easily occurs when reconstructing the specular surface in a scene by three-dimensional Gaussian sputtering and various improvement works to improve its surface reconstruction accuracy.

[0005] The purpose of the present invention is achieved by the following technical solutions:

[0006] The first aspect of the present invention: A Gaussian sputtering surface reconstruction method based on anisotropic opacity, comprising the following steps:

[0007] (1) Given multi-view images of a scene and their corresponding camera poses, initialize the Gaussian sputtering model;

[0008] (2) Optimize the attributes of the Gaussian volume with a loss function to fit the light field and surface of the scene;

[0009] (3) Fuse the scene surface depths from multiple views to obtain a mesh representation of the scene as the surface reconstruction result of the scene.

[0010] Specifically, the Gaussian sputtering model is a collection of a series of Gaussian bodies, and the attributes of each Gaussian body include: center position, opacity, color, covariance, and normal vector; in the Gaussian sputtering model, the opacity of the Gaussian body is anisotropic and is represented by order spherical harmonic functions plus a sigmoid activation function, and its expression is as follows: ;

[0011] where are spherical harmonic coefficients, are spherical harmonic basis functions in a fixed form, is a normalized direction vector.

[0012] Specifically, the expression of the loss function is as follows:

[0013] ;

[0014] where is the photometric consistency loss that measures the difference between the color rendering value and the true color value and is used for the Gaussian sputtering model to fit the scene light field; is the normal vector consistency loss and is used for the Gaussian sputtering model to fit the scene surface; is a hyperparameter.

[0015] Further, the color rendering value is obtained through the following process:

[0016] Given a camera, assuming that the z-axis points forward, in the current camera coordinate system, for pixel , let represent the index set of all Gaussian bodies that can project onto this pixel. If the depth (z coordinate) of the center of the Gaussian body with index satisfies , then the color rendering value corresponding to this pixel is: ;

[0017] ;

[0018] where is the normalized direction vector of the Gaussian body from its center to the current camera optical center, is the anisotropic color of the Gaussian body, represents the rendering weight of the -th Gaussian body at pixel , is the anisotropic color of the Gaussian body, is the projection of the Gaussian body on the imaging plane; is the -th Gaussian body at pixel Light field transmittance.

[0019] Specifically, the anisotropic color of the Gaussian body is represented by the spherical harmonic function of order

[0020] ;

[0021] where are the spherical harmonic coefficients, are the spherical harmonic basis functions in a fixed form, is the normalized direction vector.

[0022] Specifically, the normal vector consistency loss is obtained by the following formula:

[0023] ;

[0024] where represents the rendering weight of the -th Gaussian body at pixel , is the "pseudo - normal vector" of the scene surface. If the scene surface coordinates determined by the scene surface depth , and ( , are adjacent pixels of , and respectively, then , where represents the vector cross - product.

[0025] Further, the scene surface depth is obtained through the following process:

[0026] Given a camera, assuming the z - axis points forward, in the current camera coordinate system, for pixel , let represent the index set of all Gaussian bodies that can project onto this pixel. If the depth (z - coordinate) of the center of the Gaussian body with index satisfies , then the scene surface depth corresponding to this pixel is:

[0027] ;

[0028] where is the depth of the -th Gaussian body at pixel , , generally taking , is the depth transmittance of the th Gaussian body at pixel , defined as:

[0029] ;

[0030] where is the projection of the Gaussian body on the imaging plane, is the maximum opacity of the Gaussian body in each line-of-sight direction, expressed as:

[0031] ;

[0032] where is the normalized direction vector, is the anisotropic opacity of the Gaussian body, represents the set of normalized direction vectors from the center of the current Gaussian body to the optical centers of all cameras used in training.

[0033] The second aspect of the present invention: discloses a Gaussian sputtering surface reconstruction device based on anisotropic opacity, including the following modules:

[0034] Initialization model module: Given the multi-view images of the scene and their corresponding camera poses, initialize the Gaussian sputtering model;

[0035] Optimization fitting module: Use the loss function to optimize the attributes of the Gaussian body to fit the light field and surface of the scene;

[0036] Fusion reconstruction module: Fusion the surface depths of multiple views to obtain the mesh representation of the scene as the surface reconstruction result of the scene.

[0037] The third aspect of the present invention: An electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the Gaussian sputtering surface reconstruction method based on anisotropic opacity described above.

[0038] The fourth aspect of the present invention: A computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the Gaussian sputtering surface reconstruction method based on anisotropic opacity described above are implemented.

[0039] The beneficial effects of the present invention are as follows:

[0040] The depth acquisition of the present invention can be synchronized with light field rendering, and the obtained depth is consistent under different views; the present invention can significantly improve the reconstruction accuracy of the glossy surface while maintaining the reconstruction accuracy of the diffuse surface. Brief Description of the Drawings

[0041] 图1 It is a comparison diagram of the surface reconstruction effects of the present invention and other existing methods in a real scenario. Detailed Description of the Invention

[0042] Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention. The embodiments of the present invention are as follows.

[0043] Given the multi-view images of a scene and their corresponding camera poses, Gaussian splatting will optimize a series of Gaussian volumes through differentiable rendering to fit the light field and surface of the scene.

[0044] The parameters that can be optimized for the Gaussian volume include: center position, covariance, normal vector , and the - order spherical harmonic coefficients representing color: , and the - order spherical harmonic coefficients representing opacity: .

[0045] Given the normalized direction vector from the center of the Gaussian volume to the optical center of the current camera , the anisotropic color of the Gaussian volume is expressed as:

[0046] ;

[0047] where , is a spherical harmonic basis function in a fixed form, and relu is an activation function.

[0048] The anisotropic opacity of the Gaussian volume is expressed as:

[0049] ;

[0050] where , is a spherical harmonic basis function in a fixed form, is an activation function.

[0051] Given a camera, assuming the z - axis is forward, in the current camera coordinate system, all Gaussian volumes are sorted in ascending order according to the depth (z - coordinate) of their centers. For a pixel , let represent the set of indices of all Gaussian volumes that can project onto this pixel. If the depth (z - coordinate) of the center of the Gaussian volume with index satisfies , then the color of the pixel is given by the following expression:

[0052] ;

[0053] where represents the rendering weight of the -th Gaussian volume at pixel . is the projection of the Gaussian volume on the imaging plane, is the -th Gaussian volume's light field transmittance at pixel .

[0054] The scene surface normal vector corresponding to pixel is:

[0055] ;

[0056] The scene surface depth corresponding to pixel is:

[0057] ;

[0058] where is the depth of the -th Gaussian volume at pixel , , generally taking , is the depth transmittance of the -th Gaussian volume at pixel , defined as:

[0059] ;

[0060] where is the maximum opacity of the Gaussian volume in each line - of - sight direction, expressed as:

[0061] ;

[0062] where represents the set of normalized direction vectors from the center of the current Gaussian volume to the optical centers of all cameras used for training.

[0063] During the training process, the maximum opacity of the Gaussian volume remains unchanged in each epoch (going through the training set once) and is updated after the end of the current epoch.

[0064] During the training process, the parameters of the Gaussian volume are optimized by constructing the following loss function:

[0065] ;

[0066] Among them is the photometric consistency loss for measuring the difference between the color and the true color value, is a hyperparameter, is the normal vector consistency loss, that is

[0067] ;

[0068] Among them is the "pseudo-normal vector" of the scene surface. If the scene surface depth is determined by , and ( , is of the adjacent pixels), the determined scene surface coordinates are respectively , and , then , among which represents the vector cross product.

[0069] During the training process, an adaptive density control strategy consistent with three-dimensional Gaussian sputtering is adopted to adjust the spatial distribution of the Gaussian volume; a photometric consistency loss consistent with three-dimensional Gaussian sputtering is adopted to fit the light field of the scene.

[0070] After the training is completed, the depth maps rendered by each camera used in the training are fused to obtain a mesh representation of the scene as the surface reconstruction result of the scene.

[0071] Through the scene surface reconstruction result obtained in this embodiment, compared with other existing methods, the reconstruction accuracy of the glossy surface can be significantly improved, and the reconstruction accuracy of the diffuse surface can be maintained.

[0072] From 图1 the comparison diagram of the surface reconstruction effects of the present invention and other existing methods in the real scene shown, it can be seen that the present invention can significantly improve the reconstruction accuracy of the glossy surface while maintaining the reconstruction accuracy of the diffuse surface. The other existing methods in the figure include: 3D Gaussian Splatting for Real-Time Radiance Field Rendering , SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering , 2D Gaussian Splatting for Geometrically Accurate Radiance Fields , High-quality Surface Reconstruction using Gaussian Surfels .

[0073] The present invention also discloses a Gaussian sputtering surface reconstruction device based on anisotropic opacity, including the following modules:

[0074] Initialization model module: Given the multi-view images of a scene and their corresponding camera poses, initialize the Gaussian splatting model; the Gaussian splatting model is a collection of a series of Gaussian volumes, and the attributes of each Gaussian volume include: center position, opacity, color, covariance, and normal vector;

[0075] Optimization and fitting module: Optimize the attributes of the Gaussian volumes with a loss function to fit the light field and surface of the scene;

[0076] Fusion and reconstruction module: Fusion the surface depths of the multi-view scenes to obtain a mesh representation of the scene as the surface reconstruction result of the scene.

[0077] The present invention also discloses an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the Gaussian splatting surface reconstruction method based on anisotropic opacity; and discloses a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, implementing the steps of the Gaussian splatting surface reconstruction method based on anisotropic opacity.

[0078] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and what is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A Gaussian sputtering surface reconstruction method based on anisotropic opacity, characterized in that: The following steps are involved: (1) Given a multi-view image of a scene and its corresponding camera pose, a Gaussian sputtering model is initialized; the Gaussian sputtering model is a collection of a series of Gaussian bodies, and the attributes of each Gaussian body include: center position, opacity, color, covariance and normal vector; in the Gaussian sputtering model, the opacity α of the Gaussian body is anisotropic and is represented by a k-order spherical harmonic function plus a sigmoid activation function, and its expression is as follows: where {s lm |l∈[0,k],m∈[-l,l]} are spherical harmonic coefficients, {Y lm |l∈[0,k],m∈[-l,l]} is a fixed form of spherical harmonic basis function, v is the normalized direction vector; (2) Use the loss function to optimize the properties of the Gaussian to fit the light field and surface of the scene; (3) Fusing the scene surface depths from multiple perspectives to obtain a grid representation of the scene as a surface reconstruction result of the scene; the scene surface depth is obtained by the following process: Given a camera, assuming that the z-axis is facing forward, in the current camera coordinate system, for pixel x, let Represents the index set of all Gaussian bodies that can be projected to the pixel. If the depth z of the center of the Gaussian body with index i is i Satisfy z1 <z2<…<z n ,in Then the scene surface depth corresponding to the pixel is: where d i (x) is the depth of the ith Gaussian at pixel x, λ∈[0,1], with λ=0.5, is the depth transmittance of the ith Gaussian at pixel x, defined as: Among them G 2D is the projection of the Gaussian body on the imaging plane, is the maximum opacity of the Gaussian body in each viewing direction, expressed as: Where v is the normalized direction vector, α(v) is the anisotropic opacity of the Gaussian volume, Represents the set of normalized direction vectors from the center of the current Gaussian volume to the optical center of all cameras used for training.

2. The Gaussian sputtering surface reconstruction method based on anisotropic opacity according to claim 1, characterized in that: The expression of the loss function is as follows: in The photometric consistency loss is used to measure the difference between the color rendering value and the true color value, and is used to fit the scene light field with the Gaussian sputtering model. is the normal vector consistency loss, which is used to fit the Gaussian sputtering model to the scene surface; β is a hyperparameter.

3. The Gaussian sputtering surface reconstruction method based on anisotropic opacity according to claim 2, characterized in that: The color rendering value is obtained by the following process: Given a camera, assuming that the z-axis is facing forward, in the current camera coordinate system, for pixel x, let Represents the index set of all Gaussian bodies that can be projected to the pixel. If the depth z of the center of the Gaussian body with index i is i Satisfy z1 <z2<…<z n ,in The color rendering value corresponding to the pixel is: Where v is the normalized direction vector of the Gaussian body from its center to the optical center of the current camera, c(v) is the anisotropic color of the Gaussian body, represents the rendering weight of the i-th Gaussian at pixel x, α(v) is the anisotropic color of the Gaussian, G 2D is the projection of the Gaussian body on the imaging plane; is the light field transmittance of the ith Gaussian body at pixel x.

4. The Gaussian sputtering surface reconstruction method based on anisotropic opacity according to claim 3, characterized in that: The anisotropic color of the Gaussian body is represented by a q-order spherical harmonic function plus a relu activation function: where {t lm |l∈[0,q],m∈[-l,l]} are spherical harmonic coefficients, {Y lm |l∈[0,q],m∈[-l,l]} is a fixed form of spherical harmonic basis function, and v is the normalized direction vector.

5. The Gaussian sputtering surface reconstruction method based on anisotropic opacity according to claim 2, characterized in that: The normal vector consistency loss is obtained by the following formula: where w i (x) represents the rendering weight of the i-th Gaussian at pixel x, Is the "pseudo normal vector" of the scene surface. If the scene surface depth and The determined scene surface coordinates are p, p1 and p2 respectively, then Where x1, x2 are the adjacent pixels of x; × represents the vector cross product.

6. A Gaussian sputtering surface reconstruction device based on anisotropic opacity, characterized in that: Includes the following modules: Initialization model module: Given a multi-view image of a scene and its corresponding camera pose, initialize the Gaussian sputtering model; the Gaussian sputtering model is a collection of a series of Gaussian bodies, and the attributes of each Gaussian body include: center position, opacity, color, covariance and normal vector; in the Gaussian sputtering model, the opacity α of the Gaussian body is anisotropic, which is represented by a k-order spherical harmonic function plus a sigmoid activation function, and its expression is as follows: where {s lm |l∈[0,k],m∈[-l,l]} are spherical harmonic coefficients, {Y lm |l∈[0,k],m∈[-l,l]} is a fixed form of spherical harmonic basis function, v is the normalized direction vector; Optimization fitting module: Use the loss function to optimize the properties of the Gaussian body to fit the light field and surface of the scene; Fusion reconstruction module: fuses the scene surface depths from multiple perspectives to obtain a grid expression of the scene as the surface reconstruction result of the scene; the scene surface depth is obtained through the following process: Given a camera, assuming that the z-axis is facing forward, in the current camera coordinate system, for pixel x, let Represents the index set of all Gaussian bodies that can be projected to the pixel. If the depth z of the center of the Gaussian body with index i is i Satisfy z1 <z2<…<z n ,in Then the scene surface depth corresponding to the pixel is: where d i (x) is the depth of the ith Gaussian at pixel x, λ∈[0,1], with λ=0.5, is the depth transmittance of the ith Gaussian at pixel x, defined as: Among them G 2D is the projection of the Gaussian body on the imaging plane, is the maximum opacity of the Gaussian body in each viewing direction, expressed as: Where v is the normalized direction vector, α(v) is the anisotropic opacity of the Gaussian volume, Represents the set of normalized direction vectors from the center of the current Gaussian volume to the optical center of all cameras used for training.

7. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a Gaussian sputtering surface reconstruction method based on anisotropic opacity as described in any one of claims 1 to 5.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the steps of the Gaussian sputtering surface reconstruction method based on anisotropic opacity as described in any one of claims 1 to 5 are implemented.

Images