Gaussian sputtering surface reconstruction method based on anisotropic opacity

Abstract

The invention discloses a Gaussian sputtering surface reconstruction method based on anisotropic opacity, which comprises the following steps of: firstly, representing the opacity of a Gaussian body by using a spherical harmonic function to enable the opacity to be related to a sight line direction, and secondly, calculating a depth transmittance by using the maximum opacity of the Gaussian body in each sight line direction during sputtering rendering, selecting the depth of the last Gaussian body of which the depth transmittance is greater than a certain fixed value as the surface depth, and finally fusing the depths of different visual angles to obtain a scene surface reconstruction result. The method is suitable for Gaussian sputtering surface reconstruction, and the reconstruction precision of the glossy surface can be remarkably improved while the reconstruction precision of the diffusion surface is maintained.

Description

Technical Field

[0001] The invention relates to the technical field of three-dimensional reconstruction, and in particular to a Gaussian sputtering surface reconstruction method based on anisotropic opacity. Background Art

[0002] In the field of 3D reconstruction, how to reconstruct the light field and surface of a scene based on multi-view images of the scene and their corresponding camera poses is a research topic with considerable academic and application value. 3D Gaussian 用于实时辐射场渲染的点积法 , ACM Trans. Graph. 42(4)) has been widely used in the industry for its high-quality light field reconstruction and efficient training and inference. However, the reconstruction accuracy of scene surfaces by 3D Gaussian sputtering is low. To address this problem, SuGaR:表面对齐的高斯点积法 用于高效3D网格重建和高质量网格渲染 (CVPR2024), 用于几何精确辐射场的2D高斯点积法 (SIGGRAPH 2024), 使用高斯表面元素进行高质量表面重建 (SIGGRAPH 2024) proposed different solutions and achieved good results.

[0003] However, although these methods can achieve good reconstruction results on the diffuse surface of the object, serious distortion still occurs in the reconstruction of the glossy surface. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a Gaussian sputtering surface reconstruction method based on anisotropic opacity to address the distortion phenomenon that is prone to occur when three-dimensional Gaussian sputtering and various improvements to improve its surface reconstruction accuracy are used to reconstruct the glossy surface in the scene.

[0005] The objective of the present invention is achieved through the following technical solutions: The first aspect of the present invention is a Gaussian sputtering surface reconstruction method based on anisotropic opacity, comprising the following steps: (1) Given a multi-view image of a scene and its corresponding camera pose, initialize the Gaussian sputtering model; (2) Use the loss function to optimize the properties of the Gaussian to fit the light field and surface of the scene; (3) Fuse the scene surface depths from multiple perspectives to obtain a grid representation of the scene as the surface reconstruction result of the scene.

[0006] Specifically, the Gaussian sputtering model is a collection of a series of Gaussian bodies, and the attributes of each Gaussian body include: center position, opacity, color, covariance and normal vector; in the Gaussian sputtering model, the opacity of the Gaussian body With anisotropy, It is represented by the order spherical harmonic function plus the sigmoid activation function, and its expression is as follows: ; in are the spherical harmonic coefficients, are the fixed form spherical harmonic basis functions, is the normalized direction vector.

[0007] Specifically, the loss function is expressed as follows: ; in The photometric consistency loss is used to measure the difference between the color rendering value and the true color value, and is used to fit the scene light field with the Gaussian sputtering model. is the normal vector consistency loss, which is used to fit the Gaussian sputtering model to the scene surface; is a hyperparameter.

[0008] Furthermore, the color rendering value is obtained by the following process: Given a camera, assuming that the z-axis is facing forward, in the current camera coordinate system, for pixels ,make Represents the index set of all Gaussian bodies that can be projected onto the pixel. If the index is The depth (z coordinate) of the center of the Gaussian volume satisfy , then the color rendering value corresponding to the pixel is: ; in is the normalized direction vector of the Gaussian body from its center to the optical center of the current camera, is the anisotropic color of the Gaussian body, Indicates Gaussian volume in pixels The rendering weight of is the anisotropic color of the Gaussian body, is the projection of the Gaussian body on the imaging plane; For the Gaussian volume in pixels The light field transmittance.

[0009] Specifically, the anisotropic color of the Gaussian body is expressed as It is represented by the order spherical harmonic function plus the relu activation function: ; in are the spherical harmonic coefficients, are the fixed form spherical harmonic basis functions, is the normalized direction vector.

[0010] Specifically, the normal vector consistency loss is obtained by the following formula: ; in Indicates Gaussian volume in pixels The rendering weight of Is the "pseudo normal vector" of the scene surface. If the scene surface depth , and ( , for The scene surface coordinates determined by the adjacent pixels of , and ,but ,in Represents vector cross product.

[0011] Furthermore, the scene surface depth is obtained by the following process: Given a camera, assuming that the z axis is facing forward, in the current camera coordinate system, for pixels ,make Represents the index set of all Gaussian bodies that can be projected onto the pixel. If the index is The depth (z coordinate) of the center of the Gaussian volume satisfy , then the scene surface depth corresponding to the pixel is: ; in For the Gaussian volume in pixels The depth of , generally take , For the Gaussian volume in pixels The depth transmittance of is defined as: ; in is the projection of the Gaussian body on the imaging plane, is the maximum opacity of the Gaussian body in each viewing direction, expressed as: ; in is the normalized direction vector, is the anisotropic opacity of the Gaussian volume, Represents the set of normalized direction vectors from the center of the current Gaussian volume to the optical center of all cameras used for training.

[0012] The second aspect of the present invention discloses a Gaussian sputtering surface reconstruction device based on anisotropic opacity, comprising the following modules: Initialization model module: Given a multi-view image of a scene and its corresponding camera pose, initialize the Gaussian sputtering model; Optimization fitting module: Use the loss function to optimize the properties of the Gaussian body to fit the light field and surface of the scene; Fusion reconstruction module: fuses the scene surface depths from multiple perspectives to obtain the mesh expression of the scene as the surface reconstruction result of the scene.

[0013] The third aspect of the present invention: an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the Gaussian sputtering surface reconstruction method based on anisotropic opacity.

[0014] The fourth aspect of the present invention: a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of a Gaussian sputtering surface reconstruction method based on anisotropic opacity.

[0015] The beneficial effects of the present invention are as follows: The depth acquisition of the present invention can be performed synchronously with the light field rendering, and the obtained depth is consistent under different viewing angles; the present invention can significantly improve the reconstruction accuracy of the glossy surface while maintaining the reconstruction accuracy of the diffuse surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a comparison chart of the surface reconstruction effects of the present invention and other existing methods on real scenes. DETAILED DESCRIPTION

[0017] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The embodiments of the present invention are as follows.

[0018] Given multi-view images of a scene and their corresponding camera poses, Gaussian sputtering optimizes a series of Gaussian volumes through differentiable rendering to fit the light field and surface of the scene.

[0019] The parameters that can be optimized for Gaussian bodies include: center position, covariance, normal vector , expressing color Spherical harmonic coefficients of order: , expressing opacity Spherical harmonic coefficients of order: .

[0020] Given a Gaussian volume, the normalized direction vector from its center to the current camera's optical center , the anisotropic color of the Gaussian body is expressed as: ; in , is a fixed form of spherical harmonic basis function, and relu is the activation function.

[0021] The anisotropic opacity of a Gaussian volume is expressed as: ; in , are the fixed form spherical harmonic basis functions, is the activation function.

[0022] Given a camera, assuming that the z-axis is facing forward, in the current camera coordinate system, sort all Gaussian bodies according to the depth (z-coordinate) of their centers from small to large. For pixels ,make Represents the index set of all Gaussian bodies that can be projected onto the pixel. If the index is The depth (z coordinate) of the center of the Gaussian volume satisfy , then the pixel The color is expressed as follows: ; in Indicates Gaussian volume in pixels The rendering weight of is the projection of the Gaussian body on the imaging plane, For the Gaussian volume in pixels The light field transmittance.

[0023] Pixel The corresponding scene surface normal vector is: ; Pixel The corresponding scene surface depth is: ; in For the Gaussian volume in pixels The depth of , generally take , For the Gaussian volume in pixels The depth transmittance of is defined as: ; in is the maximum opacity of the Gaussian body in each viewing direction, expressed as: ; in Represents the set of normalized direction vectors from the center of the current Gaussian volume to the optical center of all cameras used for training.

[0024] The maximum opacity of the Gaussian during training It remains unchanged in each epoch (traversing the training set once) and is updated after the current epoch ends.

[0025] During the training process, the parameters of the Gaussian are optimized by constructing the following loss function: ; in To measure color The photometric consistency loss of the difference from the true color value, is a hyperparameter, is the normal vector consistency loss, that is ; in Is the "pseudo normal vector" of the scene surface. If the scene surface depth , and ( , for The scene surface coordinates determined by the adjacent pixels of , and ,but ,in Represents vector cross product.

[0026] During the training process, the spatial distribution of the Gaussian volume is adjusted using an adaptive density control strategy consistent with three-dimensional Gaussian sputtering; the photometric consistency loss consistent with three-dimensional Gaussian sputtering is used. To fit the light field of the scene.

[0027] After the training is completed, the depth maps rendered by each camera used in the training are fused to obtain a mesh expression of the scene as the surface reconstruction result of the scene.

[0028] Compared with other existing methods, the scene surface reconstruction result obtained by this embodiment can significantly improve the reconstruction accuracy of the glossy surface and maintain the reconstruction accuracy of the diffuse surface.

[0029] Depend on Figure 1 As shown in the comparison diagram of the surface reconstruction effects of the present invention and other existing methods on real scenes, it can be seen that the present invention can significantly improve the reconstruction accuracy of the glossy surface while maintaining the reconstruction accuracy of the diffuse surface. The other existing methods in the figure include: 用于实时辐射场渲染的3D高斯点积法 , SuGaR:用于高效3D网格的表面对齐高斯点积法 重建和高质量网格渲染 , 用于 几何精确辐射场的2D高斯点积法 , 高质量表面重建 使用高斯表面元素 .

[0030] The present invention also discloses a Gaussian sputtering surface reconstruction device based on anisotropic opacity, comprising the following modules: Initialization model module: given a multi-view image of a scene and its corresponding camera pose, initialize the Gaussian sputtering model; the Gaussian sputtering model is a collection of a series of Gaussian bodies, and the attributes of each Gaussian body include: center position, opacity, color, covariance and normal vector; Optimization fitting module: Use the loss function to optimize the properties of the Gaussian body to fit the light field and surface of the scene; Fusion reconstruction module: fuses the scene surface depths from multiple perspectives to obtain the mesh expression of the scene as the surface reconstruction result of the scene.

[0031] The present invention also discloses an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the Gaussian sputtering surface reconstruction method based on anisotropic opacity; and discloses a computer-readable storage medium, on which computer instructions are stored, which, when executed by the processor, implement the steps of the Gaussian sputtering surface reconstruction method based on anisotropic opacity.

[0032] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A Gaussian sputtering surface reconstruction method based on anisotropic opacity, characterized in that: The following steps are involved: (1) Given a multi-view image of a scene and its corresponding camera pose, initialize the Gaussian sputtering model; (2) Use the loss function to optimize the properties of the Gaussian to fit the light field and surface of the scene; (3) Fuse the scene surface depths from multiple perspectives to obtain a grid representation of the scene as the surface reconstruction result of the scene.

2. The Gaussian sputtering surface reconstruction method based on anisotropic opacity according to claim 1, characterized in that: The Gaussian sputtering model is a collection of a series of Gaussian bodies, and the attributes of each Gaussian body include: center position, opacity, color, covariance and normal vector; in the Gaussian sputtering model, the opacity of the Gaussian body Anisotropic, It is represented by the order spherical harmonic function plus the sigmoid activation function, and its expression is as follows: ; in are the spherical harmonic coefficients, are the fixed form spherical harmonic basis functions, is the normalized direction vector.

3. The Gaussian sputtering surface reconstruction method based on anisotropic opacity according to claim 1, characterized in that: The expression of the loss function is as follows: ； in The photometric consistency loss is used to measure the difference between the color rendering value and the true color value, and is used to fit the scene light field with the Gaussian sputtering model. is the normal vector consistency loss, which is used to fit the Gaussian sputtering model to the scene surface; is a hyperparameter.

4. The Gaussian sputtering surface reconstruction method based on anisotropic opacity according to claim 3, characterized in that: The color rendering value is obtained by the following process: Given a camera, assuming that the z-axis is facing forward, in the current camera coordinate system, for pixels ,make Represents the index set of all Gaussian bodies that can be projected onto the pixel. If the index is The depth (z coordinate) of the center of the Gaussian volume satisfy , then the color rendering value corresponding to the pixel is: ； in is the normalized direction vector of the Gaussian body from its center to the optical center of the current camera, is the anisotropic color of the Gaussian body, Indicates Gaussian volume in pixels The rendering weight of is the anisotropic color of the Gaussian body, is the projection of the Gaussian body on the imaging plane; For the Gaussian volume in pixels The light field transmittance.

5. The Gaussian sputtering surface reconstruction method based on anisotropic opacity according to claim 4, characterized in that: The anisotropic color of the Gaussian body is It is represented by the order spherical harmonic function plus the relu activation function: ； in are the spherical harmonic coefficients, are the fixed form spherical harmonic basis functions, is the normalized direction vector.

6. The Gaussian sputtering surface reconstruction method based on anisotropic opacity according to claim 3, characterized in that: The normal vector consistency loss is obtained by the following formula: ； in Indicates Gaussian volume in pixels The rendering weight of Is the "pseudo normal vector" of the scene surface. If the scene surface depth , and ( , for The scene surface coordinates determined by the adjacent pixels of , and ,but ,in Represents vector cross product.

7. The Gaussian sputtering surface reconstruction method based on anisotropic opacity according to claim 1, characterized in that: The scene surface depth is obtained by the following process: Given a camera, assuming that the z-axis is facing forward, in the current camera coordinate system, for pixels ,make Represents the index set of all Gaussian bodies that can be projected onto the pixel. If the index is The depth (z coordinate) of the center of the Gaussian volume satisfy , then the scene surface depth corresponding to the pixel is: ； in For the Gaussian volume in pixels The depth of , generally take , For the Gaussian volume in pixels The depth transmittance of is defined as: ； in is the projection of the Gaussian body on the imaging plane, is the maximum opacity of the Gaussian body in each viewing direction, expressed as: ； in is the normalized direction vector, is the anisotropic opacity of the Gaussian volume, Represents the set of normalized direction vectors from the center of the current Gaussian volume to the optical center of all cameras used for training.

8. A Gaussian sputtering surface reconstruction device based on anisotropic opacity, characterized in that: Includes the following modules: Initialization model module: Given a multi-view image of a scene and its corresponding camera pose, initialize the Gaussian sputtering model; Optimization fitting module: Use the loss function to optimize the properties of the Gaussian body to fit the light field and surface of the scene; Fusion reconstruction module: fuses the scene surface depths from multiple perspectives to obtain the mesh expression of the scene as the surface reconstruction result of the scene.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a Gaussian sputtering surface reconstruction method based on anisotropic opacity as described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of a Gaussian sputtering surface reconstruction method based on anisotropic opacity as described in any one of claims 1 to 7 are implemented.

Images