Three-dimensional surface reconstruction method based on multi-resolution three-plane coding

Abstract

The invention provides a three-dimensional surface reconstruction method based on multi-resolution three-plane coding. The method comprises the following steps: S1, acquiring a plurality of two-dimensional images shot by the same object at different camera visual angles; s2, constructing an initial model, wherein the model comprises a sampling module, a coding module, a first multi-layer perceptron and a second multi-layer perceptron; s3, performing multi-stage training on the initial model by adopting a plurality of two-dimensional images, and taking a coding module, a first multi-layer perceptron and a second multi-layer perceptron obtained after training as target models; s4, taking three-dimensional space points corresponding to all vertexes of each cube in a three-dimensional space formed by a plurality of cubes as input, and predicting the color and signed distance function value of each three-dimensional space point by adopting a target model, and reconstructing the three-dimensional grid representation of the object in the two-dimensional image based on the predicted color of each three-dimensional space point and the signed distance function value.

Description

A 3D Surface Reconstruction Method Based on Multi-Resolution Three-Plane Encoding Technical Field

[0001] This invention relates to the field of computer graphics, specifically to 3D reconstruction technology in the field of computer graphics, and more specifically, to a 3D surface reconstruction method based on multi-resolution three-plane encoding. Background Technology

[0002] In recent years, 3D reconstruction technology has been increasingly widely used in fields such as computer vision, graphics, virtual reality (VR), augmented reality (AR), autonomous driving, and digital twins. Traditional methods mainly rely on multi-view stereo vision (MVS) and technologies such as structured light and laser scanning. However, these methods generally suffer from high computational complexity, strong dependence on camera calibration accuracy, and sensitivity to scene lighting and surface texture, which limits their application in real-world complex scenarios.

[0003] With the development of deep learning, neural rendering methods have brought breakthroughs to 3D reconstruction. Techniques such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting can reconstruct high-quality 3D scenes from sparse 2D images and generate realistic new perspective rendering effects. Among them, NeRF uses volume density to implicitly represent scene geometry. Although it supports the extraction of explicit surfaces through methods such as Marching Cubes, the lack of a clear definition of the geometric surface makes the reconstruction results highly dependent on the choice of density threshold, leading to unstable reconstruction quality.

[0004] To overcome the aforementioned problems, the Neural Implicit Surface Reconstruction (NeuS) method introduces a signed distance function (SDF) as a geometric representation. It uses differentiable volume rendering to pass the 2D image supervision signal to the geometry optimization process, thus significantly improving surface reconstruction quality. However, NeuS uses MLP and spherical harmonic functions for scene modeling, which has limited representational capabilities. The reconstruction results are often too smooth, making it difficult to recover high-frequency geometric details. Subsequent studies have attempted to introduce explicit feature encoding to enhance the model's expressive power; for example, PET-NeuS uses triplane encoding, while NeuS2 and Neuralangelo introduce multi-resolution hash grids. Nevertheless, these methods still face problems such as high-frequency artifacts and hash collisions, limiting further improvements in reconstruction accuracy and visual quality.

[0005] In addition, issues such as sparse viewpoints and inconsistent lighting in the input image also pose challenges to reconstruction. Some methods attempt to introduce external prior information for supervision, such as coarse geometry, depth information, or symmetry constraints generated by Structure of Motion (SfM). Among these, normal priors are readily available from monocular images and have high accuracy, and are used by methods such as NeuRIS and MonoSDF to enhance the reconstruction capabilities of complex surfaces. However, monocular normal estimation itself has uncertainties, and erroneous priors will directly affect the reconstruction quality. Although NeuRIS proposes a normal filtering strategy based on multi-view consistency, its judgment ability is still insufficient in areas with complex textures or significant lighting variations, making it difficult to completely avoid the negative impact of erroneous priors.

[0006] In summary, while current neural rendering-based 3D reconstruction methods have made significant progress in terms of representation and reconstruction quality, they still face challenges such as insufficient geometric detail restoration, artifacts and conflicts in explicit encoding, and limited reliability of external priors.

[0007] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solutions of the present invention, and does not imply that the relevant information is necessarily prior art. In the absence of evidence indicating that the relevant information was disclosed before the filing date of this invention, the relevant information should not be considered prior art. Summary of the Invention

[0008] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a three-dimensional surface reconstruction method based on multi-resolution three-plane encoding.

[0009] The objective of this invention is achieved through the following technical solutions.

[0010] This invention provides a three-dimensional surface reconstruction method based on multi-resolution three-plane coding, used to construct a three-dimensional mesh representation of an object in a two-dimensional image. The method includes: Step S1, acquiring multiple two-dimensional images of the same object taken from different camera perspectives, wherein each two-dimensional image includes multiple pixels, and each two-dimensional image corresponds to a reference normal prior image, each reference normal prior image including the reference normal vector of the object surface corresponding to each pixel in its own two-dimensional image; Step S2, constructing an initial model, which includes a sampling module, an encoding module, a first multilayer perceptron, and a second multilayer perceptron, wherein: the sampling module is used to sample multiple... The sampling module generates a ray from the camera origin to each sampled pixel, using the camera's viewpoint of the 2D image to sample multiple 3D spatial points from each ray. The encoding module stores multiple tri-plane codes of different resolutions and extracts high-frequency feature codes for each 3D spatial point sampled by the sampling module based on these tri-plane codes. Each tri-plane code consists of three mutually perpendicular feature planes, and each feature plane includes multiple feature values. The first multilayer perceptron predicts the color of each 3D spatial point based on the high-frequency feature codes extracted by the encoding module, and predicts the color of each 3D spatial point based on the same ray. The color of the point is used to predict the color of the pixel emitting the ray; the second multilayer perceptron is used to predict the signed distance function value of each three-dimensional spatial point based on the high-frequency feature encoding of each three-dimensional spatial point extracted by the encoding module, and based on the signed distance function value of each three-dimensional spatial point on the same ray, predicts the normal vector of the object surface corresponding to the pixel emitting the ray and the rendering depth value of the pixel, where the rendering depth value represents the distance from the camera origin to the object surface corresponding to the pixel; step S3: the initial model is trained in multiple stages using multiple two-dimensional images, and the encoding module, the first multilayer perceptron and the second multilayer perceptron obtained after training are used as the target model. Each training stage includes multiple iterations, and each iteration randomly selects one 2D image from multiple 2D images for training. Different loss functions are used in different training stages to calculate the loss in order to update the parameters of the first multilayer perceptron and the second multilayer perceptron, as well as the feature values ​​of each three-plane encoding in the encoding module. Step S4: Take the 3D spatial points corresponding to all vertices of each cube in the 3D space composed of several cubes as input, use the target model to predict the color and signed distance function value of each 3D spatial point, and reconstruct the 3D mesh representation of the object in the 2D image based on the predicted color and signed distance function value of each 3D spatial point.

[0011] According to some embodiments of the present invention, in step S3, the initial model is sequentially subjected to a first stage of training, a second stage of training, a third stage of training, a fourth stage of training, and a fifth stage of training, wherein: in the first stage of training, the encoding module is configured to extract the high-frequency feature code of each three-dimensional spatial point in ascending order, starting from the three-plane encoding with the lowest resolution, and gradually introducing three-plane encoding with higher resolution according to a fixed number of iterations to extract the high-frequency feature code of each three-dimensional spatial point from the introduced three-plane encodings, until all three-plane encodings are introduced; wherein, in the first stage of training, a first loss function is used to calculate the loss to update the parameters of the first multilayer perceptron and the second multilayer perceptron and the feature value of each three-plane encoding in the encoding module; in the second stage of training, a second loss function is used to calculate the loss to update the parameters of the first multilayer perceptron and the second multilayer perceptron and the feature value of each three-plane encoding in the encoding module; in the third stage of training, a third loss function is used. The loss is calculated to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane code in the encoding module. In the fourth training phase, multiple normal training and perturbation training sessions are performed alternately, each including multiple iterations. During normal training, a third loss function is used to calculate the loss to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane code in the encoding module. During perturbation training, a fourth loss function is used to calculate the loss to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane code in the encoding module. In the fifth training phase, the third loss function is used to calculate the loss to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane code in the encoding module. During the second, third, fourth, and fifth training phases, the encoding module is configured to extract high-frequency feature codes for each three-dimensional spatial point from all three-plane codes.

[0012] According to some embodiments of the present invention, in the first stage of training, the encoding module is configured to extract the high-frequency feature code of each three-dimensional spatial point in the following manner: the three-dimensional spatial point is projected onto each introduced three-plane encoding, and the query feature code of the three-dimensional spatial point on each introduced three-plane encoding is extracted by bilinear interpolation; and the query feature codes of the three-dimensional spatial point on each introduced three-plane encoding are fused according to a preset fusion rule to obtain the high-frequency feature code of the three-dimensional spatial point, wherein the encoding module is configured with a first three-plane encoding, a second three-plane encoding, a third three-plane encoding and a fourth three-plane encoding, and the resolution of the second three-plane encoding is greater than the resolution of the first three-plane encoding, the resolution of the third three-plane encoding is greater than the resolution of the second three-plane encoding, and the resolution of the fourth three-plane encoding is greater than the resolution of the third three-plane encoding.

[0013] According to some embodiments of the present invention, the preset fusion rule is as follows:

[0014]

[0015] in, High-frequency feature encoding representing points in three-dimensional space This represents the query feature encoding of a three-dimensional spatial point on the first and third planes. This represents the query feature encoding of a three-dimensional spatial point on the second and third planes. This represents the query feature encoding of a point in three-dimensional space on the third plane. This represents the query feature encoding of a point in three-dimensional space on the fourth plane. Indicates the first variable weight. Indicates the second variable weight. Indicates the third variable weight. This represents the fourth variable weight; where, before the introduction of the second three-plane encoding, The value is fixed at 1. , and The values ​​are all 0; after introducing the second and third plane encoding, The value decreases linearly from 1 to 0.5 with the number of iterations. The value increases linearly from 0 to 0.5 with the number of iterations. and The value remains 0; after introducing the third three-plane encoding, The value is fixed at 0.5. The value decreases linearly from 0.5 to 0.25 with the number of iterations. The value increases linearly from 0 to 0.25 with the number of iterations. The value remains 0; after introducing the fourth three-plane encoding, The value is fixed at 0.5. The value is fixed at 0.25. The value decreases linearly from 0.25 to 0.125 with each iteration and then remains fixed. The value increases linearly from 0 to 0.125 with the number of iterations and then becomes fixed.

[0016] According to some embodiments of the present invention, the method includes: in the first stage of training, calculating the loss using a first loss function in each iteration and calculating the original update gradient back to the encoding module based on the loss; and determining the update gradient of each three-plane encoding according to a preset gradient transformation rule based on the original update gradient to update the feature values ​​of the corresponding three-plane encoding, wherein the preset gradient transformation rule is: before the introduction of the second three-plane encoding, the update gradient of the first three-plane encoding is equal to the original update gradient, and the update gradients of the second, third, and fourth three-plane encodings are equal to 0; after the introduction of the second three-plane encoding, the update gradient of the first three-plane encoding decays to one-sixteenth of the original update gradient, and the update gradient of the second three-plane encoding decays to one-sixteenth of the original update gradient. The new gradient is set to the original update gradient, and the update gradients of the third and fourth three-plane encodings remain 0. After introducing the third three-plane encoding, the update gradient of the first three-plane encoding remains one-sixteenth of the original update gradient, the update gradient of the second three-plane encoding decays to one-quarter of the original update gradient, the update gradient of the third three-plane encoding is set to the original update gradient, and the update gradient of the fourth three-plane encoding remains 0. After introducing the fourth three-plane encoding, the update gradient of the first three-plane encoding remains one-sixteenth of the original update gradient, the update gradient of the second three-plane encoding remains one-quarter of the original update gradient, the update gradient of the third three-plane encoding remains the original update gradient, and the update gradient of the fourth three-plane encoding is set to four times the original update gradient.

[0017] According to some embodiments of the present invention, the first loss function is:

[0018]

[0019] in,

[0020]

[0021]

[0022]

[0023] in, Denotes the first loss function. Indicates color loss. This represents the set of all pixels sampled from a two-dimensional image. Represents a set The total number of pixels contained therein. Represents a set Any pixel in the image, Represents pixels The predicted color Represents pixels The true color, Denotes the square of the L2 norm. Indicates the normalized loss in the normal direction. Represents pixels The predicted normal vector corresponding to the object's surface. express L2 norm, Indicates the first normal prior loss. Represents the pixel points in the prior image corresponding to the reference normal of the two-dimensional image. The reference normal vector corresponding to the object's surface. express The L2 norm.

[0024] According to some embodiments of the present invention, the second loss function is:

[0025]

[0026] in,

[0027]

[0028] in, This represents the second loss function. Indicates the second normal prior loss. This represents the fifth variable weight. The value of decreases linearly from 1 to 0 with the number of iterations during the second stage of training. This indicates the result calculated according to the preset reliability determination rules. The corresponding normal prior reliability mask, Used to indicate the pixel in the reference normal prior image corresponding to the two-dimensional image. Is the reference normal vector corresponding to the object's surface reliable? The value can be 0 or 1. When the value is 0, it means Unreliable When the value is 1, it means reliable.

[0029] According to some embodiments of the present invention, the preset reliability determination rule is as follows: the normal prior reliability mask of the reference normal vector of the object surface corresponding to each pixel in the reference normal prior map of the two-dimensional image is calculated as follows: compare the reference normal prior map of the current two-dimensional image with the reference normal prior map of any other two-dimensional image from a different camera perspective, and see if the normal angle loss between two pixel blocks corresponding to the same area of ​​the object surface is less than or equal to a threshold. If it is less than or equal to the threshold, it means that the reference normal vector of the object surface corresponding to each pixel in the current two-dimensional image reference normal prior map is reliable; otherwise, it is unreliable.

[0030] According to some embodiments of the present invention, the third loss function is:

[0031]

[0032] in, This represents the third loss function.

[0033] According to some embodiments of the present invention, the fourth loss function is:

[0034]

[0035] in, This represents the fourth loss function. This represents the unreliable pixel block in the reference normal prior map of a two-dimensional image, which is composed of the pixels corresponding to all unreliable reference normal vectors. express Any pixel in the image, This represents the signed distance field function. Represents pixels The specific location in three-dimensional space. Represents pixels The predicted rendering depth value, Indicates and Weighted graphs of the same size, based on a two-dimensional Gaussian distribution. Represents pixels The predicted color express The mean of the predicted colors of all pixels in the array. Represents pixels The predicted normal vector corresponding to the object's surface. express The mean of the predicted normal vectors of the object surface corresponding to all pixels in the image.

[0036] Compared with the prior art, the advantages of the present invention are: (1) it introduces multiple three-plane codes with different resolutions and designs a progressive training strategy for the multi-resolution three-plane codes to gradually integrate scene details from coarse to fine, thereby effectively achieving high-quality and high-precision geometric surface reconstruction; (2) it proposes a consistency detection of multi-view normal priors and a strategy for perturbing and fine-tuning unreliable normal regions, utilizing the sensitivity of convex surfaces to occlusion relationships to effectively correct erroneous reconstructions caused by insufficient supervision, thereby significantly improving the reconstruction quality of difficult-to-reconstruct regions, and without negatively affecting the originally correct reconstruction results. Attached Figure Description

[0037] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0038] Figure 1 is a flowchart illustrating a three-dimensional surface reconstruction method based on multi-resolution three-plane encoding according to an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of a progressive training strategy framework for multi-resolution three-plane encoding according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of a multi-scale feature sampling example according to an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram illustrating an example of the fine structure reconstruction effect according to an embodiment of the present invention;

[0042] Figure 5 is a schematic diagram of the framework of the disturbance and fine-tuning strategy for unreliable regions according to an embodiment of the present invention;

[0043] Figure 6 is a schematic diagram of local and global reliability masks according to an embodiment of the present invention;

[0044] Figure 7 is a schematic diagram illustrating an example of unreliable region reconstruction according to an embodiment of the present invention;

[0045] Figure 8 is a visualization comparison of the reconstruction quality of multiple samples in the NeRF-synthetic dataset according to an embodiment of the present invention;

[0046] Figure 9 is a visualization comparison of the reconstruction quality of two samples in the NeRF-synthetic dataset according to an embodiment of the present invention;

[0047] Figure 10 is a visualization comparison of the reconstruction quality of multiple samples in the DTU dataset according to an embodiment of the present invention;

[0048] Figure 11 is a visualization comparison of the reconstruction quality of two samples in the DTU dataset according to an embodiment of the present invention;

[0049] Figure 12 is a visual comparison of two samples in the NeRF-synthetic dataset according to an embodiment of the present invention;

[0050] Figure 13 is a visual comparison chart of indoor datasets according to an embodiment of the present invention;

[0051] Figure 14 is a visual comparison of the single-resolution and multi-resolution three-plane coding reconstruction results according to an embodiment of the present invention;

[0052] Figure 15 is a comparison diagram of the rendered normal map according to an embodiment of the present invention;

[0053] Figure 16 is a comparison diagram of the normal prior reconstruction effect according to an embodiment of the present invention;

[0054] Figure 17 is a comparison of the consistency detection of multi-view normal prior according to an embodiment of the present invention with NeuRIS.

[0055] Figure 18 is a comparison of reconstruction quality after applying perturbation and fine-tuning to the unreliable region in the normal direction according to an embodiment of the present invention;

[0056] Figure 19 is a training variation trend diagram of multi-level multi-resolution three-plane encoding according to an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0058] As mentioned in the background section, although current neural rendering-based 3D reconstruction methods have made significant progress in terms of representation and reconstruction quality, they still face problems such as insufficient geometric detail restoration, artifacts and conflicts in explicit encoding, and limited reliability of external priors.

[0059] To address the aforementioned issues, the inventors propose improving the model's ability to fit details and suppress high-frequency artifacts by fusing three-plane encodings at different resolutions. They also design a progressive learning strategy to fuse scene details from coarse to fine granular, thereby making the model training process more stable and resulting in higher-quality reconstructed 3D meshes. Furthermore, to prevent incorrect normal priors from impairing the reconstruction, a consistency detection method for multi-view normal priors is proposed to determine the reliability of reference normal vectors. Even further, during training, a perturbation and fine-tuning strategy is proposed for unreliable normal regions to improve the reconstruction quality in these areas, ultimately leading to higher-quality reconstructed 3D meshes.

[0060] In summary, as shown in Figure 1, this invention proposes a 3D surface reconstruction method based on multi-resolution three-plane coding, used to construct a 3D mesh representation of an object in a 2D image. The method includes: Step S1, acquiring multiple 2D images of the same object taken from different camera perspectives, wherein each 2D image includes multiple pixels, and each 2D image corresponds to a reference normal prior image, each reference normal prior image including the reference normal vector of the object surface corresponding to each pixel in its corresponding 2D image; Step S2, constructing an initial model, which includes a sampling module, an encoding module, a first multilayer perceptron, and a second multilayer perceptron, wherein: the sampling module is used to obtain... Multiple pixels are sampled from a 2D image, and a ray is emitted from the camera origin towards each sampled pixel from the camera's viewpoint of the 2D image to sample multiple 3D spatial points from each ray. The encoding module stores multiple tri-plane codes of different resolutions. The encoding module is used to extract high-frequency feature codes for each 3D spatial point sampled by the sampling module based on these tri-plane codes. Each tri-plane code consists of three mutually perpendicular feature planes, and each feature plane includes multiple feature values. A first multilayer perceptron is used to predict the color of each 3D spatial point based on the high-frequency feature codes extracted by the encoding module, and to predict the color of each 3D spatial point based on the same ray. The color of each 3D spatial point is used to predict the color of the pixel emitting the ray; the second multilayer perceptron is used to predict the signed distance function value of each 3D spatial point based on the high-frequency feature encoding of each 3D spatial point extracted by the encoding module, and based on the signed distance function value of each 3D spatial point on the same ray, predicts the normal vector of the object surface corresponding to the pixel emitting the ray and the rendering depth value of the pixel, where the rendering depth value represents the distance from the camera origin to the object surface corresponding to the pixel; step S3, the initial model is trained in multiple stages using multiple 2D images, and the encoding module, the first multilayer perceptron and the second multilayer perceptron obtained after training are used as the target model. The training model includes multiple iterations in each stage, and each iteration randomly selects one two-dimensional image from multiple two-dimensional images for training. Different loss functions are used to calculate the loss in different training stages to update the parameters of the first multilayer perceptron and the second multilayer perceptron, as well as the feature values ​​of each three-plane encoding in the encoding module. Step S4: Take the three-dimensional spatial points corresponding to all vertices of each cube in the three-dimensional space composed of several cubes as input, use the target model to predict the color and signed distance function value of each three-dimensional spatial point, and reconstruct the three-dimensional mesh representation of the object in the two-dimensional image based on the predicted color and signed distance function value of each three-dimensional spatial point.

[0061] Before providing a detailed description of the embodiments of the present invention, we will briefly introduce the basics of neural radiation fields, neural implicit surface reconstruction (NeuS), and three-plane coding.

[0062] The neural radiation field (NRF) uses a fully connected network to reconstruct the scene. The network's input is scene sampling points, and its output is volume density and color values. The NRF is supervised by the difference between the accumulated color of a pixel and its true color. The NRF samples 3D points in the scene using rays emitted from the camera center to the image pixels, and uses volume rendering to accumulate the output of each point into a pixel value. The formula for volume rendering is as follows:

[0063]

[0064] in, Indicates light rays The cumulative color, that is, the color value of the corresponding pixel. Indicates along the light ray Total number of sampling points Indicates the first The volume density predicted from each sampling point Indicates the first The color predicted from each sampling point Indicates the first The sampling point and the first The distance between sampling points ( (Indicates the distance between adjacent sampling points) This indicates the distance of light from the starting point to the sampling point. The cumulative density along the previously traversed path.

[0065] The Neural Implicit Surface Reconstruction (NeuS) method introduces a signed distance function (SDF) into the neural radiation field to improve the quality of surface reconstruction. The Neural Implicit Surface Reconstruction method derives a conversion formula from signed distance function (SDF) values ​​to density values, incorporating the signed distance function values ​​as an optimizable parameter into volume rendering. The conversion formula is expressed as follows:

[0066]

[0067] in, With coefficient The sigmoid function, It is the signed distance function value (SDF value) predicted by the network. The opaque density proposed by the neural implicit surface reconstruction method is equivalent to the density in the neural radiation field. Since the signed distance function value has a well-defined definition for the surface (SDF > 0 means the point is outside the surface, SDF < 0 means the point is inside the surface, and SDF = 0 means the point is exactly on the surface), the neural implicit surface reconstruction method is superior to the neural radiation field in terms of the quality of the reconstructed mesh.

[0068] Triplane encoding is a popular explicit feature representation that has been proposed in existing research and combined with neural radiation fields. Triplane encoding explicitly stores features in three orthogonal planes, and the features of a 3D spatial point can be obtained by projecting the point onto each orthogonal plane and then using bilinear interpolation.

[0069] To better understand the present invention, the implementation of each step will be described in detail below with reference to specific embodiments, and comparative experiments and ablation experiments will be used to demonstrate the advantages of the present invention over the prior art.

[0070] I. Step S1

[0071] In step S1, multiple two-dimensional images of the same object are acquired from different camera perspectives. Each two-dimensional image includes multiple pixels and corresponds to a reference normal prior image. Each reference normal prior image includes the reference normal vector of the object surface corresponding to each pixel in its own two-dimensional image.

[0072] II. Step S2

[0073] In step S2, an initial model is constructed, which includes a sampling module, an encoding module, a first multilayer perceptron, and a second multilayer perceptron.

[0074] The sampling module is used to sample multiple pixels from a 2D image. It then emits a ray from the camera origin towards each sampled pixel from the camera's viewpoint in the 2D image to sample multiple 3D spatial points from each ray. It should be noted that during each training iteration, the sampling module randomly samples 512 pixels (the number of pixels can be determined according to actual needs). Initially, the sampling module samples 3D spatial points uniformly. As training progresses, the sampling module automatically adjusts its sampling strategy based on the scene fit to enhance sampling efficiency.

[0075] The encoding module stores multiple three-plane codes with different resolutions. The encoding module is used to extract the high-frequency feature code of each three-dimensional spatial point sampled by the sampling module based on the multiple three-plane codes with different resolutions. Each three-plane code consists of three mutually perpendicular feature planes, and each feature plane includes multiple feature values.

[0076] The first multilayer perceptron is used to predict the color of each three-dimensional spatial point based on the high-frequency feature encoding of each three-dimensional spatial point extracted by the encoding module, and to predict the color of the pixel emitting the ray based on the color of each three-dimensional spatial point on the same ray.

[0077] The first multilayer perceptron is used to predict the color of each three-dimensional spatial point based on the high-frequency feature encoding of each three-dimensional spatial point extracted by the encoding module, and to predict the color of the pixel emitting the ray based on the color of each three-dimensional spatial point on the same ray.

[0078] III. Step S3

[0079] In step S3, multiple two-dimensional images are used to train the initial model in multiple stages, and the encoding module, the first multilayer perceptron and the second multilayer perceptron obtained after training are used as the target model. Each stage of training includes multiple iterations, and each iteration randomly selects one two-dimensional image from multiple two-dimensional images for training. Different loss functions are used in different stages of training to calculate the loss in order to update the parameters of the first multilayer perceptron and the second multilayer perceptron and the feature values ​​of each three-plane encoding in the encoding module.

[0080] According to one embodiment of the present invention, in step S3, the initial model is sequentially subjected to a first stage of training, a second stage of training, a third stage of training, a fourth stage of training, and a fifth stage of training. Each training stage is described below.

[0081] 3.1 First Training Phase

[0082] In the first training phase, the inventors proposed a progressive training strategy for multi-resolution three-plane coding, as shown in Figure 2. In this strategy, three-plane coding of different resolutions is gradually introduced in order from small to large, so as to achieve the fusion of scene details from coarse to fine.

[0083] In simple terms, as shown in the left image of Figure 2, in the first stage of training, the encoding module extracts the high-frequency feature codes of each 3D spatial point sequentially from the lowest-resolution three-plane encoding in ascending order. Then, according to a fixed number of iterations, higher-resolution three-plane encodings are gradually introduced to extract the high-frequency feature codes of each 3D spatial point from the introduced three-plane encodings, until all three-plane encodings have been introduced. In the first stage of training, a first loss function is used to calculate the loss to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane encoding in the encoding module. As shown in Figure 3, simultaneously projecting 3D spatial points onto three-plane encodings of different resolutions allows for querying and displaying features within local areas at different scales, and fusing these features to obtain high-frequency feature codes. This feature sampling method can fuse information from different spatial scales, which is beneficial not only for detail reconstruction but also for preventing the occurrence of high-frequency artifacts.

[0084] It's important to note that the reason for gradually introducing higher-resolution tri-plane encoding in ascending order is that high-resolution images, with their greater pixel count, tend to sample training light sources that focus more on local features, resulting in a poorer grasp of macroscopic details. This makes it difficult for the network to converge, especially for randomly initialized networks. In contrast, low-resolution training images, with their smaller pixel count, allow the network to sample more diverse perspectives in the early stages of training, thus improving early model fitting. Therefore, a progressive training strategy is used in the first training phase to gradually introduce higher-resolution tri-plane encoding until all tri-plane encodings have been incorporated.

[0085] To better understand the first training phase, the following explanation will be provided with reference to specific implementation examples.

[0086] According to an embodiment of the present invention, as shown in the right figure of FIG2, the encoding module is configured to extract the high-frequency feature code of each three-dimensional spatial point in the following manner: the three-dimensional spatial point is projected onto each introduced three-plane code, and the query feature code of the three-dimensional spatial point on each introduced three-plane code is extracted by bilinear interpolation; and the query feature codes of the three-dimensional spatial point on each introduced three-plane code are fused according to a preset fusion rule to obtain the high-frequency feature code of the three-dimensional spatial point. The encoding module is configured with a first three-plane code, a second three-plane code, a third three-plane code, and a fourth three-plane code, and the resolution of the second three-plane code is greater than the resolution of the first three-plane code, the resolution of the third three-plane code is greater than the resolution of the second three-plane code, and the resolution of the fourth three-plane code is greater than the resolution of the third three-plane code.

[0087] It should be noted that the second and third plane codes are obtained by upsampling the first and third plane codes, the third and fourth plane codes are obtained by upsampling the second and third plane codes, and so on. For example, assuming the resolution of the first and third plane codes is 100*100, upsampling it will yield a second and third plane code with a resolution of 200*200. Further, upsampling the second and third plane codes will yield a third and third plane code with a resolution of 400*400, and so on, upsampling the third and third plane codes will yield a fourth and third plane code with a resolution of 800*800.

[0088] It should also be noted that, to ensure the correlation between the three-plane codes at different resolutions, the second three-plane code is obtained by upsampling the first three-plane code obtained after multiple iterations (the initial first three-plane code is obtained through random initialization), the third three-plane code is obtained by upsampling the second three-plane code obtained after multiple iterations, and the fourth three-plane code is obtained by upsampling the third three-plane code obtained after multiple iterations. For example, with a fixed number of iterations of 20k, higher resolution three-plane codes are gradually introduced. After 20k iterations, the second three-plane code is introduced, which is obtained by upsampling the first three-plane code obtained after 20k iterations; iteration continues until 40k iterations are completed, then the third three-plane code is introduced, which is obtained by upsampling the second three-plane code obtained after 20k iterations; iteration continues until 60k iterations are completed, then the fourth three-plane code is introduced, which is obtained by upsampling the third three-plane code obtained after 20k iterations.

[0089] As described in the foregoing embodiments, higher-resolution three-plane encoding is gradually introduced in the first stage of training. This is achieved by projecting a 3D spatial point onto each orthogonal plane of each introduced three-plane encoding, extracting the query feature encoding of that 3D spatial point on each of the introduced three-plane encodings, and then fusing the query feature encodings of that 3D spatial point on each three-plane encoding to obtain the high-frequency feature encoding corresponding to that 3D spatial point. It should be noted that the query feature encodings of a 3D spatial point on different three-plane encodings cannot be directly fused (directly added), because directly introducing new query feature encodings into the model may potentially disrupt the stability of the currently trained model, thereby impairing its previously obtained fitting ability. Therefore, the inventors proposed a progressive mixing fusion rule, gradually adding new query feature encodings to the model training to ensure the stability of the model training.

[0090] According to one embodiment of the present invention, the preset fusion rule is:

[0091]

[0092] in, High-frequency feature encoding representing points in three-dimensional space This represents the query feature encoding of a three-dimensional spatial point on the first and third planes. This represents the query feature encoding of a three-dimensional spatial point on the second and third planes. This represents the query feature encoding of a point in three-dimensional space on the third plane. This represents the query feature encoding of a point in three-dimensional space on the fourth plane. Indicates the first variable weight. Indicates the second variable weight. Indicates the third variable weight. This represents the fourth variable weight; before the introduction of the second three-plane coding, The value is fixed at 1. , and The values ​​are all 0; after introducing the second and third plane encoding, The value decreases linearly from 1 to 0.5 with the number of iterations. The value increases linearly from 0 to 0.5 with the number of iterations. and The value remains 0; after introducing the third three-plane encoding, The value is fixed at 0.5. The value decreases linearly from 0.5 to 0.25 with the number of iterations. The value increases linearly from 0 to 0.25 with the number of iterations. The value remains 0; after introducing the fourth three-plane encoding, The value is fixed at 0.5. The value is fixed at 0.25. The value decreases linearly from 0.25 to 0.125 with each iteration and then remains fixed. The value increases linearly from 0 to 0.125 with the number of iterations and then becomes fixed.

[0093] To better understand the fusion rule proposed in this invention, the following explanation uses 20k as a fixed number of iterations to illustrate how feature fusion is performed.

[0094] In the 0-20k iterations, each three-dimensional spatial point uses the query feature code extracted from its own first three-plane encoding as the high-frequency feature code of that three-dimensional spatial point.

[0095] In iterations 20001-40k, each 3D spatial point fuses the query feature code extracted from the first three-plane encoding with the query feature code extracted from the second three-plane encoding to obtain the high-frequency feature code of that 3D spatial point. , The value will decrease linearly from 1 to 0.5 with each iteration. The value increases linearly from 0 to 0.5 with the number of iterations (in 20k iterations from 20001 to 40k). The value decreases linearly from 1 to 0.5. The value then increases linearly from 0 to 0.5.

[0096] In iterations 40001-60k, each 3D spatial point fuses the query feature codes extracted from the first three-plane encoding, the second three-plane encoding, and the third three-plane encoding to obtain the high-frequency feature code of that 3D spatial point. , The value decreases linearly from 0.5 to 0.25 with the number of iterations. The value increases linearly from 0 to 0.25 with the number of iterations (in 20k iterations from 40001 to 60k). The value decreases linearly from 0.5 to 0.25. The value then increases linearly from 0 to 0.25.

[0097] In iterations 60001-80k, each 3D spatial point fuses the query feature codes extracted from the first three-plane encoding, the second three-plane encoding, the third three-plane encoding, and the fourth three-plane encoding to obtain the high-frequency feature code corresponding to that 3D spatial point. , The value decreases linearly from 0.25 to 0.125 with the number of iterations. The value increases linearly from 0 to 0.125 with the number of iterations (in 20k iterations from 60001 to 80k). The value decreased linearly from 0.25 to 0.125. The value then increases linearly from 0 to 0.125.

[0098] After 80k iterations (No longer changing), in subsequent training, this fusion rule is followed to fuse the query feature codes obtained by each three-dimensional spatial point on the three-plane encoding at different resolutions.

[0099] According to one embodiment of the present invention, in the first stage of training, a first loss function is used to calculate the loss in each iteration, and the original update gradient back to the encoding module is calculated based on the loss. The update gradient of each three-plane encoding is determined according to a preset gradient transformation rule based on the original update gradient to update the feature values ​​of the corresponding three-plane encoding. The preset gradient transformation rule is as follows: before the introduction of the second three-plane encoding, the update gradient of the first three-plane encoding is equal to the original update gradient, and the update gradients of the second, third, and fourth three-plane encodings are equal to 0; after the introduction of the second three-plane encoding, the update gradient of the first three-plane encoding decays to one-sixteenth of the original update gradient, and the update gradient of the second three-plane encoding... The update gradients of the first and fourth three-plane encodings are set to the original gradient, and the update gradients of the third and fourth three-plane encodings are still 0. After the introduction of the third three-plane encoding, the update gradient of the first three-plane encoding is still one-sixteenth of the original update gradient, the update gradient of the second three-plane encoding is reduced to one-quarter of the original update gradient, the update gradient of the third three-plane encoding is set to the original update gradient, and the update gradient of the fourth three-plane encoding is still 0. After the introduction of the fourth three-plane encoding, the update gradient of the first three-plane encoding is still one-sixteenth of the original update gradient, the update gradient of the second three-plane encoding is still one-quarter of the original update gradient, the update gradient of the third three-plane encoding is still the original update gradient, and the update gradient of the fourth three-plane encoding is set to four times the original update gradient.

[0100] It should be noted that, as seen in the foregoing embodiments, during feature fusion in the first stage of training, the weights before the higher-resolution three-plane encoding are smaller. Since the gradient backpropagated to the encoding module during training is the same, this gradient value, after being multiplied by different weights, will result in a relatively small update gradient for the high-resolution three-plane encoding. This leads to a significant decrease in the optimization speed of the model after adding the high-resolution three-plane encoding, and slower learning of high-frequency details. Therefore, to alleviate the problem of slow learning of high-frequency details, the inventors propose to compensate for the update gradient of the high-resolution three-plane encoding. That is, as described in the foregoing embodiments, after introducing higher-resolution three-plane encoding, the update gradient of each introduced three-plane encoding is adjusted so that the introduced higher-resolution three-plane encoding has a higher update gradient.

[0101] According to an embodiment of the present invention, the first loss function is:

[0102]

[0103] in,

[0104]

[0105]

[0106]

[0107] in, Denotes the first loss function. Indicates color loss. This represents the set of all pixels sampled from a two-dimensional image. Represents a set The total number of pixels contained therein. Represents a set Any pixel in the image, Represents pixels The predicted color Represents pixels The true color, Denotes the square of the L2 norm. Indicates the normalized loss in the normal direction. Represents pixels The predicted normal vector corresponding to the object's surface. express L2 norm, Indicates the first normal prior loss. Represents the pixel points in the prior image corresponding to the reference normal of the two-dimensional image. The reference normal vector corresponding to the object's surface. express The L2 norm.

[0108] Based on the first stage of training, this invention introduces multiple three-plane codes with different resolutions and designs a progressive training strategy for multi-resolution three-plane codes to gradually integrate scene details from coarse to fine granular, thereby effectively achieving high-quality and high-precision geometric surface reconstruction.

[0109] 3.2 Second Training Phase

[0110] In the second phase of training, a second loss function is used to calculate the loss to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane code in the encoding module. In the second phase of training, the encoding module is configured to extract the high-frequency feature code of each three-dimensional spatial point from all three-plane codes.

[0111] Despite employing multi-resolution triplane encoding, the quality of 3D surface reconstruction remains affected by the camera's viewpoint distribution. Some scene surfaces exhibit sparse viewpoints and inconsistent lighting in image datasets, making reconstruction particularly challenging. While introducing normal priors as additional supervision can improve reconstruction quality, the effectiveness of prior supervision hinges on the accuracy of the prior information. Even state-of-the-art normal estimation methods, being essentially monocular image-based predictions, may produce inconsistent normal predictions across multiple viewpoints. Furthermore, normal prediction is influenced by similar textures. When surfaces with different actual normals but similar textures are placed side-by-side in an image, their normals may be incorrectly predicted as identical values. Directly using these erroneously estimated normal priors as supervision leads to a decline in reconstruction quality. Moreover, predicted normal priors often ignore texture information and are relatively smooth, which detracts from the reconstruction of detailed geometry.

[0112] Therefore, to effectively supervise the use of normal priors, the inventors proposed a reliability judgment rule. This rule evaluates the reliability of normal priors through consistency detection of multi-view normal priors, and only reliable normal priors are used for supervision in the second stage of training. Simultaneously, combined with a progressive learning strategy, the weights of normal supervision are dynamically adjusted during the training process to avoid the damage to reconstruction caused by the overly smoothing of normal priors when reconstructing detailed features in the later stages of training.

[0113] According to one embodiment of the present invention, the second loss function is:

[0114]

[0115] in,

[0116]

[0117] in, This represents the second loss function. Indicates the second normal prior loss. This represents the fifth variable weight. The value of decreases linearly from 1 to 0 with the number of iterations during the second stage of training. This indicates the result calculated according to the preset reliability determination rules. The corresponding normal prior reliability mask, Used to indicate the pixel in the reference normal prior image corresponding to the two-dimensional image. Is the reference normal vector corresponding to the object's surface reliable? The value can be 0 or 1. When the value is 0, it means Unreliable When the value is 1, it means reliable.

[0118] According to an embodiment of the present invention, the preset reliability determination rule is as follows: the normal prior reliability mask of the reference normal vector of the object surface corresponding to each pixel in the reference normal prior map of the two-dimensional image is calculated as follows: compare the reference normal prior map of the current two-dimensional image with the reference normal prior map of any other two-dimensional image from a different camera perspective, and see if the normal angle loss between two pixel blocks corresponding to the same area of ​​the object surface is less than or equal to a threshold. If it is less than or equal to the threshold, it means that the reference normal vector of the object surface corresponding to each pixel in the current two-dimensional image reference normal prior map is reliable; otherwise, it is unreliable.

[0119] The normal angle loss between two pixel blocks is calculated as follows:

[0120]

[0121] in, Indicates in pixel block From the camera's perspective From the camera's perspective Normal angle loss, Indicates camera perspective The corresponding pixel block Indicates camera perspective Next pixel The corresponding reference normal vector, Indicates camera perspective Next pixel The rendering depth value, Indicates camera perspective Next pixel The corresponding reference normal vector, Represents pixel blocks The number of pixels in Indicates vector normalization, Indicates the perspective from a given camera. To a given camera perspective The homography transformation function, Indicates pixels Projected into camera view The corresponding pixel coordinates obtained on the image plane Indicates camera perspective Next pixel coordinates The reference normal vector at that location.

[0122] in,

[0123]

[0124] in, Represents a given camera viewpoint The intrinsic parameter matrix of the lower camera, Represents a given camera viewpoint Rotation matrix of the lower camera, Represents a given camera viewpoint The translation vector of the lower camera. Represents a given camera viewpoint The inverse of the intrinsic parameter matrix of the lower camera. Represents a given camera viewpoint The inverse of the rotation matrix of the lower camera. Represents a given camera viewpoint The translation vector of the lower camera, Indicates camera perspective Next pixel The transpose of the corresponding reference normal vector, Indicates camera perspective The transpose of the direction vector pointing from the camera center to the pixel unit.

[0125] It should be noted that the normal prior used in this invention is defined in the world coordinate system, which means that the normal of the same spatial point should be consistent under different camera views. The physical meaning of (normal angle loss) is the difference in predicted normal angles between adjacent cameras for sampling points near the same object surface; the smaller the difference, the greater the difference. This indicates that the prior normals of adjacent cameras to the same object surface are more similar. This invention provides... Set a threshold This is used to measure whether corresponding pixel blocks in the same area of ​​an object's surface are consistent in two adjacent cameras. When pixel blocks... Between two adjacent cameras Value less than or equal to threshold At that time, the pixel block can be determined. The predicted normals are consistent from both camera viewpoints. Therefore, it can be concluded that, from a given camera viewpoint, a pixel block... Between the perspectives of one or more neighboring cameras All less than When that happens, the pixel block is determined from the camera's perspective. The normal prior within is reliable, but conversely, it is unreliable.

[0126] It should also be noted that, since the smoothness of the normal prior can impair the reconstruction quality of high-precision surfaces, reference normal vectors are only used for supervision in the second training phase, allowing the network to quickly learn macroscopic scene normals. Simultaneously, to eliminate the obstacle of smooth normals to detail reconstruction, the weights of normal supervision decay to 0 with each iteration. The value of decreases linearly from 1 to 0 with the number of iterations. As shown in Figure 4, even with a relatively smooth normal prior, the present invention can still effectively reconstruct high-precision geometric details.

[0127] 3.3 Third Training Phase

[0128] In the third phase of training, a third loss function is used to calculate the loss to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane code in the encoding module; wherein, in the third phase of training, the encoding module is configured to extract the high-frequency feature code of each three-dimensional spatial point from all three-plane codes.

[0129] According to an embodiment of the present invention, the third loss function is:

[0130]

[0131] in, This represents the third loss function.

[0132] 3.4 Fourth Training Phase

[0133] In the fourth stage of training, multiple rounds of normal training and perturbation training are performed alternately. Both normal and perturbation training include multiple iterations. During normal training, a third loss function is used to calculate the loss and update the parameters of the first and second multilayer perceptrons, as well as the feature values ​​of each three-plane encoding in the encoding module. During perturbation training, a fourth loss function is used to calculate the loss and update the parameters of the first and second multilayer perceptrons, as well as the feature values ​​of each three-plane encoding in the encoding module. In the fourth stage of training, the encoding module is configured to extract high-frequency feature codes for each 3D spatial point from all three-plane encodings. It should be noted that the number of executions of normal and perturbation training in the fourth stage can be set based on actual needs, and this invention does not impose any special limitations. For example, it can be set to alternate between normal and perturbation training 10 times (each normal and perturbation training is executed 10 times separately), with each normal and perturbation training iteration including 1000 iterations.

[0134] Although consistency detection of multi-view normal priors in the second training phase can provide reliable normal supervision for some poorly reconstructed regions, underfitting may still occur in regions with unreliable normals. Research has found that, with limited viewpoint coverage, some surfaces may be reconstructed as concave due to insufficient supervision. If the normal priors for these regions are also unreliable, the model will be unable to correct these concave regions. Therefore, to address this issue, this invention proposes a perturbation and fine-tuning strategy for unreliable regions, as shown in Figure 5, to help the model re-optimize these unreliable regions. The following explanation is based on Figures 5 and 6.

[0135] As shown in Figure 5, the reliability of the normal prior is evaluated by the consistency detection of the normal prior from multiple camera views. This yields local normal reliability masks for the corresponding reference normal prior maps under different camera views (as shown in the middle layer of Figure 5). In each local normal reliability mask, black marks unreliable parts, and white marks reliable parts. Furthermore, by traversing the local normal reliability masks for the corresponding reference normal prior maps under all camera views, a global normal reliability mask is obtained (as shown in the bottom layer of Figure 5). Simply put, for a given region, if the local normal reliability masks from all camera views consider the normal prior for that region to be unreliable, then that region is marked as an unreliable normal region in the global normal reliability mask.

[0136] Specifically, taking Figure 6 as an example, in the normal priors in the upper left corner of Figure 6, the normal priors of some sloping roofs (marked with black boxes) are almost identical to those of walls perpendicular to the ground, while the normals of others (marked with white boxes) are almost identical to those of the ground. Clearly, these are incorrect priors. Therefore, through consistency checks of multi-view normal priors, these areas are determined to be unreliable in the local normal reliability mask, and are marked as unreliable (black) in the upper-middle local normal reliability mask. However, the normal priors of these areas may be correct in other views, as shown in the lower left of Figure 6, where they are determined to be reliable (white) in the lower-middle local normal mask. By traversing the local normal reliability masks of all views, globally unreliable normal regions can be found, and they are marked as unreliable (black) in the global normal reliability mask.

[0137] The global normal reliability mask identifies globally unreliable regions. These regions are deemed unreliable in the reference normal prior maps corresponding to all camera views, meaning no normal supervision is applied to them. If surface reconstruction is difficult in these regions due to factors such as sparse supervision views or differences in lighting conditions, the model cannot correctly learn these regions through ordinary optimization, often resulting in underfitting. To improve the reconstruction quality of potentially underfitted surfaces within globally unreliable regions, this invention collects pixels within the black (unreliable) areas of the global normal reliability mask to form pixel blocks, which are then processed using a fourth loss function.

[0138] According to one embodiment of the present invention, the fourth loss function is:

[0139]

[0140] in, This represents the fourth loss function. This represents the unreliable pixel block in the reference normal prior map of a two-dimensional image, which is composed of the pixels corresponding to all unreliable reference normal vectors. express Any pixel in the image, This represents the signed distance field function. Represents pixels The specific location in three-dimensional space. Represents pixels The predicted rendering depth value, Indicates and Weighted graphs of the same size, based on a two-dimensional Gaussian distribution. Represents pixels The predicted color express The mean of the predicted colors of all pixels in the array. Represents pixels The predicted normal vector corresponding to the object's surface. express The mean of the predicted normal vectors of the object surface corresponding to all pixels in the image.

[0141] In the fourth loss function, the first term penalizes the signed distance function (SDF) value of the pixel block, pushing the signed distance function value at the corresponding rendering depth towards a negative value. This causes the surface of unreliable regions to expand outward and form a convex shape. The weights are based on a Gaussian distribution. ( The weights of the loss function smoothly decrease from the center of the block outwards, aiming to maximize the impact of perturbations on the region at the center, resulting in a more natural convex shape. Higher weights are assigned to the center of the block, and other similar weight functions can be used to control the extent of expansion within the same block. The last two terms of the loss function constrain the smoothness of color and normal within the region, ensuring that the convexity is continuous and smooth during optimization. Regardless of the reconstruction quality of unreliable regions, the fourth loss function will push them outwards to form convex shapes. This perturbation will be applied separately in the later stages of training and alternated with the original training process to prevent the network from collapsing due to excessive perturbation. After several iterations of perturbation, the unreliable regions become convex, and then the unreliable loss is discontinued. The overall training reverts to the original loss function, focusing on re-optimizing these perturbated convex surfaces. During re-optimization, the model will give the unreliable regions a larger sampling rate for the normals, that is, concentrate more training rays on these convex surfaces to accelerate the refitting of these surface reconstructions.

[0142] It's important to note that globally reliable regions with reliable global normals can typically achieve correct surface reconstruction with the help of normal supervision, while globally unreliable regions may reconstruct poor-quality surfaces. Studies show that poorly reconstructed surfaces often appear in globally unreliable regions and are usually concave. As shown in Figure 7, the leftmost image displays a well-reconstructed surface, while the middle and right images show concave and convex surfaces reconstructed in unreliable regions, respectively. The black lines in Figure 7 represent the reconstructed surfaces, while the blue arrows indicate camera light during training. Figure 7 shows that when a surface is reconstructed as convex (as shown in the right image), it blocks camera light, leading to amplified image loss and a larger gradient, thus driving the incorrect convex surface to be optimized by the model. Conversely, concave surfaces lack this occlusion relationship, and 2D image supervision cannot identify such incorrectly reconstructed surfaces. While normal priors can help improve the reconstruction quality of this type of surface, this is contingent on the reliability of the normal prior. In regions where the normal direction is unreliable, such concave surfaces are difficult to repair. For these regions where the normal direction is unreliable, this invention proposes a fourth loss function to perturb and fine-tune the surface in order to effectively improve the reconstruction quality.

[0143] It should also be noted that well-reconstructed surfaces exist even in unreliable regions. Although the perturbations and fine-tuning of this method may disturb these surfaces, they will not negatively impact the reconstruction results. This is because even if these surfaces become convex during the perturbation process, the subsequent fine-tuning process will still correctly fit them. Since these surfaces originally possess the ability to be accurately reconstructed, they can still be correctly learned after being perturbed into convex surfaces. Conversely, surfaces initially incorrectly reconstructed as concave have a high probability of being correctly reconstructed during fine-tuning after perturbation.

[0144] Based on the fourth training phase, this invention proposes a consistency detection method for multi-view normal priors and a method for perturbing and fine-tuning unreliable normal regions. By utilizing the sensitivity of convex surfaces to occlusion relationships, it effectively corrects erroneous reconstructions caused by insufficient supervision, thereby significantly improving the reconstruction quality of difficult-to-reconstruct regions without negatively impacting the originally correct reconstruction results.

[0145] 3.5 Fifth Training Phase

[0146] In the fifth stage of training, a third loss function is used to calculate the loss to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane code in the encoding module; wherein, in the fifth stage of training, the encoding module is configured to extract the high-frequency feature code of each three-dimensional spatial point from all three-plane codes.

[0147] IV. Step S4

[0148] In step S4, the three-dimensional spatial points corresponding to all vertices of each cube in a three-dimensional space composed of several cubes are used as input. The target model is used to predict the color and signed distance function value of each three-dimensional spatial point, and the three-dimensional mesh representation of the object in the two-dimensional image is reconstructed based on the predicted color and signed distance function value of each three-dimensional spatial point. Specifically, after calculating the SDF value of each three-dimensional spatial point, zero-iso-value points are calculated based on the SDF value of each three-dimensional spatial point where the SDF value alternates between positive and negative. All zero-iso-value points are turned into vertices of the mesh, and every three vertices are connected to form a triangular facet to obtain the reconstructed three-dimensional mesh representation of the object in the two-dimensional image.

[0149] V. Experimental Analysis

[0150] To verify that the present invention achieves better reconstruction results compared to existing technologies, the inventors conducted comparative and ablation experiments to demonstrate the advantages of the present invention over existing technologies. In these experiments, grayscale images were used to represent the reconstructed 3D mesh to showcase the reconstruction quality.

[0151] 5.1 Experimental Setup

[0152] In the experiments, this invention uses four tri-plane codes with resolutions ranging from low to high: 200*200, 400*400, 800*800, and 1600*1600. The implementation of this invention is based on the Neural Implicit Surface Reconstruction (NeuS) method, with a batch size of 512 (the number of pixels sampled during each training iteration), and employs the same signed distance function and color network configuration. This invention uses a first loss function, a second loss function, a third loss function, and a fourth loss function for supervision. All experiments were conducted on an RTX 3090 with 24GB of VRAM, and the extracted mesh resolution was 512. The training steps of this invention include 400k iterations. The first stage of training executes 100k iterations (0-100k), and tri-plane codes are gradually introduced for training in the 20k, 40k, and 60k iterations. The second stage of training also executes 100k iterations (100001-200k), and settings are configured in the second stage of training. The threshold is 0.9848 (approximately equal to...) The training process involves 100k iterations (200001-300k) in the third phase and 20k iterations (alternating between normal and perturbation training 10 times, with each normal and perturbation training session including 1000 iterations) in the fourth phase. The training process involves 80k iterations (320001-400k) in the fifth phase. The entire training process takes approximately 16 hours and the maximum memory usage is approximately 23GB.

[0153] This invention was tested on the existing DTU dataset and the NeRF-synthetic dataset. In the DTU dataset, each scene is a real-world image captured at a resolution of 1600×1200, while in the NeRF-synthetic dataset, each scene is a synthetically rendered image at a resolution of 800×800. Furthermore, to compare the consistency detection method of multi-view normal priors with NeuRIS, the inventors added three indoor dataset scenes used by NeuRIS. All datasets were downsampled three times to construct training images at different resolutions to match a four-layer, three-plane configuration, with each subsequent resolution being one-quarter of the original resolution.

[0154] In the experiments, the inventors selected NeuS, NeuS2, PET-NeuS, LoD-NeuS, NeuRIS, and 2DGS methods as comparison methods. These methods are state-of-the-art reconstruction methods based on SDF implicit field (or Gaussian ellipsoid) representations. In particular, NeuRIS uses a normal prior for supervision, while the other works do not. Simultaneously, this method selects Peak Signal-to-Noise Ratio (PSNR) as a quantitative indicator to evaluate rendering quality and uses Chamfer Distance (CD) to evaluate the accuracy of the reconstructed mesh. PSNR measures the reconstruction quality of signals (such as images and 3D models), with higher values ​​indicating less distortion; Chamfer Distance evaluates the geometric accuracy of 3D point cloud or surface reconstruction, measuring the degree of matching between two sets of point clouds, with lower values ​​being better.

[0155] 5.2 Comparative Experiment

[0156] After conducting a comparative experiment based on the experimental setup, the experimental results shown in Figure 8-13 and Table 1-4 were obtained.

[0157] Figure 8 shows a visualization comparison of the reconstruction quality of multiple samples in the NeRF-synthetic dataset, and the quantitative comparison results of the chamfer distance corresponding to Figure 8 are shown in Table 1. Combining Figure 8 and Table 1, it can be seen that the present invention has better reconstruction results in geometric reconstruction compared to other reconstruction methods. Figure 9 shows a visualization comparison of the two scenes in Figure 8 using different reconstruction methods. As can be seen from Figure 9, the reconstruction method proposed in this invention has advantages in reconstructing fine-grained geometric details. This is due to multi-resolution tri-plane encoding, which not only fits fine-grained details but also considers a large range of local spatial information.

[0158] Figure 10 shows a visual comparison of the reconstruction quality of multiple samples in the DTU dataset, and the quantitative comparison results of the chamfer distance corresponding to Figure 10 are shown in Table 2. Combining Figure 10 and Table 2, it can be seen that the present invention has better reconstruction results in geometric reconstruction compared to other reconstruction methods. Figure 11 shows a visual comparison of the two scenes in Figure 10 using different reconstruction methods. As can be seen from Figure 11, the reconstruction method proposed in this invention can correctly reconstruct complex geometric details in real-world scenes, such as roof tiles and windows. The comparison methods show varying performance in terms of detail loss; for example, NeuS and 2DGS both exhibit smooth transitions. Even NeuRIS, which also uses a normal prior, loses details in the house scene, and due to its poor judgment of the reliability of the normal prior, the reconstructed surface is affected to some extent by erroneous normals, thus impairing the reconstruction quality.

[0159] Figure 12 shows a visualization comparison of two samples from the NeRF-synthetic dataset using different reconstruction methods. As can be seen from Figure 12, although LoD-NeuS uses multi-scale triplane representation to reconstruct high-fidelity geometry, it simultaneously optimizes the triplanes at different scales, making training more difficult and affecting the final reconstruction result. The progressively growing triplane encoding and compatible progressive feature hybridization strategy proposed in this invention are more conducive to training stability, thus better capturing geometric details and ultimately obtaining more refined geometric reconstruction results.

[0160] Figure 13 shows a visual comparison of the indoor dataset using different reconstruction methods. The top three images show the reconstruction results of NeuRIS, and the bottom three images show the reconstruction results of our proposed method. As can be seen from Figure 13, our invention achieves better reconstruction quality than NeuRIS in terms of both refined structure and surface smoothness.

[0161] Table 3 shows the quantitative comparison results of peak signal-to-noise ratio on the DTU dataset. As can be seen from Table 3, the present invention has better rendering quality than other reconstruction methods.

[0162] Table 4 shows the quantitative comparison results of the indoor dataset. As can be seen from Table 4, the present invention has better reconstruction quality than NeuRIS.

[0163] Table 1

[0164]

[0165] Table 2

[0166]

[0167] Table 3

[0168]

[0169] Table 4

[0170]

[0171] 5.3 Ablation Experiment

[0172] To verify the effectiveness of the various strategies proposed in this invention, a series of ablation experiments were conducted, yielding experimental results as shown in Figures 14-18 and Table 5. In the ablation experiments, a single-resolution triplane coding method combined with the original NeuS was used as the baseline method, labeled "ST". Several ablation techniques were progressively tested, including multi-resolution triplane coding "MT", progressive triplane coding feature mixing "Pt", progressive training set resolution scaling "Pd", and consistency detection using multi-view normal priors "MVN" employing "NCC" from NeuRIS and "NAL" from this method, respectively. Finally, the complete version of this invention was presented. Table 5 shows the chamfer distance ablation experiment results on the DTU dataset.

[0173] Table 5

[0174]

[0175] In the ablation experiments, the advantages of multi-resolution three-plane encoding over single-resolution three-plane encoding were first verified. For univariate comparison, a progressive training strategy was not used in multi-resolution three-plane encoding, resulting in a visual comparison of the reconstruction results of single-resolution and multi-resolution three-plane encoding, as shown in Figure 14. In Figure 14, the left image shows the reconstruction result of single-resolution three-plane encoding, while the right image shows the reconstruction result of multi-resolution three-plane encoding. As shown in Figure 14, multi-resolution three-plane encoding eliminates high-frequency artifacts caused by the discreteness of high-resolution three-plane encoding because low-resolution three-plane encoding focuses on a larger local area, resulting in a smoother surface. The relevant experiments are labeled "ST" and "MT" in Table 5. Table 5 shows that compared to single-resolution three-plane encoding "ST", the multi-resolution three-plane encoding "MT" of this invention has a smaller CD value and better reconstruction performance.

[0176] Figure 15 shows a comparison of the model's rendered normal maps when the highest resolution three-plane encoding is first added to the training iterations. The normal map rendered using the progressive training strategy in the early training stages is shown on the left side of Figure 15, while the normal map rendered without the progressive training strategy is shown on the right side. As shown in Figure 15, the progressive training strategy ensures that the proposed method's multi-resolution three-plane effectively fits the scene from coarse to fine granular in the early training stages without generating significant noise (such as the noise points in the lower left and lower right of the right image), which contributes to the overall training of the network. Related experiments are labeled "MT+Pt" and "MT+Pt+Pd" in Table 5. The results of "MT+Pt" show some improvement compared to "MT," indicating that the progressive blending of the three-plane features in this invention can enhance the reconstruction effect. The results of "MT+Pt+Pd" show further improvement, indicating that the progressive increase in the resolution of the training dataset also has a positive impact on reconstruction.

[0177] This invention uses multi-view normal prior consistency detection to obtain a local normal reliability mask for filtering in normal supervision. In some scenarios, such as scan40 of the DTU dataset, the camera view used for supervision is mostly concentrated in the front, while the side view is limited, making side reconstruction challenging. Worse still, lighting and shadows change dramatically with camera movement, significantly impacting surface reconstruction. As shown in Figure 16, after using multi-view normal prior consistency detection to filter the normal prior, the reconstruction quality is significantly improved, as evidenced by the results of "MT+Pt+Pd" and "MT+Pt+Pd+MVN(NAL)" in Table 5.

[0178] In the ablation experiments, the performance of two different multi-view normal prior consistency detection methods, namely the present invention and NeuRIS, was compared. Figure 17 shows the comparison results of the multi-view normal prior consistency detection in the present invention and NeuRIS. In Figure 17, the black mask represents the unreliable local normal region. The DTU dataset has inconsistent illumination under different views, as shown by the shadow changes in the red box in the first row, which leads to the confusion of the mask in NeuRIS. The multi-view normal prior consistency detection of the present invention avoids the influence of illumination condition changes and obtains more reasonable results. The relevant experiments are labeled "MVN(NCC)" and "MVN(NAL)" in Table 5, respectively. For fairness, the performance of "NCC" and "NAL" is compared only in the implementation of the present invention, and the confidence threshold of the "NCC" scheme is adjusted to the robust value of 0.66 provided by NeuRIS, while other settings remain unchanged. The experimental results show that the present invention can more effectively judge the accuracy of the normal prior, thereby improving the reconstruction effect.

[0179] For regions where the normal prior is unreliable, this invention proposes a perturbation and fine-tuning strategy. This strategy generates large gradients through perturbation, focusing on optimizing these regions where reconstruction errors may exist. As shown in Figure 18, since the normal prior cannot be used as supervision in regions with unreliable normals, relying solely on 2D image loss makes it difficult to correct underfitted surfaces, resulting in poor reconstruction quality for these surfaces (within the red boxes in the two left figures). After applying perturbation and fine-tuning to these regions, the reconstruction quality is significantly improved (within the red boxes in the two right figures). Related experiments are labeled "MT+Pt+Pd+MVN(NAL)" and "This Invention" in Table 5.

[0180] To verify the advantages of the 4-level multi-resolution triplane coding used in this invention (4 levels means including 4 triplane codes with different resolutions, and the same applies to other levels), ablation experiments were set up to compare the differences between level 1, level 2, level 3, and level 4 multi-resolution triplane coding, and the experimental results are shown in Figure 19. In Figure 19(a), the CD values ​​are shown under different levels of multi-resolution triplane coding, and in Figure 19(b), the PSNR values ​​are shown under different levels of multi-resolution triplane coding. Blue, green, red, and yellow represent levels 4 to 1, respectively. As shown in Figure 19, regardless of the metric, the 4-level multi-resolution triplane coding accelerates convergence in the early training phase and ultimately achieves better reconstruction and rendering quality.

[0181] The beneficial effects of the present invention are as follows: (1) Multiple three-plane codes with different resolutions are introduced, and a progressive training strategy is designed for the multi-resolution three-plane codes to gradually integrate scene details from coarse to fine, thereby effectively achieving high-quality and high-precision geometric surface reconstruction; (2) A consistency detection of multi-view normal priors and a strategy for perturbing and fine-tuning unreliable normal regions are proposed. By utilizing the sensitivity of convex surfaces to occlusion relationships, the erroneous reconstruction caused by insufficient supervision is effectively corrected, thereby significantly improving the reconstruction quality of difficult-to-reconstruct regions, and without negatively affecting the originally correct reconstruction results.

[0182] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0183] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0184] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0185] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A three-dimensional surface reconstruction method based on multi-resolution three-plane coding, used to construct a three-dimensional mesh representation of objects in a two-dimensional image, characterized in that, The method includes: Step S1, acquiring multiple two-dimensional images of the same object captured from different camera perspectives, wherein each two-dimensional image includes multiple pixels, each two-dimensional image corresponds to a reference normal prior image, and each reference normal prior image includes the reference normal vector of the object surface corresponding to each pixel in its own two-dimensional image; Step S2, constructing an initial model, which includes a sampling module, an encoding module, a first multilayer perceptron, and a second multilayer perceptron, wherein: the sampling module is used to sample multiple pixels from the two-dimensional image, and emit a beam from the camera origin to each sampled pixel from the camera perspective of the two-dimensional image. The line samples multiple three-dimensional spatial points from each ray; the encoding module stores multiple three-plane codes of different resolutions, and the encoding module is used to extract the high-frequency feature code of each three-dimensional spatial point sampled by the sampling module based on the multiple three-plane codes of different resolutions; wherein, each three-plane code consists of three mutually perpendicular feature planes, and each feature plane includes multiple feature values; the first multilayer perceptron is used to predict the color of each three-dimensional spatial point based on the high-frequency feature code of each three-dimensional spatial point extracted by the encoding module, and to predict the color of the pixel emitting the ray based on the color of each three-dimensional spatial point on the same ray; the second multilayer perceptron is used to predict the signed distance function value of each three-dimensional spatial point based on the high-frequency feature code of each three-dimensional spatial point extracted by the encoding module, and to predict the normal vector of the object surface corresponding to the pixel emitting the ray and the rendering depth value of the pixel based on the signed distance function value of each three-dimensional spatial point on the same ray, wherein the rendering depth value represents the distance from the camera origin to the object surface corresponding to the pixel; step S3, using multiple two-dimensional images to perform multi-stage training on the initial model, and using the encoding module, the first multilayer perceptron and the second multilayer perceptron obtained after training as the target model, wherein, in each stage of training, The process involves multiple iterations, with each iteration randomly selecting one 2D image from multiple 2D images for training. Different loss functions are used to calculate the loss at different training stages to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane encoding in the encoding module. Step S4: Using the 3D spatial points corresponding to all vertices of each cube in the 3D space composed of several cubes as input, the target model is used to predict the color and signed distance function value of each 3D spatial point, and the 3D mesh representation of the object in the 2D image is reconstructed based on the predicted color and signed distance function value of each 3D spatial point.

2. The method according to claim 1, characterized in that, In step S3, the initial model is sequentially trained in five stages: the first stage, the second stage, the third stage, the fourth stage, and the fifth stage. Specifically: In the first stage, the encoding module is configured to extract high-frequency feature codes for each 3D spatial point, starting with the lowest-resolution three-plane encoding from the lowest-resolution three-plane encoding. A fixed number of iterations is then introduced to gradually extract high-frequency feature codes for each 3D spatial point from these multiple three-plane encodings, until all three-plane encodings are introduced. In the first stage, a first loss function is used to calculate the loss and update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane encoding in the encoding module. In the second stage, a second loss function is used to calculate the loss and update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane encoding in the encoding module. In the third stage, a third loss function is used to calculate the loss and update the feature values ​​of each three-plane encoding. The parameters of the new first and second multilayer perceptrons and the feature values ​​of each three-plane code in the encoding module are used. In the fourth stage of training, multiple normal training and perturbation training are performed alternately. Both normal training and perturbation training include multiple iterations. When performing normal training, the third loss function is used to calculate the loss to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane code in the encoding module. When performing perturbation training, the fourth loss function is used to calculate the loss to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane code in the encoding module. In the fifth stage of training, the third loss function is used to calculate the loss to update the parameters of the first and second multilayer perceptrons and the feature values ​​of each three-plane code in the encoding module. In the second, third, fourth and fifth stages of training, the encoding module is configured to extract the high-frequency feature code of each three-dimensional spatial point from all three-plane codes.

3. The method according to claim 2, characterized in that, In the first stage of training, the configuration encoding module extracts the high-frequency feature encoding of each three-dimensional spatial point in the following way: the three-dimensional spatial point is projected onto each of the introduced three-plane encodings, and the query feature encoding of the three-dimensional spatial point on each of the introduced three-plane encodings is extracted by bilinear interpolation. And according to the preset fusion rules, the query feature codes of the three-dimensional spatial point on each introduced three-plane code are fused to obtain the high-frequency feature code of the three-dimensional spatial point. The encoding module is configured with a first three-plane code, a second three-plane code, a third three-plane code and a fourth three-plane code, and the resolution of the second three-plane code is greater than the resolution of the first three-plane code, the resolution of the third three-plane code is greater than the resolution of the second three-plane code, and the resolution of the fourth three-plane code is greater than the resolution of the third three-plane code.

4. The method according to claim 3, characterized in that, The preset fusion rules are as follows: in, High-frequency feature encoding representing points in three-dimensional space This represents the query feature encoding of a three-dimensional spatial point on the first and third planes. This represents the query feature encoding of a three-dimensional spatial point on the second and third planes. This represents the query feature encoding of a point in three-dimensional space on the third plane. This represents the query feature encoding of a point in three-dimensional space on the fourth plane. Indicates the first variable weight. Indicates the second variable weight. Indicates the third variable weight. This represents the fourth variable weight; where, before the introduction of the second three-plane encoding, The value is fixed at 1. 、 and The values ​​are all 0; after introducing the second and third plane encoding, The value decreases linearly from 1 to 0.5 with the number of iterations. The value increases linearly from 0 to 0.5 with the number of iterations. and The value remains 0; after introducing the third three-plane encoding, The value is fixed at 0.

5. The value decreases linearly from 0.5 to 0.25 with the number of iterations. The value increases linearly from 0 to 0.25 with the number of iterations. The value remains 0; after introducing the fourth three-plane encoding, The value is fixed at 0.

5. The value is fixed at 0.

25. The value decreases linearly from 0.25 to 0.125 with each iteration and then remains fixed. The value increases linearly from 0 to 0.125 with the number of iterations and then becomes fixed.

5. The method according to claim 4, characterized in that, The method includes: in the first stage of training, calculating the loss using a first loss function in each iteration and calculating the original update gradient back to the encoding module based on the loss; and determining the update gradient of each three-plane encoding according to a preset gradient transformation rule based on the original update gradient to update the feature values ​​of the corresponding three-plane encoding, wherein the preset gradient transformation rule is: before the introduction of the second three-plane encoding, the update gradient of the first three-plane encoding is equal to the original update gradient, and the update gradients of the second, third, and fourth three-plane encodings are equal to 0; after the introduction of the second three-plane encoding, the update gradient of the first three-plane encoding is decayed to one-sixteenth of the original update gradient, and the update gradient of the second three-plane encoding is set to... The original update gradient, the update gradients of the third and fourth three-plane encodings are still 0; after introducing the third three-plane encoding, the update gradient of the first three-plane encoding is still one-sixteenth of the original update gradient, the update gradient of the second three-plane encoding is reduced to one-quarter of the original update gradient, the update gradient of the third three-plane encoding is set to the original update gradient, and the update gradient of the fourth three-plane encoding is still 0; after introducing the fourth three-plane encoding, the update gradient of the first three-plane encoding is still one-sixteenth of the original update gradient, the update gradient of the second three-plane encoding is still one-quarter of the original update gradient, the update gradient of the third three-plane encoding is still the original update gradient, and the update gradient of the fourth three-plane encoding is set to four times the original update gradient.

6. The method according to claim 5, characterized in that, The first loss function is: in, in, Denotes the first loss function. Indicates color loss. This represents the set of all pixels sampled from a two-dimensional image. Represents a set The total number of pixels contained therein. Represents a set Any pixel in the image, Represents pixels The predicted color Represents pixels The true color, Denotes the square of the L2 norm. Indicates the normalized loss in the normal direction. Represents pixels The predicted normal vector corresponding to the object's surface. express L2 norm, Indicates the first normal prior loss. Represents the pixel points in the prior image corresponding to the reference normal of the two-dimensional image. The reference normal vector corresponding to the object's surface. express The L2 norm.

7. The method according to claim 6, characterized in that, The second loss function is: in, in, This represents the second loss function. Indicates the second normal prior loss. This represents the fifth variable weight. The value of decreases linearly from 1 to 0 with the number of iterations during the second stage of training. This indicates the result calculated according to the preset reliability determination rules. The corresponding normal prior reliability mask, Used to indicate the pixel in the prior reference normal image corresponding to the two-dimensional image. Is the reference normal vector corresponding to the object's surface reliable? The value can be 0 or 1. When the value is 0, it means Unreliable When the value is 1, it means reliable.

8. The method according to claim 7, characterized in that, The preset reliability determination rule is as follows: calculate the normal prior reliability mask of the reference normal vector of the object surface corresponding to each pixel in the reference normal prior map of the two-dimensional image as follows: compare the reference normal prior map of the current two-dimensional image with the reference normal prior map of any other two-dimensional image from a different camera perspective, and see if the normal angle loss between two pixel blocks corresponding to the same area of ​​the object surface is less than or equal to a threshold. If it is less than or equal to the threshold, it means that the reference normal vector of the object surface corresponding to each pixel in the reference normal prior map of the current two-dimensional image is reliable; otherwise, it is unreliable.

9. The method according to claim 8, characterized in that, The third loss function is: in, This represents the third loss function.

10. The method according to claim 9, characterized in that, The fourth loss function is: in, This represents the fourth loss function. This represents the unreliable pixel block in the reference normal prior map of a two-dimensional image, which is composed of the pixels corresponding to all unreliable reference normal vectors. express Any pixel in the image, This represents the signed distance field function. Represents pixels The specific location in three-dimensional space. Represents pixels The predicted rendering depth value, Indicates and Weighted graphs of the same size, based on a two-dimensional Gaussian distribution. Represents pixels The predicted color express The mean of the predicted colors of all pixels in the array. Represents pixels The predicted normal vector corresponding to the object's surface. express The mean of the predicted normal vectors of the object surface corresponding to all pixels in the image.

11. A computer-readable storage medium, characterized in that, It contains a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1-10.

12. An electronic device, characterized in that, include: One or more processors, and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method according to any one of claims 1-10 by executing the executable instructions.

Images