Three-dimensional surface reconstruction method, device and apparatus

Abstract

Embodiments of the present application provide a three-dimensional surface reconstruction method, device and equipment, the method comprising: obtaining multiple RGB images of a target object; performing sharpening and denoising on each RGB image respectively to determine a normal vector map corresponding to each pre-processed RGB image and an uncertainty map corresponding to each normal vector map; training a hybrid geometric representation neural network model; determining a three-dimensional reconstruction surface of the target object according to a zero level surface of a directed distance field obtained by the trained hybrid geometric representation neural network model, thereby realizing three-dimensional surface reconstruction of the target object. By introducing image sharpening and denoising technology, the image quality is enhanced; using hybrid representation to encode geometric surfaces can more effectively express low-frequency and high-frequency regions, so that the quality of the reconstructed three-dimensional surface is higher and has more abundant details.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method, apparatus, and device for three-dimensional surface reconstruction based on hybrid representation and prior enhancement. Background Technology

[0002] 3D surface reconstruction refers to establishing a mathematical model of a 3D object suitable for computer representation and processing. It forms the basis for processing, manipulating, and analyzing the properties of objects in a computer environment and is a key technology for creating virtual reality representations of the objective world within a computer. It has promising applications in many fields, including but not limited to robotics, games, film and television, and virtual reality. Therefore, high-quality 3D surface reconstruction technology is an important research direction in the field of computer vision.

[0003] Numerous studies have been conducted on 3D surface reconstruction. For example, some techniques combine neural implicit functions and volume rendering to reconstruct 3D objects and scenes from a series of 2D images. NeuS technology uses a neural network to encode a Signed Distance Field (SDF) and a color field, and uses volume rendering to obtain 2D images from different viewpoints. The neural network is trained by calculating a loss function with ground truth images, and finally, the reconstructed surface of a single object is obtained by extracting the zero isosurface of the SDF. NeuRIS and MonoSDF technologies, building upon NeuS, introduce additional scene prior knowledge (i.e., monocular predicted normal maps or depth maps) to assist in obtaining more reasonable reconstructed surfaces. While existing technologies can reconstruct complete and smooth contours for flat areas (such as floors and walls), they struggle to reconstruct fine surfaces for high-frequency, fine-grained areas (such as small items on a table or delicate furniture), resulting in low-quality 3D surface reconstruction that fails to meet practical application requirements. Summary of the Invention

[0004] This invention provides a three-dimensional surface reconstruction method, apparatus, and device to solve the problem of low quality in three-dimensional surface reconstruction in the prior art.

[0005] In a first aspect, embodiments of the present invention provide a three-dimensional surface reconstruction method, comprising:

[0006] Obtain multiple RGB images of the target object;

[0007] Each RGB image is sharpened and denoised separately to obtain multiple pre-processed RGB images of the target object;

[0008] Determine the normal vector map corresponding to each preprocessed RGB image and the uncertainty map corresponding to each normal vector map;

[0009] The hybrid geometric representation neural network model is trained using multiple preprocessed RGB images, the normal vector maps corresponding to the multiple preprocessed RGB images, and the uncertainty map corresponding to each normal vector map. The trained hybrid geometric representation neural network model includes a geometric module and a color module. The geometric module includes a multilayer perceptron representation module, a three-plane feature representation module, and a decoder.

[0010] The zero isosurface of the directed distance field obtained from the trained hybrid geometric representation neural network model is used to determine the three-dimensional reconstructed surface of the target object.

[0011] In one embodiment, sharpening each RGB image includes: convolving the RGB image with a 3×3 Laplacian convolution kernel; denoising each RGB image includes: performing median filtering on the RGB image.

[0012] In one embodiment, determining the normal vector map corresponding to each preprocessed RGB image and the uncertainty map corresponding to each normal vector map includes:

[0013] Monocular prediction technology is used to determine the normal vector map corresponding to each preprocessed RGB image;

[0014] Based on the visual basic model, the high-dimensional semantic features corresponding to each preprocessed RGB image are determined;

[0015] Each preprocessed RGB image, its corresponding normal vector map, and its corresponding high-dimensional semantic features are input into a pre-trained uncertainty estimation module to obtain the uncertainty map corresponding to each normal vector map.

[0016] In one embodiment, a hybrid geometric representation neural network model is trained using multiple preprocessed RGB images, normal vector maps corresponding to the multiple preprocessed RGB images, and uncertainty maps corresponding to each normal vector map, including:

[0017] Based on the camera pose parameters corresponding to a preprocessed RGB image, determine the light ray emitted from the camera center to a pixel in the image, and sample a preset number of points at equal intervals on the light ray;

[0018] Input the coordinates of each point into the multilayer perceptron representation module of the geometry module to obtain the directed distance field value and latent vector corresponding to each point;

[0019] Input the coordinates of each point into the three-plane feature representation module and the branch of the decoder in the geometry module to obtain the directed distance field residual and latent vector residual corresponding to each point;

[0020] The sum of the directed distance field value and the corresponding directed distance field residual is determined as the final directed distance field value output by the geometry module, and the sum of the latent vector and the corresponding latent vector residual is determined as the final latent vector output by the geometry module.

[0021] The gradient of the directed distance field value finally output by the geometry module is calculated to obtain its corresponding normal vector value;

[0022] Input the coordinates, normal vector values, and implicit vectors of each point into the color module to obtain the radiance of each point.

[0023] Based on the directional distance field value, radiance and normal vector value of each point, the volume rendering equation is used to render the pixel color value and rendering normal vector value corresponding to the light.

[0024] The gradient descent algorithm is used to optimize all learnable parameters of the hybrid geometric representation neural network model with the goal of minimizing the preset loss function. The preset loss function is determined based on the rendered pixel color value and rendered normal vector value corresponding to the light, the real pixel color value and real normal vector value corresponding to the light, and the prior uncertainty of the normal vector corresponding to the light.

[0025] Repeat the above training process until the hybrid geometric representation neural network model converges.

[0026] In one embodiment, before training the hybrid geometric representation neural network model, the method further includes:

[0027] The learnable parameters of the multilayer perceptron representation module and the three-plane feature representation module are initialized such that the initial directed distance field value output by the multilayer perceptron representation module is approximately equal to the directed distance field value of a spherical shell with a radius of 1 in space, and the initial directed distance field value output by the three-plane feature representation module is approximately 0.

[0028] In one embodiment, the preset loss function is determined according to the following expression:

[0029]

[0030] Where L represents the preset loss function, r represents the ray, and C(r) represents the color value of the rendered pixel corresponding to ray r. This represents the actual pixel color value corresponding to the ray r, and M represents the preset quantity. Let represent the normal vector value corresponding to the i-th point, U(r) represent the prior uncertainty of the normal vector corresponding to ray r, and N(r) represent the rendered normal vector value corresponding to ray r. This represents the true normal vector value corresponding to ray r.

[0031] In one embodiment, determining the 3D reconstructed surface of the target object based on the zero isosurface of the directed distance field obtained from a trained hybrid geometric representation neural network model includes:

[0032] All parameters of the pre-trained hybrid geometric representation neural network model remain unchanged;

[0033] The coordinates of 1024×1024×1024 points are uniformly sampled within a unit cube with a side length of 1 and input into the geometric module of the trained hybrid geometric representation neural network model to obtain the directed distance field value corresponding to each point, thus forming a directed distance field.

[0034] Extract the zero isosurface of the directed distance field to obtain the normalized three-dimensional surface;

[0035] The normalized 3D surface is reverse normalized and scaled to obtain the 3D reconstructed surface of the target object.

[0036] In a second aspect, embodiments of the present invention provide a three-dimensional surface reconstruction apparatus, comprising:

[0037] The acquisition module is used to acquire multiple RGB images of the target object;

[0038] The preprocessing module is used to sharpen and denoise each RGB image separately to obtain multiple preprocessed RGB images of the target object;

[0039] The determination module is used to determine the normal vector map corresponding to each preprocessed RGB image and the uncertainty map corresponding to each normal vector map;

[0040] The training module is used to train the hybrid geometric representation neural network model using multiple preprocessed RGB images, the normal vector maps corresponding to the multiple preprocessed RGB images, and the uncertainty map corresponding to each normal vector map, to obtain a trained hybrid geometric representation neural network model. The hybrid geometric representation neural network model includes a geometric module and a color module.

[0041] The reconstruction module is used to determine the three-dimensional reconstructed surface of the target object based on the zero isosurface of the directed distance field obtained from the trained hybrid geometric representation neural network model.

[0042] Thirdly, embodiments of the present invention provide an electronic device, comprising:

[0043] At least one processor and memory;

[0044] The memory stores instructions that the computer executes;

[0045] At least one processor executes computer execution instructions stored in memory, causing the at least one processor to perform the three-dimensional surface reconstruction method as described in any of the first aspects.

[0046] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the three-dimensional surface reconstruction method as described in any of the first aspects.

[0047] The three-dimensional surface reconstruction method, apparatus, and device provided in this invention acquire multiple RGB images of a target object; sharpen and denoise each RGB image to obtain multiple preprocessed RGB images of the target object; determine the normal vector map corresponding to each preprocessed RGB image and the uncertainty map corresponding to each normal vector map; train a hybrid geometric representation neural network model using the multiple preprocessed RGB images, the corresponding normal vector maps, and the uncertainty maps corresponding to each normal vector map to obtain a trained hybrid geometric representation neural network model, which includes a geometric module and a color module. The geometric module includes a multilayer perceptron representation module, a three-plane feature representation module, and a decoder; and determine the three-dimensional reconstructed surface of the target object based on the zero isosurface of the directed distance field obtained from the trained hybrid geometric representation neural network model, thereby realizing the three-dimensional surface reconstruction of the target object. By introducing image sharpening and denoising techniques to preprocess the acquired RGB images, the image quality is enhanced, which can improve the accuracy of the predicted normal vector prior knowledge and help improve the accuracy of the reconstruction results. Furthermore, by using hybrid representation to encode the geometric surface, the low-frequency region (floor, walls, etc.) and high-frequency region (delicate furniture, small items on the table, etc.) of the indoor scene surface can be expressed more effectively. The reconstructed 3D surface is of higher quality and has richer details, thus improving the quality of 3D surface reconstruction. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0049] Figure 1 A flowchart of a three-dimensional surface reconstruction method provided in an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the process architecture for prior enhancement and hybrid representation provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the reconstruction effect provided in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the structure of a three-dimensional surface reconstruction device provided in an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0054] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0055] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0056] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0057] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0058] While 3D surface reconstruction technology has been applied in some fields with the continuous development of computer vision technology, high-quality 3D surface reconstruction of indoor scenes remains extremely challenging. This is because indoor scenes contain both low-frequency, large-area flat areas (such as floors and walls) and high-frequency, fine-grained areas (such as small items on tables and delicate furniture). Existing methods can reconstruct complete and smooth contours in the floor and wall areas of indoor scenes, but it is difficult to reconstruct surfaces with high-frequency, fine, and complex structures (the reconstruction results are often too smooth or contain imperfections such as floating objects), thus limiting the application of 3D surface reconstruction technology in indoor scenes.

[0059] The key factors preventing current technologies from performing fine-grained, high-fidelity reconstruction are as follows:

[0060] One factor is that existing technologies typically use a large multilayer perceptron (MLP) neural network to approximate the implicit geometric functions (i.e., directed distance fields) of the entire indoor scene. While such MLP neural networks are sufficient to represent small, individual objects, they are inadequate for representing complex, multi-layered indoor scenes. Research has found that MLP networks exhibit a smoothness bias, making them well-suited for encoding flat and low-frequency surfaces. However, indoor scenes also contain many high-frequency, fine-grained regions, making it difficult to accurately approximate the geometry of these areas using only a single MLP network.

[0061] Another factor is that the additional scene prior knowledge introduced in existing technologies is often inaccurate in structurally complex regions. These scene priors are usually estimated by neural networks and are prone to producing noisy or erroneous results on fine and delicate structures. Existing technologies directly use these imperfectly accurate priors as supervision, causing the model to be misled and reconstruct incorrect geometry.

[0062] To address the shortcomings of existing technologies, this application improves the quality of 3D surface reconstruction by enhancing the accuracy of geometric surface encoding and improving the quality of scene priors. It proposes a high-quality 3D surface reconstruction method based on hybrid representation and prior enhancement. This method requires only multiple RGB images to achieve high-quality, fine-grained, and detailed 3D surface reconstruction, enabling high-quality 3D surface reconstruction even in indoor scenes with both low-frequency, large-area flat regions and high-frequency, fine-grained regions. Specifically, this application employs a novel hybrid representation to encode geometric surfaces. Compared to the multilayer perceptron (MLP) representation used in existing technologies, the hybrid representation can more effectively represent low-frequency regions (floors, walls, etc.) and high-frequency regions (delicate furniture, small items on tabletops, etc.) of indoor scene surfaces. Image enhancement techniques are introduced, using sharpening and denoising operations to improve the accuracy of normal vector prior knowledge, thereby enhancing the accuracy of the reconstruction results. The uncertainty of the prior for each normal vector map is estimated, and a loss function is designed accordingly to guide the model to focus on normal vector priors with lower uncertainty during optimization, avoiding being misled by normal vector priors with high uncertainty. The three-dimensional surface reconstructed using the method provided in this application is of higher quality and has richer details. Specific embodiments will be used to describe this application in detail below.

[0063] Figure 1 This is a flowchart illustrating a three-dimensional surface reconstruction method according to an embodiment of the present invention. Figure 1 As shown, the three-dimensional surface reconstruction method provided in this embodiment may include:

[0064] S101. Obtain multiple RGB images of the target object.

[0065] In this embodiment, the RGB image of the target object can be obtained from a pre-stored storage device, or the RGB image of the target object can be acquired in real time using an image acquisition device. The target object in this embodiment can be any object requiring 3D surface reconstruction, such as an indoor scene containing both low-frequency, large-area flat areas (floors, walls, etc.) and high-frequency, fine-grained areas (small items on a table, delicate furniture, etc.). The number of required RGB images can be set according to actual needs; this embodiment does not impose any limitations on this.

[0066] S102. Sharpen and denoise each RGB image separately to obtain multiple pre-processed RGB images of the target object.

[0067] To improve the accuracy of prior knowledge of normal vectors predicted from RGB images during the 3D surface reconstruction of the target object, this embodiment introduces image sharpening and denoising techniques. Multiple RGB images of the acquired target object are preprocessed to enhance image quality. The prior enhancement process can be referenced... Figure 2 As shown.

[0068] Specifically, for sharpening, a 3×3 Laplacian convolution kernel can be used to convolve the image, meaning that for any position in the image, the new pixel value is equal to nine times the old pixel value minus the values ​​of the surrounding eight pixels.

[0069]

[0070] Where p is the old pixel value, p ′ It's the new pixel value. It is the eight pixels of the p-neighborhood.

[0071] For noise reduction, median filtering can be used, where for any location in the image, the new pixel value is equal to the median of its neighboring pixels.

[0072]

[0073] The median() function takes the median.

[0074] Therefore, in one optional implementation, sharpening each RGB image may specifically include: convolving the RGB image with a 3×3 Laplacian convolution kernel; denoising each RGB image may specifically include: performing median filtering on the RGB image.

[0075] S103. Determine the normal vector map corresponding to each preprocessed RGB image and the uncertainty map corresponding to each normal vector map.

[0076] After obtaining multiple preprocessed RGB images, monocular prediction techniques can be used to obtain the normal vector map corresponding to each RGB image. A visual foundation model can be used to extract high-dimensional semantic features corresponding to each preprocessed RGB image, such as 16-dimensional semantic features corresponding to each preprocessed RGB image.

[0077] In this embodiment, a pre-trained uncertainty estimation module can be used to estimate the prior uncertainty of each normal vector map. The uncertainty estimation module can adopt a U-shaped structure, specifically including four downsampling modules and four upsampling modules. Each downsampling module can specifically include: one 2D max pooling layer, two 2D convolutional layers, two 2D batch normalization layers, and two ReLU activation layers; each upsampling module can specifically include: one 2D transposed convolutional layer, two 2D convolutional layers, two 2D batch normalization layers, and two ReLU activation layers. For 3D surface reconstruction of indoor scenes, a large number (e.g., 300) of indoor 3D scenes are first used as training data to train the uncertainty estimation module. Specifically, the difference between the normal vector predicted using monocular prediction technology and the normal vector rendered from the real 3D scene is calculated, and the difference is normalized to between 0 and 1, serving as the ground truth to supervise the uncertainty map predicted by the uncertainty estimation module. After training, the parameters of the uncertainty estimation module are fixed and will not be changed in subsequent processes. Finally, each preprocessed RGB image, its corresponding normal vector map, and its corresponding high-dimensional semantic features are input into the pre-trained uncertainty estimation module to obtain the uncertainty map corresponding to each normal vector map.

[0078] S104. The hybrid geometric representation neural network model is trained using multiple preprocessed RGB images, the normal vector maps corresponding to the multiple preprocessed RGB images, and the uncertainty map corresponding to each normal vector map, to obtain a trained hybrid geometric representation neural network model. The hybrid geometric representation neural network model includes a geometric module and a color module. The geometric module includes a multilayer perceptron representation module, a three-plane feature representation module, and a decoder.

[0079] To address the challenge of accurately approximating geometric surfaces containing numerous high-frequency, fine-grained regions using a single multilayer perceptron network, this embodiment employs a hybrid geometric representation neural network model. This model can more effectively represent both low-frequency and high-frequency regions of an indoor scene. The structure of the hybrid geometric representation neural network model in this embodiment can be found in [reference needed]. Figure 2As shown, the system specifically includes a geometry module and a color module. The geometry module uses a hybrid representation, consisting of an 8-layer MLP representation module, a 256-resolution three-plane feature representation module, and a decoder. The MLP is responsible for representing flat outer contour surfaces, while the three-plane features are responsible for representing surfaces with fine high-frequency structures. The color module uses a single representation, comprising a 4-layer MLP. Using a hybrid representation to encode geometric surfaces, compared to the single multilayer perceptron (MLP) representation used in other technologies, can more effectively represent low-frequency regions (floors, walls, etc.) and high-frequency regions (delicate furniture, small items on tabletops, etc.) of indoor scene surfaces, thus helping to improve the accuracy of 3D surface reconstruction results.

[0080] The hybrid geometric representation neural network model is trained using multiple preprocessed RGB images, the normal vector maps corresponding to the multiple preprocessed RGB images, and the uncertainty map corresponding to each normal vector map, thereby optimizing all learnable parameters of the hybrid geometric representation neural network model.

[0081] S105. Determine the three-dimensional reconstructed surface of the target object based on the zero isosurface of the directed distance field obtained from the trained hybrid geometric representation neural network model.

[0082] For a trained hybrid geometric representation neural network model, all its parameters are fixed. By extracting the zero isosurface of the SDF field predicted by the geometric modules of the hybrid geometric representation neural network model, a high-quality 3D reconstructed surface of the target object can be obtained.

[0083] In one optional implementation, the three-dimensional reconstructed surface of the target object is determined based on the zero isosurface of the directed distance field obtained from the trained hybrid geometric representation neural network model. Specifically, this may include: keeping all parameters of the trained hybrid geometric representation neural network model unchanged; uniformly sampling the coordinates of 1024×1024×1024 points within a unit cube with a side length of 1 and inputting them into the geometric module of the trained hybrid geometric representation neural network model to obtain the directed distance field value corresponding to each point, forming a directed distance field; using the Marching Cubes algorithm to extract the zero isosurface of the directed distance field to obtain a normalized three-dimensional surface; and performing inverse normalization scaling on the normalized three-dimensional surface to obtain the three-dimensional reconstructed surface of the target object.

[0084] The 3D surface reconstruction method provided in this embodiment acquires multiple RGB images of the target object; sharpens and denoises each RGB image to obtain multiple preprocessed RGB images of the target object; determines the normal vector map corresponding to each preprocessed RGB image and the uncertainty map corresponding to each normal vector map; trains a hybrid geometric representation neural network model using the multiple preprocessed RGB images, the corresponding normal vector maps, and the uncertainty maps corresponding to each normal vector map to obtain a trained hybrid geometric representation neural network model. The hybrid geometric representation neural network model includes a geometric module and a color module. The geometric module includes a multilayer perceptron representation module, a three-plane feature representation module, and a decoder; based on the zero isosurface of the directed distance field obtained from the trained hybrid geometric representation neural network model, the 3D reconstructed surface of the target object is determined, thus realizing the 3D surface reconstruction of the target object. By introducing image sharpening and denoising techniques to preprocess the acquired RGB images, the image quality is enhanced, which can improve the accuracy of the predicted normal vector prior knowledge and help improve the accuracy of the reconstruction results. Furthermore, by using hybrid representation to encode the geometric surface, the low-frequency region (floor, walls, etc.) and high-frequency region (delicate furniture, small items on the table, etc.) of the indoor scene surface can be expressed more effectively. The reconstructed 3D surface is of higher quality and has richer details, thus improving the quality of 3D surface reconstruction.

[0085] Based on the above embodiments, the specific implementation of step S104 will be described in detail below. In the three-dimensional surface reconstruction method provided in this embodiment, multiple preprocessed RGB images, normal vector maps corresponding to the multiple preprocessed RGB images, and uncertainty maps corresponding to each normal vector map are used to train a hybrid geometric representation neural network model. Specifically, this may include:

[0086] S1041. Determine the light ray emitted from the camera center to a pixel in the image based on the camera pose parameters corresponding to a preprocessed RGB image, and sample a preset number of points at equal intervals on the light ray.

[0087] The camera pose parameters can be obtained by calibrating the RGB images of the target object. For example, image calibration tools such as OpenMVG and OpenMVS can be used, following the official documentation provided by these tools. After calibration, the camera's intrinsic parameter matrix and the extrinsic parameter matrix for each RGB image can be obtained. The calibrated camera pose parameter matrix is ​​then normalized and scaled so that the camera position in all viewpoints is within a unit sphere of radius 1.

[0088] Randomly select a pixel in an image, calculate the parametric equation of the ray r emitted from the camera center to that pixel based on the camera pose parameters corresponding to the image, and sample M points at equal intervals along this ray r.

[0089] S1042. Input the coordinates of each point into the multilayer perceptron representation module of the geometry module to obtain the directed distance field value and latent vector corresponding to each point.

[0090] Let each point x i The coordinates are input into the MLP branch of the geometry module to obtain the SDF values ​​s corresponding to each point. i and latent vector f i .

[0091] S1043. Input the coordinates of each point into the three-plane feature representation module and the branch where the decoder is located in the geometry module to obtain the directed distance field residual and the latent vector residual corresponding to each point.

[0092] Let each point x i The coordinate input geometry module's three-plane feature representation plus decoder branch yields the SDF residual Δs corresponding to each point. i and the implicit vector residual Δf i .

[0093] S1044. The sum of the directed distance field value and the corresponding directed distance field residual is determined as the final directed distance field value output by the geometry module, and the sum of the latent vector and the corresponding latent vector residual is determined as the final latent vector output by the geometry module.

[0094] Adding the outputs of the two branches yields the final SDF value s of the geometry module. i ′=s i +Δs i and latent vector f i ′=f i +Δf i .

[0095] S1045. Calculate the gradient of the directed distance field value finally output by the geometry module to obtain its corresponding normal vector value.

[0096] Calculate the gradient of the SDF value to obtain its corresponding normal vector value.

[0097] S1046. Input the coordinates, normal vector values, and implicit vectors of each point into the color module to obtain the radiance corresponding to each point.

[0098] Let each point x i coordinates, normal vector n i Latent vector f i Input the color module to obtain the radiance c of each point.i .

[0099] S1047. Based on the directional distance field value, radiance and normal vector value of each point, the volume rendering equation is used to render the pixel color value and rendering normal vector value corresponding to the light.

[0100] According to the rendering equation of volume rendering, from these M sampling points, we obtain the pixel color value C(r) and normal vector value N(r) corresponding to the ray r:

[0101]

[0102] in, Φ is the Sigmoid function, and τ is a learnable parameter.

[0103] S1048. The gradient descent algorithm is adopted to optimize all learnable parameters of the hybrid geometric representation neural network model with the goal of minimizing the preset loss function. The preset loss function is determined based on the rendered pixel color value and rendered normal vector value corresponding to the light, the real pixel color value and real normal vector value corresponding to the light, and the prior uncertainty of the normal vector corresponding to the light.

[0104] The gradient descent algorithm is used to optimize all learnable parameters of the hybrid geometric representation neural network model, minimizing a preset loss function. In one optional implementation, the preset loss function is determined according to the following expression:

[0105]

[0106] Where L represents the preset loss function, r represents the ray, and C(r) represents the color value of the rendered pixel corresponding to ray r. This represents the actual pixel color value corresponding to the ray r, and M represents the preset quantity. Let represent the normal vector value corresponding to the i-th point, U(r) represent the prior uncertainty of the normal vector corresponding to ray r, and N(r) represent the rendered normal vector value corresponding to ray r. This represents the true normal vector value corresponding to ray r.

[0107] S1049. Repeat the above training process until the hybrid geometric representation neural network model converges.

[0108] Go to S1041 and repeat the above process until the model converges.

[0109] The three-dimensional surface reconstruction method provided in this embodiment, based on the above embodiment, further employs an uncertainty estimation module to estimate the prior uncertainty of each normal vector map, and designs a loss function based on the estimated uncertainty. Based on this loss function, in the optimization process of the hybrid geometric representation neural network model, the model can be guided to focus on the normal vector prior with lower uncertainty, avoid being misled by the prior with high uncertainty, and help improve the accuracy of the reconstruction results.

[0110] Understandably, the learnable parameters of a hybrid geometric representation neural network model need to be initialized before training it. Building upon the above embodiments, to further improve the model's convergence speed and shorten its training time, the 3D surface reconstruction method provided in this embodiment initializes the learnable parameters of the multilayer perceptron representation module and the three-plane feature representation module before training the hybrid geometric representation neural network model. This ensures that the initially output directed distance field value of the branch containing the multilayer perceptron representation module is approximately equal to the directed distance field value of a spherical shell with a radius of 1 in space, and that the initially output directed distance field value of the branch containing the three-plane feature representation module is approximately 0.

[0111] In summary, the 3D surface reconstruction method provided in this application enhances image quality and improves the accuracy of prior knowledge of normal vectors by introducing image enhancement techniques during the 3D surface reconstruction process. Through image sharpening and denoising operations, it improves image quality and increases the accuracy of prior knowledge of normal vectors, thereby contributing to the improvement of reconstruction accuracy. Using hybrid representation to encode geometric surfaces can more effectively represent low-frequency regions (floors, walls, etc.) and high-frequency regions (delicate furniture, small items on tabletops, etc.) of indoor scene surfaces, resulting in reconstructed 3D surfaces with richer details. An uncertainty estimation module is used to estimate the uncertainty of the prior knowledge of each normal vector map, and a loss function is designed based on the estimated uncertainty. This loss function guides the model to focus on normal vector priors with lower uncertainty during the optimization of the hybrid geometric representation neural network model, avoiding being misled by high-uncertainty priors, thus contributing to the improvement of reconstruction accuracy. The 3D surfaces reconstructed using the method provided in this application are of higher quality and have richer details. The reconstruction effect can be referenced. Figure 3 As shown. To illustrate the effectiveness of this application, it was evaluated and compared with existing technologies on the publicly available benchmark dataset ScanNet. The evaluation metrics were accuracy, completeness, precision, recall, and F-score, and the definitions of these metrics are shown in Table 1 (P and P). * These refer to the set of sampling points on the reconstructed surface and the set of sampling points on the truth surface, respectively.

[0112] Table 1

[0113]

[0114] Existing technologies used for comparison include NeuS, NeuRIS, and MonoSDF technologies. The evaluation and comparison results are shown in Table 2.

[0115] Table 2

[0116] Accuracy ↓ Completeness ↓ Accuracy ↑ Recall rate ↑ F-score↑ NeuS technology 0.179 0.208 0.313 0.275 0.291 NeuRIS technology 0.051 0.048 0.720 0.674 0.696 MonoSDF technology 0.035 0.048 0.799 0.681 0.733 This application <![CDATA[ 0.033 ]]> <![CDATA[ 0.041 ]]> <![CDATA[ 0.814 ]]> <![CDATA[ 0.737 ]]> <![CDATA[ 0.773 ]]>

[0117] As can be seen, the reconstruction results obtained in this application surpass those of existing technologies in all aspects, which means that the three-dimensional surface reconstructed by this invention is of high quality. In particular, the improvement in accuracy and completeness indicators shows that the reconstruction results of this application have richer and more accurate details.

[0118] Figure 4 This is a schematic diagram of the structure of a three-dimensional surface reconstruction device provided in an embodiment of the present invention. Figure 4 As shown, the three-dimensional surface reconstruction device 40 provided in this embodiment may include: an acquisition module 401, a preprocessing module 402, a determination module 403, a training module 404, and a reconstruction module 405.

[0119] The acquisition module 401 is used to acquire multiple RGB images of the target object;

[0120] The preprocessing module 402 is used to perform sharpening and denoising processing on each RGB image to obtain multiple preprocessed RGB images of the target object.

[0121] The determination module 403 is used to determine the normal vector map corresponding to each preprocessed RGB image and the uncertainty map corresponding to each normal vector map;

[0122] The training module 404 is used to train the hybrid geometric representation neural network model using multiple preprocessed RGB images, the normal vector maps corresponding to the multiple preprocessed RGB images, and the uncertainty map corresponding to each normal vector map, to obtain a trained hybrid geometric representation neural network model, which includes a geometric module and a color module.

[0123] The reconstruction module 405 is used to determine the three-dimensional reconstructed surface of the target object based on the zero isosurface of the directed distance field obtained from the trained hybrid geometric representation neural network model.

[0124] The apparatus of this embodiment can be used to perform Figure 1 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0125] In one optional implementation, the preprocessing module 402 is used to sharpen each RGB image, specifically including: convolving the RGB image with a 3×3 Laplacian convolution kernel; the preprocessing module 402 is used to denoise each RGB image, specifically including: performing median filtering on the RGB image.

[0126] In one optional implementation, the determining module 403 is used to determine the normal vector map corresponding to each preprocessed RGB image and the uncertainty map corresponding to each normal vector map, specifically including:

[0127] Monocular prediction technology is used to determine the normal vector map corresponding to each preprocessed RGB image;

[0128] Based on the visual basic model, the high-dimensional semantic features corresponding to each preprocessed RGB image are determined;

[0129] Each preprocessed RGB image, its corresponding normal vector map, and its corresponding high-dimensional semantic features are input into a pre-trained uncertainty estimation module to obtain the uncertainty map corresponding to each normal vector map.

[0130] In one optional implementation, the training module 404 is used to train the hybrid geometric representation neural network model using multiple preprocessed RGB images, normal vector maps corresponding to the multiple preprocessed RGB images, and uncertainty maps corresponding to each normal vector map, including:

[0131] Based on the camera pose parameters corresponding to a preprocessed RGB image, determine the light ray emitted from the camera center to a pixel in the image, and sample a preset number of points at equal intervals on the light ray;

[0132] Input the coordinates of each point into the multilayer perceptron representation module of the geometry module to obtain the directed distance field value and latent vector corresponding to each point;

[0133] Input the coordinates of each point into the three-plane feature representation module and the branch of the decoder in the geometry module to obtain the directed distance field residual and latent vector residual corresponding to each point;

[0134] The sum of the directed distance field value and the corresponding directed distance field residual is determined as the final directed distance field value output by the geometry module, and the sum of the latent vector and the corresponding latent vector residual is determined as the final latent vector output by the geometry module.

[0135] The gradient of the directed distance field value finally output by the geometry module is calculated to obtain its corresponding normal vector value;

[0136] Input the coordinates, normal vector values, and implicit vectors of each point into the color module to obtain the radiance of each point.

[0137] Based on the directional distance field value, radiance and normal vector value of each point, the volume rendering equation is used to render the pixel color value and rendering normal vector value corresponding to the light.

[0138] The gradient descent algorithm is used to optimize all learnable parameters of the hybrid geometric representation neural network model with the goal of minimizing the preset loss function. The preset loss function is determined based on the rendered pixel color value and rendered normal vector value corresponding to the light, the real pixel color value and real normal vector value corresponding to the light, and the prior uncertainty of the normal vector corresponding to the light.

[0139] Repeat the above training process until the hybrid geometric representation neural network model converges.

[0140] In one optional implementation, the training module 404 is further configured to initialize the learnable parameters of the multilayer perceptron representation module and the three-plane feature representation module before training the hybrid geometric representation neural network model, such that the initially output directed distance field value of the multilayer perceptron representation module is approximately equal to the directed distance field value of a spherical shell with a radius of 1 in space, and the initially output directed distance field value of the three-plane feature representation module is approximately 0.

[0141] In one optional implementation, the preset loss function is determined according to the following expression:

[0142]

[0143] Where L represents the preset loss function, r represents the ray, and C(r) represents the color value of the rendered pixel corresponding to ray r. This represents the actual pixel color value corresponding to the ray r, and M represents the preset quantity. Let represent the normal vector value corresponding to the i-th point, U(r) represent the prior uncertainty of the normal vector corresponding to ray r, and N(r) represent the rendered normal vector value corresponding to ray r. This represents the true normal vector value corresponding to ray r.

[0144] In one optional implementation, the reconstruction module 405 is used to determine the three-dimensional reconstructed surface of the target object based on the zero isosurface of the directed distance field obtained from the trained hybrid geometric representation neural network model, including:

[0145] All parameters of the pre-trained hybrid geometric representation neural network model remain unchanged;

[0146] The coordinates of 1024×1024×1024 points are uniformly sampled within a unit cube with a side length of 1 and input into the geometric module of the trained hybrid geometric representation neural network model to obtain the directed distance field value corresponding to each point, thus forming a directed distance field.

[0147] Extract the zero isosurface of the directed distance field to obtain the normalized three-dimensional surface;

[0148] The normalized 3D surface is reverse normalized and scaled to obtain the 3D reconstructed surface of the target object.

[0149] This invention also provides an electronic device, please refer to [link to relevant documentation]. Figure 5 As shown, the embodiments of the present invention are only used as examples. Figure 5 The examples are provided for illustration only and do not imply that the invention is limited to these examples. Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. Figure 5 As shown, the electronic device 50 provided in this embodiment may include: a memory 501, a processor 502, and a bus 503. The bus 503 is used to connect the various components.

[0150] The memory 501 stores a computer program, which, when executed by the processor 502, can implement the technical solutions of any of the above method embodiments.

[0151] The memory 501 and the processor 502 are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines, such as bus 503. The memory 501 stores a computer program for implementing a three-dimensional surface reconstruction method, including at least one software functional module that can be stored in the memory 501 in the form of software or firmware. The processor 502 executes various functional applications and data processing by running the software program and modules stored in the memory 501.

[0152] The memory 501 may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 501 stores programs, and the processor 502 executes the programs after receiving execution instructions. Furthermore, the software programs and modules within the memory 501 may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.) and can communicate with various hardware or software components to provide an operating environment for other software components.

[0153] Processor 502 can be an integrated circuit chip with signal processing capabilities. The aforementioned processor 502 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. It is understood that... Figure 5 The structure shown is for illustrative purposes only and may include more... Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown. Figure 5 The components shown can be implemented in hardware and / or software.

[0154] This invention also provides a computer-readable storage medium storing a computer program thereon, which is executed by a processor to implement the technical solutions of any of the above method embodiments.

[0155] The various embodiments in this disclosure are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0156] The scope of protection of this disclosure is not limited to the embodiments described above. Those skilled in the art can make various modifications and variations to this disclosure without departing from its scope and spirit. If such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A three-dimensional surface reconstruction method, characterized in that, include: Obtain multiple RGB images of the target object; Each RGB image is sharpened and denoised to obtain multiple pre-processed RGB images of the target object. Determine the normal vector map corresponding to each preprocessed RGB image and the uncertainty map corresponding to each normal vector map; The hybrid geometric representation neural network model is trained using the multiple preprocessed RGB images, the normal vector maps corresponding to the multiple preprocessed RGB images, and the uncertainty maps corresponding to each normal vector map, to obtain a trained hybrid geometric representation neural network model. The hybrid geometric representation neural network model includes a geometric module and a color module. The geometric module includes a multilayer perceptron representation module, a three-plane feature representation module, and a decoder. The three-dimensional reconstructed surface of the target object is determined based on the zero isosurface of the directed distance field obtained from the trained hybrid geometric representation neural network model. The step of training the hybrid geometric representation neural network model using the multiple preprocessed RGB images, the normal vector maps corresponding to the multiple preprocessed RGB images, and the uncertainty map corresponding to each normal vector map includes: Based on the camera pose parameters corresponding to a preprocessed RGB image, determine the light ray emitted from the camera center to a pixel in the image, and sample a preset number of points at equal intervals on the light ray; Input the coordinates of each point into the multilayer perceptron representation module of the geometry module to obtain the directed distance field value and latent vector corresponding to each point; Input the coordinates of each point into the three-plane feature representation module and the branch of the decoder of the geometry module to obtain the directed distance field residual and latent vector residual corresponding to each point; The sum of the directed distance field value and the corresponding directed distance field residual is determined as the final directed distance field value output by the geometry module, and the sum of the latent vector and the corresponding latent vector residual is determined as the final latent vector output by the geometry module. The gradient of the directed distance field value finally output by the geometric module is calculated to obtain its corresponding normal vector value; Input the coordinates, normal vector values, and implicit vectors of each point into the color module to obtain the radiance of each point; Based on the directional distance field value, radiance and normal vector value of each point, the volume rendering equation is used to render the pixel color value and rendering normal vector value corresponding to the light ray. The gradient descent algorithm is used to optimize all learnable parameters of the hybrid geometric representation neural network model with the goal of minimizing the preset loss function. The preset loss function is determined based on the rendered pixel color value and rendered normal vector value corresponding to the ray, the real pixel color value and real normal vector value corresponding to the ray, and the prior uncertainty of the normal vector corresponding to the ray. Repeat the above training process until the hybrid geometric representation neural network model converges.

2. The method according to claim 1, characterized in that, Sharpening each RGB image includes: convolving the RGB image with a 3×3 Laplacian convolution kernel; denoising each RGB image includes: performing median filtering on the RGB image.

3. The method according to claim 1, characterized in that, The process of determining the normal vector map corresponding to each preprocessed RGB image and the uncertainty map corresponding to each normal vector map includes: Monocular prediction technology is used to determine the normal vector map corresponding to each preprocessed RGB image; Based on the visual basic model, the high-dimensional semantic features corresponding to each preprocessed RGB image are determined; Each preprocessed RGB image, its corresponding normal vector map, and its corresponding high-dimensional semantic features are input into a pre-trained uncertainty estimation module to obtain the uncertainty map corresponding to each normal vector map.

4. The method according to claim 1, characterized in that, Before training the hybrid geometric representation neural network model, the method further includes: The learnable parameters of the multilayer perceptron representation module and the three-plane feature representation module are initialized such that the initially output directed distance field value of the multilayer perceptron representation module is approximately equal to the directed distance field value of a spherical shell with a radius of 1 in space, and the initially output directed distance field value of the three-plane feature representation module is approximately 0.

5. The method according to claim 1, characterized in that, The preset loss function is determined according to the following expression: in, Indicates the preset loss function. Indicates light, Indicates light rays The corresponding rendered pixel color value, Indicates light rays The corresponding real pixel color value, Indicates the preset quantity. This represents the normal vector value corresponding to the i-th point. Indicates light rays The corresponding prior uncertainty of the normal vector Indicates light rays The corresponding rendering normal vector value, Indicates light rays The corresponding true normal vector value.

6. The method according to claim 1, characterized in that, The step of determining the 3D reconstructed surface of the target object based on the zero isosurface of the directed distance field obtained from the trained hybrid geometric representation neural network model includes: Keep all parameters of the trained hybrid geometric representation neural network model unchanged; 1024 samples will be uniformly sampled within a unit cube with a side length of 1. 1024 The coordinates of 1024 points are input into the geometric module of the trained hybrid geometric representation neural network model to obtain the directed distance field value corresponding to each point, forming a directed distance field; Extract the zero isosurface of the directed distance field to obtain a normalized three-dimensional surface; The normalized 3D surface is then reverse-normalized and scaled to obtain the 3D reconstructed surface of the target object.

7. A three-dimensional surface reconstruction device, characterized in that, include: The acquisition module is used to acquire multiple RGB images of the target object; The preprocessing module is used to sharpen and denoise each RGB image separately to obtain multiple preprocessed RGB images of the target object; The determination module is used to determine the normal vector map corresponding to each preprocessed RGB image and the uncertainty map corresponding to each normal vector map; The training module is used to train the hybrid geometric representation neural network model using the multiple preprocessed RGB images, the normal vector maps corresponding to the multiple preprocessed RGB images, and the uncertainty map corresponding to each normal vector map, to obtain a trained hybrid geometric representation neural network model, wherein the hybrid geometric representation neural network model includes a geometric module and a color module; The step of training the hybrid geometric representation neural network model using the multiple preprocessed RGB images, the normal vector maps corresponding to the multiple preprocessed RGB images, and the uncertainty map corresponding to each normal vector map includes: Based on the camera pose parameters corresponding to a preprocessed RGB image, determine the light ray emitted from the camera center to a pixel in the image, and sample a preset number of points at equal intervals on the light ray; Input the coordinates of each point into the multilayer perceptron representation module of the geometry module to obtain the directed distance field value and latent vector corresponding to each point; Input the coordinates of each point into the three-plane feature representation module and the branch of the decoder of the geometry module to obtain the directed distance field residual and latent vector residual corresponding to each point; The sum of the directed distance field value and the corresponding directed distance field residual is determined as the final directed distance field value output by the geometry module, and the sum of the latent vector and the corresponding latent vector residual is determined as the final latent vector output by the geometry module. The gradient of the directed distance field value finally output by the geometric module is calculated to obtain its corresponding normal vector value; Input the coordinates, normal vector values, and implicit vectors of each point into the color module to obtain the radiance of each point; Based on the directional distance field value, radiance and normal vector value of each point, the volume rendering equation is used to render the pixel color value and rendering normal vector value corresponding to the light ray. The gradient descent algorithm is used to optimize all learnable parameters of the hybrid geometric representation neural network model with the goal of minimizing the preset loss function. The preset loss function is determined based on the rendered pixel color value and rendered normal vector value corresponding to the ray, the real pixel color value and real normal vector value corresponding to the ray, and the prior uncertainty of the normal vector corresponding to the ray. Repeat the above training process until the hybrid geometric representation neural network model converges; The reconstruction module is used to determine the three-dimensional reconstructed surface of the target object based on the zero isosurface of the directed distance field obtained from the trained hybrid geometric representation neural network model.

8. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the three-dimensional surface reconstruction method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the three-dimensional surface reconstruction method as described in any one of claims 1-6.

Images