Indoor three-dimensional reconstruction method and device based on signed distance field

Abstract

The application discloses an indoor three-dimensional reconstruction method and device based on a signed distance field, and when a signed distance field network and a neural radiance field network are trained, three-dimensional sparse point clouds corresponding to pixel points with large pixel value differences between rendered images and real images are introduced to introduce a detail correction loss, and meanwhile, an overall supervision loss constructed based on the rendered images and the real images is cooperated to optimize the signed distance field network and the neural radiance field network, so that the network prediction performance is stronger, the geometry shape and the illumination appearance of a high-quality reconstruction scene can be considered, and meanwhile, the loss of details such as furniture in an indoor scene can be prevented.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional reconstruction, and particularly relates to an indoor three-dimensional reconstruction method and device based on a signed distance field. BACKGROUND

[0002] Scene modeling has always been a research hotspot in the field of computer vision, and high-precision three-dimensional scene modeling is a prerequisite for the implementation of technologies such as robot perception and virtual reality. Three-dimensional reconstruction generally includes three parts. First, a handheld camera is used to scan the target to be reconstructed from multiple perspectives. Then, the features of the scanned multiple frames of pictures are extracted, matched, and the camera pose is estimated. Finally, the mapping from two-dimensional pixels to three-dimensional coordinate points is completed through stereo vision technology to obtain the final reconstructed model.

[0003] With the development of artificial intelligence and deep learning algorithms, three-dimensional visual reconstruction technology based on neural networks has gradually developed. For example, the patent document CN115147543A discloses a face three-dimensional reconstruction method based on a deep neural network, which includes preprocessing, feature extraction, retrieval and matching steps, and can learn the data distribution law directly from three-dimensional point clouds to perform three-dimensional reconstruction. For another example, the patent document CN109410321A discloses a three-dimensional reconstruction method based on a convolutional neural network, which includes: collecting a two-dimensional face image, obtaining a corrected face image through a face correction module, and obtaining face 3D data through a three-dimensional reconstruction module. The neural three-dimensional reconstruction algorithm disclosed in the two patent documents is an end-to-end automatic optimization algorithm, which does not require excessive human intervention in the optimization process, can save human resources for modeling practitioners, and automatically reproduces realistic reproduction effects that are difficult for humans to manually simulate. In addition, the spatial occupancy size of the neural network representing the three-dimensional scene is usually much smaller than the traditional triangular mesh, and has good spatial continuity.

[0004] However, the reconstruction quality of existing three-dimensional visual reconstruction technology based on neural networks still needs to be improved. In the three-dimensional reconstruction of indoor scenes, there are often problems of missing furniture and missing details. SUMMARY

[0005] In view of the above, the purpose of the present application is to provide an indoor three-dimensional reconstruction method and device based on a signed distance field, which improves the quality of indoor three-dimensional reconstruction and prevents the loss of details such as furniture in indoor scenes.

[0006] To achieve the above-mentioned purpose of the application, an embodiment provides an indoor three-dimensional reconstruction method based on a signed distance field, comprising the following steps:

[0007] Constructing a signed distance field network and a neural radiance field network;

[0008] The three-dimensional points in the scene are input into the signed distance field network and the neural radiance field network respectively for geometry prediction and illumination prediction, and volume rendering is performed according to the geometry prediction result and the illumination prediction result to obtain a rendered image.

[0009] A three-dimensional sparse point cloud of the scene is calculated according to the camera parameters and the depth map, a pixel-by-pixel error between the rendered image and a real image as a supervision label is calculated, and a target three-dimensional sparse point corresponding to a pixel with a high pixel error is screened, geometry prediction is performed on the target three-dimensional sparse point by using the signed distance field network, and a detail correction loss is constructed according to an absolute value of a geometry prediction result corresponding to the target three-dimensional sparse point.

[0010] An overall supervision loss is constructed according to a pixel-by-pixel error between the rendered image and the real image as the supervision label, and the signed distance field network and the neural radiance field network are parameter-optimized by using the detail correction loss and the overall supervision loss.

[0011] Geometry prediction and illumination prediction are performed by using the parameter-optimized signed distance field network and the neural radiance field network, and three-dimensional reconstruction and volume rendering are performed according to the geometry prediction result and the illumination prediction result.

[0012] In one embodiment, the signed distance field network is represented as a signed distance function defined as a minimum value of distances of any three-dimensional point in the scene to all surfaces in the scene, the geometry prediction result is represented as a signed distance function value, and a point at which the signed distance function value is zero is represented as a zero point of the signed distance function.

[0013] The three-dimensional point is input into the neural radiance field network, and an RGB value is calculated and output, which is used as the illumination prediction result.

[0014] In one embodiment, the volume rendering is performed according to the geometry prediction result and the illumination prediction result to obtain the rendered image, including:

[0015] According to a peak point of the Laplace surface distribution being located at the zero point of the signed distance function, the signed distance function is corresponded to a density function of the Laplace surface distribution, a density value is calculated according to the corresponding relationship and the signed distance function value as the geometry prediction result, and a final color value of each pixel is obtained by performing numerical integration on the RGB value as the illumination prediction result and the density value to form the rendered image.

[0016] In one embodiment, the camera parameters and the depth map are used to calculate the three-dimensional sparse point cloud of the scene, including:

[0017] According to the intrinsic parameters, the extrinsic parameters of the camera and the depth map of the scene, a three-dimensional point coordinate corresponding to a depth pixel value of the depth map is calculated by using an inverse projection algorithm, and all three-dimensional point coordinates form the three-dimensional sparse point cloud of the scene.

[0018] In an embodiment, the method further comprises: inputting the three-dimensional points in the scene into the signed distance field network for geometry prediction, and then performing volume rendering to obtain depth information; constructing a depth supervision loss function according to the depth information and a depth map as a supervision label; and performing parameter optimization on the signed distance field network and the neural radiance field network by using the depth supervision loss function, the detail correction loss, and the overall supervision loss.

[0019] In an embodiment, the method further comprises: inputting the three-dimensional points in the scene into the signed distance field network for geometry prediction, and then performing volume rendering to obtain normal information; constructing a normal supervision loss function according to the normal information and a normal map as a supervision label; performing parameter optimization on the signed distance field network and the neural radiance field network by using the normal supervision loss function, the detail correction loss, and the overall supervision loss, or performing parameter optimization on the signed distance field network and the neural radiance field network by using the normal supervision loss function, the depth supervision loss function, the detail correction loss, and the overall supervision loss.

[0020] In an embodiment, the three-dimensional reconstruction according to the geometry prediction result comprises:

[0021] An isosurface extraction algorithm is used to extract isosurfaces of the signed distance function value as the geometry prediction result, to extract the explicit geometric structure defined by the signed distance field, and to reproduce the explicit geometric structure in the form of a triangular mesh.

[0022] To achieve the above-mentioned purposes, the embodiments provide an indoor three-dimensional reconstruction device based on a signed distance field, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned indoor three-dimensional reconstruction method based on a signed distance field when executing the computer program.

[0023] Compared with the prior art, the present application has at least the following beneficial effects:

[0024] During training of the signed distance field network and the neural radiance field network, the detail correction loss is introduced based on three-dimensional sparse point clouds corresponding to pixel points with large pixel value differences between the rendered image and the real image, and the overall supervision loss is constructed based on the rendered image and the real image, so as to optimize the signed distance field network and the neural radiance field network, so that the network prediction performance is stronger, the geometry shape and the lighting appearance of the reconstructed scene can be considered, and the loss of details such as furniture in the indoor scene can be prevented. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0026] Figure 1 is a flowchart of the indoor three-dimensional reconstruction method based on the signed distance field provided by the embodiment;

[0027] Figure 2 is a flowchart of network training provided by the embodiment;

[0028] Figure 3 is an application flowchart provided by the embodiment. DETAILED DESCRIPTION

[0029] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0030] For the three-dimensional reconstruction of the indoor scene in the prior method, the problems of missing furniture and missing details often occur. The embodiment of the present application proposes an indoor three-dimensional reconstruction method and device based on the signed distance field, which completes and optimizes the missing object details in the reconstruction on the signed distance field, so as to realize a high-quality indoor scene reconstruction result.

[0031] Figure 1 is a flowchart of the indoor three-dimensional reconstruction method based on the signed distance field provided by the embodiment, Figure 2 is a flowchart of network training provided by the embodiment. As shown in Figure 1 and Figure 2 The indoor three-dimensional reconstruction method based on the signed distance field provided by the embodiment includes the following steps:

[0032] Step 1, constructing a signed distance field network and a neural radiation field network.

[0033] In the embodiment, a neural network such as a multi-layer perceptron is used as a network to build a signed distance field network and a neural radiance field network. The signed distance field network is used to express the geometry of a three-dimensional scene, and specifically, a geometric prediction is performed according to three-dimensional point coordinates to obtain a geometric prediction result. The fitting target of the three-dimensional scene is a signed distance field, which is defined based on a signed distance function defined on a surface, that is, the signed distance field network is represented as a signed distance function defined as the minimum distance from any three-dimensional point in the scene to all surfaces of the scene. The geometric prediction result is represented as a signed distance function value. When the signed distance function value is zero, it indicates a zero point of the signed distance field. When the signed distance function value is greater than zero, it indicates that the three-dimensional point is inside the scene. When the signed distance function value is less than zero, it indicates that the three-dimensional point is outside the scene. The three-dimensional points with a signed distance function value of zero are extracted to form a geometric surface.

[0034] The neural radiance field network is used to express the illumination of the scene, and specifically, an illumination prediction is performed according to three-dimensional point coordinates to output a third RGB value as an illumination prediction result.

[0035] In step 2, the three-dimensional points in the scene are input into the signed distance field network and the neural radiance field network for geometric prediction and illumination prediction, and volume rendering is performed according to the geometric prediction result and the illumination prediction result to obtain a rendered image.

[0036] In the embodiment, the importance sampling method is used to sample a plurality of three-dimensional points on the transmitted light in the field of view, and the three-dimensional point coordinates are input into the signed distance field network and the neural radiance field network for geometric prediction and illumination prediction. Then, volume rendering is performed according to the geometric prediction result and the illumination prediction result to obtain a rendered image. The specific process of volume rendering includes:

[0037] According to the fact that the peak point of the Laplace surface distribution is located at the zero point of the signed distance function, the signed distance function is corresponded to the density function of the Laplace surface distribution. The density value is calculated according to the corresponding relationship and the signed distance function value as the geometric prediction result. The RGB value as the illumination prediction result is numerically integrated with the density value to obtain the final color value of each pixel to form a rendered image.

[0038] In the embodiment, the depth information and the normal information of the scene are superimposed using the volume rendering method. Specifically, the gradient of the signed distance function and the density function of the Laplace surface distribution are weighted and accumulated to obtain the normal information. The distance from the sampling point to the camera position and the density function of the Laplace surface distribution are weighted and accumulated to obtain the depth information.

[0039] In step 3, a three-dimensional sparse point cloud of the scene is calculated according to the camera parameters and the depth map.

[0040] In the embodiment, according to the intrinsic parameters, extrinsic parameters and depth map of the camera, the corresponding three-dimensional point coordinates are calculated from the depth pixel values of the depth map by an inverse projection algorithm, and all the three-dimensional point coordinates form a three-dimensional sparse point cloud of the scene.

[0041] In step 4, a detail correction loss is constructed by using the three-dimensional sparse point cloud.

[0042] In the embodiment, the pixel-wise error between the rendered image and the real image as the supervision label is calculated, and the target three-dimensional sparse point corresponding to the pixel with high pixel-wise error is screened, the signed distance function value is supervised by using the target three-dimensional sparse point, and the signed distance field network is optimized, that is, the target three-dimensional sparse point is geometrically predicted by using the signed distance field network, and the detail correction loss is constructed according to the absolute value of the geometric prediction result corresponding to the target three-dimensional sparse point, so as to minimize the detail correction loss as the training target to make the target three-dimensional sparse point located at the zero point of the signed distance field.

[0043] In step 5, an overall supervision loss is constructed according to the rendered image, and parameter optimization is performed.

[0044] In the embodiment, the overall supervision loss is constructed according to the pixel-wise error between the rendered image and the real image as the supervision label, and the signed distance field network and the neural radiance field network are parameter optimized by using the detail correction loss and the overall supervision loss by using the gradient descent algorithm, and the optimization target of the overall supervision loss is to minimize the pixel-wise error.

[0045] In one possible implementation, a depth supervision loss function is constructed according to the depth information and the depth map as the supervision label; the signed distance field network and the neural radiance field network are parameter optimized by using the depth supervision loss function, the detail correction loss and the overall supervision loss by using the gradient descent algorithm, and the optimization target of the depth supervision loss function is to minimize the difference between the depth information and the depth map.

[0046] In one possible implementation, a normal supervision loss function is constructed according to the normal information and the normal map as the supervision label; the signed distance field network and the neural radiance field network are parameter optimized by using the normal supervision loss function, the detail correction loss and the overall supervision loss by using the gradient descent algorithm, and the optimization target of the normal supervision loss function is to minimize the difference between the normal information and the normal map.

[0047] In one possible implementation, the signed distance field network and the neural radiance field network are parameter optimized by using the normal supervision loss function, the depth supervision loss function, the detail correction loss and the overall supervision loss by using the gradient descent algorithm.

[0048] Any one of the above parameter optimization methods can achieve zero point optimization, i.e., optimizing the lost furniture on the signed distance field, to improve the prediction accuracy of each detail of the network, because the detail correction loss is used in each method.

[0049] Step 6, performing geometry prediction and illumination prediction by using the parameter-optimized signed distance field network and the neural radiance field network, and performing three-dimensional reconstruction and volume rendering according to the geometry prediction result and the illumination prediction result.

[0050] In the embodiment, after obtaining the parameter-optimized signed distance field network and the neural radiance field network, as shown in Figure 3 the signed distance field network and the neural radiance field network are used to perform geometry prediction and illumination prediction of three-dimensional points in a new view, and three-dimensional reconstruction and volume rendering are performed according to the geometry prediction result and the illumination prediction result.

[0051] For three-dimensional reconstruction, an isosurface extraction algorithm is used to extract the isosurface of the signed distance function value as the geometry prediction result, extract the explicit geometric structure defined by the signed distance field, and reproduce it in the form of a triangular mesh.

[0052] For volume rendering, according to the corresponding relationship between the signed distance function and the density function of the Laplace surface distribution, the density value corresponding to the signed distance function value output by the signed distance field network is calculated, and then the density value and the RGB value output by the neural radiance field network are numerically integrated to superimpose the RGB value to obtain the final color value of each pixel to form a rendering image in a new view.

[0053] The indoor three-dimensional reconstruction method based on the signed distance field provided by the embodiment can reconstruct a three-dimensional model of an indoor scene with high quality and without loss, and render a realistic scene image.

[0054] Based on the same inventive concept, the embodiment further provides an indoor three-dimensional reconstruction device based on a signed distance field, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the above-mentioned indoor three-dimensional reconstruction method based on the signed distance field when executing the computer program, comprising the following steps:

[0055] Step 1, constructing a signed distance field network and a neural radiance field network;

[0056] Step 2, inputting three-dimensional points in the scene into the signed distance field network and the neural radiance field network respectively to perform geometry prediction and illumination prediction, and performing volume rendering according to the geometry prediction result and the illumination prediction result to obtain a rendering image;

[0057] Step 3, calculating a three-dimensional sparse point cloud of the scene according to camera parameters and a depth map;

[0058] Step 4, constructing a detail correction loss by using a three-dimensional sparse point cloud;

[0059] Step 5, constructing an overall supervision loss according to a rendered image, and performing parameter optimization;

[0060] Step 6, performing geometry prediction and illumination prediction by using a parameter-optimized signed distance field network and a neural radiance field network, and performing three-dimensional reconstruction and volume rendering according to the geometry prediction result and the illumination prediction result.

[0061] The memory can be a volatile memory, such as a RAM, at a near end, can also be a non-volatile memory, such as a ROM, a FLASH, a floppy disk, a mechanical hard disk, etc., and can also be a storage cloud at a far end. The processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), i.e., the steps of the indoor three-dimensional reconstruction method based on the signed distance field can be implemented by these processors.

[0062] The above detailed description of the specific embodiments has described the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application, and is not intended to limit the present application. Any modification, supplement, and equivalent replacement, etc. made within the principle range of the present application shall be included in the protection scope of the present application.

Claims

1. A method for indoor 3D reconstruction based on signed distance field, characterized in that, The method comprises the following steps: constructing a signed distance field network and a neural radiance field network, wherein the signed distance field network is represented as a signed distance function defined as the minimum value of the distance of any three-dimensional point in the scene to all surfaces of the scene, the geometry prediction result is represented as a signed distance function value, and the signed distance function value of zero represents a zero point of the signed distance function; inputting the three-dimensional points in the scene into the signed distance field network and the neural radiance field network for geometry prediction and illumination prediction, and performing volume rendering based on the geometry prediction result and the illumination prediction result to obtain a rendered image; calculating a three-dimensional sparse point cloud of the scene according to camera parameters and a depth map, calculating a pixel-by-pixel error between the rendered image and a real image as a supervision label, and screening target three-dimensional sparse points corresponding to pixels with high pixel errors; performing geometry prediction on the target three-dimensional sparse points by using the signed distance field network, and constructing a detail correction loss based on the absolute value of the geometry prediction result corresponding to the target three-dimensional sparse points; constructing an overall supervision loss based on the pixel-by-pixel error between the rendered image and the real image as the supervision label, and optimizing parameters of the signed distance field network and the neural radiance field network by using the detail correction loss and the overall supervision loss; performing geometry prediction and illumination prediction by using the signed distance field network and the neural radiance field network with the optimized parameters, and performing three-dimensional reconstruction and volume rendering based on the geometry prediction result and the illumination prediction result; wherein the volume rendering based on the geometry prediction result and the illumination prediction result to obtain the rendered image comprises: corresponding the signed distance function to a density function of a Laplace surface distribution based on the fact that a peak point of the Laplace surface distribution is located at the zero point of the signed distance function, calculating a density value based on the corresponding relationship and the signed distance function value as the geometry prediction result, and performing numerical integration on the density value and the illumination prediction result to superimpose the illumination prediction result to obtain a final color value of each pixel to form the rendered image; wherein the three-dimensional reconstruction based on the geometry prediction result comprises: performing isosurface extraction on the signed distance function value as the geometry prediction result by using an isosurface extraction algorithm, extracting an explicit geometric structure defined by the signed distance field, and reproducing the explicit geometric structure in the form of a triangular mesh.

2. The method of claim 1, wherein, The three-dimensional points are input into the neural radiance field network to output an RGB value, which is used as the illumination prediction result.

3. The method of claim 1, wherein, The camera parameters and the depth map are used to calculate the three-dimensional sparse point cloud of the scene, comprising: calculating three-dimensional point coordinates corresponding to depth pixel values of the depth map from the depth map by using an inverse projection algorithm based on intrinsic and extrinsic parameters of a camera and a depth map of the scene, and all the three-dimensional point coordinates form the three-dimensional sparse point cloud of the scene.

4. The method of claim 1, wherein, Further comprising: inputting the three-dimensional points in the scene into the signed distance field network for geometry prediction, and then performing volume rendering to obtain depth information; constructing a depth supervision loss function based on the depth information and a depth map as a supervision label; and optimizing parameters of the signed distance field network and the neural radiance field network by using the depth supervision loss function, the detail correction loss, and the overall supervision loss.

5. The method according to claim 1 or 4, characterized in that, Further comprising: The three-dimensional points in the scene are input to a signed distance field network for geometry prediction, and then body rendering is performed to obtain normal information; a normal supervision loss function is constructed according to the normal information and a normal map as a supervision label; or a normal supervision loss function, a depth supervision loss function, a detail correction loss and an overall supervision loss are used to optimize parameters of the signed distance field network and the neural radiance field network.

6. An indoor three-dimensional reconstruction apparatus based on a signed distance field, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the computer program to realize the indoor three-dimensional reconstruction method based on the signed distance field in any one of claims 1-4. The processor implements the computer program to realize the indoor three-dimensional reconstruction method based on the signed distance field in any one of claims 1-4.

Images