Indoor three-dimensional scene reconstruction method and system based on implicit coding and geometric prior

Abstract

The invention discloses an implicit coding and geometric prior-based indoor three-dimensional scene reconstruction method and system. The method comprises the following steps of: performing data processing on image data to be reconstructed to obtain corresponding pre-processed data; calculating rays under a world coordinate system corresponding to pixels needing to be trained on the image data according to internal and external parameters of the camera; inputting a three-dimensional coordinate code and an observation angle code of a sampling point on a ray and an implicit appearance code of a corresponding image into the neural radiation field model to obtain a predicted density value and a predicted color value, and performing volume rendering prediction based on a symbolic distance function to obtain a pixel value so as to obtain a trained neural radiation field model; and obtaining a standard implicit appearance code based on the density value and the color value, and inputting data at least comprising the standard implicit appearance code into the trained neural radiation field model to obtain scene geometric information. According to the method, the dependence of the existing reconstruction scheme on the three-dimensional consistency of the acquired data can be reduced, and the robustness and representation capability of the algorithm in a complex scene can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of infrastructure, and in particular to a method and system for indoor three-dimensional scene reconstruction based on implicit coding and geometric prior. Background Art

[0002] Scene reconstruction based on multi-view stereo vision: It can generate dense 3D point clouds. This method relies on multi-view stereo vision. Figure 1 The consistency assumption is to estimate the 3D structure of the scene by matching features between views. Therefore, it performs poorly in occluded areas, weak texture areas, and strong reflection areas, and it is difficult to obtain high-confidence depth values, resulting in large holes in the final output point cloud, affecting the integrity of scene reconstruction.

[0003] Reconstruction methods based on deep neural networks: Use neural networks to predict the properties of points in three-dimensional space and construct surfaces by fusing depth maps. This type of method does not need to estimate the single-view depth separately on each keyframe, but directly reconstructs the local surface of each video clip through sparse volume sequence without depth fusion. These technologies can predict the geometric shape of the scene more accurately, but do not pay attention to the rendering quality of the scene, cannot output high-quality rendered images, and are not suitable for direct application in scene drawing.

[0004] The solution based on pure neural radiance field and signed distance function: Combining the advantages of two implicit representations, neural radiance field and signed distance function, can accurately reconstruct the geometric surface of the scene while rendering high-resolution images. Compared with the traditional multi-view stereo vision algorithm, the implicit representation contains the information of the entire scene, so it has the potential to reconstruct the three-dimensional structure of low-texture areas, and achieves amazing reconstruction results in small-scale and rich-texture scenes, but often produces poor results in large-scale indoor scenes with textureless flat areas.

[0005] Disadvantages of existing technology:

[0006] 1) Poor reconstruction effect for textureless areas: Existing reconstruction schemes are based on feature matching and constrained reconstruction of scene geometry between multiple views. Indoor scenes have large textureless areas, such as walls, where it is difficult to perform accurate feature extraction and matching, and do not contain sufficient visual features required for pixel-level optimization. Therefore, existing schemes perform poorly in textureless areas and have difficulty in restoring accurate geometric information. Most neural methods cannot reconstruct indoor scenes with a large number of textureless areas.

[0007] 2) Require strong photometric consistency of data: Existing methods are all based on multi-view consistency to complete 3D reconstruction tasks. They assume that the appearance of the scene in different pictures is consistent, which is also called the photometric consistency assumption. However, in real situations, the color of the same point in the scene often changes in different pictures. This is due to different imaging processes (such as exposure, aperture size, white balance, etc.) and different acquisition times. In this situation where the breadth consistency assumption is not met, the effect of existing algorithms will be greatly degraded. Summary of the invention

[0008] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0009] To this end, in response to the problems existing in the current technical solutions for three-dimensional reconstruction of indoor scenes, namely, reconstruction of textureless areas and data photometric consistency problems, the present invention proposes a method for reconstructing indoor three-dimensional scenes based on implicit coding and geometric priors. This method is based on implicit coding and geometric priors, and utilizes a neural radiation field model to reconstruct the scene, which significantly improves the reconstruction effect and overcomes the limitations of existing methods.

[0010] Another object of the present invention is to propose an indoor three-dimensional scene reconstruction system based on implicit coding and geometric prior.

[0011] To achieve the above object, the present invention proposes, on one hand, a method for indoor three-dimensional scene reconstruction based on implicit coding and geometric prior, comprising:

[0012] Processing the image data to be reconstructed to obtain corresponding preprocessed data; wherein the preprocessed data at least includes camera internal and external parameters and initialized implicit appearance coding;

[0013] Calculate the ray in the world coordinate system corresponding to the pixel to be trained on the image data according to the camera internal and external parameters;

[0014] The three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray and the implicit appearance encoding of the corresponding image are input into the neural radiation field model to obtain the predicted density value and color value, and the pixel value is predicted by volume rendering based on the signed distance function to obtain the trained neural radiation field model;

[0015] A standard implicit appearance code is obtained based on the density value and the color value, and data including at least the standard implicit appearance code is input into a trained neural radiance field model to obtain scene geometry information for scene reconstruction.

[0016] The indoor 3D scene reconstruction method based on implicit coding and geometric prior in the embodiment of the present invention may also have the following additional technical features:

[0017] In one embodiment of the present invention, data processing is performed on the image data to be reconstructed to obtain corresponding preprocessed data, including:

[0018] Acquire image data to be reconstructed; wherein the image data to be reconstructed includes an RGB image of an indoor three-dimensional scene to be reconstructed;

[0019] The camera intrinsic and extrinsic parameters of each RGB image are calculated using the SfM motion recovery structure technology;

[0020] Input all RGB images into a pre-trained depth estimation network and a normal estimation network to obtain corresponding depth maps and normal maps, and use the depth maps and normal maps as supervisory signals to provide geometric prior information for the training of the neural radiation field model;

[0021] A learnable implicit appearance code is randomly initialized for each RGB image.

[0022] In one embodiment of the present invention, a multilayer perceptron is used as the neural radiation field model, and the method further includes:

[0023] Inputting the sampling points of each ray into a neural radiation field model based on a signed distance function and implicit appearance coding for volume rendering to obtain a network estimation result; wherein the network estimation result includes an estimated RGB value, an estimated depth value, and an estimated normal value on a final imaging plane;

[0024] Obtaining an RGB true value label corresponding to the RGB image, a depth pseudo label corresponding to the depth map, and a normal pseudo label of the normal map;

[0025] Based on the network estimation result, the RGB true value label, the depth pseudo label and the normal pseudo label are used for supervision, and the parameters of the multilayer perceptron and the value of the implicit appearance encoding are updated through back propagation to train the multilayer perceptron.

[0026] In one embodiment of the present invention, inputting data at least including the standard implicit appearance coding into a trained neural radiance field model to obtain scene geometry information for scene reconstruction includes:

[0027] Obtain the three-dimensional coordinate code and observation angle code of the sampling point on the ray corresponding to the RGB image used for testing;

[0028] Input the three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray corresponding to the RGB image for testing and the standard implicit appearance encoding into the trained neural radiation field to extract the zero potential surface of the signed distance field through the level set method;

[0029] The zero potential surface is used as the geometric surface of the indoor three-dimensional scene to reconstruct the indoor three-dimensional scene.

[0030] In one embodiment of the present invention, the step of obtaining the image data to be reconstructed includes:

[0031] The indoor scene to be reconstructed is photographed by a camera to acquire an RGB image of the indoor three-dimensional scene to be reconstructed; wherein the acquired RGB image covers all object surfaces of the entire scene, and multiple images of different angles are acquired for the same object surface.

[0032] To achieve the above object, the present invention further proposes an indoor three-dimensional scene reconstruction system based on implicit coding and geometric prior, comprising:

[0033] A data preprocessing module, used for performing data processing on the image data to be reconstructed to obtain corresponding preprocessed data; wherein the preprocessed data at least includes camera internal and external parameters and initialized implicit appearance coding;

[0034] The ray data calculation module is used to calculate the rays in the world coordinate system corresponding to the pixels to be trained on the image data according to the internal and external parameters of the camera;

[0035] The network model training module is used to input the three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray and the implicit appearance encoding of the corresponding image into the neural radiation field model to obtain the predicted density value and color value, and use the volume rendering prediction based on the signed distance function to obtain the pixel value, so as to obtain the trained neural radiation field model;

[0036] The indoor scene reconstruction module is used to obtain a standard implicit appearance code based on the density value and the color value, and input data including at least the standard implicit appearance code into a trained neural radiation field model to obtain scene geometry information for scene reconstruction.

[0037] The indoor 3D scene reconstruction method and system based on implicit coding and geometric prior of the embodiment of the present invention first uses a pre-trained model to provide more geometric priors, and introduces additional geometric prior information for 3D reconstruction by using pre-trained depth estimation neural network and normal estimation neural network, thereby improving the modeling effect of large texture-free areas in the middle of indoor scenes and improving the accuracy and completeness of geometric reconstruction results. Then, by introducing implicit appearance coding, the appearance differences that may exist between different collected pictures are modeled, which can reduce the reliance of existing 3D reconstruction schemes on the 3D consistency of collected data, so as to improve the robustness and representation ability of the algorithm in complex scenarios.

[0038] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0040] Figure 1 is a flow chart of a method for indoor three-dimensional scene reconstruction based on implicit coding and geometric prior according to an embodiment of the present invention;

[0041] Figure 2 is a block diagram illustrating the use of implicit coding according to an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of a neural radiation field training process based on a signed distance function and latent coding according to an embodiment of the present invention;

[0043] Figure 4 4 is a structural diagram of an indoor three-dimensional scene reconstruction system based on implicit coding and geometric prior according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] First, the technical terms proposed by the present invention are introduced:

[0047] Signed distance function: refers to a continuous function. Given the coordinates of a three-dimensional point, the function outputs the distance from the point to the nearest object surface. When the point is inside the object, the sign of the distance is negative and outside is positive:

[0048] S={x∈R 3 |f(x)=0}

[0049] The point set S with a signed distance function value of 0 is considered to be the surface of the object, so it can be said that the function implicitly represents the geometric information of the object.

[0050] Neural Radiance Field: With the development of implicit representation technology, researchers have deeply realized the potential of using neural networks to represent implicit surfaces. Neural networks can efficiently store geometric representations with infinite resolution and are widely used in related tasks. Neural Radiance Field technology uses multi-layer perceptrons to model scenes, represents scenes as continuous implicit functions of position and observation angle, and renders them using volume rendering, producing high-quality scene geometry reconstruction results and image rendering results.

[0051] Scene reconstruction: It is an important research direction in the field of computer vision and graphics. It involves the process of recovering a three-dimensional (3D) scene from a series of images or sensor data. The goal of scene reconstruction is to create a digital model that represents real-world objects, environments, and their spatial relationships as accurately as possible. Scene reconstruction has a wide range of applications in many fields, including virtual reality (VR), augmented reality (AR), robot navigation, game development, cultural heritage protection, architecture, and urban planning.

[0052] The following describes an indoor three-dimensional scene reconstruction method and system based on implicit coding and geometric priors according to an embodiment of the present invention with reference to the accompanying drawings.

[0053] Figure 1 is a flow chart of a method for indoor three-dimensional scene reconstruction based on implicit coding and geometric prior according to an embodiment of the present invention, such as Figure 1 As shown, including:

[0054] S1, obtaining corresponding preprocessed data by performing data processing on the image data to be reconstructed; wherein the preprocessed data at least includes camera internal and external parameters and an initialized implicit appearance code.

[0055] S2, calculates the ray in the world coordinate system corresponding to the pixel to be trained on the image data according to the internal and external parameters of the camera.

[0056] S3, the three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray and the implicit appearance encoding of the corresponding image are input into the neural radiation field model to obtain the predicted density value and color value, and the pixel value is predicted using the volume rendering based on the signed distance function to obtain the trained neural radiation field model.

[0057] S4, obtains a standard implicit appearance code based on the density value and the color value, and inputs at least the standard implicit appearance code into a trained neural radiance field model to obtain scene geometry information for scene reconstruction.

[0058] Firstly, the present invention needs to collect relevant image data as a training data set for subsequent use.

[0059] Specifically, the user needs to use a camera to collect complete and dense pictures or shoot videos of the indoor scene to be reconstructed.

[0060] Specifically, the present invention requires that the collected data can cover all surfaces of objects in the entire scene, and collect multiple pictures from different angles for the same surface as much as possible. The overall data size varies from hundreds to thousands of pictures depending on the size of the scene.

[0061] Furthermore, the collected image data is preprocessed for use in subsequent model training.

[0062] Exemplarily, the camera intrinsic and extrinsic parameters of each image are calculated using a technique such as SfM for the collected images or processed video frames.

[0063] Exemplarily, all images are input into a pre-trained depth estimation network and a normal estimation network, and the corresponding depth maps and normal maps are obtained as supervisory signals to provide geometric prior information for the training of the neural radiation field.

[0064] Exemplarily, for each image, a learnable latent code is randomly initialized.

[0065] Specifically, the training process of the neural radiation field of the present invention is as follows: Figure 3 As shown. First, select one from all the training images, randomly select 2048 pixels to be trained on the image, and calculate the ray in the world coordinate system corresponding to the pixel based on the camera's internal and external parameters, and sample 128 sampling points on each ray. The three-dimensional coordinate encoding and observation angle encoding of the sampling point and the implicit appearance encoding of the corresponding image are input into the multi-layer perceptron to obtain the predicted density value and color value.

[0066] Exemplarily, the present invention adopts a multilayer perceptron as a neural radiation field model, and for each ray: the sampling point of each ray is input into a neural radiation field model based on a signed distance function and implicit appearance coding for volume rendering to obtain a network estimation result; wherein the network estimation result includes an estimated RGB value, an estimated depth value, and an estimated normal value on the final imaging plane; the RGB true value label corresponding to the RGB image, the depth pseudo label corresponding to the depth map, and the normal pseudo label of the normal map are obtained; based on the network estimation result, the RGB true value label, the depth pseudo label, and the normal pseudo label are used for supervision, and the parameters of the multilayer perceptron and the values ​​of the implicit appearance coding are updated through back propagation to train the multilayer perceptron.

[0067] Specifically, for each ray, a volume rendering framework combined with a signed distance function is used to obtain the network estimation result, which includes the estimated RGB value, estimated depth value and estimated normal value on the final imaging plane. For the network estimation result, the corresponding RGB true value label, predicted depth pseudo label and normal pseudo label are used for supervision. The parameters of the multi-layer perceptron and the value of the implicit appearance code are updated through back propagation.

[0068] It is understandable that the present invention adopts a new geometric prior regularization scheme to address the problem of difficulty in reconstructing textureless areas. Using a pre-trained neural network, the present invention generates a depth map and a normal map for each image, which is specifically used to provide geometric prior information for textureless areas, so that accurate reconstruction can be achieved in areas that are originally difficult to handle, thereby significantly improving the quality of reconstructed geometry.

[0069] It is understandable that the present invention uses the signed distance function as a three-dimensional representation of the scene based on the neural radiation field, providing a smoother geometric surface and richer texture details for scene reconstruction. At the same time, the representation of the signed distance function also has the advantages of infinite representation resolution, standardization, and easy conversion to other three-dimensional data forms such as grids and point clouds.

[0070] like Figure 2 As shown, the implicit appearance code l in the present invention is defined as a learnable 48-dimensional vector assigned to each training image separately. The implicit appearance code is part of the entire neural network and its parameters can be updated by back propagation. In the initialization phase, the initial value of the implicit code is obtained by random sampling from a normal distribution with a mean of 0 and a variance of 1. In the training phase, the implicit appearance code is input into the neural network together with the coordinate code x and the angle code v to obtain the estimated density value σ and color value c:

[0071] [σ,c]=F θ (x,v,l)

[0072] Furthermore, in the inference phase, the mean of all implicit appearance codes is calculated As a standard implicit appearance encoding of the entire scene, it is used as a data input in the reasoning stage.

[0073] It is understandable that the use of implicit appearance coding per view models the slight appearance differences between different images, thereby reducing the conflicts and performance degradation caused by multi-view inconsistency in the 3D reconstruction process. At the same time, it also reduces the requirements of the present invention for the photometric consistency of the data set, greatly reduces the difficulty of data collection, and improves the reconstruction efficiency.

[0074] For example, in the neural radiation field training stage, for a randomly selected image pixel, the ray r corresponding to the pixel imaging can be calculated by the camera internal and external parameters. For each ray, 128 three-dimensional points are sampled. For each sampling point p i The density σ obtained by neural network prediction i and color c i , the present invention can use a volume rendering framework based on a signed distance function to obtain a predicted pixel value C:

[0075]

[0076] The present invention then uses the collected pixel RGB true value The output of the neural network is supervised and the loss function is defined as follows:

[0077]

[0078] Depth geometry prior regularization: The present invention uses a depth estimation network to estimate a depth map for each image. At the same time, the present invention can calculate the predicted depth value / expected depth value D on each ray through the volume rendering framework:

[0079]

[0080] It is understandable that the present invention uses a pre-trained depth estimation neural network to predict a dense depth map for each frame of the captured image, and uses the depth map as a deep pseudo-label to supervise the process of 3D reconstruction of the scene. Using a deep prior to supervise the training process can reduce the shape ambiguity problem of indoor scenes. Dense depth prior supervision is used in the optimization framework of the neural radiation field.

[0081] Similarly, the present invention uses the depth value of the corresponding pixel on the depth map obtained in the preprocessing As deep pseudo labels, they are deeply supervised:

[0082]

[0083] Normal geometry prior regularization: Since the neural radiation field in the present invention uses the signed distance function f(p) as a geometric representation, the present invention can simply calculate the expected normal n by taking the derivative of the signed distance function i :

[0084]

[0085] It can be understood that the present invention innovatively uses a pre-trained normal estimation network to predict normal maps for the collected images, and uses the normal maps as normal pseudo labels to supervise the optimization process of the neural radiation field. Surface normals are important for 3D scene understanding. They are invariant to translation and scaling, and have better multi-viewing performance than depth prior information. Figure 1 Consistency can provide more robust texture and plane information.

[0086] The present invention uses the normal value of the corresponding pixel on the normal map obtained in preprocessing As normal pseudo-labels, normal supervision is performed on them:

[0087]

[0088] Finally, the present invention extracts the zero potential surface of the signed distance field as the geometric surface of the scene through the level set method.

[0089] Specifically, the three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray corresponding to the RGB image used for testing are obtained; the three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray corresponding to the RGB image used for testing and the standard implicit appearance encoding are input into the trained neural radiation field to extract the zero potential surface of the signed distance field through the level set method; the zero potential surface is used as the geometric surface of the indoor three-dimensional scene to reconstruct the indoor three-dimensional scene.

[0090] It can be understood that the present invention innovatively combines global generative implicit coding technology with neural radiation field technology, providing it with the ability to model the appearance differences between different pictures. This makes the present invention more robust to color changes caused by factors such as exposure, white balance, and time differences between different photos. Furthermore, through this explicit modeling of appearance changes, the neural radiation field can control and change the appearance of the scene, such as lighting, weather, etc., to a certain extent during the rendering stage, adding a certain lighting function to this 3D reconstruction technology.

[0091] According to the indoor 3D scene reconstruction method based on implicit coding and geometric prior according to the embodiment of the present invention, by introducing geometric prior information and implicit appearance coding model, the present invention not only realizes accurate reconstruction in texture-free areas, but also successfully solves the problem of data photometric consistency, so that high-quality reconstruction results can be obtained under real imaging conditions. The solution of the present invention has great potential in the field of indoor scene 3D reconstruction and provides a new way to solve the limitations of existing methods.

[0092] In order to implement the above embodiment, Figure 4 As shown, this embodiment also provides an indoor three-dimensional scene reconstruction system 10 based on implicit coding and geometric prior, including:

[0093] The data preprocessing module 100 is used to process the image data to be reconstructed to obtain corresponding preprocessed data; wherein the preprocessed data at least includes camera internal and external parameters and initialized implicit appearance coding;

[0094] A ray data calculation module 200 is used to calculate rays in a world coordinate system corresponding to pixels to be trained on the image data according to camera internal and external parameters;

[0095] The network model training module 300 is used to input the three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray and the implicit appearance encoding of the corresponding image into the neural radiation field model to obtain the predicted density value and color value, and use the volume rendering prediction based on the signed distance function to obtain the pixel value, so as to obtain the trained neural radiation field model;

[0096] The indoor scene reconstruction module 400 is used to obtain a standard implicit appearance code based on the density value and the color value, and input data including at least the standard implicit appearance code into a trained neural radiance field model to obtain scene geometry information for scene reconstruction.

[0097] Exemplarily, the data preprocessing module 100 includes:

[0098] An image data acquisition unit, used to acquire image data to be reconstructed; wherein the image data to be reconstructed includes an RGB image of an indoor three-dimensional scene to be reconstructed;

[0099] The camera parameter calculation unit is used to calculate the camera intrinsic and extrinsic parameters of each RGB image using the SfM motion recovery structure technology;

[0100] A supervisory signal determination unit, used for inputting all RGB images into a pre-trained depth estimation network and a normal estimation network to obtain corresponding depth maps and normal maps, and using the depth maps and normal maps as supervisory signals to provide geometric prior information for the training of the neural radiation field model;

[0101] The implicit code creation unit is used to randomly initialize a learnable implicit appearance code for each RGB image.

[0102] Exemplarily, a multi-layer perceptron is used as the neural radiation field model, and the network model training module 300 is also used for:

[0103] Inputting the sampling points of each ray into a neural radiation field model based on a signed distance function and implicit appearance coding for volume rendering to obtain a network estimation result; wherein the network estimation result includes an estimated RGB value, an estimated depth value, and an estimated normal value on a final imaging plane;

[0104] Obtaining an RGB true value label corresponding to the RGB image, a depth pseudo label corresponding to the depth map, and a normal pseudo label of the normal map;

[0105] Based on the network estimation result, the RGB true value label, the depth pseudo label and the normal pseudo label are used for supervision, and the parameters of the multilayer perceptron and the value of the implicit appearance encoding are updated through back propagation to train the multilayer perceptron.

[0106] Exemplarily, the indoor scene reconstruction module 400 is further used for:

[0107] Obtain the three-dimensional coordinate code and observation angle code of the sampling point on the ray corresponding to the RGB image used for testing;

[0108] Input the three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray corresponding to the RGB image for testing and the standard implicit appearance encoding into the trained neural radiation field to extract the zero potential surface of the signed distance field through the level set method;

[0109] The zero potential surface is used as the geometric surface of the indoor three-dimensional scene to reconstruct the indoor three-dimensional scene.

[0110] Exemplarily, the image data acquisition unit is further used for:

[0111] The indoor scene to be reconstructed is photographed by a camera to acquire an RGB image of the indoor three-dimensional scene to be reconstructed; wherein the acquired RGB image covers all object surfaces of the entire scene, and multiple images of different angles are acquired for the same object surface.

[0112] According to the indoor 3D scene reconstruction system based on implicit coding and geometric prior according to the embodiment of the present invention, a pre-trained model is first used to provide more geometric priors. By using pre-trained depth estimation neural network and normal estimation neural network, additional geometric prior information is introduced for 3D reconstruction, thereby improving the modeling effect of large texture-free areas in the middle of indoor scenes and improving the accuracy and completeness of geometric reconstruction results. Then, by introducing implicit appearance coding, the appearance differences that may exist between different collected pictures are modeled, which reduces the reliance of existing 3D reconstruction schemes on the 3D consistency of collected data and improves the robustness and representation ability of the algorithm in complex scenarios.

[0113] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0114] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A method for indoor three-dimensional scene reconstruction based on implicit coding and geometric prior, characterized in that: include: Processing the image data to be reconstructed to obtain corresponding preprocessed data; wherein the preprocessed data at least includes camera internal and external parameters and initialized implicit appearance coding; Calculate the ray in the world coordinate system corresponding to the pixel to be trained on the image data according to the camera's internal and external parameters; The three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray and the implicit appearance encoding of the corresponding image are input into the neural radiation field model to obtain the predicted density value and color value, and the pixel value is predicted by volume rendering based on the signed distance function to obtain the trained neural radiation field model; A standard implicit appearance code is obtained based on the density value and the color value, and data including at least the standard implicit appearance code is input into a trained neural radiance field model to obtain scene geometry information for scene reconstruction.

2. The method according to claim 1, characterized in that The image data to be reconstructed is processed to obtain corresponding preprocessed data, including: Acquire image data to be reconstructed; wherein the image data to be reconstructed includes an RGB image of an indoor three-dimensional scene to be reconstructed; The camera intrinsic and extrinsic parameters of each RGB image are calculated using the SfM motion recovery structure technology; Input all RGB images into a pre-trained depth estimation network and a normal estimation network to obtain corresponding depth maps and normal maps, and use the depth maps and normal maps as supervisory signals to provide geometric prior information for the training of the neural radiation field model; A learnable implicit appearance code is randomly initialized for each RGB image.

3. The method according to claim 2, characterized in that A multilayer perceptron is used as the neural radiation field model, and the method further includes: Inputting the sampling points of each ray into a neural radiation field model based on a signed distance function and implicit appearance coding for volume rendering to obtain a network estimation result; wherein the network estimation result includes an estimated RGB value, an estimated depth value, and an estimated normal value on a final imaging plane; Obtaining an RGB true value label corresponding to the RGB image, a depth pseudo label corresponding to the depth map, and a normal pseudo label of the normal map; Based on the network estimation result, the RGB true value label, the depth pseudo label and the normal pseudo label are used for supervision, and the parameters of the multilayer perceptron and the value of the implicit appearance encoding are updated through back propagation to train the multilayer perceptron.

4. The method according to claim 3, characterized in that: Inputting data including at least the standard implicit appearance encoding into a trained neural radiance field model to obtain scene geometry information for scene reconstruction, including: Obtain the three-dimensional coordinate code and observation angle code of the sampling point on the ray corresponding to the RGB image used for testing; Input the three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray corresponding to the RGB image for testing and the standard implicit appearance encoding into the trained neural radiation field to extract the zero potential surface of the signed distance field through the level set method; The zero potential surface is used as the geometric surface of the indoor three-dimensional scene to reconstruct the indoor three-dimensional scene.

5. The method according to claim 2, characterized in that: The step of obtaining the image data to be reconstructed comprises: The indoor scene to be reconstructed is photographed by a camera to acquire an RGB image of the indoor three-dimensional scene to be reconstructed; wherein the acquired RGB image covers all object surfaces of the entire scene, and multiple images of different angles are acquired for the same object surface.

6. An indoor three-dimensional scene reconstruction system based on implicit coding and geometric prior, characterized in that: include: A data preprocessing module, used for performing data processing on the image data to be reconstructed to obtain corresponding preprocessed data; wherein the preprocessed data at least includes camera internal and external parameters and initialized implicit appearance coding; The ray data calculation module is used to calculate the rays in the world coordinate system corresponding to the pixels to be trained on the image data according to the internal and external parameters of the camera; The network model training module is used to input the three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray and the implicit appearance encoding of the corresponding image into the neural radiation field model to obtain the predicted density value and color value, and use the volume rendering prediction based on the signed distance function to obtain the pixel value, so as to obtain the trained neural radiation field model; The indoor scene reconstruction module is used to obtain a standard implicit appearance code based on the density value and the color value, and input data including at least the standard implicit appearance code into a trained neural radiation field model to obtain scene geometry information for scene reconstruction.

7. The system according to claim 6, characterized in that The data preprocessing module comprises: An image data acquisition unit, used to acquire image data to be reconstructed; wherein the image data to be reconstructed includes an RGB image of an indoor three-dimensional scene to be reconstructed; The camera parameter calculation unit is used to calculate the camera intrinsic and extrinsic parameters of each RGB image using the SfM motion recovery structure technology; A supervisory signal determination unit, used for inputting all RGB images into a pre-trained depth estimation network and a normal estimation network to obtain corresponding depth maps and normal maps, and using the depth maps and normal maps as supervisory signals to provide geometric prior information for the training of the neural radiation field model; The implicit code creation unit is used to randomly initialize a learnable implicit appearance code for each RGB image.

8. The system according to claim 7, characterized in that A multi-layer perceptron is used as the neural radiation field model, and the network model training module is also used for: Inputting the sampling points of each ray into a neural radiation field model based on a signed distance function and implicit appearance coding for volume rendering to obtain a network estimation result; wherein the network estimation result includes an estimated RGB value, an estimated depth value, and an estimated normal value on a final imaging plane; Obtaining an RGB true value label corresponding to the RGB image, a depth pseudo label corresponding to the depth map, and a normal pseudo label of the normal map; Based on the network estimation result, the RGB true value label, the depth pseudo label and the normal pseudo label are used for supervision, and the parameters of the multilayer perceptron and the value of the implicit appearance encoding are updated through back propagation to train the multilayer perceptron.

9. The system according to claim 8, characterized in that The indoor scene reconstruction module is further used for: Obtain the three-dimensional coordinate code and observation angle code of the sampling point on the ray corresponding to the RGB image used for testing; Input the three-dimensional coordinate encoding and observation angle encoding of the sampling points on the ray corresponding to the RGB image for testing and the standard implicit appearance encoding into the trained neural radiation field to extract the zero potential surface of the signed distance field through the level set method; The zero potential surface is used as the geometric surface of the indoor three-dimensional scene to reconstruct the indoor three-dimensional scene.

10. The system according to claim 7, characterized in that The image data acquisition unit is further used for: The indoor scene to be reconstructed is photographed by a camera to acquire an RGB image of the indoor three-dimensional scene to be reconstructed; wherein the acquired RGB image covers all object surfaces of the entire scene, and multiple images of different angles are acquired for the same object surface.

Images