Sparse-view indoor reconstruction method based on uncertainty-aware depth supervision

Abstract

The application discloses a sparse-view indoor reconstruction method based on uncertainty-aware depth supervision, which comprises the following steps: acquiring multi-view image data; solving camera pose based on SFM; constructing a neural radiance field model based on uncertainty awareness, modeling the volume density, color and depth uncertainty parameters of a space point; introducing a depth uncertainty synthesis formula in a volume rendering framework; constructing a photometric loss fused with random structure similarity; designing an adaptive depth optimization mechanism based on uncertainty awareness, optimizing the training process through a progressive uncertainty learning strategy; and applying the trained model to a sparse-view indoor scene to generate high-quality three-dimensional reconstruction results and uncertainty quantification graphs. The application effectively solves the problems of poor reconstruction quality and missing details under sparse views by introducing a fusion method of uncertainty awareness and adaptive depth supervision, and significantly improves the accuracy and robustness of indoor scene three-dimensional reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction, and is applied to the field of indoor three-dimensional reconstruction under sparse view angles, in particular to a sparse view angle indoor reconstruction method based on uncertainty perception depth supervision. BACKGROUND

[0002] Traditional indoor three-dimensional reconstruction methods mainly rely on multi-view geometric principles, such as Structure from Motion (SfM) and Multi-View Stereo (MVS) techniques. These methods extract feature points from images captured from multiple views and use geometric constraints such as triangulation to recover the three-dimensional structure of the scene. Although these methods can achieve high-precision reconstruction under ideal conditions, they still have significant defects in practical applications. First, traditional methods are highly dependent on image quality and texture richness. In scenes with missing texture, uneven lighting, or repeated structures, feature matching errors are likely to occur, resulting in incomplete or distorted reconstruction results. Second, these methods usually require dense view coverage, and under sparse view conditions, the reconstruction accuracy is significantly reduced due to insufficient geometric constraints. In addition, traditional methods are difficult to effectively handle dynamic scenes or complex occlusion situations, and have high computational complexity, making it difficult to meet real-time requirements. Therefore, the application of traditional three-dimensional reconstruction methods in actual indoor scenes is greatly limited, and a new method that can overcome the above defects is urgently needed.

[0003] In recent years, Neural Radiance Fields (NeRF) as a new three-dimensional scene representation method has attracted widespread attention in the field of three-dimensional reconstruction. NeRF implicitly represents the geometry and appearance information of the scene through a neural network, and can generate high-quality three-dimensional reconstruction results from sparse two-dimensional images, especially in view synthesis tasks. Its core idea is to use a multi-layer perceptron (MLP) to model the radiance field of the scene, and generate realistic new view images through volume rendering technology. However, the NeRF method also has some limitations in practical applications. First, the training process of NeRF requires a large amount of computational resources and time, and has high requirements for the view distribution and number of input images. In the case of sparse view angles, overfitting is likely to occur, resulting in loss of details or inaccurate geometric structure in the reconstruction results. Second, NeRF lacks modeling of uncertainty in the reconstruction process, and cannot evaluate the reliability of the reconstruction results, which limits its application in scenes that require high precision and robustness. In addition, NeRF still performs poorly in complex indoor scenes, especially in the presence of occlusions, reflections, or transparent objects, and the reconstruction quality will decrease significantly.

[0004] To solve the above problems, in recent years some research works try to introduce deep supervision mechanism, by using additional depth information or geometric prior to constrain the training process of neural network, so as to improve the reconstruction accuracy and robustness. Further method (Roessle B, Barron J T, Mildenhall B, et al. Dense depth priors for neural radiance fields from sparse input views[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 12892-12901.) uses a pre-trained depth prediction network that fuses uncertainty to constrain depth supervision, but its method needs to train an uncertainty-aware depth prediction network, which brings unnecessary additional training overhead. Further method (Kim S. DäRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation[C] / / Neural Information Processing Systems (NeurIPS). 2023.) uses adaptive monocular depth optimization and achieves good results, but its method cannot solve the problem of noise in depth prior information, which leads to the problem of blurred local scene edges. Therefore, how to effectively introduce depth supervision and improve the reconstruction performance under sparse view is still a problem to be solved.

[0005] Uncertainty is a measure of the confidence of the model in the prediction result, which is divided into cognitive uncertainty and accidental uncertainty, which are respectively caused by model limitations and data noise. In neural radiation field (NeRF), uncertainty modeling can quantify the reliability of the reconstruction result, help identify unreliable areas and guide optimization, especially in sparse view or complex scene to improve reconstruction accuracy and robustness. For example, through probability modeling or depth integration method, NeRF can output geometric information and uncertainty distribution, provide reliability evaluation for dynamic scene reconstruction, new view synthesis and other tasks, and enhance its practicability in robot navigation, virtual reality and other fields. SUMMARY

[0006] The present application aims to provide an uncertainty-aware depth supervision based sparse view indoor reconstruction method which can be implemented, to solve the problem of poor reconstruction quality, missing details and blurred local scene edges in sparse view indoor reconstruction method.

[0007] To achieve the above object, the technical scheme adopted by the present application is as follows:

[0008] A sparse view indoor reconstruction method based on uncertainty perception deep supervision, comprising the following steps:

[0009] Step 1, acquiring multi-view image data for indoor three-dimensional reconstruction;

[0010] Step 2, solving the position and pose of the multi-view image data for indoor three-dimensional reconstruction based on SFM (Structure from Motion).

[0011] Step 3, modeling the radiance field in three-dimensional space based on the uncertainty perception neural radiance field model, modeling the attributes of the space points as volume density, color and depth uncertainty parameters.

[0012] Step 3.1, construction of neural radiance field input. By learning the radiance field representation of the three-dimensional scene from the image data and camera pose from different views, the neural radiance field can efficiently perform three-dimensional reconstruction and more accurately capture the lighting and geometric features in complex scenes. In the processing of complex and high-detail scenes, the neural radiance field can provide higher quality image reconstruction. The input multi-view image data and the corresponding camera pose are input into the preprocessing module to generate the direction of the corresponding light and the direction vector and the starting point of the input ray. Each ray is then uniformly or adaptively sampled into several points , wherein represents the position at different depths on the ray. The positions of these sampling points and the corresponding direction information are embedded as input into the neural network, providing a basis for subsequent color and density estimation.

[0013] Step 3.2, construction of uncertainty perception neural radiance field model and pre-training monocular depth estimation model. In the construction of the neural radiance field, an uncertainty perception module is introduced, which operates as follows: the positions and directions of the sampling points are input into the neural network, and the neural network outputs the color , volume density and corresponding uncertainty prediction of the sampling point after learning.

[0014] Step 4, introducing the synthesis formula of depth uncertainty on the classical volume rendering framework to further improve the expression ability for geometric blur areas. Specifically, when performing color synthesis along each ray, in addition to the conventional color and depth weighted accumulation to obtain color and depth , the model also models the data uncertainty of the depth value of each sampling point .

[0015] Step 5, construct the photometric loss, introduce the random structure similarity loss function module. The traditional photometric loss is extended to the combination of pixel-level error and structure-aware error, where the structure-aware part adopts the random structure similarity loss function (S3IM). The complete photometric loss is represented as:

[0016]

[0017] where, is

[0018]

[0019] Step 6, this step introduces a new uncertainty-aware fusion adaptive depth supervision mechanism to bidirectionally enhance the collaborative learning between NeRF and monocular depth estimation (MDE). For this purpose, three loss modules are designed:

[0020] Step 6.1, design a monocular depth regularization loss module with fusion uncertainty-aware for seen views.

[0021]

[0022] is the pre-trained monocular depth estimation model predicts the pseudo-depth map of the training image . represents the depth value of the p-th pixel on the depth map . is the depth map obtained by rendering the training image with NeRF, represents the depth value of the p-th pixel on the depth map . is the scaling factor obtained by fitting the depth map and by least squares. is the pixel-level depth uncertainty value obtained by rendering. Where: represents the stop-gradient operation, which does not return the gradient; is an adaptive gradient masking function, the gradient of the supervision term in the high uncertainty area will be masked.

[0023] Step 6.2, design a monocular depth regularization loss module with fusion uncertainty-aware for unseen views.

[0024]

[0025] and These are from any unseen perspective Render the corresponding RGB image, depth image, and depth uncertainty value. For pre-trained monocular depth estimation models Predicted rendered image Pseudo-depth map, In the depth map The depth value of pixel p. Similarly, the scaling factor is obtained by least squares fitting. and offset .

[0026] Step 6.3: Design a monocular depth adaptive loss module that incorporates uncertainty awareness. This involves introducing scene geometry priors from the optimized NeRF model to guide the MDE model in adaptive training, thereby enhancing the MDE's ability to model absolute depth structures. A two-branch depth consistency loss is designed: the first term is an absolute depth alignment term without scale adjustment, used to improve the MDE's ability to model scene scales; the second term is a patch-based scale alignment consistency term, incorporating pixel-level depth uncertainty from the NeRF rendering output. To achieve adaptive weight adjustment, a stop-gradient operation is applied to the uncertainty estimation to prevent the uncertainty term from being interfered with by the supervision error of the MDE. The complete loss function is as follows:

[0027]

[0028] Among them, the first item For absolute depth alignment, In the depth map The depth value of the p-th pixel. In the depth map The depth value of the p-th pixel. This indicates a stop-gradient operation, where no gradient is returned. (Second term) To integrate uncertainty-aware scale alignment consistency terms, This represents the pixel-level depth uncertainty value obtained from rendering. Similarly, the scaling factor is obtained through least squares fitting. and offset .

[0029] Finally, the loss function for the uncertainty depth is obtained:

[0030]

[0031] Step S7, an incremental uncertainty learning strategy is implemented. The scheme proposes an incremental uncertainty learning strategy, which realizes stable convergence of the training process by dynamically adjusting the weight of uncertainty optimization

[0032] From the above technical solutions, the present application relates to the field of indoor three-dimensional reconstruction technology in neural radiation field, and discloses a sparse view indoor reconstruction method based on uncertainty perception depth supervision. The method comprises: acquiring multi-view image data for indoor three-dimensional reconstruction; constructing an uncertainty perception neural radiation field model; constructing a depth uncertainty rendering synthesis; using S3IM-based photometric loss; introducing a new uncertainty perception adaptive depth optimization mechanism, designing an uncertainty perception-based monocular depth regularization loss module for seen views, designing a monocular depth regularization loss module for unseen views, and designing an uncertainty perception-based monocular depth adaptive loss module. The present application improves the depth supervision loss function used in DäRF (Kim S. DäRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation [C] / / Neural Information Processing Systems (NeurIPS). 2023.) by introducing an uncertainty fusion adaptive depth supervision method, effectively solves the problem of poor reconstruction quality caused by the ambiguity of depth prior in depth supervision neural radiation field, and improves the accuracy of neural radiation field three-dimensional reconstruction under sparse conditions. At the same time, the structural perception similarity loss (S3IM) is combined to improve the reconstruction robustness and geometric alignment ability of the model in complex structure areas. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings needed in the embodiments.

[0034] Figure 1 is the overall flowchart of the method;

[0035] Figure 2 is the structure diagram of the neural radiation field model constructed based on evidence deep learning proposed by the method;

[0036] Figure 3 is the rendering result comparison diagram of the method and other methods. DETAILED DESCRIPTION

[0037] As Figure 1 shown, in order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0038] The sparse view indoor reconstruction method based on uncertainty perception depth supervision of the embodiment of the application comprises the following steps: acquiring multi-view image data; solving camera pose based on SFM; constructing a neural radiance field model based on uncertainty perception, modeling the volume density, color and depth uncertainty parameters of a space point; introducing a depth uncertainty synthesis formula in a volume rendering framework; constructing a photometric loss that fuses random structure similarity; designing an adaptive depth optimization mechanism with uncertainty perception, including a depth regularization loss module for seen views and unseen views and a monocular depth adaptive loss module; optimizing the training process through a progressive uncertainty learning strategy; applying the trained model to a sparse view indoor scene to generate high-quality three-dimensional reconstruction results and uncertainty quantization maps. By introducing the fusion method of uncertainty perception and adaptive depth supervision, the application effectively solves the problems of poor reconstruction quality and missing details under sparse views, significantly improving the accuracy and robustness of indoor scene three-dimensional reconstruction.

[0039] The method comprises the following steps:

[0040] Step 1: acquiring multi-view indoor scene image data to provide basic input for subsequent processing;

[0041] Step 2: based on the image data obtained in step 1, using a motion recovery structure (SFM) method to recover the camera pose of each image;

[0042] Step 3: using the camera pose recovered in step 2, constructing an uncertainty perception neural radiance field model, and introducing a pre-trained monocular depth estimation model to provide depth prior;

[0043] Step 4: in the volume rendering framework, rendering the image and the depth, and combining the depth estimation result obtained in step 3 and the uncertainty modeling, calculating and generating the depth uncertainty map of the rendered image through the uncertainty synthesis formula;

[0044] Step 5: based on the difference between the rendered image generated in step 4 and the real image in step 1, constructing a photometric loss term that fuses random structure similarity (S3IM);

[0045] Step 6: combining the depth uncertainty map obtained in step 4 and the rendered depth map, designing an adaptive depth optimization loss term with uncertainty perception;

[0046] Step 7: through a progressive uncertainty learning strategy, jointly optimizing the total loss function obtained from step 5 and step 6 to train the neural radiance field and depth estimation model.

[0047] The following is further described:

[0048] Step S1: Obtain multi-view image data for indoor three-dimensional reconstruction. Use a camera device to capture image data of multiple views in an indoor scene. In order to ensure the accuracy of the reconstruction, it is necessary to cover the entire indoor space as much as possible and ensure that there is sufficient overlap between adjacent views.

[0049] Step S2: Solve camera pose based on SFM method. After obtaining multi-view image data, SFM algorithm is used to recover the camera position and attitude corresponding to each image. SFM is a vision-based three-dimensional reconstruction technology that can estimate the internal and external parameters of the camera and the three-dimensional coordinates of the feature points in the scene by analyzing the correspondence of feature points in the image sequence.

[0050] Step S3: As shown in Figure 2 , construct an uncertainty-aware neural radiance field model and a pre-trained monocular depth estimation model.

[0051] Specifically, the following steps are included:

[0052] Step S3.1, input construction of neural radiance field model. The input construction stage of the neural radiance field model is based on pixels and rays, combining multi-view image data with camera view direction to generate sampling points in three-dimensional space. The input multi-view image data and the corresponding camera pose are input into the preprocessing module to generate the direction of the corresponding rays and the direction vector and the starting point of the input rays. Then, sampling is performed along each ray to obtain a series of sampling points , where represents the jth sampling point on the ith ray, represents the different depth positions of the jth sampling point on the ray. These sampling points will be used as input for the neural network for subsequent color and volume density estimation.

[0053] Step S3.2, construct a depth uncertainty-aware neural radiance field. An uncertainty-aware module is introduced to better capture the geometric and lighting features in the three-dimensional scene. The specific method is as follows: after obtaining the position and direction of the sampling points through step S3.1, the position information and direction information of these sampling points are input into the neural network model after position encoding. Through the training of this neural network model, the model can learn to output the following three attributes of each sampling point: color represents the color information of the sampling point, volume density represents the volume density of the sampling point, which describes the geometric structure in the scene, and the corresponding uncertainty prediction This indicates the degree of uncertainty regarding the depth information of the sampling point. This modeling approach, which incorporates uncertainty awareness, helps the neural radiation field better capture detailed features in complex scenes, thereby improving the quality of 3D reconstruction. The formula is as follows:

[0054]

[0055] Step S3.3 Construct a pre-trained monocular depth estimation model A pre-trained depth estimation model is constructed using MidasNet without a monocular depth prediction model. MidasNet is a monocular depth estimation model based on a convolutional neural network that can predict the depth information of a scene from a single input image. Specifically, given an input image... A pseudo depth map is obtained.

[0056] Step S4: Render the RGB values ​​of each ray. Depth value and deep uncertainty values Specifically, based on classic volume rendering, in addition to the usual color and depth weighted summation, a cumulative weighted summation is used to obtain depth uncertainty. Deep uncertainty This reflects the potential ambiguity or uncertainty in depth information of certain areas within the scene (such as edges and occlusions). Incorporating this uncertain prediction into the final volume rendering formula allows the rendering result to better reflect the uncertainty of depth information. The specific compositing formula is as follows:

[0057]

[0058] in, This represents the weight, specifically the contribution weight of the i-th sample point to the final rendered color. More specifically: , It is the cumulative projection rate. It is opacity, in which, Similarly, the values ​​of depth and depth uncertainty can be calculated.

[0059]

[0060]

[0061] This discrete summation method allows for the efficient calculation of final color, depth, and uncertainty information along the ray integral. This volumetric rendering framework, which introduces uncertainty prediction, can better describe the geometric details and lighting characteristics in complex scenes.

[0062] Step S5, construct the photometric loss function of the fusion random structure similarity term (S3IM). The traditional L2 loss pays more attention to the pixel-level difference, but these methods cannot capture the structural information of the image. S3IM can better capture the local texture and structure advantages of the image by comparing small block regions randomly selected in the image while calculating the region structure similarity. The method combines S3IM loss and L2 loss, and the specific loss function is as follows:

[0063]

[0064] Wherein,

[0065]

[0066] Step S6, build an uncertainty fusion-based adaptive depth optimization loss module to enhance the collaborative learning between NeRF and monocular depth estimation (MDE) in both directions. The depth predicted by MDE usually has scale ambiguity, and in weak texture areas or occlusion boundaries, geometric ambiguity (such as false depth smoothing assumption) is easy to occur, while existing methods do not explicitly model this uncertainty, which leads to NeRF incorrectly absorbing noise information from MDE, thereby affecting the quality and accuracy of three-dimensional reconstruction. At the same time, the uncertainty used by existing depth-supervised NeRF comes from the model uncertainty of MDE, which requires modifying the model structure of MDE, thereby bringing unnecessary time overhead for retraining MDE. This study proposes to build an uncertainty fusion-based adaptive depth optimization loss module, which realizes the mutual enhancement of NeRF and MDE through a bidirectional collaborative learning mechanism. The module includes three key sub-modules: Sub-module one: uncertainty perception-based monocular depth regularization loss module for seen views; Sub-module two: fusion uncertainty perception-based monocular depth regularization loss module for unseen views; Sub-module three: design of fusion uncertainty perception-based monocular depth adaptive loss module. First, by establishing the correlation between the depth uncertainty modeling capability of NeRF and the depth map generated by MDE, sub-module one and sub-module two can explicitly capture and quantify the geometric ambiguity of MDE output, thereby improving the model's ability to represent depth estimation ambiguity; second, sub-module three uses the depth map generated by the NeRF rendering process and its corresponding uncertainty map as a supervision signal to design an adaptive optimization mechanism based on uncertainty perception, dynamically adjusting the training process of the MDE model. This bidirectional interaction mechanism not only enhances the learning ability of NeRF for MDE uncertainty, but also enables MDE to significantly improve its perception and reconstruction accuracy of complex scene structures based on the geometric consistency constraints provided by NeRF under sparse supervision conditions. ​

[0067] Submodule one: uncertainty-aware monocular depth regularization loss module for seen views. As shown in Figure 2 To alleviate the scale error and uncertainty interference problem of monocular depth prior in local regions, an uncertainty-aware geometric alignment loss function is introduced to optimize the consistency between the NeRF rendered depth map and the monocular estimated depth map. Specifically, a pre-trained monocular depth estimation model is used to predict the monocular estimated depth map for each input image. In the local patch range, the scaling factor and the offset are obtained by least squares fitting, which are aligned to absolute depth to match the NeRF rendered depth map . Further, the pixel-level depth uncertainty value estimated in the NeRF rendering process is introduced to construct the following loss function:

[0068]

[0069] Where: represents the stop-gradient operation, which does not return the gradient; is an adaptive gradient masking function, and the gradient of the supervision term in the high uncertainty area will be masked.

[0070] Submodule two: fusion uncertainty-aware monocular depth regularization loss module for unseen views. Few-shot NeRF is often affected by artifacts at unseen views, although the images rendered by MDE models can provide new depth supervision signals at unseen views, the supervision signals generated by MDE models are still unreliable, and the influence of artifacts will bring noise. The addition of uncertainty can make the NeRF model adaptively learn the noise of new depth supervision information, thereby increasing the robustness of the model at unseen views. From any unseen view , the corresponding RGB image and depth map are rendered, and are input into the pre-trained monocular depth estimator to obtain the pseudo-depth map . Similarly, the scaling factor and the offset are obtained by least squares fitting. At the same time, combined with the corresponding uncertainty map rendered from the unseen view , the uncertainty-aware is designed as follows:

[0071] ​

[0072] Sub-module 3: Design a monocular depth adaptive loss module that integrates uncertainty perception. For example... Figure 2 of This paper introduces a method to back-guide the MDE model for adaptive training using the scene geometry priors inherent in the optimized NeRF model, thereby enhancing the MDE's ability to model absolute depth structures. A two-branch depth consistency loss is designed: the first term is an absolute depth alignment term without scale adjustment, used to improve the MDE's ability to model scene scales; the second term is a patch-based scale alignment consistency term, and pixel-level depth uncertainty in the NeRF rendering output is introduced. To achieve adaptive weight adjustment, a stop-gradient operation is applied to the uncertainty estimation to prevent the uncertainty term from being interfered with by the supervision error of the MDE. The loss function is as follows:

[0073]

[0074] Among them, the first item For absolute depth alignment, In the depth map The depth value of the p-th pixel. In the depth map The depth value of the p-th pixel. This indicates a stop-gradient operation, where no gradient is returned. (Second term) To integrate uncertainty-aware scale alignment consistency terms, This represents the pixel-level depth uncertainty value obtained from rendering. Similarly, the scaling factor is obtained through least squares fitting. and offset .

[0075] Finally, by summing the loss functions, we obtain the loss function for the uncertainty of one iteration depth as follows:

[0076]

[0077] in, Set to 0.01, 0.02, 0.01.

[0078] Step S7 implements a progressive uncertainty learning strategy. In the collaborative training of Neural Radiation Field (NeRF) and Monocular Depth Estimation (MDE), directly introducing uncertainty modeling may lead to instability in the early stages of training, especially when depth prediction noise is high or geometric priors are insufficient. Therefore, this scheme proposes a progressive uncertainty learning strategy, which achieves stable convergence of the training process by dynamically adjusting the weights of uncertainty optimization.

[0079] Specifically, in the first 2000 rounds of training (t<2000), only the geometry and appearance networks of NeRF and the depth prediction branch of the MDE model are used. At this time, the loss function only contains the basic rendering loss and depth alignment loss:

[0080]

[0081] where, is the RGB rendering loss of NeRF, specifically 0.1, is the depth loss, specifically:

[0082] After the model is preliminarily converged, between the 2000th round and the 10000th round of training (2000 ) overfitting phase, uncertainty optimization is gradually introduced, and uncertainty loss is gradually introduced. u(t) is the weight that controls and increases with the number of training rounds:

[0083]

[0084] Finally, in the later stage of training (after 50%), uncertainty optimization is completely enabled, and the loss function contains the complete uncertainty perception term:

[0085]

[0086] Step s8, according to the total loss , the network parameters of NeRF and MDE model are updated by gradient descent. The total number of iterations is a fixed value of 30000 rounds, and when the number of iterations reaches the preset value, it means that the three-dimensional reconstruction task is completed.

[0087] As shown in Table 1, in order to verify the effectiveness of the present scheme, in the Scannet dataset, 20 pictures are selected as training data and 8 pictures are selected as test data to evaluate the performance of the model in real indoor scenes. This sparse view setting (only 20 training images) aims to simulate the common data limited scene in actual application, and also provides an ideal test environment for verifying the proposed uncertainty perception depth fusion mechanism.

[0088] The present application selects the training condition of 30000 rounds of model full fitting, and uses PSNR, SSIM and LPIPS as the difference between the predicted image and the real image. The present application is compared with the DäRF method (Kim S. DäRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation [C] / / Neural Information Processing Systems (NeurIPS). 2023.), the SCADE method (Uy M A, Martin-Brualla R, Guibas L, et al. Scade: Nerfs from space carving with ambiguity-aware depth estimates [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 16518-16527.), and the EMD-NeRF method (Rau A, Aklilu J, Christopher Holsinger F, et al. Depth-Guided NeRF Training via Earth Mover's Distance [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 1-17.). The comparison results are shown in Table 1, and it can be seen that the present application is superior to other methods in terms of PSNR and LPIPS indicators. At the same time, it can be seen from the reference Figure 3 It can be seen that the quality of the final reconstructed image of the present application is better than that of the existing method.

[0089] Table 1 is the experimental results

[0090]

[0091] In another aspect, the present application also discloses a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the above method.

[0092] In another aspect, the present application also discloses a computer device, which comprises a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the above method.

[0093] It can be understood that the system, device and storage medium provided by the embodiments of the present application correspond to the method provided by the embodiments of the present application, and the explanations, examples and beneficial effects of the related content can refer to the corresponding parts in the above method.

[0094] In the above embodiments, the system, device and storage medium provided by the embodiments of the present application can be realized by software, hardware, firmware or any combination thereof, in whole or in part. When realized by software, it can be realized in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD), or semiconductor media (for example, Solid State Disk (SSD)) and the like.

[0095] It should be noted that in this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0096] The various embodiments in the specification are described in a related manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0097] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A sparse-view indoor reconstruction method based on uncertainty-aware depth supervision, characterized in that, Includes the following steps: Step 1: Acquire multi-view indoor scene image data to provide basic input for subsequent processing; Step 2: Based on the image data obtained in Step 1, the camera pose of each image is recovered using the structure-of-motion reconstruction method; Step 3: Using the camera pose recovered in Step 2, construct an uncertainty-perceiving neural radiation field model and introduce a pre-trained monocular depth estimation model to provide depth priors; Step 4: In the volume rendering framework, render the image and depth. At the same time, combine the depth estimation results obtained in Step 3 with uncertainty modeling, and calculate and generate the depth uncertainty map of the rendered image through the uncertainty synthesis formula. Step 5: Based on the difference between the rendered image generated in Step 4 and the real image in Step 1, construct a photometric loss term that fuses random structural similarity; Step 6: Combining the depth uncertainty map and the rendered depth map obtained in Step 4, design an uncertainty-aware adaptive depth optimization loss term; Step 7: Train the neural radiation field and depth estimation model by jointly optimizing the total loss function obtained from Step 5 and Step 6 through a progressive uncertainty learning strategy. Step 6 specifically includes, We construct an uncertainty fusion-based adaptive depth optimization loss module to bidirectionally enhance the collaborative learning between NeRF (Neural Radiation Field) and monocular depth estimation. An adaptive depth optimization loss module based on uncertainty fusion is constructed, and the neural radiation field (NeRF) and monocular depth estimation (MDE) are mutually enhanced through a bidirectional collaborative learning mechanism. The adaptive depth optimization loss module based on uncertainty fusion consists of three key sub-modules: Submodule 1: Monocular depth regularization loss module based on uncertainty perception for the seen view; Submodule 2: Monocular depth regularization loss module for fusion uncertainty perception of unseen perspectives; Sub-module 3: Design a monocular depth adaptive loss module that integrates uncertainty perception; First, by establishing the correlation between the depth uncertainty modeling capability of the Neural Radiation Field (NeRF) and the depth map generated by Monocular Depth Estimation (MDE), Submodule 1 and Submodule 2 can explicitly capture and quantify the geometric ambiguity of the monocular depth estimation (MDE) output, thereby improving the model's ability to represent the ambiguity of depth estimation. Secondly, submodule three uses the depth map and its corresponding uncertainty map generated by the NeRF rendering process as supervision signals to design an adaptive optimization mechanism based on uncertainty perception to dynamically adjust the training process of the monocular depth estimation MDE model. Step 6 specifically also includes, Submodule 1 includes the introduction of an uncertainty-aware geometric alignment loss function to optimize the consistency between the NeRF-rendered depth map and the monocular estimated depth map; Specifically, it utilizes a pre-trained monocular depth estimation model. Predict each input Monocular estimated depth map ; The scaling factor is obtained by least-squares fitting within the local patch. With offset Align it to absolute depth to match the NeRF rendering depth map. ; Furthermore, pixel-level depth uncertainty values ​​estimated during the NeRF rendering process are introduced. Construct the following loss function: in: This indicates a stop-gradient operation, which does not return gradients. As an adaptive gradient masking function, the gradient of the supervision term in the high uncertainty region will be masked; Submodule two includes perspectives from any unseen viewpoint Render the corresponding RGB image and depth map and will Input pre-trained monocular depth estimator Obtain a pseudo-depth map ; The scaling factor was obtained by least squares fitting. and offset At the same time, combined with a previously unseen perspective Render the corresponding uncertainty graph in the middle. Design uncertainty perception : ； Submodule 3 includes introducing the use of scene geometry priors implied by the optimized NeRF model to back-guide the MDE model for adaptive training, thereby enhancing the MDE's ability to model absolute depth structures; A two-branch depth consistency loss is designed: the first term is an absolute depth alignment term without scale adjustment, used to improve MDE's ability to model scene scale; the second term is a patch-based scale alignment consistency term, which introduces pixel-level depth uncertainty in the NeRF rendering output. To achieve adaptive weight adjustment; The stop-gradient operation is applied to the uncertainty estimation, and the loss function is as follows: Among them, the first item For absolute depth alignment, In the depth map The depth value of the p-th pixel. In the depth map The depth value of the p-th pixel. This indicates a stop-gradient operation, which does not return gradients; the second term To integrate uncertainty-aware scale alignment consistency terms, This represents the pixel-level depth uncertainty value obtained from rendering. The scaling factor was obtained by least squares fitting. and offset ; Finally, by summing the loss functions, we obtain the loss function for the uncertainty of one iteration depth as follows: in, Set to 0.01, 0.02, 0.

01.

2. The sparse viewpoint indoor reconstruction method based on uncertainty-aware depth supervision according to claim 1, characterized in that... The construction of the uncertainty-perceiving neural radiation field model in step 3 specifically includes: Step S3.1: Input and construction of the neural radiation field model; The input construction stage of the neural radiation field model is based on pixels and light rays, combining multi-view image data with camera viewpoint direction to generate sampling points in three-dimensional space; Input multi-view image data The corresponding camera pose is input into the preprocessing module to generate the direction of the corresponding ray and the direction vector of the input ray. and starting point ; Then, samples are taken along each ray to obtain a series of sampling points. ,in, This represents the j-th sampling point on the i-th ray. This represents the different depth positions of the j-th sampling point on the light beam; these sampling points will subsequently be used as input to the neural network for subsequent color and volume density estimation. Step S3.2: Construct a neural radiation field for depth uncertainty perception by introducing an uncertainty perception module. The specific method is as follows: After obtaining the location of the sampling point through step S3.1... and direction Then, the position and orientation information of these sampling points are encoded and fed into the neural network model. Through training this neural network model, the model can learn to output the following three attributes for each sampling point: color. This indicates the color information and volume density of the sampling point. This represents the volume density at the sampling point, used to describe the geometry in the scene and the corresponding uncertainty prediction. This indicates the degree of uncertainty regarding the depth information of the sampling point; Step S3.3: Construct a pre-trained monocular depth estimation model MidasNet is used as a monocular depth prediction model to construct a pre-trained depth estimation model.

3. The sparse viewpoint indoor reconstruction method based on uncertainty-aware depth supervision according to claim 2, characterized in that... Step 4 specifically includes, Render the RGB values ​​on each ray. Depth value and deep uncertainty values ; Building upon classic volume rendering, in addition to the conventional color and depth weighted summation, a cumulative weighted summation is used to obtain depth uncertainty. Deep uncertainty This reflects the ambiguity or uncertainty in the depth information of certain areas in the scene; this uncertain prediction value is incorporated into the final volume rendering formula, and the specific composition formula is as follows: in, This represents the weight, that is, the contribution weight of the i-th sample point to the final rendered color; , It is the cumulative projection rate. It is opacity, in which, ; The values ​​for depth and depth uncertainty are obtained as follows: 。 4. The sparse viewpoint indoor reconstruction method based on uncertainty-aware depth supervision according to claim 3, characterized in that... Step 5 specifically includes, Step S5: Construct the photometric loss function of the fused random structural similarity term S3IM; combine the fused random structural similarity term S3IM loss and L2 loss, the specific loss function is as follows: in, for 。 5. The sparse viewpoint indoor reconstruction method based on uncertainty-aware depth supervision according to claim 1, characterized in that... Step 7 specifically includes the following: In the collaborative training of Neural Radiation Field (NeRF) and Monocular Depth Estimation (MDE), directly introducing uncertainty modeling will lead to instability in the early stages of training. Specifically, this includes... In the first 2000 epochs of training, only the geometry and appearance networks of the Neural Radiation Field (NeRF) and the depth prediction branch of the monocular depth estimation (MDE) model are used. At this time, the loss function only includes the basic rendering loss and depth alignment loss. in, For NeRF's RGB rendering loss, Specifically, it is 0.

1. For depth loss, specifically: After the model initially converges, during the transition phase between 2000 and 10000 training epochs, uncertainty optimization is gradually introduced, along with uncertainty loss, where u(t) is the control... and The weights increase with the number of training rounds: Finally, in the later stages of training, after 50% completion, uncertainty optimization is fully enabled, and the loss function includes a complete uncertainty-aware term: 。

Images