A High-Fidelity Reconstruction Method for Indoor Scenes Based on Normal Deflection Networks

Abstract

This invention discloses a high-fidelity reconstruction method for indoor scenes based on a normal deflection network, achieving accurate recovery of indoor 3D surfaces from an image set with known poses. The method constructs a normal deflection network within a volume rendering framework based on SDF values. This network stores the angular deviation between scene normals and normal priors. The invention establishes an indoor surface volume rendering reconstruction model including the deflection network, utilizing volume rendering techniques to synthesize the rendered color, rendering depth, rendered normals, and rendering deflection quaternions of light rays. Based on the deviations learned by the proposed normal deflection network, the method adaptively utilizes prior samples with different characteristics and performs targeted optimization for areas with large deviations, solving the problems of traditional multi-view stereo vision frameworks and conventional implicit reconstruction methods that rely on priors. Experiments on numerous indoor environments demonstrate that the method achieves advanced performance in recovering dense indoor surfaces.

Description

Technical Field

[0001] This invention relates to the field of 3D surface reconstruction, and specifically to a high-fidelity surface reconstruction method for indoor scenes based on a normal deflection network. Background Technology

[0002] The goal of 3D surface reconstruction is to reconstruct a watertight 3D model. One approach is to reconstruct accurate 3D scene geometry and appearance from multi-view images, which is an important research problem in computer vision and graphics. Reconstructed dense surfaces have a wide range of applications, including but not limited to localization, measurement, animation, games, virtual / augmented reality (VR / AR), and indoor robot navigation.

[0003] Traditional implicit approaches, such as the Marching Cubes algorithm, extract triangular mesh surfaces from estimated implicit representation functions (SDF / TSDF); global approaches, such as Poisson surface reconstruction methods, directly estimate dense surfaces from point cloud data. Multi-view stereo geometry (MVS) algorithms, which directly reconstruct 3D models from a set of images, typically involve sparse reconstruction to calculate pose, stereo matching to estimate depth and normal maps, inverse projection to fuse dense point clouds, and then Poisson reconstruction to restore dense surfaces. MVS methods can achieve highly detailed and accurate reconstruction results. However, MVS algorithms rely on photometric consistency and multi-view constraints, making them unable to handle homogeneous, textureless regions and ambiguous observations. This can lead to erroneous surfaces or lost structures, problems particularly prominent in indoor scenes with numerous low-texture areas.

[0004] Neural networks demonstrate powerful performance and simplicity in encoding the latent structure and appearance of scenes. The combination of neural implicit reconstruction methods such as VolSDF and NeuS with rendering achieves remarkable performance. Compared to traditional MVS methods, the inductive smoothing property of neural networks allows implicit methods to obtain smoother and more complete surfaces. Indoor scenes often involve complex lighting and environments. To handle highlight and low-frequency regions (walls, floors, etc.), subsequent works such as Manhattan-SDF propose combining the Manhattan hypothesis with semantic information for optimization, resulting in better performance. Other methods, such as MonoSDF and NeuRIS, propose using auxiliary data to improve reconstruction quality, including monocular depth and monocular normal cues output by pre-trained models.

[0005] Problems with existing methods: 1) Inability to simultaneously guarantee smoothness and detail: Traditional MVS algorithms or ordinary implicit reconstruction methods such as VolSDF cannot handle indoor scenes containing a large number of homogeneous, textureless regions. Implicit methods such as MonoSDF, which introduce monocular depth and normal cues obtained from pre-trained models, have achieved considerable performance improvements in indoor reconstruction. However, the prior information is not accurate, especially in high-frequency complex structural regions. Incorrect supervision leads to overly smoothed surfaces and loss of a large amount of detail. 2) Dependence on various priors and poor adaptability: Some methods attempt to filter out erroneous geometric priors through semantic, visual features, or even prior uncertainty information. Among these, semantic information and prior uncertainty rely more on accurate prior models, while visual features have poor adaptability in large scenes and cannot distinguish high-frequency smooth regions. An adaptive and robust indoor reconstruction method is urgently needed. 3) Inability to recover fine structures: Current SDF-based implicit reconstruction algorithms have inherent bias problems, leading to the disappearance of small or fine structures in the scene. Summary of the Invention

[0006] To address the problems in existing technologies, this invention provides a high-fidelity reconstruction method for indoor scenes based on a normal deflection network. This method constructs an indoor surface volume rendering and reconstruction model comprising a position encoder, a hash encoder, a scene geometry network, a color network, and a normal deflection network. The normal deflection network is a neural network that uses MLP reparameterization to store the deviation between scene geometry and prior geometry. Its inputs are sampling point coordinates, ray direction, scene normals, and scene geometric features. Its output is a deflection quaternion of the sampling points, used to deflect the rendering normals.

[0007] The high-fidelity reconstruction method for indoor scenes based on normal deflection networks proposed in this invention includes the following steps:

[0008] Step 1: Preprocess all RGB images in the multi-view image set of the scene to be reconstructed with known poses and camera intrinsics, and obtain monocular depth and monocular normal cues for each image;

[0009] Step 2: Construct an indoor surface volume rendering and reconstruction model, which includes a position encoder, a hash encoder, a scene geometry network, a color network, and a normal deflection network;

[0010] The pose and camera intrinsic parameters corresponding to the multi-view images of the scene to be reconstructed are input into the indoor surface volume rendering and reconstruction model. The indoor surface volume rendering and reconstruction model implements a surface volume rendering optimization process based on SDF values. The rendering color, rendering depth, rendering normal and rendering deflection quaternion of the sampled light in the scene are synthesized using volume rendering technology.

[0011] Step 3: Use rendering deflection quaternions to deflect the rendering normals. By aligning the deflected rendering normals with the prior normals, learn the deviation between the scene geometry and the prior geometry implied by the normal priors. Calculate the angle difference between the deflected rendering normals and the rendering normals before deflection, and construct an adaptive deflection angle prior consistency loss to dynamically adjust the utilization of the prior of samples in different characteristic regions.

[0012] Step 4: Based on the angle difference between the deflected rendering normal and the undeflected rendering normal obtained in Step 3, first construct a pixel-by-pixel deflection angle map for each RGB image. During the training process, dynamically maintain the deflection angle map according to the angle difference; achieve multi-sampling of high-amplitude deflection angles to indicate high-frequency regions and multi-sampling of low-amplitude deflection angles to indicate low-texture regions.

[0013] Then, the deflection angle map is used to query the deflection angle of the sampled light during the training process, and the deflection angle of the sampled light and the unbiased rendering transformation function are combined to perform unbiased rendering of small or fine structures.

[0014] Then, the deflection angle-weighted photometric consistency loss calculated during the training process is utilized;

[0015] Step 5: Construct the total loss function of the indoor surface volume rendering and reconstruction model; train the indoor surface volume rendering and reconstruction model: after training, extract the zero level set of the scene geometry network, which is the final reconstructed dense 3D surface of the indoor scene.

[0016] According to a preferred embodiment of the present invention, step 1 is as follows: cropping or scaling all RGB images in the image set, converting them into the image input format required by the monocular cue pre-training model and transforming the camera intrinsic parameters, and using the monocular cue pre-training model to predict the monocular depth and monocular normal cue for each image.

[0017] According to a preferred embodiment of the present invention, step 2, the surface volume rendering optimization process based on SDF values, includes:

[0018] The coordinates of interest are sampled along the light lines in the scene; these coordinates are then input to a position encoder and a hash encoder to obtain position codes and local scene geometric feature codes; combined with these encoded features, a scene geometry network f is used. g Color network f c and normal deflection network f d Query to obtain the SDF value, color, deflection quaternion, scene normal, and scene implicit geometric features of the sampling points.

[0019] The total loss function for constructing the indoor surface volume rendering and reconstruction model is as follows:

[0020] Constructing a color loss based on deflection angle Eikonal standard loss Adaptive prior loss based on deflection angle and Total loss function For the multi-resolution hash grid γ L Scene geometric network f g Color network f c and deflection field network f d Training is performed to obtain a high-fidelity three-dimensional surface;

[0021] The Eikonal normalized loss It is the loss of the SDF value in the gauge space;

[0022] The color loss based on deflection angle Re-weighting the rendering colors using the deflection angle The loss of true color C(r);

[0023] The adaptive prior loss based on deflection angle and The prior loss of samples with different characteristics is adaptively calculated using the deflection angle;

[0024] Total loss function Represented as:

[0025]

[0026] Wherein, λ1, λ2, and λ3 are weighting coefficients used to balance the contributions of each loss.

[0027] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0028] 1) This invention improves robustness to inaccurate priors: By introducing a Normal Deflection network, this invention utilizes the neural network to reparameterize the deviation between scene geometry and prior geometry, estimating the angle difference between the geometric normal and the prior normal at any point. This addresses the shortcomings of existing methods in effectively handling erroneous priors, effectively addressing the problem of inaccurate reconstruction under erroneous prior supervision in implicit reconstruction methods with priors, and improving the robustness of implicit reconstruction methods to prior cues. The Normal Deflection network can learn the deviation between scene geometry and the prior geometry implied by the normal prior. Utilizing the learned deviation information, it significantly improves the adaptability of prior information, achieving more accurate reconstruction under arbitrary prior supervision. The adaptive deflection angle prior loss constructed accordingly effectively filters out erroneous priors, achieving dynamic prior supervision and accurate surface restoration.

[0029] 2) This invention solves the bias problem in surface volume rendering: By combining learned deflection angle information with an unbiased rendering transformation function, this invention achieves unbiased rendering optimization for high-frequency structures implied by deflection angles, effectively alleviating the bias problem in surface volume rendering. This significantly improves the recovery of small or fine structures and promotes the integrity of complex indoor scene reconstruction.

[0030] 3) This invention improves the quality of surface reconstruction in complex indoor scenes: It utilizes deflection angle information to adaptively adjust the prior utilization level of different characteristic regions, achieving automatic differentiation and supervision of high- and low-frequency regions. Furthermore, by using angular deviation to divide structures with different levels of detail, this invention specifically improves the sampling rate and photometric consistency loss in fine-grained regions, further promoting the recovery of more surface details and significantly improving the quality of indoor surface reconstruction. Experiments on ScanNet and ScanNet++ indoor datasets, as well as validation on TanksandTemples, demonstrate the advantages of this invention compared to existing state-of-the-art methods, proving its robustness and effectiveness in handling complex indoor scenes. Attached Figure Description

[0031] Figure 1 This is an overall framework diagram of the present invention;

[0032] Figure 2 This is a network structure diagram of the present invention;

[0033] Figure 3 This is the qualitative result of the reconstruction on the ScanNet dataset by the present invention, and it is compared with other solutions;

[0034] Figure 4 This is the qualitative result of the reconstruction on the TanksandTemples dataset of this invention. Detailed Implementation

[0035] The following description, in conjunction with the accompanying drawings, provides a detailed account of the present invention. The technical features of each embodiment of the present invention can be combined appropriately without conflict.

[0036] This invention is a high-fidelity reconstruction method for indoor scenes based on a normal deflection network. It learns the deviations between the actual normal geometry and prior geometry of points in the scene, and then dynamically adjusts the prior utilization of samples in different regions using angular deviation information. Subsequently, the method further combines deviation information to improve the sampling rate and photometric loss of more detailed structures, and applies an unbiased transformation function of volume density to small or fine structures, thereby achieving high-fidelity, fine-grained and highly smooth indoor surface reconstruction.

[0037] like Figure 1 As shown, the method specifically includes:

[0038] Step 1: Preprocess all RGB images in the multi-view image set of the scene to be reconstructed with known poses and camera intrinsics, and obtain monocular depth and monocular normal cues for each image;

[0039] Step 2: Construct an indoor surface volume rendering and reconstruction model, which includes a position encoder, a hash encoder, a scene geometry network, a color network, and a normal deflection network;

[0040] The pose and camera intrinsic parameters corresponding to the multi-view images of the scene to be reconstructed are input into the indoor surface volume rendering and reconstruction model. The indoor surface volume rendering and reconstruction model implements a surface volume rendering optimization process based on SDF values. The rendering color, rendering depth, rendering normal and rendering deflection quaternion of the sampled light in the scene are synthesized using volume rendering technology.

[0041] Step 3: Use rendering deflection quaternions to deflect the rendering normals. By aligning the deflected rendering normals with the prior normals, learn the deviation between the scene geometry and the prior geometry implied by the normal priors. Calculate the angle difference between the deflected rendering normals and the rendering normals before deflection, and construct an adaptive deflection angle prior consistency loss to dynamically adjust the utilization of the prior of samples in different characteristic regions.

[0042] Step 4: Based on the angle difference between the deflected rendering normal and the undeflected rendering normal obtained in Step 3, first construct a pixel-by-pixel deflection angle map for each RGB image. During the training process, dynamically maintain the deflection angle map according to the angle difference; achieve multi-sampling of high-amplitude deflection angles to indicate high-frequency regions and multi-sampling of low-amplitude deflection angles to indicate low-texture regions.

[0043] Then, the deflection angle map is used to query the deflection angle of the sampled light during the training process, and the deflection angle of the sampled light and the unbiased rendering transformation function are combined to perform unbiased rendering of small or fine structures.

[0044] Then, the deflection angle-weighted photometric consistency loss calculated during the training process is utilized;

[0045] Step 5: Construct the total loss function of the indoor surface volume rendering and reconstruction model; train the indoor surface volume rendering and reconstruction model: after training, extract the zero level set of the scene geometry network, which is the final reconstructed dense 3D surface of the indoor scene.

[0046] like Figure 2 As shown, the indoor surface volume rendering and reconstruction model of this invention consists of the following structure: a multi-resolution hash grid γ that encodes the local geometric features of the scene. L Geometric network f encoding implicit directed metric field g The color network f of the encoded scene radiation cThe normal deflection network f encodes the deflection quaternion representing the deviation between scene geometry and prior geometry. d All networks are constructed using MLP. The normal deflection network f proposed in this invention... d Parameterized as:

[0047] q = f d (x,v,n,z)

[0048] Where q=(q 0 ,q 1 ,q 2 ,q 3 ) represents any sampling point x from network f d The retrieved deflection quaternion (normalized by default) is used, where v is the direction of the ray at the sampling point, n is the scene normal at the sampling point, and z is the implicit geometric feature at the sampling point. All these tensors are concatenated sequentially and then input into the deflection network f. d This allows for the prediction of deflection quaternions.

[0049] The steps of the present invention are described in detail below:

[0050] Step 1: Monocular Clue Estimation and Initialization

[0051] First, before training, preprocess all RGB image data and related parameters in a multi-view image set containing known poses and camera intrinsics of the scene to be reconstructed: crop or scale all RGB images in the image set, convert them to the image input format required by the monocular cue pre-training model, and transform the camera intrinsic parameters. Input all images into a monocular cue pre-training model network (such as Omnidata) to predict the monocular depth and monocular normal cues for each image. Confirm that the poses of all images are known and that the scene is within a finite unit bounding box or bounding sphere. If it exceeds this range, the scene needs to be transformed back to within the bounding box and the pose data adjusted.

[0052] Step 2: Sampling, Scene Representation, and Surface Volume Rendering

[0053] The poses and camera intrinsic parameters corresponding to the multi-view images of the scene to be reconstructed are input into the indoor surface volume rendering and reconstruction model. The indoor surface volume rendering and reconstruction model implements a surface volume rendering optimization process based on SDF values: sampling the coordinates of interest on the light lines sampled in the scene; inputting the coordinates to the position encoder and hash encoder to obtain the position code and local scene geometric feature code; combining the encoded features, using the scene geometric network f g Color network f c and deflection network f d Query to obtain the SDF value, color, deflection quaternion, scene normal, and scene implicit geometric features of the sampling points.

[0054] The rendering color, rendering depth, rendering normal, and rendering deflection quaternion of the sampled light rays in the scene are synthesized using volume rendering technology; the synthesized geometric properties are used in the subsequent optimization process.

[0055] Step 3: Learn the bias and dynamically adjust the prior utilization rate

[0056] This step uses the rendering deflection quaternion obtained in step 2 to deflect the rendering normals, aiming to learn the deviation between scene geometry and prior geometry. Quaternion multiplication is used to apply the rendering deflection quaternion to the rendering deflection normals, resulting in the deflected rendering normals. The goal is to align the deflected rendering normals with the prior normals, thereby automatically learning the deviation between the scene set and the prior set. The optimized deviation is stored in the normal deflection network. The learned deviations can help dynamically adjust the prior utilization rate for different characteristic regions, achieving high utilization of high-confidence priors while filtering out inaccurate priors.

[0057] Step 4: Use deflection angle information to guide detailed optimization

[0058] The preceding steps generated deflection angle information, which is then used in this step to guide further surface detail optimization. The optimization is based on the fact that the deflection angle implies different structural characteristics within the indoor scene; higher deflection angles correspond to more complex structures, while lower angles correspond to textureless structures. Therefore, adjusting the light sampling rate and volume rendering photometric consistency loss in different areas according to the deflection angle promotes more sampling of high-frequency structures in the image and applies more photometric optimization. Further fusing the deflection angle information and unbiased rendering transformation functions alleviates biased rendering problems of specific structures, promoting the recovery of more small or fine structures.

[0059] Step 5: Construct the total loss function and train the model.

[0060] Based on the aforementioned optimization methods, a total loss function for the indoor surface volume rendering and reconstruction model is constructed; the indoor surface volume rendering and reconstruction model is trained; after training, the zero-level set of the scene geometry network is extracted, which is the final reconstructed dense 3D surface of the indoor scene.

[0061] As a preferred embodiment of the present invention, the surface volume rendering optimization process based on SDF values ​​in step 2 is specifically as follows: assuming the light set obtained from this iteration sampling is... The parameterized equation for a point on the ray is expressed as r(t) = o + tv, where o is the viewpoint, v is the ray direction, and t is the distance of the sampling point from the viewpoint. N points x are sampled on the ray. i =o+t i v, i ∈ {1, 2, ... N}. All sampling points of this ray are input to the position encoder PE and the multi-resolution hash grid encoder γ. LThe location features and hash grid encoded scene local set features are obtained from the encoding process. These features are then combined and input into the scene geometry network f. g The SDF value s(x) and implicit geometric feature z(x) are obtained from this:

[0062]

[0063] Where PE(x) is a positional encoding based on sine and cosine functions, γ L (x) represents the feature of the first and last concatenation of the multi-resolution hash grid encoding output.

[0064] Next, the scene normal n at the sampling point location is calculated using the gradient analysis method. The calculation formula is as follows:

[0065]

[0066] in, This indicates taking the partial derivative with respect to the sampling point x;

[0067] Construct a color network f c and normal deflection network f d The inputs include the coordinates of the sampling point, the direction of the ray, the scene normal, and the implicit geometric features of the scene. These tensors are concatenated and input into the color network and the deflection network to obtain the color c and the deflection quaternion q of the sampling point.

[0068] c = f c (x,v,n,z),q=f d (x,v,n,z)

[0069] Color networks and deflection networks, such as Figure 2 The images shown all consist of two layers of multilayer perceptrons containing 256 neurons each, with ReLU activation function. This step aims to query the neural field properties of all sampling points, including color, normals, and the deflection quaternion proposed in this invention, in preparation for the next step of volume rendering and synthesis.

[0070] As a preferred embodiment of the present invention, the SDF-based volume rendering synthesis method in step 2 specifically involves: using the Laplacian cumulative density function to convert the SDF value into the volume density required for volume rendering, and the calculation formula is as follows:

[0071]

[0072] Among them, Ψ β σ(x) is the Laplace cumulative density function, β is the variance of the Laplace cumulative density function, σ(x) is the volume density at sampling point x, and s(x) is the SDF value output by the geometric network.

[0073] Next, volume rendering techniques are used to synthesize the neural field properties of pixels (light rays), including rendering colors. Rendering depth Rendering normals The rendering deflection quaternion Q(r) proposed in this invention is calculated using the following formula:

[0074]

[0075] Among them, c i σ i These represent the upsampling points x and x of the light rays, respectively. i Color and volume density, α i Indicates sampling point x i and sampling point x i-1 The opacity of light between T i Indicates sampling point x i and sampling point x i-1 The light transmission rate between the light rays, δ i =t i -t i-1 It is the distance between adjacent sampling points; n i q i These represent the upsampling points x and x of the light rays, respectively. i The scene normal and deflection quaternion of the position.

[0076] Using monocular depth and monocular normal cues to supervise the rendering depth and rendering normal optimization surface, the original normal loss is constructed. and the original depth loss The calculation formula is:

[0077]

[0078]

[0079] Where N(r) and D(r) are the monocular depth and monocular normal cues output by the preprocessing process, w is the scaling parameter and q is the offset parameter.

[0080] As a preferred embodiment of the present invention, the deflection method and the method for dynamically adjusting the prior utilization rate in step 3 are specifically as follows: the rendering normal is deflected using the deflection quaternion synthesized by the rendering, and the calculation is performed using quaternion multiplication:

[0081]

[0082] in, Q represents the rendered normal after deflection. -1 (r) denotes the inverse of Q(r), which is equivalent to conjugate in quaternions. It is a quaternion multiplication operation;

[0083] By minimizing the difference between the deflected rendered normals and the prior normals, the deviation between scene geometry and prior geometry can be automatically learned, and the deflection normal loss is constructed as follows:

[0084]

[0085] The bias learned during training is defined as the angle difference between the rendered normal and the rendered normal after deflection. The angle between the normals before and after deflection is calculated using the dot product method.

[0086]

[0087] Where Δθ∈[0,π] represents the angular difference of the rendered normal before and after deflection, i.e., the deflection angle. Furthermore, by adaptively adjusting the prior utilization of different characteristic regions using the deflection angle, we can construct the monocular depth and monocular normal prior loss with adaptive deflection angle as follows:

[0088]

[0089]

[0090] Where g d (Δθ) and g(Δθ) are modulation functions that adjust the weights of the rendering normal loss term after deflection and the original rendering normal loss term based on the deviation angle, thereby achieving dynamic differentiation and utilization of prior information. The modulation function is constructed using a translated Logistic function:

[0091]

[0092] g(Δθ)=1-g d (Δθ)

[0093] In this function, the independent variable is the deflection angle Δθ, and s0 is the control variable g. d The scaling factor for steepness, θ0 is the offset term, set s0 = 12.5.

[0094] As a preferred embodiment of the present invention, step 4, which involves using deflection angle information to guide detailed optimization, specifically includes:

[0095] For a ray sampling strategy based on deflection angle, a deflection angle map for each image is dynamically maintained during training. The maintenance method utilizes the old deflection angle diagram. The update strategy for iteratively updating the deflection angle Δθ(r) calculated in this iteration is as follows:

[0096]

[0097] Where γ decayThis represents the decay factor of the old deflection angle map, and Δθ(r) is the deflection angle calculated in the current iteration step. The update means taking the maximum value between the decayed deflection angle map and the latest deflection angle.

[0098] During the sampling phase, a probability map is obtained by calculating the sampling probability of each pixel using the deflection angle map. Then, inverse sampling is performed based on the probability map, with more rays being sampled in areas with larger deflection angles. A scaled and translated Logistic function is used to calculate the sampling probability p(r) of each pixel. i ):

[0099]

[0100] in, This is a deflection angle map dynamically maintained for each image during training, where s1 is the scaling factor controlling the steepness of the function, and θ1 is the offset term; it can be seen that p(r i )∈[1,t1], the steepness is controlled by s1, and inverse sampling is performed after normalizing the probability graph:

[0101]

[0102] For an unbiased rendering transformation function based on deflection angle, in order to achieve partial unbiased rendering of specific structures implied by the deflection angle, a biased confidence cfd(r) is calculated using a translated Logistic function based on the deflection angle:

[0103]

[0104] Where s3 is the scaling factor that controls the steepness of the biased confidence function, and θ3 is the offset term;

[0105] Based on the above biased confidence level, a partial unbiased transformation of the structure based on the deflection angle is achieved:

[0106]

[0107] Where r(t) is the parameterized representation of the coordinate point x sampled by ray r, f g It is a scene geometry network, which omits the position encoding PE(x) and hash encoding γ for convenience. L (x), f g (x,PE(x),γ L (x) is abbreviated as f g (r(t)); f′ g (r(t)) represents the derivative with respect to t, and the result of the derivative is n·v, which is equal to the dot product of the scene normal and the ray.

[0108] For photometric optimization guided by deflection angle, the method is to reweight the color loss of different samples according to the deflection angle, and then use another scaled and translated Logistic function to calculate the color loss weights based on the deflection angle.

[0109]

[0110] Where s2 is the scaling factor that controls the steepness of the weighting function, and θ2 is the offset term;

[0111] set up

[0112] Color loss based on deflection angle Represented as:

[0113]

[0114] Where C(r) is the actual image pixel color, w color It is a weighted function.

[0115] The comprehensive optimization of the loss across all networks in this invention includes: a color loss reweighted based on the deflection angle. Eikonal standard loss and adaptive prior loss based on deflection angle and

[0116] The Eikonal normalized loss The loss is the SDF value in the gauge space:

[0117]

[0118] The color loss based on deflection angle Re-weighting the rendering colors using the deflection angle The loss of true color C(r);

[0119] The adaptive prior loss based on deflection angle and The prior loss of samples with different characteristics is adaptively calculated using the deflection angle;

[0120] Total loss function Represented as:

[0121]

[0122] Wherein, λ1, λ2, and λ3 are weighting coefficients used to balance the contributions of each loss. The color loss, reweighted according to the deflection angle, specifically optimizes high-frequency structures with large deviations, improving surface detail. The adaptive prior loss based on the deflection angle dynamically adjusts the prior utilization rate of different samples according to the deflection angle, improving robustness to prior cues. By optimizing the above losses and implementing the deflection angle-based optimization method described in step 4, this invention reconstructs high-fidelity indoor surfaces.

[0123] Specifically, this invention utilizes the total loss function For the multi-resolution hash grid γ L Scene geometric network f g Color network f c and deflection field network f d After training and network convergence, the Marching Cubes algorithm is used to extract the dense surface implied by the implicit distance field.

[0124] Example

[0125] To demonstrate the effectiveness of this invention, this embodiment comprehensively evaluates the ND-SDF (Normal Deflection Signed Distance Function) reconstruction method on standard indoor datasets (such as ScanNet and ScanNet++), and conducts qualitative and quantitative comparisons with previous state-of-the-art methods such as Monosdf and HelixSurf to assess its performance and effectiveness in standard indoor scenes. Simultaneously, this embodiment will also perform qualitative validation on the large dataset TanksandTemples.

[0126] Implementation details:

[0127] This embodiment primarily trains the ND-SDF model on the ScanNet dataset. Training intervals are set to lr=1e-3, with 4096 rays sampled per iteration, for a total of 120,000 training steps until the network converges. The model was trained for 10 hours on a single RTX3090 processor.

[0128] Evaluation indicators:

[0129] In this embodiment, a triangular mesh surface is restored. Therefore, distance and score metrics are used to evaluate the difference between the reconstructed surface and the real surface.

[0130] Distance metrics refer to the calculation of the distance between the reconstructed surface and the real surface. This is done by cloudifying the surface points and then calculating the distance between the two point clouds. There are three calculation methods: Accuracy, Completeness, and Chamfer-Distance, with units of mm or cm, abbreviated as Acc, Comp, and Chamfer, respectively.

[0131] The scoring metrics first calculate the distance from each point in the point cloud to another point cloud, then count the proportion of points with distances less than a specified threshold. This allows for the calculation of metrics such as precision, recall, and F-score. For example, precision represents the proportion of points on the reconstructed surface that are less than 5 cm from the real surface. It's worth noting that among all metrics, the F-score is the most important for evaluating the accuracy of the reconstructed surface because it considers both accuracy and completeness. Precision is abbreviated as Pre.

[0132] Experiment 1: Comparison of reconstructed surfaces in the ScanNet dataset

[0133] The ScanNet dataset contains 1513 indoor scenes captured using a Kinect V1 RGB-D camera. Camera pose and surface reconstruction were obtained using the BundleFusion algorithm. This invention adopts the Manhattan-SDF evaluation strategy, defining the four selected indoor scenes 0050_00, 0084_00, 0580_00, and 0616_00. This embodiment will compare the results with the classic multi-view stereo vision algorithm COLMAP and the most advanced implicit reconstruction methods in recent years, such as VolSDF and MonoSDF. The results are shown in Table 1 below.

[0134] Table 1: Quantitative evaluation of reconstructed surfaces on the ScanNet dataset

[0135]

[0136]

[0137] The quantitative results in the table indicate that the traditional COLMAP method recovers a limited number of reliable surfaces. Common implicit reconstruction methods that do not utilize auxiliary data, such as VolSDF and NeuS, fail to reconstruct the correct surface at all, achieving the lowest F-score. While recent advanced indoor methods such as Monosdf, NeuRIS, and HelixSurf perform well, the reconstruction method based on the normal deflection field in this invention significantly outperforms them, with an F-score improvement of 7.2% compared to MonoSDF and 4.1% compared to the current state-of-the-art HelixSurf.

[0138] Figure 3 The qualitative comparison results of surface reconstruction on the ScanNet dataset are also presented. It can be observed that the method of this invention can accurately capture difficult high-frequency structures in indoor scenes, such as stool legs, table legs, and light rings, demonstrating the superiority of our proposed deflection field and a series of strategies combining the deflection field. Both qualitative and quantitative results are optimal.

[0139] Experiment 2: Testing on the TanksandTemples large dataset

[0140] TankandTemples is a large-scale 3D reconstruction dataset, including high-resolution (4K video) outdoor scenes and indoor environments. To evaluate the performance and generalization ability of our proposed method in large and complex scenes, this embodiment selects four large indoor scenes from TankandTemples. The evaluation method involves submitting the reconstructed surfaces to T&T's website for F-score calculation. This embodiment is compared with the state-of-the-art implicit reconstruction method MonoSDF. By running the open-source code of MonoSDF, experiments show that MonoSDF cannot reconstruct relatively accurate surfaces in the four scenes of T&T, and our proposed method can reconstruct more details, demonstrating the effectiveness of our proposed method. In the T&T subset, our proposed method improves the F-score from 6.58 for the state-of-the-art MonoSDF to 9.35. Specific experimental data are shown in Table 2 below.

[0141] Table 2: Quantitative assessment of restored surfaces (F-score) on Tanks and Temples.

[0142]

[0143] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A high-fidelity reconstruction method for indoor scenes based on a normal deflection network, characterized in that, Includes the following steps: Step 1: Preprocess all RGB images in the multi-view image set of the scene to be reconstructed with known poses and camera intrinsics, and obtain monocular depth and monocular normal cues for each image; Step 2: Construct an indoor surface volume rendering and reconstruction model, which includes a position encoder, a hash encoder, a scene geometry network, a color network, and a normal deflection network; The pose and camera intrinsic parameters corresponding to the multi-view images of the scene to be reconstructed are input into the indoor surface volume rendering and reconstruction model, and the indoor surface volume rendering and reconstruction model implements the surface volume rendering optimization process based on SDF value. The rendering color, rendering depth, rendering normal, and rendering deflection quaternion of sampled light rays in the scene are synthesized using volume rendering technology. Step 3: Use rendering deflection quaternions to deflect the rendering normals. By aligning the deflected rendering normals with the prior normals, learn the deviation between the scene geometry and the prior geometry implied by the normal priors. Calculate the angle difference between the deflected rendering normals and the rendering normals before deflection, and construct an adaptive deflection angle prior consistency loss to dynamically adjust the utilization of the prior of samples in different characteristic regions. Step 4: Based on the angle difference between the deflected rendering normal and the undeflected rendering normal obtained in Step 3, first construct a pixel-by-pixel deflection angle map for each RGB image. During the training process, dynamically maintain the deflection angle map according to the angle difference; achieve multi-sampling of high-amplitude deflection angles to indicate high-frequency regions and multi-sampling of low-amplitude deflection angles to indicate low-texture regions. Then, the deflection angle map is used to query the deflection angle of the sampled light during the training process, and the deflection angle of the sampled light and the unbiased rendering transformation function are combined to perform unbiased rendering of small or fine structures. Then, the deflection angle-weighted photometric consistency loss calculated during the training process is utilized; Step 5: Construct the total loss function of the indoor surface volume rendering and reconstruction model; train the indoor surface volume rendering and reconstruction model: after training, extract the zero level set of the scene geometry network, which is the final reconstructed dense 3D surface of the indoor scene.

2. The high-fidelity reconstruction method for indoor scenes based on normal deflection networks according to claim 1, characterized in that, Step 1 is as follows: crop or scale all RGB images in the image set, convert them to the image input format required by the monocular cue pre-training model, transform the camera's internal parameters, and use the monocular cue pre-training model to predict the monocular depth and monocular normal cue for each image.

3. The method for high-fidelity reconstruction of indoor scenes based on normal deflection networks according to claim 1, characterized in that, In step 2, the surface volume rendering optimization process based on SDF values ​​includes: The coordinates of interest are sampled along the light lines in the scene; these coordinates are then input to a position encoder and a hash encoder to obtain position codes and local scene geometric feature codes; combined with these encoded features, a scene geometry network f is used. g Color network f c and normal deflection network f d Query to obtain the SDF value, color, deflection quaternion, scene normal, and scene implicit geometric features of the sampling points.

4. The method for high-fidelity reconstruction of indoor scenes based on normal deflection networks according to claim 1 or 3, characterized in that, The surface volume rendering optimization process based on SDF values ​​is as follows: The parameterized equation for a point on a ray is expressed as r(t) = o + tv, where o is the viewpoint, v is the ray direction, and t is the distance of the sampling point from the viewpoint. N points x are sampled on the ray. i =o+t i v, i ∈ {1, 2... N}; using the scene geometry network f g Predict the SDF value for all sample points: Where PE(x) is a positional encoding based on sine and cosine functions, γ L z(x) represents the local scene geometric features of the first and last stitched output of the multi-resolution hash grid encoding, z(x) represents the implicit geometric features of the sampling point location, and s(x) represents the SDF value of the sampling point. The scene normal at the sampling point location is calculated using the gradient analysis method. The calculation formula is as follows: in, This indicates taking the partial derivative with respect to the sampling point x; Construct a color network f c and normal deflection network f d The inputs, including the coordinates of the sampling point, the direction of the light ray, the scene normal, and the implicit geometric features of the scene at the sampling point position, are fed into the color network and the normal deflection network to obtain the color c retrieved from the sampling point x and the deflection quaternion q. c=f c (x,v,n,z) q=f d (x,v,n,z)。 5. The high-fidelity reconstruction method for indoor scenes based on a normal deflection network according to claim 4, characterized in that, The process of using volumetric rendering technology to synthesize the rendered color, rendered depth, rendered normal, and rendered deflection quaternions of light sampled in the scene is as follows: The SDF value is converted into the volume density required for volume rendering using the Laplacian cumulative density function. The calculation formula is as follows: Among them, Ψ β σ(x) is the Laplace cumulative density function, β is the variance of the Laplace cumulative density function, σ(x) is the volume density at sampling point x, and s(x) is the SDF value output by the geometric network. Synthesizing the color of light using volume rendering technology Among them, c i σ i These represent the upsampling points x and x of the light rays, respectively. i Color and volume density, α i Indicates sampling point x i and sampling point x i-1 The opacity of light between T i Indicates sampling point x i and sampling point x i-1 The light transmission rate between the light rays, δ i =t i -t i-1 It is the distance between adjacent sampling points; Calculate the depth of the composite rendering Rendering normals And the rendering deflection quaternion Q(r): Where, n i q i These represent the upsampling points x and x of the light rays, respectively. i The scene normal and deflection quaternion of the position; Using monocular depth and monocular normal cues to supervise the rendering depth and rendering normal optimization surface, the original normal loss is constructed. and the original depth loss The calculation formula is: Where N(r) and D(r) are the monocular depth and monocular normal cues output by the preprocessing process, w is the scaling parameter and q is the offset parameter.

6. The method for high-fidelity reconstruction of indoor scenes based on normal deflection networks according to claim 5, characterized in that, In step 3, the rendering normals are deflected using rendering deflection quaternions. This is achieved by aligning the deflected rendering normals with the prior normals to learn the deviation between the scene geometry and the prior geometry implied by the normals. Specifically: Use rendering deflection quaternions to deflect rendering normals: in, Q represents the rendered normal after deflection. -1 (r) denotes the inverse of Q(r), which is equivalent to conjugate in quaternions. It is a quaternion multiplication operation; By automatically learning the deviation between scene geometry and prior geometry by minimizing the difference between the deflected rendered normals and the prior normals, a deflection normal loss is constructed. for:

7. The method for high-fidelity reconstruction of indoor scenes based on normal deflection networks according to claim 6, characterized in that, In step 3, the angle difference between the rendered normal after deflection and the rendered normal before deflection is calculated, and an adaptive deflection angle prior consistency loss is constructed to dynamically adjust the utilization degree of prior information for samples in different characteristic regions. Specifically: The bias learned during training is defined as the angle difference between the rendered normal and the rendered normal after deflection. The angle between the normals before and after deflection is calculated using the dot product method. Where Δθ∈[0,π] represents the angle difference of the rendered normal before and after deflection, i.e., the deflection angle; the magnitude of the deviation is equal to the magnitude of the deflection angle, and the degree of prior utilization of different characteristic regions is adaptively adjusted using the deflection angle; the monocular depth prior loss and monocular normal prior loss for adaptive deflection angle are constructed as follows: in, It is the collection of sampled rays, g d (Δθ) and g(Δθ) are modulation functions used to adjust the weights of the rendering normal loss term after deflection and the original rendering normal loss term according to the deviation angle, so as to realize the dynamic differentiation and utilization of prior knowledge.

8. The high-fidelity reconstruction method for indoor scenes based on a normal deflection network according to claim 7, characterized in that, Step 4 specifically involves: Dynamically maintain the deflection angle map for each image The maintenance method utilizes the old deflection angle diagram. Together with the deflection angle Δθ calculated in this iteration, the system is iteratively updated. Where γ decay This represents the attenuation factor of the old deflection angle diagram; For unbiased rendering transformation based on deflection angle, the bias is neutralized by multiplying the normal and the ray direction. The specific unbiased transformation formula is as follows: Where r(t) is the parameterized representation of the coordinate point x sampled by ray r, f g It is a scene geometry network, where cfd(r) represents the biased confidence level; f′ g (r(t)) represents the derivative with respect to t, and the result of the derivative is n·v, which is equal to the dot product of the scene normal and the ray. For the color loss guided by the deflection angle, the color loss based on the deflection angle is reweighted using the deflection angle calculated during training: in, For color loss based on deflection angle, C(r) is the true image pixel color, w color It is a weighted function.

9. The method according to claim 8, characterized in that, The total loss function for constructing the indoor surface volume rendering and reconstruction model is as follows: Constructing a color loss based on deflection angle Eikonal Standard Loss Adaptive prior loss based on deflection angle and Total loss function For the multi-resolution hash grid γ L Scene geometric network f g Color network f c and deflection field network f d Training is performed to obtain a high-fidelity three-dimensional surface; The Eikonal normalized loss The loss is the SDF value in the gauge space: The color loss based on deflection angle Re-weighting the rendering colors using the deflection angle The loss of true color C(r); The adaptive prior loss based on deflection angle and The prior loss of samples with different characteristics is adaptively calculated using the deflection angle; Total loss function Represented as: Wherein, λ1, λ2, and λ3 are weighting coefficients used to balance the contributions of each loss.

Images