Neural radiance field reconstruction method for weak texture scenes based on adaptive curvature regularization

Abstract

The application discloses a kind of weak texture scene-oriented neural radiance field reconstruction method based on sight tube geometric constraint. In view of the problem that existing NeRF technology is prone to geometric collapse and floating artifact in the area lacking texture features, an adaptive geometric constraint strategy is proposed. The method first uses image gradient entropy and luminosity variance to generate a texture confidence mask, accurately locating the weak texture area. Then, a second-order curvature consistency regularization model based on sight tube sampling is constructed, and the object surface is constrained to maintain smooth surface characteristics by minimizing the divergence of the local normal vector field. With the dynamic annealing training strategy, the application effectively improves the reconstruction geometric precision and new view synthesis quality of smooth surface objects without relying on the plane assumption.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and computer graphics, specifically relating to a 3D scene reconstruction method based on implicit neural representation. More specifically, this invention relates to a neural radiation field (NeRF) reconstruction technique based on line-of-sight geometric constraints for scenes with weak or no texture. This technique aims to recover the continuous 3D geometric structure and appearance information of a scene from multi-view 2D images using deep learning models, particularly for areas lacking photometric features such as white walls and monochromatic objects. This invention can be widely applied to virtual reality (VR) content generation, augmented reality (AR) interaction, digital twin city modeling, and high-precision 3D inspection of industrial parts. Background Technology

[0002] In recent years, with the rapid development of computer vision and deep learning technologies, 3D scene reconstruction technology has been widely applied in fields such as virtual reality, augmented reality, autonomous driving, and digital cultural heritage protection. Among them, Neural Radiation Field (NeRF), as an emerging implicit 3D representation method, has become a research hotspot in this field due to its superior synthetic quality from a novel perspective. The core idea of ​​NeRF is to use a multilayer perceptron to represent a scene as a continuous five-dimensional function, outputting the scene's volume density and RGB colors. By performing volume rendering integration along rays, NeRF can recover realistic 3D scenes from sparse 2D images. Its training process mainly relies on the photometric consistency assumption, that is, optimizing network parameters by minimizing the pixel color error between the rendered image and the real image. However, although NeRF performs excellently in textured scenes, it faces significant technical bottlenecks when processing weakly textured or textureless regions.

[0003] Existing technologies suffer from the following main shortcomings: Geometric collapse based on pure photometric constraints: In weakly textured regions, due to the lack of spatial variation in pixel color, the photometric consistency loss function cannot provide effective geometric constraint signals. This leads to extremely severe shape ambiguity in the network during optimization, often resulting in the placement of geometric surfaces near the camera plane or the generation of cloud-like floating artifacts in space, leading to uneven surfaces and severely distorted geometric structures in the reconstructed 3D model.

[0004] Limitations of Existing Regularization Methods: To address the aforementioned issues, existing techniques typically introduce additional regularization constraints. One type of method introduces a depth smoothing prior, forcing adjacent pixels to maintain consistent depth values. However, this simple smoothing often leads to blurred object edges, loss of high-frequency geometric details, and difficulty in distinguishing object boundaries from smooth surfaces. Another type of method introduces planar geometric constraints, assuming the scene is primarily composed of planes. While these methods perform well in regular indoor scenes, they struggle with non-planar, weakly textured objects with complex curvature. Forcibly imposing planar constraints can flatten curved objects, causing a secondary loss of geometric accuracy. In summary, how to effectively suppress geometric collapse and noise while accurately restoring the original smooth surface structure of objects without over-smoothing object edges, in the absence of texture information, is a critical challenge that urgently needs to be addressed in current neural radiation field reconstruction techniques. Summary of the Invention

[0005] This invention proposes a neural radiation field reconstruction method based on line-of-sight (NeRF) geometric constraints for weakly textured scenes. Addressing the challenge of existing NeRF techniques in accurately recovering geometric structures in regions lacking photometric features, this invention innovatively introduces an adaptive texture confidence mask based on gradient entropy and a second-order curvature consistency regularization model based on line-of-sight sampling by constructing a multi-resolution hybrid coding network. This method effectively eliminates geometric collapse and floating artifacts by utilizing the local smoothness constraints of the normal vector field within the line-of-sight, without relying on planar assumptions, thus achieving high-precision 3D reconstruction of weakly textured scenes containing complex surfaces.

[0006] To achieve the above objectives, the specific technical solution adopted by this invention is as follows: First, data acquisition and network initialization. A multi-view RGB image sequence of the target scene is acquired, and the camera pose is calculated using the Structure for Motion Restoration (SfM) algorithm to generate a sparse point cloud. A neural radiation field network based on Multi-resolution Hash Encoding is constructed to map spatial coordinates to volume density and spatial coordinates and viewpoint direction to RGB colors, thereby achieving efficient scene representation.

[0007] The second step involves constructing an adaptive recognition mechanism for weak texture regions. For the input image, a texture analysis algorithm based on multi-feature fusion is designed. This algorithm quantifies pixel-level texture richness by calculating the photometric variance (representing color changes) and gradient histogram entropy (representing structural disorder) within a local sliding window. Based on this, a continuous-value texture confidence mask is generated. This mask can accurately distinguish between high-frequency texture regions, flat weak texture regions, and non-flat weak texture regions, providing a basis for subsequent differential constraints.

[0008] The third step is to construct a second-order geometric constraint model based on the ray bundle. This is the core of the invention. During ray casting rendering, a ray bundle sampling strategy is adopted for regions identified as having weak textures by the mask. That is, multiple accompanying rays are randomly sampled within a conical neighborhood centered on the principal ray. The normal vector field at the intersection of these rays on the object surface is calculated using the volume density gradient output by the network. Subsequently, a curvature consistency loss function is constructed. By minimizing the divergence of the normal vector field within the ray bundle (i.e., the difference between the normal vector of the principal ray and the normal vector of the accompanying rays), the object surface is forced to maintain second-order geometric continuity locally (i.e., a smooth surface), thereby suppressing geometric noise.

[0009] The fourth step involves multi-task joint optimization and dynamic training. A general objective function is constructed, comprising photometric reconstruction loss, distortion regularization loss, and the aforementioned curvature consistency loss. During training, a dynamic annealing strategy is employed: initially, the curvature loss is given a large weight to strongly correct shape collapse in the early stages of geometry formation; as training progresses, this weight is gradually reduced, allowing the network to recover subtle surface wrinkle details while maintaining geometric stability. Finally, the converged network is used for high-quality synthesis of novel perspectives.

[0010] The beneficial effects of this invention are as follows:

[0011] 1. This invention overcomes the limitations of the "planar assumption," significantly improving the reconstruction accuracy of non-planar weakly textured objects. Existing techniques (such as methods based on the Manhattan world assumption) typically force weakly textured regions to fit as a plane. This can lead to incorrectly "flattening" of the geometry or producing step-like artifacts when processing objects with curvature (such as smooth ceramics, streamlined car bodies, cylinders, etc.). The second-order curvature consistency constraint based on the line-of-sight tube proposed in this invention does not force parallel normal vectors, but rather constrains the smooth transition of normal vectors within their local neighborhood. This allows the invention to be compatible with both planes and curved surfaces, accurately restoring the original smooth geometry of objects and greatly expanding the applicability of weakly textured reconstruction algorithms.

[0012] 2. Adaptive processing of textured regions is achieved, effectively avoiding excessive smoothing of high-frequency details. Existing techniques often apply uniform regularization constraints to the entire image, which can easily lead to the loss of details in texture-rich areas (such as text and patterns). This invention constructs a texture confidence mask based on the fusion of photometric variance and gradient entropy, which can accurately distinguish between high-frequency textured regions and weak textured regions. The algorithm applies strong geometric constraints only to weak textured regions with low confidence, while preserving the dominance of RGB photometric loss in texture-rich regions, thereby restoring smooth surfaces while preserving the high-frequency texture details of the scene to the greatest extent.

[0013] 3. Effectively suppresses geometric collapse and floating object artifacts, improving the visual quality of novel perspective synthesis. In regions lacking texture features, traditional NeRF is prone to geometric surface depressions or generating cloud-like noise in space. This invention utilizes a line-of-sight sampling mechanism, examining the geometric consistency of light beams in three-dimensional space to provide the network with strong geometric supervision signals in addition to color. Experiments show that this mechanism can effectively eliminate erroneous geometric structures, resulting in cleaner and more continuous model surfaces, significantly improving the peak signal-to-noise ratio and structural similarity of novel perspective synthesized images.

[0014] The overall solution of this invention significantly improves the reconstruction quality and detail representation capability of NeRF under weak texture input conditions, providing technical support for applications such as virtual reality, augmented reality, and autonomous driving, and has broad application prospects. Attached Figure Description

[0015] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0016] Figure 1 Network structure diagram; Detailed Implementation

[0017] The following detailed description provides further details on specific implementation methods.

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail. This invention proposes a neural radiation field reconstruction method based on line-of-sight geometric constraints for weakly textured scenes. Its core idea is to address the geometric collapse problem caused by the lack of photometric constraints in low-texture regions of traditional neural radiation fields (NeRF). By introducing an adaptive texture mask based on image statistical features and a curvature consistency regularization term based on line-of-sight micro-differential sampling, high-precision reconstruction of complex curved weakly textured scenes can be achieved without relying on planar assumptions.

[0019] The HFA-NeRF 3D reconstruction method based on NeRF includes the following steps:

[0020] S1. In the initial stage of this embodiment, it is first necessary to build a high-quality data foundation suitable for neural radiation field training. Specifically, for the target static scene to be reconstructed, a camera is used to take multiple angle shots around the object or scene center to obtain a set of two-dimensional RGB image sequences containing rich viewpoint information. These images are recommended to maintain a high resolution to ensure that small details in the scene can be captured. After acquiring the image data, it is not directly input into the network, but first the structure-of-motion (SOMO) algorithm is used to perform complex geometric calculations on the image sequence. This process first extracts robust feature points such as SIFT in each image, establishes the correspondence between images through feature matching, and then uses bundle adjustment joint optimization to accurately estimate the camera intrinsic parameter matrix and the camera extrinsic parameter pose matrix in the world coordinate system corresponding to each frame image.

[0021] After pose estimation, to adapt to the input range of the subsequent neural network, the effective bounding box of the scene needs to be determined based on the sparse point cloud generated by SfM, and a spatial normalization operation is performed to linearly map all 3D coordinates of the scene to the interior of a unit cube or unit sphere. Subsequently, observation rays for volume rendering are generated based on the pinhole camera model. For each pixel on the image plane, based on its 2D pixel coordinates and the camera intrinsic matrix, the direction vector in the camera coordinate system is first back-projected, and then transformed to the world coordinate system by combining the camera extrinsic rotation matrix. Finally, a ray is constructed that originates from the camera optical center, passes through the center of the pixel, and shines into the depth of the scene. This ray is uniquely determined by the origin coordinate vector and the normalized direction vector, providing a geometric reference for subsequent ray sampling and integral rendering.

[0022] S2. This method overcomes the shortcomings of traditional frequency-position coding, which involves high computational cost and difficulty in capturing high-frequency details, by constructing a highly efficient multi-resolution hash coding network architecture. This architecture first divides the 3D scene space into several layers of different resolutions, typically 16 layers, with the resolution increasing geometrically from coarse to fine. For any continuous coordinate point in space, the network does not process it directly, but instead maps it to one of these 16 mesh layers. Within each layer, a spatial hash function is used to find the corresponding corner feature vector, and trilinear interpolation is used to synthesize the local features of that coordinate point at that layer. The hash mapping formula is specifically defined as:

[0023]

[0024] in This represents the XOR bitwise operation. Let be a large prime number, and T be the size of the hash table. Through table lookup and trilinear interpolation, the network obtains multi-scale geometric feature vectors f(x). Subsequently, two multilayer perceptrons (MLPs) are used to predict volume density and color, respectively. The density network F...σ Output volume density σ And geometric features g, color network F c Combining the spherical harmonic encoding SH(d) of the viewing direction d with the output RGB color c, the mapping relationship can be expressed as:

[0025]

[0026] S3. To address the geometric collapse problem caused by the lack of features in weakly textured regions, these regions must first be accurately identified. This embodiment proposes an adaptive mask generation mechanism based on image statistical features. Instead of simply performing binary segmentation based on color thresholds, this mechanism generates a continuous confidence weight map. Specifically, the algorithm defines a sliding window of a preset size to traverse every pixel position in the input image. Within each window, the standard deviation of the color intensity of all pixels is first calculated. This index, as a luminance dispersion feature, can intuitively reflect the flatness of the color within a local area. Second, the Sobel operator is used to calculate the gradient magnitude of the pixels within the window, and the information entropy of the gradient direction histogram is statistically analyzed. This entropy value, as a structural disorder feature, is used to distinguish between disordered noise and ordered texture.

[0027] After obtaining the two features mentioned above, the algorithm normalizes and weights them to obtain a comprehensive texture score. To convert this score into weights usable for network training, this invention uses an inverse sigmoid nonlinear activation function to map the score to a closed interval between 0 and 1, thereby generating the final texture confidence mask. In this mask, regions with values ​​close to 1 correspond to weak texture areas such as white walls and single-color blocks, meaning the network will heavily rely on geometric constraints in these areas; while regions with values ​​close to 0 correspond to texture-rich areas, meaning the network should primarily rely on photometric loss for optimization. This soft mask mechanism effectively avoids boundary artifacts caused by hard thresholding, achieving a smooth transition from strong to weak texture regions.

[0028] S4. Calculation of Second-Order Geometric Curvature Constraints Based on Line-of-Sight Sampling. For the weakly textured regions identified in step S3, this invention innovatively introduces a second-order geometric constraint model based on a line-of-sight sampling system to replace the traditional planar assumption. During the forward propagation of network training, for a selected principal ray, the system constructs a conical spatial neighborhood centered on that ray. Within this neighborhood, several accompanying rays are randomly sampled according to a Gaussian distribution probability. These accompanying rays, together with the principal ray, constitute a tiny line-of-sight sampling system. For each ray within the line-of-sight sampling system, the gradient vector of the network output volume density with respect to spatial coordinates is calculated using automatic differentiation techniques and normalized to accurately obtain the normal vector at the intersection point of the ray on the object surface.

[0029] Subsequently, this embodiment constructs a curvature consistency loss function, the core logic of which lies in measuring the divergence of the normal vector field inside the line-of-sight tube. In specific calculations, the algorithm does not force all normal vectors within the line-of-sight tube to be perfectly parallel. Instead, it calculates the cosine similarity difference between the normal vector of the principal ray and the normal vectors of the surrounding accompanying rays, and performs a weighted sum based on the spatial distance between the accompanying rays and the principal ray. This loss function aims to minimize drastic abrupt changes in local normal vectors, forcing the object surface to maintain second-order geometric continuity within a local range. In other words, this constraint allows for smooth gradients in the normal vectors, thus perfectly fitting smooth curved surface structures such as spheres and cylinders, while strongly suppressing random geometric noise and surface unevenness caused by insufficient data.

[0030] For the weakly textured regions indicated by the mask M(p), this embodiment employs a line-of-sight sampling mechanism to constrain the second-order curvature of the surface. First, an automatic differentiation mechanism is used to calculate the gradient of the volume density σ with respect to spatial coordinate x, thereby obtaining the normal vector n(x) of the object's surface:

[0031]

[0032] Next, for the principal ray r c Sample K accompanying rays within its conical neighborhood. This invention constructs a curvature consistency loss function L. curve The aim is to minimize the principal ray normal vector n c With the accompanying ray normal vector n k The divergence between them. The specific mathematical expression of this loss function is:

[0033]

[0034] in, For Gaussian weights based on spatial distance, (1 - n) c· n k The cosine similarity is used to measure the directional difference of the normal vectors. This constraint forces the local normal vector field to maintain a smooth transition, thereby achieving accurate fitting of smooth surfaces.

[0035] S5. In the final optimization stage of the network, this embodiment constructs a joint objective function with multiple constraints and employs a dynamic training strategy to ensure convergence stability. The total loss function consists of three parts: first, the photometric reconstruction loss, typically using the Charbonnier loss function to measure the color error between the rendered image and the real image, which is the basis for the network to learn the appearance of the scene; second, the distortion regularization loss, used to suppress the false density accumulation caused by light in space and eliminate floating artifacts; and finally, the aforementioned curvature consistency loss. These three are added together with weighted coefficients to guide the gradient descent direction of the network.

[0036] To balance the restoration of geometric structure with the detailed representation of high-frequency textures, this invention designs a dynamic annealing strategy for weights. In the initial training phase, weights with a large curvature consistency loss are assigned because, in the early stages of training, the geometric contours of the scene are not yet fully formed and are prone to getting trapped in local optima. Strong geometric constraints guide the network to quickly establish the correct macroscopic shape. As the number of training iterations increases, these weights are gradually reduced according to the cosine annealing curve. In the later stages of training, the geometric structure has stabilized, and reducing the constraint strength allows the network to fine-tune subtle surface wrinkles, thereby maximizing the restoration of high-frequency texture details while ensuring geometric accuracy. The entire training process is performed end-to-end iteratively on a high-performance GPU until the loss function converges.

[0037] During network training, this embodiment employs volume rendering technology to project 3D information onto a 2D image plane. N points are sampled along the ray r, and their corresponding pixel colors are... The result is obtained through the discretization integral formula:

[0038]

[0039] To achieve end-to-end optimization, a total objective function was constructed that includes photometric loss, distortion loss, and curvature loss.

[0040]

[0041] In order to balance geometric convergence and detail restoration, the weight of curvature loss is... With training iterations Dynamic changes, following the cosine annealing strategy:

[0042]

[0043] Through this dynamic weighting mechanism, the network uses strong geometric constraints to suppress collapse in the early stages of training, and reduces constraints in the later stages to finely fit high-frequency textures, ultimately achieving high-quality 3D reconstruction.

Claims

1. A method for reconstructing the neural radiation field of a weakly textured scene based on adaptive curvature regularization, characterized in that, Includes the following steps: S1. Multi-view data acquisition and preprocessing: Acquire multi-view RGB image sequences of the target static scene, remove blurry or abnormally exposed image frames; use the structure-of-motion reconstruction algorithm to sparsely reconstruct the filtered image sequence, solve the camera intrinsic parameter matrix and extrinsic parameter pose matrix of each frame image, and generate sparse point cloud of the scene to determine the effective bounding box of the scene. S2. Construct a multi-resolution hybrid coding neural radiation field network; establish a deep learning model based on implicit neural representation, which includes a position coding module and a multilayer perceptron (MLP) decoding module to map the input three-dimensional spatial coordinates and observation viewpoint into corresponding volume density values ​​and RGB color values. S3. Generate an adaptive confidence mask for weak texture regions. Perform local texture feature analysis on the input image based on a sliding window. By jointly calculating the photometric statistical features and gradient statistical features of the local region, quantify the texture richness of each pixel and generate a pixel-level adaptive confidence mask to distinguish between high-frequency texture regions and weak texture smooth regions in the scene. S4. Construct a second-order geometric constraint model based on the line-of-sight tube; during the ray casting rendering process, for the weak texture area indicated by the confidence mask, the line-of-sight tube sampling strategy is used to obtain the geometric properties of the central ray and its accompanying rays in the spatial neighborhood, and a curvature regularization term is constructed based on the consistency of the local normal vector field to constrain the geometric smoothness of the object surface and suppress noise. S5. Construct a multi-task joint optimization objective function; construct a joint loss function including photometric reconstruction loss, distortion regularization loss and the curvature regularization term, and introduce a dynamic weight adjustment mechanism; S6. Network training and new perspective synthesis: The neural radiation field network is trained end-to-end using the joint loss function; After the network converges, given the camera pose of any new perspective, a high-quality two-dimensional image is synthesized or a three-dimensional mesh model is extracted using the volume rendering integral formula.

2. The method according to claim 1, characterized in that, The specific construction method of the multi-resolution hybrid coding neural radiation field network in step S2 includes: configuring a multi-resolution hash encoder as a position coding module, dividing the three-dimensional space into multiple grid levels of different resolutions, using a hash function to retrieve the feature vector of each level and performing linear interpolation, concatenating the interpolated multi-level feature vectors, and inputting them into the density decoder to predict the volume density. The output features of the density decoder are concatenated with the viewpoint direction vector encoded by spherical harmonics and input into the color decoder to predict the viewpoint-related RGB color. Both the density decoder and the color decoder are fully connected neural networks, and the density decoder has fewer layers than the color decoder to achieve fast inference.

3. The method according to claim 1, characterized in that, The specific process of generating the adaptive confidence mask in step S3 includes: defining a sliding window of a preset size, traversing each pixel position of the input image, calculating the standard deviation of the gray values ​​of all pixels within the window as the first texture feature, used to characterize the drastic degree of light intensity change; calculating the histogram information entropy of the gradient directions of all pixels within the window as the second texture feature, used to characterize the disorder of the texture structure; normalizing the first and second texture features respectively, and performing weighted fusion to obtain a comprehensive texture score; finally, using a nonlinear activation function to map the comprehensive texture score to a continuous interval from 0 to 1 to obtain the confidence mask value; the closer the mask value is to 1, the weaker the texture of the region, and the stronger the geometric constraints need to be applied.

4. The method according to claim 1, characterized in that, The specific implementation of the line-of-sight sampling strategy in step S4 is as follows: For a pixel to be rendered on the image plane, a main ray is generated that starts from the optical center of the camera and passes through the center of the pixel. A conical spatial neighborhood is set with the main ray as the axis. Finally, within the conical neighborhood, several accompanying rays are randomly sampled according to a Gaussian distribution probability. The accompanying rays and the main ray together constitute a line-of-sight tube, which is used to detect small geometric changes in the local area of ​​the object surface.

5. The method according to claim 4, characterized in that, The specific calculation method for the curvature regularization term is as follows: the gradient of the volume density of the neural radiation field output with respect to spatial coordinates is calculated using an automatic differentiation mechanism, and the normal vector of the intersection point of the principal ray and the accompanying ray on the scene surface is approximated by a normalization operation. The cosine similarity distance between the normal vector corresponding to the principal ray and the normal vectors corresponding to each accompanying ray is calculated. The cosine similarity distances of all accompanying rays within the line of sight are weighted and summed. The weighting factors include the spatial distance between the accompanying ray and the principal ray and the value of the confidence mask. Finally, by minimizing the weighted summation result, the local normal vector field of the weak texture region is forced to change gradually rather than being chaotic.

6. The method according to claim 1, characterized in that, The dynamic weight adjustment mechanism in step S5 is specifically configured as follows: the weight coefficient of the curvature regularization term is set to change with the number of training iterations. In the early stage of training, a large weight coefficient is assigned to the curvature regularization term to strongly suppress geometric collapse caused by weak texture in the early stage of geometry formation. In the middle and late stages of training, the weight coefficient is gradually reduced according to the cosine annealing strategy or the linear decay strategy until it is reduced to a non-zero minimum value, so as to allow the network to fit fine surface wrinkle details on the basis of geometric stability.

Images