Neural radiation field rendering method based on linear constraint

Through the neural radiation field rendering method based on linear constraints, linear features are extracted and the rendering process is accelerated, which solves the problems of high complexity of traditional three-dimensional reconstruction methods and inefficient use of computing resources, and achieves efficient and accurate three-dimensional reconstruction effect.

CN118967913APending Publication Date: 2024-11-15NANJING TECH UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410986123.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Traditional geometry-based three-dimensional reconstruction methods have high complexity, limited scalability and versatility. The NeRF framework is difficult to effectively capture geometric information when processing complex scenarios, and the computing resource usage is inefficient.

Method used

Using a neural radiation field rendering method based on linear constraints, the camera position is estimated through COLMAP software, and linear features are extracted using Hough transform and K-Means clustering, combining multi-resolution hash encoding and small MLP network to accelerate the rendering process and introduce Linear depth constraint optimizes the loss function, and training results in the implicit expression of the three-dimensional model.

Benefits of technology

It improves rendering speed and accuracy, improves the accuracy and efficiency of 3D reconstruction, and is suitable for high-precision 3D reconstruction and rendering tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967913A_ABST
    Figure CN118967913A_ABST
Patent Text Reader

Abstract

The invention provides a neural radiation field rendering method based on linear constraint, and belongs to the technical field of computer vision. In the image preprocessing stage, Hough transform is utilized to extract line information in a scene, and a straight line is mapped to a 3D space. And thirdly, clustering the point clouds through a K-Means clustering method, and fitting the three-dimensional space straight line through an RANSAC method, thereby providing additional geometric information constraints for NeRF. Besides, a multi-resolution Hash coding method is adopted, feature vectors under different resolutions are connected, a final feature vector sequence is formed, and the NeRF training process is accelerated. The invention aims to realize accurate capture of the 3D geometric structure of the scene and generate a more accurate and detailed rendering result. Meanwhile, in the training process, the higher rendering speed is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and remote sensing, and in particular relates to a neural radiation field rendering method based on straight line constraints. Background Art

[0002] The NeRF (Neural Radiance Field) framework has become a mature technology that can learn and represent 3D scenes and generate realistic images from never-before-seen perspectives. Therefore, NeRF technology can be applied to 3D generation of remote sensing images to obtain high-quality 3D reconstruction results.

[0003] Converting two-dimensional images into high-fidelity three-dimensional scenes is an important trend in the development of future visual imaging technology, especially in the fields of photogrammetry and remote sensing. High-precision three-dimensional reconstruction can not only accurately reproduce buildings, roads and infrastructure, but also more intuitively display geographic information through realistic three-dimensional terrain models. This technology has a wide range of applications in smart cities, urban planning and management, digital mapping and navigation, etc. Traditional geometry-based three-dimensional reconstruction methods have two main disadvantages: each step requires fine-tuning of parameters and processing, which increases the complexity of implementation and debugging, thereby limiting the scalability and versatility of the method. Traditional methods usually rely on manually designed features to describe the geometric structure and texture information in the scene.

[0004] The original NeRF framework mainly focuses on capturing color and depth information, which may face challenges when processing complex scenes. Improvements and optimizations to the NeRF framework may include: integrating more geometric information to enhance structure capture capabilities, optimizing the use of computing resources to improve efficiency, and extending the NeRF framework to handle larger and more complex geometric structures. These improvements will further promote the application of 3D reconstruction technology in various application fields and provide more powerful tools and solutions for future visual imaging and geographic information systems. Summary of the invention

[0005] The present invention provides a neural radiation field rendering method based on straight line constraints, thereby realizing straight line depth constraints and accelerating the original NeRF framework, thereby improving the speed and accuracy of rendering.

[0006] The present invention provides a neural radiation field rendering method based on straight line constraints, comprising the following steps:

[0007] Step 1: Image preprocessing: using COLMAP software to estimate the camera pose from n input images;

[0008] Step 2: In the input two-dimensional image, a multimodal line feature extractor is used to extract straight line features for data of different scales;

[0009] Step 3: According to the two-dimensional straight line features, when the network performs weighting and depth estimation, three-dimensional space straight line fitting is performed and converted into three-dimensional depth line features;

[0010] Step 4: Use a multi-resolution hash coding scheme to speed up the rendering process, using a small MLP network structure and a multi-resolution hash coding strategy;

[0011] Step 5: Add straight line depth constraints, optimize the loss function, and train to obtain the implicit expression of the three-dimensional model.

[0012] Step 6: Synthesize a new view based on the implicit 3D representation of the scene covered by the input image and the new camera pose parameters.

[0013] Furthermore, in the neural radiation field rendering method based on straight line constraints described in the present invention, in step 1, the COLMAP tool is used to extract feature points and feature descriptors from the data set, and the geometric consistency of these feature points and their descriptors with the original image is verified by the SIFT algorithm. The incremental structure from motion (SFM) method is used for sparse reconstruction, and the three-dimensional coordinates and camera pose information of the image are obtained by matching the mapping relationship between the view coordinates and the actual three-dimensional building.

[0014] In the neural radiation field rendering method based on straight line constraints of the present invention, in step 2, the Hough transform is used to extract structural information from the image and capture line features in the training image. By performing voting accumulation in the parameter space, the Hough transform can efficiently detect line segments and extract their position information as peaks in the image. The mathematical expression of the Hough transform is as follows:

[0015] r=x·cos(θ)+y·sin(θ)

[0016] In the neural radiation field rendering method based on straight line constraints described in the present invention, in step 3, a K-Means clustering algorithm is used, and the clustering process is determined by the following formula:

[0017]

[0018] where c i is the i-th cluster center, parameter k predefines the number of clusters, and index i identifies each cluster, ranging from 1 to k. Data point p represents the predicted 3D spatial coordinates corresponding to pixels located on the same 2D line. This algorithm is used to group the 3D spatial predicted coordinates corresponding to pixels located on the same 2D image line.

[0019] In the neural radiation field rendering method based on straight line constraints described in the present invention, in step 3, the RANSAC algorithm is used to perform line fitting on the clustered spatial coordinates, and the straight line form is as follows:

[0020]

[0021] In the straight line constraint-based neural radiation field rendering method of the present invention, in step 4, the three-dimensional scene is normalized and divided into a multi-resolution voxel grid in the range of 0-1. This step further includes mapping the feature encoding of the voxel vertex into a hash table, and finding the corresponding index values ​​of the surrounding voxel vertices at different resolution levels. According to the index value, the 8 vertex encoding features stored in the hash table are found. The resolution level is calculated by the following formula:

[0022]

[0023] b is calculated by the following maximum and minimum resolutions:

[0024]

[0025] According to the relative positions in each voxel, trilinear interpolation is used to calculate the feature vectors at different resolutions. These feature vectors are concatenated to form the final feature vector sequence, which is then input into a small neural network for inference calculation.

[0026] In the neural radiation field rendering method based on straight line constraints of the present invention, in step 5, the distance between the point and its corresponding line is minimized to ensure that all detected points are located on the same 3D space line. The final loss function is defined as:

[0027] L=L Color +λ D L Lc

[0028] Where L Color Represents color loss, L Lc Represents the straight line consistency loss. Hyperparameter λ D Used to balance the importance of color and line consistency supervision during training, L Lc It is given by the following formula:

[0029]

[0030] Where L lc is the straight line loss, N is the number of straight lines, Mi is the number of points fitting each straight line, and p is the point near the straight line.

[0031] In the straight line constraint-based neural radiation field rendering method of the present invention, in step 6, NeRF uses the principle of classical volume rendering to render the color of each light ray passing through the scene, and synthesizes the output color c and volume density σ into the image. Given a camera ray r(t)=o+td, the color C(r) of the corresponding pixel can be calculated by the following formula.

[0032]

[0033] Where o is the camera center, t n and t f are the near boundary and the far boundary respectively, and T(t) is the distance along the ray from t n The cumulative transmittance up to t can be calculated by the following formula.

[0034]

[0035] In the present invention, COLMAP software is first used to perform image preprocessing on n input images to estimate the camera pose. Subsequently, in the input two-dimensional image, a multimodal line feature extractor is used to extract straight line features for data of different scales. Based on these two-dimensional straight line features, weighting and depth estimation are performed through the network to achieve three-dimensional space straight line fitting and generate three-dimensional depth line features. Furthermore, a multi-resolution hash coding scheme is adopted to accelerate the rendering process, and a small MLP network structure and a multi-resolution hash coding strategy are combined. By adding straight line depth constraints and optimizing the loss function, an implicit expression of the three-dimensional model is obtained through training. Finally, the synthesis of new views is achieved based on the implicit three-dimensional representation of the scene covered by the input image and the new camera pose parameters. The present invention can combine traditional image processing technology and modern neural radiation field rendering technology to improve accuracy and rendering speed, and is suitable for efficient three-dimensional reconstruction and rendering tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a flow chart of a neural radiation field rendering method based on straight line constraints provided by an embodiment of the present invention;

[0037] Figure 2 Comparison of DTU data set experimental results of the straight line constraint-based neural radiation field rendering method provided in the embodiment of the present invention;

[0038] Figure 3 A diagram showing the experimental results of a DTU dataset of a neural radiation field rendering method based on straight line constraints provided in an embodiment of the present invention;

[0039] Figure 4 Comparison of experimental results of the UseGeo dataset for the neural radiation field rendering method based on straight line constraints provided in an embodiment of the present invention;

[0040] Figure 5A diagram showing experimental results of a UseGeo dataset for a neural radiation field rendering method based on straight line constraints provided by an embodiment of the present invention;

[0041] Figure 6 Comparison of experimental results on the DFC2019 dataset of the straight line constraint-based neural radiation field rendering method provided in an embodiment of the present invention;

[0042] Figure 7 A graph of experimental results on the DFC2019 dataset of the straight line constraint-based neural radiation field rendering method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] This paper aims to explore a neural radiance field rendering method based on straight line constraints, extract line information using Hough transform and integrate it into NeRF's rendering mechanism, use K-means clustering and RANSAC algorithm to establish geometric constraints on 3D line directions. Finally, a multi-resolution hash coding scheme is used to accelerate the rendering process.

[0044] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0045] This embodiment provides a neural radiation field rendering method based on straight line constraints, which mainly includes the following steps:

[0046] Step 1: Image preprocessing: using COLMAP software to estimate the camera pose from n input images;

[0047] Step 2: In the input two-dimensional image, a multimodal line feature extractor is used to extract straight line features for data of different scales;

[0048] Step 3: According to the two-dimensional straight line features, when the network performs weighting and depth estimation, three-dimensional space straight line fitting is performed and converted into three-dimensional depth line features;

[0049] Step 4: Use a multi-resolution hash coding scheme to speed up the rendering process, using a small MLP network structure and a multi-resolution hash coding strategy;

[0050] Step 5: Add straight line depth constraints, optimize the loss function, and train to obtain the implicit expression of the three-dimensional model.

[0051] Step 6: Synthesize a new view based on the implicit 3D representation of the scene covered by the input image and the new camera pose parameters.

[0052] See also Figure 1, a neural radiation field rendering method based on straight line constraints provided in this embodiment mainly includes the following steps: First, the COLMAP software is used to perform image preprocessing on the input n images to estimate the camera pose. Subsequently, a multimodal line feature extractor is used to extract straight line features from two-dimensional image data of different scales. Based on these two-dimensional straight line features, weighted and depth estimation are performed through the network to achieve fitting of three-dimensional space straight lines and generate three-dimensional depth line features. Furthermore, a multi-resolution hash coding scheme is adopted to accelerate the rendering process, combining a small MLP network structure and a multi-resolution hash coding strategy. By introducing a straight line depth constraint and optimizing the loss function, an implicit expression of the three-dimensional model is obtained through training. Finally, based on the implicit three-dimensional representation of the scene covered by the input image and the new camera pose parameters, the synthesis of the new view is achieved. The following is an explanation of several main parts of this embodiment.

[0053] 1.1 3D space line feature extraction

[0054] Hough transform is a powerful tool for extracting structural information from images. It is used to capture line features in training images. By using voting accumulation in parameter space, Hough transform can efficiently detect line segments and extract their position information as peaks in the image. The Hough transform formula is as follows:

[0055] r=x·cos(θ)+y·sin(θ)

[0056] The K-Means clustering algorithm is used to group the predicted 3D spatial coordinates corresponding to the pixels that should be located on the same 2D image line. The RANSAC algorithm is used to perform line fitting on the clustered spatial coordinates. The straight line form is as follows:

[0057]

[0058] 1.2 Multi-resolution hash coding strategy

[0059] Normalize the 3D scene and divide it into a multi-resolution voxel grid in the range of 0-1. Map the voxel vertex encoding features into a hash table. Find the corresponding index values ​​of the surrounding voxel vertices at different resolutions and find the corresponding 8 vertex encoding features stored in the hash table according to the index values. The resolution level is calculated by the following formula:

[0060]

[0061] b is calculated by the following maximum and minimum resolutions:

[0062]

[0063] According to the relative positions in each voxel, trilinear interpolation is used to calculate the feature vectors at different resolutions. The feature vectors at different resolutions are connected to form the final feature vector sequence, and the feature vectors are input into a small neural network for inference calculation.

[0064] 1.3 Straight Line Constraint Loss

[0065] Minimize the distance between a point and its corresponding line to ensure that all detected points are located on the same 3D space line. The final loss function is defined as:

[0066] L=L Color +λ D L Lc

[0067] Where L Color Represents color loss, L Lc Represents the straight line consistency loss. Hyperparameter λ D Used to balance the importance of color and line consistency supervision during training, L Lc It is given by the following formula:

[0068]

[0069] Where L lc is the straight line loss, N is the number of straight lines, M i is the number of points for fitting each straight line, and p is the number of points near the straight line.

[0070] In order to verify the effect of the present invention, the following is a description in conjunction with experimental results.

[0071] According to the requirements, the present invention designed comparative experiments on three data sets. The results of the comparative experiments all show that the effect of the neural radiation field rendering method based on straight line constraints is better than other methods. The comparative experimental results of the DTU data set are as follows: Figure 2 and Figure 3 As shown in the figure, the results show the absolute error of depth estimation (Abs err) and the depth reconstruction accuracy index under the error threshold of 0.02 (Acc(0.02)) and 0.05 (Acc(0.05)). To ensure a fair comparison, all methods were trained for 5000 iterations on the same training image set. The comparative experimental results of the UseGeo dataset are shown in the figure. Figure 4 and Figure 5 As shown in the figure, the comparative experimental results of the DFC2019 dataset are as follows Figure 6 and Figure 7As shown, these metrics are evaluated without and with linear constraints. This comparative analysis highlights the performance differences between the various methods and emphasizes the impact of introducing linear constraints on the accuracy of depth estimation. Upward arrows indicate higher values ​​for higher accuracy, while downward arrows indicate lower values ​​for higher accuracy. The present invention consistently outperforms the original baseline method across a variety of metrics and scenarios.

[0072] Finally, it should be noted that the above specific implementation methods are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, a person of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the present invention, which should be covered by the scope of the patent claims of the present invention.

Claims

1. A neural radiation field rendering method based on straight line constraints, characterized in that: The following steps are included Step 1: Image preprocessing: using COLMAP software to estimate the camera pose from n input images; Step 2: In the input two-dimensional image, a multimodal line feature extractor is used to extract straight line features for data of different scales; Step 3: According to the two-dimensional straight line features, when the network performs weighting and depth estimation, three-dimensional space straight line fitting is performed and converted into three-dimensional depth line features; Step 4: Use a multi-resolution hash coding scheme to speed up the rendering process, using a small MLP network structure and a multi-resolution hash coding strategy; Step 5: Add straight line depth constraints, optimize the loss function, and train to obtain the implicit expression of the three-dimensional model. Step 6: Synthesize a new view based on the implicit 3D representation of the scene covered by the input image and the new camera pose parameters.

2. The straight line constraint-based neural radiation field rendering method according to claim 1, characterized in that: In step 2, the multimodal line feature extractor uses Canny edge detection, Gaussian filter, LSD local extremum detection and cumulative Hough transform to extract the line features of the two-dimensional image. The Hough transform formula is as follows: r=x·cos(θ)+y·sin(θ) 3. The neural radiation field rendering method based on straight line constraints according to claim 1 is characterized in that: In step 3, the K-Means algorithm is used to cluster the three-dimensional depth point cloud. The clustering process is determined by the following formula: where c i is the i-th cluster center, the parameter k predefines the number of clusters, and the index i identifies each cluster, ranging from 1 to k. The data point p represents the predicted three-dimensional spatial coordinates corresponding to the pixels lying on the same two-dimensional line. The RANSAC algorithm is used to generate three-dimensional straight lines for different categories of depth point clouds. The straight line form is as follows:

4. The straight line constraint-based neural radiation field rendering method according to claim 1, characterized in that: In step 4, the voxel vertices are hash-encoded and the voxel vertex encoding features are mapped into the hash table. The resolution level is calculated by the following formula: b is calculated by the following maximum and minimum resolutions: When performing hash queries, trilinear interpolation is used to calculate feature vectors at different resolutions based on their relative positions in each voxel.

5. The neural radiation field rendering method based on straight line constraints according to claim 1, characterized in that: The specific expression of the straight line constraint in step 5 is Where L lc is the straight line loss, N is the number of straight lines, M i is the number of points for fitting each straight line, and p is the number of points near the straight line.

Citation Information

Cited By

  • Neural radiation field DSM generation method based on two-dimensional triangulation network constraint

    CN120374898A