A monocular dense SLAM map construction method based on image enhancement and NeRF

Abstract

A monocular dense SLAM map construction method based on image enhancement and NeRF, the steps include: first, inputting a low-light RGB image into a low-light image enhancement algorithm based on the fusion Retinex theory and attention mechanism, which is used to enhance the brightness and contrast of the low-light image and retain the image detail information, then using the quadtree method to homogenize the feature points so that the feature points are evenly distributed in the entire image, and using a grid-based motion statistics algorithm to evaluate the reliability of feature matching, combining the random sample consistency RANSAC algorithm to further eliminate mismatched features, based on camera motion estimation and triangulation to obtain the camera pose, and through coordinate transformation to obtain sparse point cloud information, finally based on neural radiation field technology, combined with color and depth loss functions, using the obtained camera pose, input image and 3D point cloud information to complete the construction of a three-dimensional dense map. The present invention combines the low-light image enhancement algorithm based on the fusion Retinex theory and attention mechanism with the quadtree method and the grid-based motion statistics algorithm and neural radiation field technology, which can improve the accuracy of trajectory tracking and mapping, and the present invention has achieved remarkable results in achieving accurate dense map construction.

Description

Technical Field

[0001] The present invention is applicable to the field of simultaneous localization and mapping (SLAM). Specifically, it relates to a low-light image enhancement algorithm based on the fusion of Retinex theory and attention mechanism, a visual odometer based on the quadtree method to homogenize feature points and a grid-based motion statistics algorithm (Grid-based Motion Statistics, GMS), and a dense map construction algorithm based on neural radiation field technology. These technologies show superiority when processing low-light RGB images, and can improve the quality of feature points, enhance the matching ability of feature points, and improve the robustness and accuracy of the system. In addition, the present invention combines color and depth supervision to obtain more accurate dense map construction results. Through the application of these innovative technologies, the present invention has achieved remarkable results in achieving accurate trajectory tracking and dense map construction. Background Art

[0002] SLAM is an important interdisciplinary field of robotics and computer vision. It is widely used in fields such as autonomous driving, drone driving and robot navigation, and has attracted much research attention in recent years.

[0003] A low-light image enhancement algorithm based on the fusion of Retinex theory and attention mechanism can better improve the quality of low-light images. Multi-scale feature extraction is performed on low-light images through a feature extraction module based on the Unet architecture to generate a high-dimensional multi-scale feature map. An attention mechanism module is established to highlight feature information of different scales that is beneficial to the enhanced image, and a weighted high-dimensional feature map is obtained. Finally, the Retinex theory is used to establish a network model in the reflection estimation module to generate the final enhanced image through the high-dimensional feature map.

[0004] The visual odometry based on the quadtree method to homogenize feature points and the grid-based motion statistics algorithm (Grid-based Motion Statistics, GMS) uses the quadtree method to evenly distribute feature points throughout the image, and then uses a combination of the grid-based motion statistics algorithm and RANSAC (random sample consistency) to remove mismatched features.

[0005] The dense map construction based on neural radiation field technology first obtains the color and density of the sampling points in the space through existing photos and 3D point cloud information, and then uses the data of these particles to render photos from any perspective. Finally, color and density loss optimization is used to obtain a better map reconstruction effect. Summary of the invention

[0006] The map constructed by the traditional ORB-SLAM2 system is a sparse map, which cannot realize dense map construction. The present invention proposes a monocular dense SLAM map construction method based on image enhancement and NeRF, which combines a low-light image enhancement algorithm based on the fusion of Retinex theory and attention mechanism, a quadtree method, a grid-based motion statistics algorithm and a neural radiation field technology. Compared with other traditional image enhancement technologies, the low-light image enhancement algorithm based on the fusion of Retinex theory and attention mechanism can enhance the brightness, color and contrast while retaining rich image detail information. At the same time, the quadtree method and the grid-based motion statistics algorithm are added. Uniform feature points are obtained by using the quadtree method, and the GMS algorithm is used to estimate the number of matching pairs in the neighborhood. Subsequently, the random sample consistency RANSAC algorithm is used to further eliminate mismatched features, provide more accurate feature point matching, and then the neural radiation field technology is used to construct a dense three-dimensional map. Compared with other reconstruction methods such as TSDF, this method does not have holes and has better detail processing.

[0007] Specific content of the invention:

[0008] A monocular dense SLAM map construction method based on image enhancement and NeRF, characterized by comprising the following steps:

[0009] Step 1) A low-light image enhancement algorithm based on the fusion of Retinex theory and attention mechanism is used to enhance image brightness and contrast while retaining image detail information, providing higher quality images for subsequent feature point extraction and feature point matching tasks, including the following steps:

[0010] Step 1.1) Input the low-light image into a feature extraction module consisting of 18 convolutional layers, 4 downsampling and 4 upsampling steps. The convolutional layer uses a 3×3 convolution kernel, a stride and padding of 1, and a ReLU activation function; downsampling is achieved by 2×2 maximum pooling with a stride of 2, and upsampling is achieved by deconvolution with a stride of 2; the downsampled feature map is cascaded to the upsampled feature map of the same resolution through a skip connection mechanism, and finally a 32-channel high-dimensional multi-scale feature map is output;

[0011] Step 1.2) Input the output 32-dimensional feature map into the attention mechanism module, first undergo a global average pooling operation, and compress the extracted feature map x into a channel descriptor z, the formula of which is as follows:

[0012]

[0013] where x c is the feature map of the cth channel of the input, z cis the feature descriptor of the cth channel, H and W represent the length and width of the channel feature map respectively, and then the feature descriptor z c Using two fully connected operations F sq , the ReLU activation function is used between the two fully connected layers for nonlinear processing, and finally the Sigmoid activation function is used to output the weight of each dimension. The fully connected operation formula is as follows:

[0014] s c =F sq (z c ,w)=σ(w2δ(w1z c ))

[0015] In the above formula, w1 and w2 represent two fully connected layers, σ and δ represent Sigmoid function and ReLU function, and s c Represents the weight of the cth channel of the output, and finally the channel weight s c The feature map x output by the feature extraction module c The recalibration operation is performed to obtain a weighted feature map with 32 channels;

[0016] Step 1.3) Input the feature map into the reflection estimation module consisting of 6 symmetrically cascaded convolutional layers and 1 output layer. Each convolutional layer consists of 32 convolution operations with a kernel size of 3×3 and a stride of 1 and a ReLU activation function. The activation function of the last layer is Tanh. The output layer outputs the inverse of the illumination component with a channel number of 3. Finally, the final enhanced image is obtained by pixel-by-pixel multiplication of the input image and the inverse of the illumination component.

[0017] Step 2) extracting feature points from the enhanced image and homogenizing the feature points based on the quadtree method to achieve uniform distribution of the feature points in the entire image, including the following steps:

[0018] Step 2.1) Before extracting feature points, first allocate the number of feature points to be extracted in each layer of the image pyramid. The feature points are allocated according to the principle that the number of feature points extracted per unit area is the same. Assume that the initial image area is S0 and the image scaling factor is s. The total area of ​​the n-layer image is:

[0019]

[0020] Assume the number of feature points is N, and the number of feature points extracted per unit area is:

[0021]

[0022] The feature points assigned to the i-th layer are:

[0023]

[0024] Step 2.2) Divide the image into regions using a 30×30 pixel grid, and then perform FAST key point extraction in each grid;

[0025] Step 2.3) Select the extracted feature points in each grid using the quadtree method, and finally retain the feature point with the largest Harris response value in each area;

[0026] Step 3) The reliability of feature matching is evaluated by using the grid-based motion statistics algorithm GMS, and then the random sample consistency RANSAC algorithm is used to further eliminate mismatched features, thereby effectively improving the accuracy of feature matching, including the following steps:

[0027] Step 3.1) Perform brute force matching on the feature points based on the Hamming distance of the binary descriptor to obtain the initial feature point matching pair. The formula for calculating the Hamming distance is as follows:

[0028]

[0029] Where n is the length of the string, a i and b i The i-th characters of strings a and b respectively, 1(a i ≠b i ) is the indicator function, when a i ≠b i The value is 1 when it is, otherwise the value is 0;

[0030] Step 3.2) Use the GMS algorithm to preliminarily screen the initial feature point matching pairs to reduce the number of matches;

[0031] Step 3.3) using a random sample consistency algorithm to estimate the transformation model between the two images, identifying the inlier matching pairs and the outlier matching pairs, and only retaining the inlier matching pairs as the correct feature matching information;

[0032] Step 4) obtaining the camera pose through camera motion estimation and triangulation, and obtaining sparse point cloud information through coordinate transformation, including the following steps:

[0033] Step 4.1) For the matched feature point pairs, calculate their essential matrix E, and decompose the rotation matrix R and translation vector t from E, and obtain the 3D position of the spatial point in the current camera coordinate system through triangulation;

[0034] Step 4.2) All spatial points are unified into the world coordinate system through coordinate transformation, thereby constructing a sparse point cloud map;

[0035] Step 5) Based on the neural radiance field algorithm, combined with the color and depth loss functions, the obtained camera pose, input image and 3D point cloud information are used to complete the construction of the 3D dense map, including the following steps:

[0036] Step 5.1) Use the linear interpolation method to generate 64 uniformly distributed sampling points between the near end and the far end of the ray, and combine the ray projection and camera model technology to generate the sampling point pose;

[0037] Step 5.2) Hash code the obtained sampling point pose. The specific formula is as follows:

[0038]

[0039] The sampling point pose is assumed to be (x i ,y i ,z i ), d represents the dimension of the pose, here d = 3, π1, π2, π3 are 1, 2654435761, 805459861, It means bitwise XOR, mod T means modulo operation, here T = 2 14 -1;

[0040] Step 5.3) Send the encoded pose information to the fully connected network to obtain the color density information of the sampling point;

[0041] Step 5.4) Use volume rendering technology to obtain any ray r(t)=o+tD obs At the boundary [t n ,t f ], where o is the camera position and the viewing direction is D obs , its rendering equation is as follows:

[0042]

[0043] The cumulative transmittance Indicates that the light from t n The probability of propagating to t without being blocked is, where σ(r(t)) and c(r(t),D obs ) represents the volume density and color, dt represents the differential distance traveled by the light ray in each integration step;

[0044] Step 5.5) The color loss is represented by the square of the Euclidean distance between the predicted color C(r) of the light r and the actual observed color C(r); the depth loss is represented by the KL divergence of the depth value estimated by NeRF technology rendering and the depth value of the 3D sparse point cloud. The formulas are as follows:

[0045]

[0046] Where R(P) represents the set of rays generated by a specific camera parameter P, and E r∈R(P) It means to calculate the expectation of each ray r in the ray set, X represents the set of all 3D key points {X:x1,x2,x3}, represents the subset of 3D keypoints visible from camera position j, x i ∈X j Represents the 3D key point x i Visible keypoints belonging to camera j. represents all key points x visible from camera position j i The expected value of represents the sum of the sampling point index k along the direction of the light, h(k) represents the weight function of the sampling point, t k is the depth value of the kth sampling point rendered, D ij According to the depth value of the 3D point cloud, represents the average reprojection error between views where key points are detected, Δt k Indicates the interval of sampling points.

[0047] The present invention has the following advantages and beneficial effects:

[0048] The low-light image enhancement algorithm based on the fusion of Retinex theory and attention mechanism is combined to process low-light images. By entering the feature extraction module, attention mechanism module and reflection estimation module in sequence, it helps to improve the contrast and details of low-light images, thereby improving the accuracy of the system in processing low-light sequences;

[0049] The combination of the quadtree method and the grid-based motion statistics algorithm aims to improve the quality of feature points and the reliability of feature matching. The quadtree method ensures that the feature points are evenly distributed in the image, so that the SLAM system can extract useful information from the entire image when estimating the camera pose. By eliminating unreliable matching pairs, GMS improves the accuracy of feature matching, thereby improving the accuracy of trajectory estimation and the refinement of map construction;

[0050] The purpose of neural radiation field technology is to solve the problem of traditional monocular SLAM systems in reconstructing sparse maps. By using rendering technology, combining color and depth loss, and using the skip voxel strategy combined with 3D information, the map construction effect is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is the overall technical flow chart of the monocular dense SLAM map construction method based on image enhancement and NeRF of the present invention; DETAILED DESCRIPTION

[0052] The advantages and purposes of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the description herein is only for explaining the present invention and is not intended to limit the present invention.

[0053] The monocular dense SLAM map construction method based on image enhancement and NeRF proposed in the present invention first combines the low-light image enhancement algorithm, and processes the low-light image through the feature extraction module, the attention mechanism module and the reflection estimation module to obtain enhanced image information. Then, the quadtree method is used to ensure the uniform distribution of feature points in the image, and the unreliable matching pairs are eliminated through the grid-based motion statistics algorithm (GMS), thereby improving the accuracy of feature matching and the quality of feature points. Next, a high-precision dense point cloud map is constructed through a volume rendering method based on neural radiation field technology, combined with color and depth loss and the skip voxel strategy of 3D information. Appendix Figure 1 The overall flow chart of the SLAM system of the present invention specifically includes the following steps: A monocular dense SLAM map construction method based on image enhancement and NeRF, characterized in that it includes the following steps:

[0054] Step 1) A low-light image enhancement algorithm based on the fusion of Retinex theory and attention mechanism is used to enhance image brightness and contrast while retaining image detail information, providing higher quality images for subsequent feature point extraction and feature point matching tasks, including the following steps:

[0055] Step 1.1) Input the low-light image into a feature extraction module consisting of 18 convolutional layers, 4 downsampling and 4 upsampling steps. The convolutional layer uses a 3×3 convolution kernel, a stride and padding of 1, and a ReLU activation function; downsampling is achieved by 2×2 maximum pooling with a stride of 2, and upsampling is achieved by deconvolution with a stride of 2; the downsampled feature map is cascaded to the upsampled feature map of the same resolution through a skip connection mechanism, and finally a 32-channel high-dimensional multi-scale feature map is output;

[0056] Step 1.2) Input the output 32-dimensional feature map into the attention mechanism module, first undergo a global average pooling operation, and compress the extracted feature map x into a channel descriptor z, the formula of which is as follows:

[0057]

[0058] where x c is the feature map of the cth channel of the input, z c is the feature descriptor of the cth channel, H and W represent the length and width of the channel feature map respectively, and then the feature descriptor z c Using two fully connected operations F sq, the ReLU activation function is used between the two fully connected layers for nonlinear processing, and finally the Sigmoid activation function is used to output the weight of each dimension. The fully connected operation formula is as follows:

[0059] s c =F sq (z c ,w)=σ(w2δ(w1z c ))

[0060] In the above formula, w1 and w2 represent two fully connected layers, σ and δ represent Sigmoid function and ReLU function, and s c Represents the weight of the cth channel of the output, and finally the channel weight s c The feature map x output by the feature extraction module c The recalibration operation is performed to obtain a weighted feature map with 32 channels;

[0061] Step 1.3) Input the feature map into the reflection estimation module consisting of 6 symmetrically cascaded convolutional layers and 1 output layer. Each convolutional layer consists of 32 convolution operations with a kernel size of 3×3 and a stride of 1 and a ReLU activation function. The activation function of the last layer is Tanh. The output layer outputs the inverse of the illumination component with a channel number of 3. Finally, the final enhanced image is obtained by pixel-by-pixel multiplication of the input image and the inverse of the illumination component.

[0062] Step 2) extracting feature points from the enhanced image and homogenizing the feature points based on the quadtree method to achieve uniform distribution of the feature points in the entire image, including the following steps:

[0063] Step 2.1) Before extracting feature points, first allocate the number of feature points to be extracted in each layer of the image pyramid. The feature points are allocated according to the principle that the number of feature points extracted per unit area is the same. Assume that the initial image area is S0 and the image scaling factor is s. The total area of ​​the n-layer image is:

[0064]

[0065] Assume the number of feature points is N, and the number of feature points extracted per unit area is:

[0066]

[0067] The feature points assigned to the i-th layer are:

[0068]

[0069] Step 2.2) Divide the image into regions using a 30×30 pixel grid, and then perform FAST key point extraction in each grid;

[0070] Step 2.3) Select the extracted feature points in each grid using the quadtree method, and finally retain the feature point with the largest Harris response value in each area;

[0071] Step 3) The reliability of feature matching is evaluated by using the grid-based motion statistics algorithm GMS, and then the random sample consistency RANSAC algorithm is used to further eliminate mismatched features, thereby effectively improving the accuracy of feature matching, including the following steps:

[0072] Step 3.1) Perform brute force matching on the feature points based on the Hamming distance of the binary descriptor to obtain the initial feature point matching pair. The formula for calculating the Hamming distance is as follows:

[0073]

[0074] Where n is the length of the string, a i and b i The i-th characters of strings a and b respectively, 1(a i ≠b i ) is the indicator function, when a i ≠b i The value is 1 when it is, otherwise the value is 0;

[0075] Step 3.2) Use the GMS algorithm to preliminarily screen the initial feature point matching pairs to reduce the number of matches;

[0076] Step 3.3) using a random sample consistency algorithm to estimate the transformation model between the two images, identifying the inlier matching pairs and the outlier matching pairs, and only retaining the inlier matching pairs as the correct feature matching information;

[0077] Step 4) obtaining the camera pose through camera motion estimation and triangulation, and obtaining sparse point cloud information through coordinate transformation, including the following steps:

[0078] Step 4.1) For the matched feature point pairs, calculate their essential matrix E, and decompose the rotation matrix R and translation vector t from E, and obtain the 3D position of the spatial point in the current camera coordinate system through triangulation;

[0079] Step 4.2) All spatial points are unified into the world coordinate system through coordinate transformation, thereby constructing a sparse point cloud map;

[0080] Step 5) Based on the neural radiance field algorithm, combined with the color and depth loss functions, the obtained camera pose, input image and 3D point cloud information are used to complete the construction of the 3D dense map, including the following steps:

[0081] Step 5.1) Use the linear interpolation method to generate 64 uniformly distributed sampling points between the near end and the far end of the ray, and combine the ray projection and camera model technology to generate the sampling point pose;

[0082] Step 5.2) Hash code the obtained sampling point pose. The specific formula is as follows:

[0083]

[0084] The sampling point pose is assumed to be (x i ,y i ,z i ), d represents the dimension of the pose, here d = 3, π1, π2, π3 are 1, 2654435761, 805459861, It means bitwise XOR, mod T means modulo operation, here T = 2 14 -1;

[0085] Step 5.3) Send the encoded pose information to the fully connected network to obtain the color density information of the sampling point;

[0086] Step 5.4) Use volume rendering technology to obtain any ray r(t)=o+tD obs At the boundary [t n ,t f ], where o is the camera position and the viewing direction is D obs , its rendering equation is as follows:

[0087]

[0088] The cumulative transmittance Indicates that the light from t n The probability of propagating to t without being blocked is, where σ(r(t)) and c(r(t),D obs ) represents the volume density and color, dt represents the differential distance traveled by the light ray in each integration step;

[0089] Step 5.5) The color loss is represented by the square of the Euclidean distance between the predicted color C(r) of the light r and the actual observed color C(r); the depth loss is represented by the KL divergence of the depth value estimated by NeRF technology rendering and the depth value of the 3D sparse point cloud. The formulas are as follows:

[0090]

[0091] Where R(P) represents the set of rays generated by a specific camera parameter P, and E r∈R(P)It means to calculate the expectation of each ray r in the ray set, X represents the set of all 3D key points {X:x1,x2,x3}, represents the subset of 3D keypoints visible from camera position j, x i ∈X j Represents the 3D key point x i Visible keypoints belonging to camera j. represents all key points x visible from camera position j i The expected value of represents the sum of the sampling point index k along the direction of the light, h(k) represents the weight function of the sampling point, t k is the depth value of the kth sampling point rendered, D ij According to the depth value of the 3D point cloud, represents the average reprojection error between views where key points are detected, Δt k Indicates the interval of sampling points.

Claims

1. A monocular dense SLAM map construction method based on image enhancement and NeRF, characterized in that The following steps are involved: Step 1) A low-light image enhancement algorithm based on the fusion of Retinex theory and attention mechanism is used to enhance image brightness and contrast while retaining image detail information, providing higher quality images for subsequent feature point extraction and feature point matching tasks, including the following steps: Step 1.1) Input the low-light image into a feature extraction module consisting of 18 convolutional layers, 4 downsampling and 4 upsampling steps. The convolutional layer uses a 3×3 convolution kernel, a stride and padding of 1, and a ReLU activation function; downsampling is achieved by 2×2 maximum pooling with a stride of 2, and upsampling is achieved by deconvolution with a stride of 2; the downsampled feature map is cascaded to the upsampled feature map of the same resolution through a skip connection mechanism, and finally a 32-channel high-dimensional multi-scale feature map is output; Step 1.2) Input the output 32-dimensional feature map into the attention mechanism module, first undergo a global average pooling operation, and compress the extracted feature map x into a channel descriptor z, the formula of which is as follows: where x c is the feature map of the cth channel of the input, z c is the feature descriptor of the cth channel, H and W represent the length and width of the channel feature map respectively, and then the feature descriptor z c Using two fully connected operations F sq , the ReLU activation function is used between the two fully connected layers for nonlinear processing, and finally the Sigmoid activation function is used to output the weight of each dimension. The fully connected operation formula is as follows: s c =F sq (z c ,w)=σ(w2δ(w1z c )) In the above formula, w1 and w2 represent two fully connected layers, σ and δ represent Sigmoid function and ReLU function, and s c Represents the weight of the cth channel of the output, and finally the channel weight s c The feature map x output by the feature extraction module c The recalibration operation is performed to obtain a weighted feature map with 32 channels; Step 1.3) Input the feature map into the reflection estimation module consisting of 6 symmetrically cascaded convolutional layers and 1 output layer. Each convolutional layer consists of 32 convolution operations with a kernel size of 3×3 and a stride of 1 and a ReLU activation function. The activation function of the last layer is Tanh. The output layer outputs the inverse of the illumination component with a channel number of 3. Finally, the final enhanced image is obtained by pixel-by-pixel multiplication of the input image and the inverse of the illumination component. Step 2) extracting feature points from the enhanced image and homogenizing the feature points based on the quadtree method to achieve uniform distribution of the feature points in the entire image, including the following steps: Step 2.1) Before extracting feature points, first allocate the number of feature points to be extracted in each layer of the image pyramid. The feature points are allocated according to the principle that the number of feature points extracted per unit area is the same. Assume that the initial image area is S0 and the image scaling factor is s. The total area of ​​the n-layer image is: Assume the number of feature points is N, and the number of feature points extracted per unit area is: The feature points assigned to the i-th layer are: Step 2.2) Divide the image into regions using a 30×30 pixel grid, and then perform FAST key point extraction in each grid; Step 2.3) Select the extracted feature points in each grid using the quadtree method, and finally retain the feature point with the largest Harris response value in each area; Step 3) The reliability of feature matching is evaluated by using the grid-based motion statistics algorithm GMS, and then the random sample consistency RANSAC algorithm is used to further eliminate mismatched features, thereby effectively improving the accuracy of feature matching, including the following steps: Step 3.1) Perform brute force matching on the feature points based on the Hamming distance of the binary descriptor to obtain the initial feature point matching pair. The formula for calculating the Hamming distance is as follows: Where n is the length of the string, a i and b i are the i-th characters of strings a and b respectively, 1(ai≠bi) is the indicator function, when a i ≠b i The value is 1 when it is, otherwise the value is 0; Step 3.2) Use the GMS algorithm to preliminarily screen the initial feature point matching pairs to reduce the number of matches; Step 3.3) using a random sample consistency algorithm to estimate the transformation model between the two images, identifying the inlier matching pairs and the outlier matching pairs, and only retaining the inlier matching pairs as the correct feature matching information; Step 4) obtaining the camera pose through camera motion estimation and triangulation, and obtaining sparse point cloud information through coordinate transformation, including the following steps: Step 4.1) For the matched feature point pairs, calculate their essential matrix E, and decompose the rotation matrix R and translation vector t from E, and obtain the 3D position of the spatial point in the current camera coordinate system through triangulation; Step 4.2) All spatial points are unified into the world coordinate system through coordinate transformation, thereby constructing a sparse point cloud map; Step 5) Based on the neural radiance field algorithm, combined with the color and depth loss functions, the obtained camera pose, input image and 3D point cloud information are used to complete the construction of the 3D dense map, including the following steps: Step 5.1) Use the linear interpolation method to generate 64 uniformly distributed sampling points between the near end and the far end of the ray, and combine the ray projection and camera model technology to generate the sampling point pose; Step 5.2) Hash code the obtained sampling point pose. The specific formula is as follows: The sampling point pose is assumed to be (x i ,y i ,z i ), d represents the dimension of the pose, here d = 3, π1, π2, π3 are 1, 2654435761, 805459861, It means bitwise XOR, mod T means modulo operation, here T = 2 14 -1; Step 5.3) Send the encoded pose information to the fully connected network to obtain the color density information of the sampling point; Step 5.4) Use volume rendering technology to obtain any ray r(t)=o+tD obs At the boundary [t n ,t f ], where o is the camera position and the viewing direction is D obs , its rendering equation is as follows: The cumulative transmittance Indicates that the light from t n The probability of propagating to t without being blocked is, where σ(r(t)) and c(r(t),D obs ) represents the volume density and color, dt represents the differential distance traveled by the light ray in each integration step; Step 5.5) The color loss is represented by the square of the Euclidean distance between the predicted color C(r) of the light r and the actual observed color C(r); the depth loss is represented by the KL divergence of the depth value estimated by NeRF technology rendering and the depth value of the 3D sparse point cloud. The formulas are as follows: Where R(P) represents the set of rays generated by a specific camera parameter P, and E r∈R(P) It means to calculate the expectation of each ray r in the ray set, X represents the set of all 3D key points {X:x1,x2,x3}, represents the subset of 3D keypoints visible from camera position j, x i ∈X j Represents the 3D key point x i Visible keypoints belonging to camera j; represents all key points x visible from camera position j i The expected value of represents the sum of the sampling point index k along the light direction, h(k) represents the weight function of the sampling point, t k is the depth value of the kth sampling point rendered, D ij According to the depth value of the 3D point cloud, represents the average reprojection error between views where key points are detected, Δt k Indicates the interval of sampling points.

Images