A Neural Implicit 3D Reconstruction SLAM Method Based on Loop Closure Detection

Abstract

The invention relates to the technical field of image three-dimensional reconstruction, and specifically discloses a neural implicit three-dimensional reconstruction SLAM method based on loop detection, comprising the following steps: step S01: implicitly reconstructing objects in a scene, decoding them by using an MLP multi-layer perceptron, obtaining feature layers, and calculating occupancy probability values ​​of the feature layers; step S02: calculating color values ​​and termination probabilities according to the occupancy probability values ​​of the feature layers, and rendering the color values ​​and depth values ​​according to the calculation results to obtain a rendered image; step S03: creating a common view optical flow matrix according to the rendered image, selecting key frames in the common view optical flow matrix, and calculating relative optical flows of the key frames; step S04: performing loop detection on the relative optical flows of the key frames, comparing them with relative optical flow thresholds, and completing key frame selection; and step S05: projecting a point on a visual ray in the feature layer in the selected key frame, calculating a projection error, and performing neural implicit three-dimensional reconstruction optimization.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional image reconstruction, and in particular to a neural implicit three-dimensional reconstruction SLAM method based on loop closure detection. Background Art

[0002] In the prior art, the methods of three-dimensional reconstruction include: the first one is to perform three-dimensional reconstruction based on traditional unordered image groups. However, due to the lack of temporal information between images, when there are repeated places, devices, and textures with similar appearance in the scene, it is easy to register some images to the wrong position, causing the model after three-dimensional reconstruction to be deformed and misplaced. The second one is to perform three-dimensional reconstruction based on point clouds, which is mostly used for RGBD sensors. However, this method depends on the shape of the trajectory and the quality of the point cloud, and the reconstruction speed is greatly affected by the scale of the point cloud; in addition, if there are structured repeated areas in the scene, point cloud alignment is prone to ambiguity, which will affect the accuracy of the reconstructed model. The third one is to perform three-dimensional reconstruction based on single-track SLAM information. This method only supports the information of a single trajectory collected at a single time, and relies on the calculation of the information covariance between the acquisition device and the acquisition trajectory in the SLAM algorithm. Therefore, this method does not support the processing of trajectories collected multiple times separately, nor does it support the optimization of the joint factor graph of trajectories obtained from multiple devices. In the prior art, the influence of computational complexity and depth loss on accuracy is considered. Summary of the invention

[0003] In view of the above problems, an object of the present invention is to provide a neural implicit 3D reconstruction SLAM method based on loop closure detection, which reduces the computational workload, performs a separate loss calculation for depth, and improves accuracy.

[0004] The technical solution adopted by the present invention is: a neural implicit three-dimensional reconstruction SLAM method based on loop detection, comprising:

[0005] Step S01: implicitly reconstructing the objects in the scene in three dimensions, decoding them using an MLP multi-layer perceptron, obtaining feature layers, and calculating occupancy probability values ​​of the feature layers;

[0006] Step S02: Calculating a color value and a termination probability according to the occupancy probability value of the feature layer, and rendering the color value and the depth value according to the calculation result to obtain a rendered image;

[0007] Step S03: creating a common view optical flow matrix according to the rendered image, selecting a key frame in the common view optical flow matrix, and calculating the relative optical flow of the key frame;

[0008] Step S04: performing loop detection on the relative optical flow of the key frame, comparing it with the relative optical flow threshold, and completing key frame selection;

[0009] Step S05: Project a point on the visual ray in the feature layer into the selected key frame, calculate the projection error, and perform neural implicit three-dimensional reconstruction optimization.

[0010] Preferably, the feature layering includes a bottom layer, a middle layer and a fine layer.

[0011] Preferably, step S01 further comprises modeling the scene appearance using the feature grid and the decoder.

[0012] Preferably, step S03 further includes: obtaining a tracking model and a three-dimensional reconstruction model using the rendered image.

[0013] Preferably, in step S03, the selecting of the key frame comprises: adding a new current frame by information gain, and when the current frame has visual overlap with a previous key frame, selecting the current frame as the key frame.

[0014] Preferably, in step S03, the calculation of the relative optical flow of the key frame includes: keeping the illumination intensity of the pixel unchanged between adjacent key frames, calculating the velocity vector of the optical flow according to the moving distance and moving time of the pixel between adjacent key frames, and obtaining the relative optical flow of the key frame.

[0015] Preferably, the relative optical flow threshold is 25.

[0016] Preferably, in step S03, when the relative optical flow of the key frame is greater than the relative optical flow threshold, the implicit 3D reconstruction is continued;

[0017] When the relative optical flow of a key frame is continuously less than the relative optical flow threshold, it is selected as the key frame to complete implicit 3D reconstruction.

[0018] Preferably, in step S05, the projection coordinates of the points in the feature layer are obtained according to the projection, compared with the predicted image coordinates, and the projection error is calculated and assigned to the predicted image coordinates for optimization.

[0019] Preferably, the method further comprises step S06: performing loss calculation on the depth value of the key frame to optimize the accuracy and complete implicit 3D reconstruction;

[0020] The loss calculation includes loss calculation for the tracking model and the three-dimensional reconstruction model.

[0021] Beneficial effects of the above technical solution:

[0022] (1) The present invention performs a separate loss calculation for depth to improve accuracy. This solves the problem in the prior art that no separate loss calculation for depth is performed. As the depth loss is not considered, the contribution of each pixel value to tracking and mapping is consistent, resulting in different deviations and uncertainties due to different sensors. When such errors accumulate to a certain size, they will cause uncertainty to the overall image.

[0023] (2) The present invention adopts loop detection for the selection and optimization of key frames, thus avoiding repeated calculations and effectively reducing the computational workload. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flow chart of a neural implicit 3D reconstruction SLAM method based on loop closure detection provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following is a further detailed description of the implementation methods of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0026] The terms "first", "second", etc. (if any) in the specification and claims are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0027] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0028] Embodiment 1

[0029] An embodiment of the present invention provides a neural implicit 3D reconstruction SLAM method based on loop closure detection, including: step S01: implicitly reconstructing objects in a scene, decoding them using an MLP multi-layer perceptron, obtaining feature layers, and calculating occupancy probability values ​​of the feature layers; step S02: calculating color values ​​and termination probabilities according to the occupancy probability values ​​of the feature layers, and rendering the color values ​​and depth values ​​according to the calculation results to obtain a rendered image; step S03: creating a common view optical flow matrix according to the rendered image, selecting key frames in the common view optical flow matrix, and calculating the relative optical flow of the key frames; step S04: performing loop closure detection on the relative optical flow of the key frames, comparing them with the relative optical flow threshold, and completing key frame selection; step S05: projecting a point on the visual ray in the feature layer into the selected key frame, calculating the projection error, and performing neural implicit 3D reconstruction optimization.

[0030] Embodiment 2

[0031] In this embodiment, in addition to the steps in the first embodiment, step S06 is further included: performing loss calculation on the depth value of the key frame to optimize the accuracy and complete implicit 3D reconstruction.

[0032] Specifically, step S01: implicitly reconstruct the objects in the scene in three dimensions, decode them using the MLP multi-layer perceptron, obtain feature layers, and calculate the occupancy probability values ​​of the feature layers. Step S02: calculate the color value and termination probability according to the occupancy probability values ​​of the feature layers, render the color value and depth value according to the calculation results, and obtain a rendered image. Step S03: create a common view optical flow matrix according to the rendered image, select key frames in the common view optical flow matrix, and calculate the relative optical flow of the key frames. Step S04: perform loop detection on the relative optical flow of the key frames, compare it with the relative optical flow threshold, and complete the optimization of key frame selection; Step S05: project a point in the feature layer in the selected key frame, calculate the projection error, and optimize the selection of the key frame. Step S06: perform loss calculation on the depth value of the key frame, optimize the accuracy, and complete implicit three-dimensional reconstruction.

[0033] Embodiment 3

[0034] An embodiment of the present invention provides a neural implicit 3D reconstruction SLAM method based on loop detection, including: inputting RGB images and camera extrinsics into a feature layer, the feature layer respectively calculating the occupancy probability values ​​and color values ​​of the scene details of the bottom layer, the middle layer and the fine layer, and using the above results to calculate the termination probability value, the depth and color of the image are rendered. Then, a key frame sequence is created, key frames are selected and the sequence is maintained, the relative optical flow of the key frames entering the sequence for each frame is calculated, and it is input into the common view optical flow matrix. By comparing the threshold, when it is found that the optical flow of consecutive frames is less than the threshold, it is considered that a loop is detected and optimized. The reprojection error of the feature points is optimized by the bundle adjustment method (BA), and depth loss is added to the tracking and mapping parts to improve the accuracy.

[0035] Embodiment 4

[0036] In this embodiment, the neural implicit 3D reconstruction SLAM method based on loop detection specifically includes:

[0037] A. Implicit 3D reconstruction based on NICE-SLAM

[0038] A1. First, four feature grids and corresponding decoders are used to represent the geometric objects in the scene. In the scene, the geometric figures will be represented by the feature grid φ l Classification is performed and the corresponding MLP multi-layer perceptron is used for decoding f θ , where θ∈{0,1,2}, corresponding to the scene details of the bottom layer, middle layer and fine layer respectively. The last feature grid and decoder g σ Used to model the appearance of a scene.

[0039] A2. Calculate the occupancy probability of the above different layers respectively, and calculate the occupancy probability values ​​of the different layers decoded by the corresponding MLP multi-layer perceptron.

[0040] For the middle layer, the occupancy probability value of any point δ for:

[0041]

[0042] in, Perform trilinear interpolation at the middle layer point δ for the feature grid.

[0043] For the fine layer, the occupancy probability value of any point δ for:

[0044]

[0045] in, is the offset added to the middle layer. Perform trilinear interpolation at the middle layer point δ for the feature grid, Perform trilinear interpolation at the fine layer point δ for the feature grid.

[0046] For the bottom layer, the occupancy probability value of any point δ for:

[0047]

[0048] in, Perform trilinear interpolation on the feature grid at the underlying point δ.

[0049] The bottom layer is used to approximate occupancy values ​​outside the observed geometry, i.e., those partially observed figures and background.

[0050] A3, Depth and Color Rendering

[0051] The colors of the above different layers are rendered to different degrees according to the occupancy probability values. By rendering the RGB image, additional information is provided for tracking and mapping, and the computational workload can be reduced.

[0052] For any point P, the color value c p The feature grid in terms of color and its corresponding decoder are defined as:

[0053]

[0054] Among them, ω is the learnable parameter of the optimization process, is the color feature, p θ The occupancy probability values ​​calculated for different feature layers, θ∈{0,1,2}, correspond to the scene details of the bottom layer, middle layer, and fine layer, respectively. The learnable parameter ω takes different values ​​in different scenes, and gradually increases with the bottom layer, middle layer, and fine layer.

[0055] For depth and color rendering, at any point P on any visual ray, the above formula can be used to calculate its occupancy probability p at different feature layers θ and color value c p , and then calculate the termination probability τ i for:

[0056] τ i =p θ (1-p θ (0))

[0057] where p θ (0) is the occupancy probability value of the first sampling point of the ray.

[0058] Finally, the depth and color for each ray can be rendered as:

[0059]

[0060] Among them, d i is the depth measurement value of each point, c i For each point color value.

[0061] B. Loopback Detection

[0062] B1. Selection of key frames

[0063] Maintain a The key frame sequence gradually adds new key frames through information gain. When the new key frame has visual overlap with the current frame, it is added to the sequence to update the key frame. This key frame selection method has a small amount of calculation, ensuring that objects outside the current view are stationary, and when performing BA global optimization, only necessary parameters need to be optimized.

[0064] B2. Key frame relative optical flow calculation

[0065] Calculate the keyframe relative optical flow for each frame entering the keyframe sequence.

[0066] Assume that the light intensity of pixel I(x,y,t) in the first frame moves to the next frame by a distance (dx,dy) and a time dt.

[0067] When calculating the optical flow, it is assumed that the illumination intensity of two adjacent key frames remains unchanged, then I(x, y, t) satisfies the following formula:

[0068] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0069] Taylor expansion of the above equation yields:

[0070]

[0071] Where δ represents the second-order infinitesimal term, we get:

[0072]

[0073] Let u and v be the velocity vectors of the optical flow along the x-axis and y-axis respectively, and we get:

[0074]

[0075] make They represent the partial derivatives of the grayscale of the pixel in the key frame along the X, Y, and T directions respectively.

[0076] Finally, we can get:

[0077] I x u+I yv+I t =0

[0078] The solution (u, v) is the relative optical flow of the key frame.

[0079] B3. Loop detection

[0080] The calculated relative optical flow of the keyframe is added to a created common-view optical flow matrix. The dimension of the common-view optical flow matrix is ​​equal to the number of keyframes in the keyframe sequence. Set the relative optical flow threshold τ, and calculate the velocity vectors (u, v) of the optical flow along the x-axis and y-axis respectively. τ=25. When u and v are both less than the optical flow threshold τ, the frame is considered to be a high common-view keyframe. Compare the optical flow of the newly entered keyframe with the threshold τ. When the optical flow of the newly entered keyframe is greater than the threshold τ, this keyframe is defined as a low common-view keyframe, which means that it is likely to be in a new environment. When the optical flow of the newly entered keyframe is less than the keyframe with the threshold τ, this keyframe is defined as a high common-view keyframe. When the relative optical flow of 5 consecutive frames is less than the threshold τ, it is considered that the loop is detected, and the neural implicit 3D reconstruction SLAM is completed, and global optimization is performed.

[0081] C. Global Optimization

[0082] C1. Optimizing reprojection error

[0083] Any three-dimensional point p ij The projection coordinates in the k-th key frame are (u ij ,v ij ), and the reprojection error of the feature points is optimized by bundle adjustment (BA). The objective function is:

[0084]

[0085] Among them, m is the mth 3D point, n is the nth key frame, P(x i ,R i ,t i ) is the predicted image point coordinate, (u ij ,v ij ) is the coordinate of the projection point obtained by triangulation, X is the optimized pose, R is the rotation vector, and T is the translation vector.

[0086] The objective function is simplified by the Levenberg-Marquardt algorithm:

[0087]

[0088] e ij =P(x i ,R i ,t i )-p ij

[0089] Among them, K i is the reprojection error, X∈R 3m , R∈R 3m , T∈R 3m .

[0090] The reprojection error is optimized using the Gauss-Newton method:

[0091] The error function is:

[0092]

[0093] P0 is the initial value, and Taylor expansion is performed at P0 to obtain:

[0094] e ij (P) = e ij (P0)+J ij Δ

[0095] Among them J ij is the Jacobian matrix, J ij ∈R 2×(3m+6n) ,Δ∈R 3m+6n , m is the 3D point dimension, n is the camera dimension.

[0096]

[0097] Linearized objective function:

[0098]

[0099]

[0100] Where Z is the obtained Hessian matrix, Z∈R (3m+6n)×(3m+6m)

[0101] Setting the partial derivative to 0 gives:

[0102] ZΔ=-γ

[0103] Solve the system of equations to get the optimized result and update it:

[0104] P←P+Δ

[0105] C2. Deep Uncertainty Optimization

[0106] The NICE-SLAM method in the prior art does not perform a separate loss calculation for depth. Since the depth loss is not considered, the contribution of each pixel value to tracking and mapping is the same, which will result in different deviations and uncertainties due to different sensors. When such errors accumulate to a certain size, they will cause overall uncertainty.

[0107] The method of the present invention adds depth loss calculation to both the tracking and mapping parts to optimize the depth.

[0108] The depth loss in the tracking part is:

[0109]

[0110] The depth loss in the mapping part is:

[0111]

[0112] in is d n The standard deviation of n is the measured depth of pixel n, where N represents the number of pixels.

[0113] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made on the basis of the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the protection scope of the present invention.

Claims

1. A neural implicit 3D reconstruction SLAM method based on loop closure detection, characterized in that: include: Step S01: Use the feature grid to classify the geometric figures of objects in the scene, and use the MLP multi-layer perceptron to decode and obtain feature layers; Calculating the occupancy probability value of the feature layer; Step S02: Rendering the color value and the depth value according to the occupancy probability value of the feature layer, calculating the color value and the termination probability, obtaining a rendered image, and completing the neural implicit 3D reconstruction of the object in the scene; Step S03: creating a key frame sequence, selecting and adding key frames therein through information gain, calculating the relative optical flow of the key frames, and adding the relative optical flow of the key frames to the created common view optical flow matrix; Step S04: performing loop detection on the relative optical flow of the key frame and comparing it with the optical flow threshold to complete the loop detection; Step S05: projecting any three-dimensional point on the visual ray in the feature layer into the selected key frame, calculating the projection error, and performing neural implicit three-dimensional reconstruction optimization; In step S03, the selecting and adding key frames includes: adding a new current frame by information gain, and when the current frame has visual overlap with the previous key frame, selecting the current frame as the key frame; In step S04, when the relative optical flow of the key frame is greater than the optical flow threshold, the implicit 3D reconstruction is continued; When the relative optical flow of the key frame is less than the optical flow threshold for five consecutive times, loop closure detection is completed.

2. The neural implicit 3D reconstruction SLAM method based on loop closure detection according to claim 1, characterized in that: The feature layering includes a bottom layer, a middle layer and a fine layer.

3. The neural implicit 3D reconstruction SLAM method based on loop closure detection according to claim 1, characterized in that: Step S01 also includes modeling the scene appearance using the feature grid and the decoder.

4. The neural implicit 3D reconstruction SLAM method based on loop closure detection according to claim 1, characterized in that: Step S02 also includes: obtaining a tracking model and a three-dimensional reconstruction model using the rendered image.

5. The neural implicit 3D reconstruction SLAM method based on loop closure detection according to claim 1, characterized in that: In step S03, the calculation of the relative optical flow of the key frame includes: keeping the illumination intensity of the pixel unchanged between adjacent key frames, calculating the velocity vector of the optical flow according to the moving distance and moving time of the pixel between adjacent key frames, and obtaining the relative optical flow of the key frame.

6. The neural implicit 3D reconstruction SLAM method based on loop closure detection according to claim 1, characterized in that: The optical flow threshold is 25.

7. The neural implicit 3D reconstruction SLAM method based on loop closure detection according to claim 1, characterized in that: In step S05, the projection coordinates of the points in the feature layer are obtained according to the projection, compared with the predicted image coordinates, and the projection error is calculated and assigned to the predicted image coordinates for optimization.

8. The neural implicit 3D reconstruction SLAM method based on loop closure detection according to claim 4 is characterized in that: The method further includes step S06: performing loss calculation on the depth value of the key frame to optimize the accuracy and complete neural implicit 3D reconstruction; The loss calculation includes loss calculation for the tracking model and the three-dimensional reconstruction model.

Images