A method for integrating multi-base deep completion network with SLAM

Abstract

The present invention proposes a method for fusing a multi-base depth completion network with SLAM, tightly coupling the dense depth results predicted by multi-base depth completion with SLAM, and effectively optimizing the trajectory error of SLAM and improving the depth consistency of the map by associating the multi-base dense depth of the key frames of the SLAM optimization window and using global depth consistency as a constraint.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and computer graphics, and in particular to a method for fusing a multi-radix-based depth completion network with SLAM. Background Art

[0002] Depth information is a crucial three-dimensional scene information source, widely used in robotics, autonomous driving, AR, and VR applications. With depth information, robots can easily perform tasks such as obstacle avoidance and grasping. In AR / VR scenarios, dense depth can be used to detect occlusion and collisions, resulting in more realistic AR / VR effects. Furthermore, integrating depth information with visual information can solve many problems that cannot be addressed by purely visual methods. For example, visual SLAM is prone to scale drift, and incorporating depth information can improve the accuracy of the SLAM system. While most SLAM systems can only generate sparse point clouds or semi-dense maps, combining dense depth allows for real-time dense reconstruction.

[0003] Traditionally, depth information is obtained through sensors such as Kinect and Realsense. However, the use of these depth sensors often increases hardware costs, and different sensors are also limited by their application scenarios. With the rise of deep learning, many works have emerged that use neural networks to predict dense depth from images, and some have also integrated depth prediction networks with SLAM systems.

[0004] Depending on the input data, depth prediction can be categorized into single-frame depth prediction, sparse depth-based depth completion, and multi-frame depth prediction. Depth completion involves generating a dense depth map from a sparse depth map. Compared to directly predicting depth from RGB images, depth completion, due to the prior knowledge of sparse depth, can obtain depth information with accurate and stable scales and achieve higher accuracy. Due to the cumulative error in SLAM systems, even when using SLAM's 3D point depth information, single-frame depth completion cannot guarantee consistent global depth accuracy. Only by incorporating predicted depth information into the joint SLAM optimization process can global consistency of depth and trajectory be better ensured. Traditional methods assume that depth prediction is reliable relative to depth, with only scale error and depth drift. However, this model is highly inaccurate and can easily lead to optimization instability. Another traditional approach uses a variational autoencoder, incorporating network coding results into the SLAM optimization process to optimize global depth. However, this approach requires the network to perform inference for each derivative, which affects system efficiency. Furthermore, this fusion approach rarely improves SLAM accuracy. Here we use the multi-layer depth base predicted by the multi-base depth completion network to add it to the SLAM optimization process, and optimize the final depth by optimizing the weights of the multi-base depth. Summary of the Invention

[0005] To address existing technical issues, the present invention proposes a method for integrating a multi-base depth completion network with SLAM. The method comprises two processes: front-end tracking and back-end mapping. The system accepts a sequence of RGB images as input. The front-end is responsible for real-time tracking, and then selects some frames and sends them to the back-end as keyframes. Multi-base dense depth completion is performed on the images sent to the back-end mapping. Then, by aligning the predicted multi-base dense depth between the back-end keyframes, the camera pose and the weights of the multi-base dense depth basis are gradually optimized, and the map is updated.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a method for fusing a multi-base depth completion network with SLAM, characterized by comprising the following steps:

[0008] Step 1: The real-time SLAM accepts RGB images as input. The SLAM front-end is responsible for real-time tracking and then selects some frames to be sent to the SLAM back-end as keyframes.

[0009] Step 2: The real-time SLAM sends the latest backend keyframe image and the sparse depth corresponding to the keyframe to the depth completion neural network based on multi-basis fitting for depth completion. If the depth completion is successful, the dense depth is associated with the keyframe. If the completion does not fail, the current frame is ignored.

[0010] Step 3: When the backend accumulates enough key frames with dense depth information, an optimization window is established, and the weight coefficient of the dense depth basis and the current pose of SLAM are optimized simultaneously through the multi-frame relative depth consistency algorithm.

[0011] As a preferred solution of the present invention, in step 1, some frames are selected and sent to the back end as key frames, specifically:

[0012] Based on the front-end visual tracking status, if the number of trackable feature points between the current frame and the previous keyframe is less than a certain threshold, it is considered that a new scene has been reached and the current frame is added to the back-end as a keyframe.

[0013] Alternatively, if a certain amount of time has passed since the last keyframe was added, the current frame is added to the backend as a keyframe.

[0014] As a preferred solution of the present invention, the depth completion neural network based on multi-basis fitting in step 2 outputs multiple depth bases, and the final dense depth is obtained by weighted summation of the depth bases; the depth completion neural network based on multi-basis fitting accepts an RGB image and the corresponding depth point cloud as input, predicts multiple depth bases as output, and the weight of each depth base is solved according to the sparse depth construction optimization problem.

[0015] Preferably, the step 3 is specifically as follows:

[0016] The dense depth result of depth completion is obtained by weighted summation of multiple dense depth bases. The depth prediction of the i-th frame image is defined to obtain n depth bases, which are recorded as The weight coefficient corresponding to each depth basis is The final depth D of the i-th frame is i It can be expressed as follows:

[0017]

[0018] Define the camera pose of the i-th frame as T i , for the 2D point m on the i-th frame image, its image coordinate system coordinate is

[0019] When the depth and camera posture are known, back projection is performed to obtain the coordinates of point m in the world coordinate system.

[0020]

[0021]

[0022] Where π is the projection matrix, π -1 is the back projection matrix, Indicates D i exist The depth value of the position; then, the three-dimensional point Projecting to the jth frame, the image coordinates of the point on the jth frame image can be obtained and the corresponding depth

[0023]

[0024]

[0025]

[0026] The depth obtained by projection The depth predicted from the jth frame The residual term r of the relative depth constraint can be obtained m :

[0027]

[0028] In addition to the relative depth constraint, in order to optimize stability, add the depth base initial value constraint.

[0029]

[0030] in is the initial weight value of the kth basis in the i-th frame; by adding the relative depth constraint and the depth basis initial value constraint to the BA optimization of SLAM, the joint optimization of the depth basis weight and the SLAM key frame pose and map points is achieved.

[0031] Compared with the existing technology, the advantages of the present invention are:

[0032] 1) The present invention uses a multi-base-based depth completion network to perform depth prediction, and represents the single-frame depth information with multi-base weights. By incorporating the weights of each depth base into the optimization of the SLAM system, the joint optimization of depth, SLAM pose, and point cloud is achieved. Previous methods that integrate depth completion networks with SLAM systems based on variational autoencoders require the use of a decoder network to derive a new depth map to implement the derivation step each time optimization is performed. Our method only requires one inference to obtain the depth base, and no longer relies on network inference in the subsequent optimization process, resulting in higher optimization efficiency.

[0033] 2) The present invention uses the weight of the depth basis as a variable to be optimized. The number of variables that need to be optimized for each frame image is relatively small, and the number of parameters to be optimized is smaller than that of the previous method based on variational autoencoder, so the solution efficiency is higher.

[0034] 3) The present invention incorporates the relative depth consistency error between key frames. Through SLAM BA optimization, the depth and pose of the key frames can be continuously optimized simultaneously. Compared with previous methods, our invention method can achieve higher depth accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic flow diagram of the method of the present invention.

[0036] Figure 2 This is a diagram showing the reconstruction effect of the present invention; the left side is the point cloud of the SLAM real-time trajectory, and the right side is the reconstruction result.

[0037] Figure 3 3 is a comparison chart of the reconstruction effects before and after the optimization of the present invention. The left side shows the reconstruction result without using the method of the present invention, and the right side shows the reconstruction result with using the method of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be described in detail below with reference to the accompanying drawings. The technical features of each embodiment of the present invention can be combined accordingly without conflict.

[0039] refer to Figure 1 The method of the present invention includes two processes: front-end tracking and back-end mapping. The system accepts a sequence of RGB images as input and runs a real-time SLAM process. It then selects frames from the SLAM sliding window and feeds them into the back-end as keyframes. Multi-base dense depth completion is then performed on the keyframe images fed into the back-end mapping process. The predicted multi-base dense depths between the back-end keyframes are then aligned to gradually optimize the camera pose and the weights of the multi-base dense depth basis, and the map is updated.

[0040] like Figure 1 As shown, in a specific embodiment of the present invention, the method of the present invention includes the following steps:

[0041] Step 1: The SLAM running in real time accepts RGB images as input. The front end of SLAM is responsible for real-time tracking, and then selects some frames and sends them to the back end of SLAM as key frames. In order to record the scenes that have been passed and the map information, the present invention needs to save some historical frames as key frames in the map. In order to avoid serious map redundancy and excessive memory usage, the selection of key frames needs to follow a certain strategy. Commonly used strategies include: based on the front-end visual tracking status, if the number of feature points that can be tracked by the current frame and the previous key frame is less than a certain threshold, it is considered that a new scene has been reached, and the current frame is added to the back end as a key frame;

[0042] Alternatively, keyframes can be added based on time. For example, if a keyframe hasn't been added for a certain period of time relative to the previous keyframe, the current frame will be added to the backend as a keyframe. In practice, the keyframe selection strategy may be managed by multiple conditions.

[0043] Step 2: The real-time SLAM sends the latest backend keyframe image and the sparse depth corresponding to the keyframe to the depth completion neural network based on multi-basis fitting (Qu C, Nguyen T, Taylor C. Depth completion via deep basis fitting[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2020:71-80.) for depth completion. If the depth completion is successful, the dense depth is associated with the keyframe; if the completion does not fail, the current frame is ignored.

[0044] The depth completion neural network based on multi-base fitting of the present invention is different from the network that directly predicts dense depth as the final output. The depth completion neural network based on multi-base fitting outputs multiple depth bases, and the final dense depth is obtained by weighted summation of the depth bases; the depth completion neural network based on multi-base fitting accepts an RGB image and the corresponding sparse depth as input, and predicts multiple depth bases as output. The weight of each depth base can be solved according to the sparse depth construction optimization problem, and the purpose of continuously optimizing the final depth can be achieved by continuously adjusting the weight of each depth base.

[0045] Step 3: When the backend accumulates enough key frames with dense depth information, an optimization window is established, and the weight coefficient of the dense depth basis and the current pose of the SLAM are optimized simultaneously through the multi-frame relative depth consistency algorithm. In a preferred embodiment, the present invention is specifically as follows to step 3:

[0046] The dense depth result of depth completion is obtained by weighted summation of multiple dense depth bases. The depth prediction of the i-th frame image is defined to obtain n depth bases, which are recorded as The weight coefficient corresponding to each depth basis is The final depth D of the i-th frame is i It can be expressed as follows:

[0047]

[0048] Define the camera pose of the i-th frame as T i , for the 2D point m on the i-th frame image, its image coordinate system coordinate is

[0049] When the depth and camera posture are known, back projection is performed to obtain the coordinates of point m in the world coordinate system.

[0050]

[0051]

[0052] Where π is the projection matrix, π -1 is the back projection matrix, Indicates D i exist The depth value of the position; then, the three-dimensional point Projecting to the jth frame, the image coordinates of the point on the jth frame image can be obtained and the corresponding depth

[0053]

[0054]

[0055]

[0056] The depth obtained by projection The depth predicted from the jth frame The residual term r of the relative depth constraint can be obtained m :

[0057]

[0058] In addition to the relative depth constraint, in order to optimize stability, add the depth base initial value constraint.

[0059]

[0060] in is the initial weight value of the kth basis in the i-th frame; by adding the relative depth constraint and the depth basis initial value constraint to the BA optimization of SLAM, the joint optimization of the depth basis weight and the SLAM key frame pose and map points is achieved.

[0061] Implementation Cases

[0062] Figure 1 This is a flow chart of the method of the present invention. The sparse depth points and images generated by the key frames in the sliding window will be fed into the multi-base-based depth completion network. The network predicts the depth base corresponding to the image, combines the depth base with the sparse depth points, and optimizes the depth of the key frames. After the key frames in the window slide out of the window, the key frames added to the map will be added to the BA optimization of the backend. Through BA optimization, the joint optimization of the key frame pose and dense depth is achieved to obtain higher-precision key frame pose and dense depth. Finally, we can use the key frame pose and dense depth to generate a dense model using the TSDF dense reconstruction algorithm, such as Figure 2 shown. Figure 2 This is a diagram showing the reconstruction effect of the present invention; the left side is the point cloud of the SLAM real-time trajectory, and the right side is the reconstruction result. Through our method, the sparse point cloud can be completed into a dense model.

[0063] The present invention conducts experimental comparisons with previous work on the Euroc dataset. The depth accuracy obtained by our method and the accuracy of camera pose estimation are significantly improved compared with previous methods.

[0064] Figure 3 This is a comparison of the reconstruction results before and after optimization using our method. The left image shows the reconstruction result without depth optimization using our method, while the right image shows the reconstruction result using our method. We can see that our method produces a more consistent model (the wall in the left model exhibits delamination).

[0065] The present invention can be applied to various SLAM / SFM systems, and the sparse point cloud generated by the SLAM / SFM algorithm can be completed with dense depth using a multi-base depth completion neural network. The obtained dense depth can be used to generate more complex AR and VR effects such as occlusion and collision; it can also be used for dense three-dimensional reconstruction. Compared with point cloud models, dense three-dimensional models have higher utilization value, such as three-dimensional object tracking. The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and there can be many variations. All variations that can be directly derived or associated with the content disclosed by ordinary technicians in this field should be considered to be within the scope of protection of the present invention.

Claims

1. A method for fusing a multi-base depth completion network with SLAM, characterized in that: The following steps are involved: Step 1: The real-time SLAM accepts RGB images as input. The SLAM front-end is responsible for real-time tracking and then selects some frames to be sent to the SLAM back-end as keyframes. Step 2: The real-time SLAM sends the latest backend keyframe image and the sparse depth corresponding to the keyframe to the depth completion neural network based on multi-basis fitting for depth completion. If the depth completion is successful, the dense depth is associated with the keyframe. If completion does not fail, the current frame is ignored; Step 3: When the backend accumulates enough keyframes with dense depth information, an optimization window is established. Through the multi-frame relative depth consistency algorithm, the weight coefficient of the dense depth basis and the current pose of the SLAM are optimized simultaneously. The step 3 is specifically as follows: The dense depth result of depth completion is obtained by weighted summation of multiple dense depth bases. The depth prediction of the i-th frame image is defined to obtain n depth bases, which are recorded as The weight coefficient corresponding to each depth basis is The final depth D of the i-th frame is i It can be expressed as follows: Define the camera pose of the i-th frame as T i , for the 2D point m on the i-th frame image, its image coordinate system coordinate is When the depth and camera posture are known, back projection is performed to obtain the coordinates of point m in the world coordinate system. Where π is the projection matrix, π -1 is the back projection matrix, Indicates D i exist The depth value of the position; then, the three-dimensional point Project to the jth frame and obtain the image coordinates of the point on the jth frame image and the corresponding depth The depth obtained by projection The depth predicted from the jth frame Get the residual term r of the relative depth constraint m : In addition to the relative depth constraint, in order to optimize stability, add the depth base initial value constraint. in is the initial weight value of the kth basis in the i-th frame; by adding the relative depth constraint and the depth basis initial value constraint to the BA optimization of SLAM, the joint optimization of the depth basis weight and the SLAM key frame pose and map points is achieved.

2. The method of fusing a multi-based depth completion network with SLAM according to claim 1, wherein: In step 1, some frames are selected and sent to the backend as key frames, specifically: Based on the front-end visual tracking status, if the number of trackable feature points between the current frame and the previous keyframe is less than a certain threshold, it is considered that a new scene has been reached and the current frame is added to the back-end as a keyframe. Alternatively, if a certain amount of time has passed since the last keyframe was added, the current frame is added to the backend as a keyframe.

3. The method of fusing a multi-based depth completion network with SLAM according to claim 1, wherein: The depth completion neural network based on multi-basis fitting in step 2 outputs multiple depth bases, and the final dense depth is obtained by weighted summation of the depth bases; the depth completion neural network based on multi-basis fitting accepts an RGB image and the corresponding depth point cloud as input, predicts multiple depth bases as output, and the weight of each depth base is solved according to the sparse depth construction optimization problem.

Images