Multi-modal visual odometer fusing RGB and depth features

Abstract

The invention provides a multi-modal visual odometer fusing RGB and depth features, which comprises the following steps: S1, analyzing depth features and RGB features of adjacent image frames to obtain an inverse depth map; s2, by calculating the minimum re-projection loss of each pixel, negative effects caused by continuous frame shielding of the monocular video are reduced, and more excellent depth estimation is obtained; s3, performing channel fusion on two different modes of information of RGB and depth information to form relative attitude output; and S4, in combination with the double-flow network, introducing a regularization optimization network model so as to achieve a better effect.

Description

Technical Field

[0001] The present invention relates to the field of visual SLAM, and in particular to a multimodal visual odometer integrating RGB and depth features. Background Art

[0002] With the rapid development of artificial intelligence, the robotics industry has also developed rapidly, playing an increasingly important role in people's production and life [1]. Mobile robots, driverless cars and other products are increasingly appearing in people's vision. To achieve autonomous movement, two problems must be solved: "simultaneous positioning" and "map building", namely SLAM (Simultaneous Localization and Mapping). SLAM is generally divided into visual SLAM and laser SLAM. Compared with the expensive equipment required by laser SLAM, the sensor of visual SLAM is a camera, which is significantly less expensive and more conducive to promotion.

[0003] With the development of deep learning in the field of computer vision, more and more researchers are combining deep learning with traditional visual SLAM, using deep learning methods to replace a certain link in traditional visual SLAM. The combination of deep learning and visual odometry has been a key research direction in recent years. The main working principle of visual odometry is to estimate the camera pose through the movement between two consecutive frames of images. With the support of a large amount of training data, the combination of deep learning and visual odometry abandons a series of processes such as front-end point tracking and back-end optimization and solution, and directly inputs the image to give the pose estimation result, which has a good estimation effect in harsh environments.

[0004] Most current work transforms the camera pose estimation problem into regression classification of color images. In the above research process, a common problem is that changes in image scenes, such as changes in illumination, can greatly affect the accuracy of pose estimation. Recent studies have shown that better camera pose estimation can be achieved by fusing information features from different sensors, and fusing RGB information features and depth features is one of the directions. Summary of the invention

[0005] The purpose of the present invention is to integrate RGB and depth features into a multimodal visual odometer. First, the inverse depth map is obtained using a self-supervised monocular depth estimation network, and then the minimum reprojection loss of each pixel is calculated to reduce the negative impact of continuous frame occlusion of the monocular video, so as to obtain a better depth estimation. Then, a channel exchange fusion-based mode is adopted to fuse the two different modal information of RGB and depth information to form a relative posture output. Finally, a dual-stream network is combined to introduce a regularized optimization network model to achieve better training results.

[0006] The technical solution of the present invention to solve the above problem is as follows: a multimodal visual odometer integrating RGB and depth features, comprising the following steps:

[0007] S1, analyze the depth features and RGB features of adjacent image frames to obtain the inverse depth map

[0008] S2, calculates the minimum re-projection loss for each pixel to obtain excellent depth estimation

[0009] S3, combines the two different modal information of RGB and depth information to form the relative posture output

[0010] S4, combined with the dual-stream network, introduces regularized optimization network model to achieve better training effect

[0011] The beneficial effects of the present invention are:

[0012] The present invention is a multimodal visual odometer that integrates RGB and depth features. First, the depth image and RGB features of each input frame are estimated through the self-motion estimation framework DLF-VO, and the inverse depth map is obtained by using the self-supervised monocular depth estimation network. The minimum reprojection loss of each pixel is calculated to reduce the negative impact of continuous frame occlusion of the monocular video, so as to obtain a better depth estimation. Then, a multi-layer fusion mode is adopted to integrate multiple feature information appearing in the middle layer of the encoder, and the two different modal information of RGB and depth information are integrated to form the final relative posture output. The posture estimation model has a dual-stream structure, and the complementary features of RGB and depth modes are retained by using the channel exchange CEN strategy. Feature elements are exchanged according to riding importance, and finally a regularized optimization network model is introduced to achieve better training results. The multimodal visual odometer that integrates RGB and depth features has great advantages over the traditional visual odometer based on single information feature analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Schematic diagram of the process of the present invention

[0014] Figure 2 The motion estimation network architecture of the present invention

[0015] Figure 3 This is the self-supervised training loss graph of the present invention

[0016] Figures 4 to 5 This is the trajectory diagram of Example 1 DETAILED DESCRIPTION

[0017] The present invention is further described below with reference to specific examples, which are only used to explain the present invention but not to limit the scope of the present invention.

[0018] S1, analyze the depth features and RGB features of adjacent image frames to obtain the inverse depth map

[0019] S2, calculates the minimum re-projection loss for each pixel to obtain excellent depth estimation

[0020] S3, combines the two different modal information of RGB and depth information to form the relative posture output

[0021] S4, combined with the dual-stream network, introduces regularized optimization network model to achieve better training effect

[0022] Furthermore, in step S1, the depth image of each input frame is first estimated, and an inverse depth map is obtained using a self-supervised monocular depth estimation network. The formula is:

[0023] D t =θ depth (I t ) (1)

[0024] where θ pose represents the depth estimation, I t represents an image pair of consecutive video frames, D t Represents the inverse depth map.

[0025] Furthermore, in step S2, the minimum reprojection loss of each pixel is calculated to reduce the negative impact of occlusion of consecutive frames of the monocular video, so as to obtain a better depth estimation, relative to the attitude estimator θ pose The relative pose between consecutive frames is predicted as the final output of ego-motion, which is formulated as:

[0026]

[0027] Here ⊕ represents the concatenation along the input channels.

[0028] Furthermore, the step S3 uses a multi-layer fusion mode to fuse the two different modalities of RGB and depth information. In order to make the model produce a singular solution, this paper proposes a new loss function.

[0029] First, prepare a training set consisting of

[0030] {I s ,I s ,T t s ,s∈{t-1,t+1}}a target image I t and its continuous source image I s composition

[0031]

[0032] Where L p Represents the luminosity loss, L gc represents the geometric consistency loss, L s Indicates the smoothness loss. t represents the input target, I s represents the source image, D t represents the depth map, T s Indicates relative motion.

[0033] Furthermore, after combining the dual-stream network, step S4 introduces a regularized network optimization model and proposes a new regularized loss to reduce the impact of redundant information on the training model.

[0034] The total self-supervised loss formula is:

[0035]

[0036] Among them, Lp represents photometric loss, Lgc represents geometric consistency loss, Ls represents smoothness loss, It represents input target, Is represents source image, Dt represents depth map, Ts represents relative motion, and λi represents respective hyperparameters.

[0037] The objective function of network training after regularizing the scale factor is:

[0038]

[0039] Where M represents the loss function, R represents the L2 regularization of the network weights, and Rs represents the sparse regularization of the neuron scale factor. In pruning, a reasonable threshold is selected and neurons with a scale factor lower than the threshold are pruned.

[0040] Let γ=(γ1,γ2,···,γ n ), and I n =(1,1,···,1)∈R n . Then the average value is:

[0041]

[0042] To prevent the scaling factor from converging to a fixed value, the polarization regularization is defined as:

[0043]

[0044] From the above formula, we can see that when all γ are equal, R s (γ) gets the maximum value when half of γ i When is 1, and the other half is equal to a, the term reaches the minimum value. This paper also uses the hyperparameter t to control the relative Weights. At the same time, t also controls the scaling factor equal to 0 under regularization.

[0045] The original norm in this paper is expanded to:

[0046]

[0047] Where γ1 represents the average value of the exchangeable scaling factor in the first BN layer. f(γ m ,l,i) represents the scaling factor of the i-th exchangeable channel of the 1st BN layer of modality m.

[0048] Example 1: To verify the performance of the visual odometry proposed in this paper and the effectiveness of the CEN channel exchange network and regularization method, both the depth and posture networks receive input images of size 640×192 pixels. The batch size is set to 12, and the model is trained for 20 epochs. In the first 10 cycles, the learning rate is set to 1e-4. For the remaining cycles, the learning rate is reduced to 5e-5. A single NVIDIA RTX 3060Ti is used to train the complete model, and the KITTI dataset is used for training and testing. The 00-08 sequence is used as the training set, and 09-10 is used as the test set to verify the results. In order to analyze the experimental data more accurately, this experiment calculates the subsequence translation error, subsequence rotation error, absolute trajectory error, relative pose error (translation), and relative pose error (rotation). The global trajectories of each method are shown in Tables 1 and 2 and Figures 4 to 5 shown.

[0049] Table 109 Sequence verification results

[0050]

[0051]

[0052] Table 210 Sequence verification results

[0053]

[0054] From Example 1, we can conclude that ORB-SLAM2 [3] The large-scale drift error on sequence 09 is shown. The method in this paper is close to ORBSLAM2 in terms of subsequence translation percentage, subsequence rotation error and absolute trajectory error, which is significantly better than SC-SFMLearner. In addition, the relative pose translation error and relative pose rotation error are much better than other experimental data, indicating that the method in this paper maintains good consistency. This shows that the visual odometry based on RGB features and depth information features in this paper has great advantages over the traditional visual odometry based on single information feature analysis.

Claims

1. A multimodal visual odometry system integrating RGB and depth features, characterized in that: The following steps are involved: S1, analyze adjacent image frames to obtain an inverse depth map; S2, calculates the minimum re-projection loss for each pixel to obtain an excellent depth estimate; S3, combines the two different modalities of RGB and depth information to form the relative posture output; S4, combined with the two-stream network, introduces a regularized optimization network model to achieve better training results.

2. The multimodal visual odometer integrating RGB and depth features according to claim 1, characterized in that: In S1, a self-supervised monocular depth estimation network is used to obtain an inverse depth map. Then, the minimum reprojection loss of each pixel is calculated to reduce the negative impact of occlusion of consecutive frames of the monocular video, so as to obtain a better depth estimation. D t =θ depth (I t ) (1) where θ pose represents the depth estimation, I t represents an image pair of consecutive video frames, D t Represents the inverse depth map.

3. The multimodal visual odometer integrating RGB and depth features according to claim 1, characterized in that: In S2, the minimum reprojection loss of each pixel is calculated to reduce the negative impact caused by the occlusion of consecutive frames of the monocular video, so as to obtain a better depth estimation relative to the pose estimator θ pose The relative pose between consecutive frames is predicted as the final output of ego-motion: where θ pose represents the relative pose estimator, Relative attitude output.

4. The multimodal visual odometer integrating RGB and depth features according to claim 1, characterized in that: In the S3, during the fusion process of the RGB feature information and the depth feature information, a channel exchange strategy is used to simultaneously retain the complementary features of the RGB and depth features, and the feature elements are exchanged according to their importance.

5. The multimodal visual odometry integrating RGB and depth features according to claim 1, characterized in that: In the process of fusing RGB feature information and depth feature information in S3, in order to generate a singular solution for the model, this paper proposes a new loss function. Prepare a training set by A target image I t and its successive source images I s Recalculate the luminosity loss L p and geometric consistency loss L gc . Among them, L p Represents the luminosity loss, L gc represents the geometric consistency loss, L s Indicates the smoothness loss. t represents the input target, I s represents the source image, D t represents the depth map, T s Indicates relative motion.

6. The multimodal visual odometer integrating RGB and depth features according to claim 1, characterized in that: After combining the dual-stream network in S4, a regularized network optimization model is introduced, and a new regularization loss is proposed to reduce the impact of redundant information on the training model. The total self-supervised loss formula is: Among them, L p Represents the luminosity loss, L gc represents the geometric consistency loss, L s Indicates the smoothness loss. t represents the input target, I s represents the source image, D t represents the depth map, T s represents relative motion, λ i denote their respective hyperparameters. The objective function of network training after regularizing the scale factor is: Where M represents the loss function, R represents the L2 regularization of the network weights, and R s Represents the sparse regularization of the neuron scale factor. In pruning, a reasonable threshold is selected and neurons with a scale factor lower than the threshold are pruned. The original norm in this paper is expanded to: Where γ1 represents the average value of the exchangeable scaling factor in the first BN layer. f(γ m ,l,i) represents the scaling factor of the i-th exchangeable channel of the 1st BN layer of modality m.

Images