Design method of monocular visual odometry based on unsupervised deep learning

Abstract

The present invention discloses a design method of a monocular visual odometer based on unsupervised deep learning. The performance of the visual odometer is improved by jointly training depth, relative posture and optical flow. A deep network and an optical flow network are used to obtain long-sequence consistent depth information and dense optical flow information. Accurate sparse optical flow sampling is performed through front-back consistency errors. The optimal tracking mode is selected by the model score, and the visual odometer with consistent scale is obtained by aligning with the depth information. The method is significantly superior to simple geometric methods and end-to-end deep learning methods in multiple error evaluation indicators by combining the geometric constraints of traditional methods and the robustness matching of deep networks. Experiments have proved that the method effectively reduces the problems of scale inconsistency and scale drift.

Description

Technical Field

[0001] The present invention relates to a design method of a monocular visual odometer based on unsupervised deep learning. Background Art

[0002] In recent years, mobile robots and autonomous driving technologies have developed rapidly, and the requirements for autonomous positioning and navigation accuracy have become increasingly higher. In indoor or weak satellite navigation environments, vision-based simultaneous localization and mapping (SLAM) technology plays a vital role. As a key link in visual SLAM, visual odometry (VO) has also attracted more and more attention and research.

[0003] Visual odometry can be divided into feature-based methods and direct methods. Feature-based methods detect feature points and extract local descriptors as intermediate representations, then perform feature matching between images and use reprojection errors to optimize the camera pose. Direct methods model the image formation process and optimize the photometric error function by assuming grayscale invariance.

[0004] Deep learning has swept the field of computer vision in recent years, and SLAM research based on deep learning has also made significant progress. At present, related work mainly focuses on sub-problems of the standard SLAM link, such as feature extraction, feature matching, outlier removal and bundle adjustment (BA). The end-to-end visual odometry framework proposes to directly regress the relative pose or positioning information of the camera from the CNN network. CNN-SLAM replaces both depth estimation and image matching with CNN network-based methods based on LSD-SLAM, but its accuracy is seriously insufficient in outdoor environments. GEN-SLAM uses a monocular RGB camera to train the network with the results of traditional geometric SLAM to obtain pose and depth estimation. SfM-Learner simultaneously trains pose and depth networks to obtain results that compete with ORB-SLAM. Deep-VO-Feat and D3VO use binocular cameras for training, and can directly obtain trajectories at real scales under monocular camera operation. However, due to the lack of multi-view geometric constraints, end-to-end deep learning methods often face a huge scale drift problem, and it is difficult to obtain results that compete with traditional VO. In the above studies, monocular visual odometry still has problems of scale drift and scale inconsistency. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a design method of a monocular visual odometer based on unsupervised deep learning, which can effectively reduce scale inconsistency and scale drift problems.

[0006] The technical solution of the present invention to solve the above technical problems is: a design method of a monocular visual odometer based on unsupervised deep learning, comprising the following steps:

[0007] Step 1: Combine the depth consistency and image similarity loss functions to obtain a scale-consistent unsupervised deep learning network, and jointly train it with the RAFT optical flow network to obtain a more robust optical flow.

[0008] Step 2: Based on the consistency error, sparse sampling is performed in the dense optical flow to obtain the corresponding relationship;

[0009] Step 3: Select the optimal tracking method based on the correspondence, and use the deep network to perform depth alignment to obtain a visual odometer with consistent scale.

[0010] In the above-mentioned design method of the monocular visual odometer based on unsupervised deep learning, in the step 1, the framework of the unsupervised deep learning network includes three parts: a deep network, a relative posture network and an optical flow network. The deep network receives a single RGB image as input and outputs an inverse depth map. The relative posture network and the optical flow network both receive two frames of images as input. The relative posture network outputs a six-degree-of-freedom relative posture between the two frames, and the optical flow network outputs a dual-channel optical flow between the two frames.

[0011] In the above-mentioned design method of monocular visual odometer based on unsupervised deep learning, in the step 1, during training, the depths of two adjacent frames of images are estimated simultaneously, and the spatial consistency constraint is used to make their depth information consistent; the input of the posture network and the optical flow network are two adjacent RGB images, and the relative posture estimation and depth estimation are combined to obtain a synthetic image, and the photometric consistency loss function and the image smoothing loss function are used to optimize the depth information and the camera posture, and the RAFT network is jointly optimized by synthesizing the optical flow;

[0012] In the absence of true depth information and optical flow information, the unsupervised deep learning network uses synthetic views and inter-frame similarity as a supervisory signal to train the network model; the deep network and the optical flow network are geometrically associated through a relative pose network, where the relative pose network is used to help constrain the deep network and the optical flow network and is only used during training;

[0013] Consider two adjacent images I k and I k+1 , I k Represents the kth image, and the relative motion T between adjacent frames is obtained through the relative posture network and the deep network k→k+1 and single view depth D k ,D k+1 , D k represents the k-th single view depth, according to the equation

[0014]

[0015] in, The pixel coordinates of the k+1th image are obtained by transforming the pixel coordinates of the kth image; is the predicted camera relative motion from sequence k to sequence k+1, is the predicted depth of the k-th image pixel; p is the image pixel coordinate, p k is the pixel coordinate of the kth image, K is the camera intrinsic parameter matrix; the input image I k Transform to get the composite image Since the image is a continuous discrete value, differentiable bilinear interpolation is used to obtain continuous pixel coordinate values, and thus a synthetic optical flow F is obtained. syn :

[0016]

[0017] Unsupervised training assumes that the surface appearance of the same object between frames is the same. On the basis of simple pixel-by-pixel difference, structural similarity loss is introduced to learn the structural information between adjacent frames, and L1 and SSIM loss are combined as the reconstructed image loss:

[0018]

[0019] Among them, L p : Loss function result; Structural similarity loss function; α = 0.85, SSIM is calculated using a 3×3 window, V is the effective common view area of ​​adjacent frames;

[0020] In low-texture scenes or uniform areas, the assumed photometric invariance will lead to prediction void problems. In order to obtain smooth depth prediction, a first-order edge smoothing loss is introduced:

[0021]

[0022] Among them, L s : Smooth loss function result; D i,j : Represents the image disparity gradient; Represents the gradient of the image edge probability map, where the subscripts i and j represent pixel coordinates; x and y represent pixel directions; Ii,j : Pixel point at position (i, j) of the image; are the first-order derivatives along the x and y directions respectively;

[0023] For dynamic objects, the joint image segmentation network performs mask processing. According to Monodepth2, a binary mask is used to ignore objects that move synchronously with the camera. The mask is automatically calculated in the forward pass of the network:

[0024]

[0025] Where ω: binary mask; L p (I k ,I k+1 ): The image reconstruction loss function mentioned in formula (3) in the main text; They represent the k-th image, the composite image of the k-th image, and the k+1-th image respectively; they are fine-tuned by jointly training the synthetic optical flow and the error of the RAFT optical flow network:

[0026]

[0027] L f : Joint fine-tuning loss function results; flow : Common effective pixel area of ​​image optical flow field; F R (p): RAFT network optical flow prediction result; F syn (p): Optical flow prediction results of synthetic optical flow network;

[0028] During training, in order to obtain structurally consistent depth prediction, D k+1 With D k Align and calculate the depth consistency loss:

[0029]

[0030] Among them, L dc : Result of depth consistency loss function; D k (p): depth of the kth image; The depth of the kth image calculated by matching the optical flow network; S is the common effective area of ​​optical flow and depth, so as to obtain a consistent depth estimate;

[0031] To sum up, the overall loss function L of the network is:

[0032] L=L p +λ s L s +λ f L f +λ dc L dc (8)

[0033] Among them, λ s , f and λ dcRepresents the weight of each loss, and all losses are applied jointly to the deep network and the optical flow network.

[0034] The above-mentioned design method of monocular visual odometer based on unsupervised deep learning, the specific steps of step 2 are:

[0035] (2-1) Consistency of forward and backward optical flow: Get the forward optical flow F from the optical flow network f and backward optical flow F b And calculate the forward and backward consistency d F =|F f (p)+F b (p+F f (p))| 2 ;

[0036] (2-2) Sparse point sampling: Divide the image into 10×10 grid regions and take d points in each region. F The first 20 groups of sparse matching points less than the threshold δ.

[0037] The above-mentioned design method of monocular visual odometer based on unsupervised deep learning, the specific steps of step three are:

[0038] (3-1) Model selection: Calculate the essential matrix and homography matrix, and then calculate the model score R F , R F =S F / (S F +S H ), S F , S H are the F and R model scores respectively; if R F >0.5, select 2D-2D tracking mode; otherwise, select 3D-3D tracking mode;

[0039] (3-2) Scale recovery: For 2D-2D tracking mode, the normalized camera motion is obtained from the essential matrix decomposition R represents the rotation matrix, represents the displacement vector, whose modulus is unit 1; then the scale is aligned using the triangulation method to restore the scale factor s, and we get represents the camera motion, represents the actual displacement length obtained by scale recovery; for the 3D-3D tracking mode, the ICP method is used to solve and obtain T k→k+1 =[R|t], [R|t] represents the movement of the camera, and t represents the camera displacement vector.

[0040] In the above-mentioned design method of monocular visual odometer based on unsupervised deep learning, in the step (3-1), inspired by the ORB-SLAM initialization method, two tracking modes, 2D-2D and 3D-3D, are considered; ORB-SLAM uses the model score method to only select the initialization model, and the tracking process uses the constant speed motion model and the PnP method to solve the motion trajectory. Since there are both 2D-2D and 3D-3D correspondences, only the model score R is used F Select the tracking method; first solve the homography matrix H cr and the essential matrix F cr :

[0041]

[0042] Among them, p c is the matching point of the previous frame in two adjacent frames, p r is the matching point of the latter frame in two adjacent frames, and then calculates S for the H model and F model respectively H and S F Score:

[0043]

[0044] Where M is H or F, ρ M : The intermediate calculation result of the model score S; d 2 : represents the symmetric transfer error; Γ: equivalent to T H , is the invalid data exclusion threshold; The i′th matching point in the current frame; The i′th matching point of the reference frame; They are the symmetrical transfer errors between the current frame and the reference frame, and between the reference frame and the current frame; the error T M is the distance threshold, referring to ORB-SLAM, let T H =5.99,T F =3.84, Γ and T H The definitions are the same;

[0045] When the 3D point cloud structure is degraded, the depth information with consistent scale is obtained through the deep network to avoid decomposing the homography matrix, and the SVD method is used to solve [R|t]:

[0046]

[0047] n: represents the number of feature matching points between two adjacent images; i′: represents the matching point sequence number; R: represents the camera rotation matrix; Represents the i′th matching point in the kth image.

[0048] The beneficial effects of the present invention are as follows: the present invention improves the performance of the visual odometer by jointly training depth, relative posture and optical flow, uses a deep network and an optical flow network to obtain long-sequence consistent depth information and dense optical flow information, performs accurate sparse optical flow sampling through front-end consistency errors, selects the optimal tracking method based on the model score, and aligns it with the depth information to obtain a scale-consistent visual odometer, combines the geometric constraints of traditional methods with the robust matching of deep networks, and is significantly superior to simple geometric methods and end-to-end deep learning methods in multiple error evaluation indicators, and experiments have proved that the method effectively reduces scale inconsistency and scale drift problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of the present invention.

[0050] Figure 2 This is an architecture diagram of the unsupervised deep learning network of the present invention.

[0051] Figure 3 Schematic diagram of depth estimation.

[0052] Figure 4 Schematic diagram of extracting sparse matching relationships from forward and backward optical flows.

[0053] Figure 5 This is the KITTI 09 sequence trajectory diagram.

[0054] Figure 6 This is the trajectory diagram of KITTI 10 sequence. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0056] like Figure 1 As shown, a design method of a monocular visual odometer based on unsupervised deep learning includes the following steps:

[0057] Step 1: Combine the depth consistency and image similarity loss functions to obtain a scale-consistent unsupervised deep learning network, and jointly train it with the RAFT optical flow network to obtain a more robust optical flow.

[0058] The key to unsupervised learning is to use image reconstruction loss to calculate the difference between the synthesized image and the target image using the estimated depth, pose, optical flow and source image. The single-view depth network, optical flow network and camera relative pose network are three separate tasks, but they have interrelated image similarity constraints. Through view synthesis, based on the existing unsupervised signals, the spatial consistency loss function and the image similarity loss function are combined to couple the training process of these three networks.

[0059] The framework of the unsupervised deep learning network consists of three parts: a deep network, a relative pose network, and an optical flow network. The deep network receives a single RGB image as input and outputs an inverse depth map. The relative pose network and the optical flow network both receive two frames of images as input. The relative pose network outputs the six-degree-of-freedom relative pose between the two frames, and the optical flow network outputs a dual-channel optical flow between the two frames.

[0060] Unsupervised deep learning network architecture Figure 2 As shown in the figure, during training, the depth of two adjacent image frames is estimated simultaneously, and the spatial consistency constraint is used to make their depth information consistent; the input of the pose network and the optical flow network are two adjacent RGB images, and the relative pose estimation and depth estimation are combined to obtain a synthetic image. The photometric consistency loss function and the image smoothing loss function are used to optimize the depth information and camera pose, and the RAFT network is jointly optimized by synthesizing the optical flow; compared with training the network separately, the combination of multi-task consistency constraints strengthens the connection between the networks, and obtains more accurate and robust depth, pose and optical flow estimates.

[0061] In the absence of true depth information and optical flow information, the unsupervised deep learning network uses synthetic views to train the network model with inter-frame similarity as a supervisory signal; the deep network and the optical flow network are geometrically associated through a relative pose network, where the relative pose network is used to help constrain the deep network and the optical flow network and is only used during training.

[0062] Consider two adjacent images I k and I k+1 , I k Represents the kth image, and the relative motion T between adjacent frames is obtained through the relative posture network and the deep network k→k+1 and single view depth D k ,D k+1 , D k represents the kth single view depth, according to the equation

[0063]

[0064] in, The pixel coordinates of the k+1th image are obtained by transforming the pixel coordinates of the kth image; is the predicted camera relative motion from sequence k to sequence k+1, is the predicted depth of the k-th image pixel; p is the image pixel coordinate, p k is the pixel coordinate of the kth image, K is the camera intrinsic parameter matrix; the input image I k Transform to get the composite image Since the image is a continuous discrete value, differentiable bilinear interpolation is used to obtain continuous pixel coordinate values, and thus a synthetic optical flow F is obtained.syn :

[0065]

[0066] Unsupervised training assumes that the surface appearance of the same object between frames is the same. On the basis of simple pixel-by-pixel difference, structural similarity loss is introduced to learn the structural information between adjacent frames, and L1 and SSIM loss are combined as the reconstructed image loss:

[0067]

[0068] Among them, L p : Loss function result; Structural similarity loss function; α = 0.85, SSIM is calculated using a 3×3 window, V is the effective common view area of ​​adjacent frames;

[0069] In low-texture scenes or uniform areas, the assumed photometric invariance will lead to prediction void problems. In order to obtain smooth depth prediction, a first-order edge smoothing loss is introduced:

[0070]

[0071] Among them, L s : Smooth loss function result; D i,j : Represents the image disparity gradient; Represents the gradient of the image edge probability map, where the subscripts i and j represent pixel coordinates; x and y represent pixel directions; Ii,j : Pixel point at position (i, j) of the image; are the first-order derivatives along the x and y directions respectively;

[0072] For dynamic objects, the joint image segmentation network performs mask processing. According to Monodepth2, a binary mask is used to ignore objects that move synchronously with the camera. The mask is automatically calculated in the forward pass of the network:

[0073]

[0074] Where ω: binary mask; L p (I k ,I k+1 ): The image reconstruction loss function mentioned in formula (3) in the main text; They represent the k-th image, the composite image of the k-th image, and the k+1-th image respectively.

[0075] We chose to use the RAFT network, which has strong generalization ability, is fast and accurate, as the optical flow backbone network. Compared with the coarse-to-fine pyramid iterative network, RAFT only maintains and updates a single optical flow field at high resolution and shares weights during iterations, overcoming the two major difficulties of the coarse-to-fine optimization network: it is difficult to correct errors at coarse resolution and it is difficult to detect the rapid movement of small objects. Fine-tuning is performed by jointly training the errors of the synthetic optical flow and the RAFT optical flow network:

[0076]

[0077] L f : Joint fine-tuning loss function results; flow : Common effective pixel area of ​​image optical flow field; F R (p): RAFT network optical flow prediction result; F syn (p): Optical flow prediction results of synthetic optical flow network;

[0078] During training, in order to obtain structurally consistent depth prediction, D k+1 With D k Align and calculate the depth consistency loss:

[0079]

[0080] Among them, L dc : Result of depth consistency loss function; D k (p): depth of the kth image; The depth of the kth image is calculated by matching the optical flow network; S is the common effective area of ​​optical flow and depth, so as to obtain a consistent depth estimate.

[0081] To sum up, the overall loss function L of the network is:

[0082] L=L p +λ s L s +λ f L f +λ dc L dc (8)

[0083] Among them, λ s , f and λ dc Represents the weight of each loss, and all losses are applied jointly to the deep network and the optical flow network.

[0084] Step 2: Based on the consistency error between the previous and next images, sparse sampling is performed in the dense optical flow to obtain the corresponding relationship.

[0085] The specific steps are:

[0086] (2-1) Consistency of forward and backward optical flow: Get the forward optical flow F from the optical flow network f and backward optical flow F b And calculate the forward and backward consistency d F =|F f (p)+F b (p+F f (p))| 2 .

[0087] Using a deep network to obtain depth information with consistent scale, and performing the triangulation alignment process independently can minimize the scale drift problem.

[0088] In order to extract sparse matches from the optical flow network, forward and backward optical flows are used simultaneously, and bidirectional consistency error filtering is used to obtain accurate sparse correspondences. Since local matching does not rely on motion models, the problem of mismatching of a large number of feature points caused by sudden motion steering is avoided.

[0089] (2-2) Sparse point sampling: Divide the image into 10×10 grid regions and take d points in each region. F The first 20 groups of sparse matching points less than the threshold δ.

[0090] Step 3: Select the optimal tracking method based on the correspondence, and use the deep network to perform depth alignment to obtain a visual odometer with consistent scale.

[0091] The specific steps are:

[0092] (3-1) Model selection: Calculate the essential matrix and homography matrix, and then calculate the model score R F , R F =S F / (S F +S H ), S F , S H are the F and R model scores respectively; if R F >0.5, the 2D-2D tracking mode is selected; otherwise, the 3D-3D tracking mode is selected.

[0093] Inspired by the ORB-SLAM initialization method, two tracking methods, 2D-2D and 3D-3D, are considered. ORB-SLAM uses the model score method to select the initialization model only, and the constant speed motion model and PnP method are used to solve the motion trajectory during the tracking process. Since there are both 2D-2D and 3D-3D correspondences, only the model score R is used. F Select the tracking method; first solve the homography matrix H cr and the essential matrix F cr :

[0094]

[0095] Among them, p c is the matching point of the previous frame in two adjacent frames, p r is the matching point of the latter frame in two adjacent frames, and then calculates S for the H model and F model respectively H and S F Score:

[0096]

[0097] Where M is H or F, ρ M : The intermediate calculation result of the model score S; d 2 : represents the symmetric transfer error; Γ: equivalent to T H , is the invalid data exclusion threshold; The i′th matching point in the current frame; The i′th matching point of the reference frame; They are the symmetrical transfer errors between the current frame and the reference frame, and between the reference frame and the current frame; the error T M is the distance threshold, referring to ORB-SLAM, let T H =5.99,T F =3.84, Γ and T H The definitions are the same;

[0098] When the 3D point cloud structure is degraded, the depth information with consistent scale is obtained through the deep network to avoid decomposing the homography matrix, and the SVD method is used to solve [R|t]:

[0099]

[0100] n: represents the number of feature matching points between two adjacent images; i′: represents the matching point sequence number; R: represents the camera rotation matrix; Represents the i′th matching point in the kth image.

[0101] (3-2) Scale recovery: For 2D-2D tracking mode, the normalized camera motion is obtained from the essential matrix decomposition R represents the rotation matrix, represents the displacement vector, whose modulus is unit 1; then the scale is aligned using the triangulation method to restore the scale factor s, and we get represents the camera motion, represents the actual displacement length obtained by scale recovery; for the 3D-3D tracking mode, the ICP method is used to solve and obtain T k→k+1 =[R|t], [R|t] represents the movement of the camera, and t represents the camera displacement vector.

[0102] Experimental verification and result analysis

[0103] The Ubuntu 20.04 system is used, the CPU is i5-10300H, the GPU is NVIDIA Geforce GTX 1660Ti, the graphics card memory is 6GB, and the device memory is 16GB. Visual odometry experiments are conducted on the KITTI dataset and compared with traditional methods and end-to-end deep learning-based methods to verify the effectiveness of the method.

[0104] Network architecture and parameter settings:

[0105] The depth estimation network is based on the general U-net network architecture, i.e., an encoder-decoder structure, with ResNet18 as the encoder network; the decoder uses a skip connection architecture, and the skip connections between network layers enable it to simultaneously fuse shallow geometric information and high-level abstract features. Since the motion between adjacent frames is small, multi-scale output is neither accurate nor necessary, and we only output single-scale depth prediction results, which also greatly saves computing resources. The optical flow network uses the RAFT network as the backbone network; the relative pose network is a pose estimation network with a ResNet18 structure, using axis angles to represent three-dimensional rotations.

[0106] The network model is implemented using the Pytorch framework and trained in two stages using the Adam optimizer. The first stage is trained for 20 cycles and the learning rate is set to 10 -4 , the batch size is 8. The second stage is trained for 100 cycles, and the learning rate is set to 10 -5 , the batch size is 4. In training, we set λ s =0.4,λ f =0.4,λ dc =0.1, resize the image sequence to 480×640.

[0107] Visual Odometry:

[0108] The KITTI dataset provides a total of 22 sets of sequence data, of which 0-10 provide real trajectories. The experiment is trained on 0-8 sequences and evaluated on 9-10 sequences. It is also compared with ORB-SLAM2 and end-to-end deep learning methods. Since the monocular visual odometer cannot obtain the scale of the real world, in order to make a fair comparison, the results are aligned with the scale of the real trajectory. The qualitative trajectory results are as follows: Figure 5 and Figure 6As shown in Figure 2, compared with ORB-SLAM2, SfMLearner, SC-SfMLerner, and Depth-VO-Feat, our method has significantly less trajectory translation drift, thanks to our scale-consistent depth estimation. Although Depth-VO-Feat is trained with a stereo camera and can obtain results consistent with the scale of the real world, its scale drift problem is the most serious. On the contrary, due to the extraction of accurate matching relationships, after scale alignment, our method is more consistent with the real trajectory.

[0109] Table 1 KITTI 09&10 sequence comparison

[0110] Table 1 Comparison of KITTI sequence 09&10

[0111]

[0112] Using the translation error (t err ), rotation error (r err ), relative pose error RPE (m / °) and absolute trajectory error (ATE) for a more detailed analysis. The bold font indicates the best result in this evaluation. As shown in Table 1, our method outperforms traditional methods and pure deep learning methods in most indicators. ORB-SLAM2 has a better performance in rotation error. This is because the vehicle mainly travels at a constant speed, and its motion model also assumes that there is a constant motion between two frames, resulting in a smaller error, and the gap between our method is also very small. SC-SfMLearner also uses deep consistency constraints during training. Its absolute trajectory error on sequence 9 is slightly better than ours, and there is a large gap with other methods. However, since it does not impose multi-view geometry constraints in pose estimation, it is inferior to us in most other indicators. This paper extracts sparse matching relationships of multi-structural features and uses epipolar geometry constraints. It shows obvious advantages in other evaluations and has better overall performance.

Claims

1. A design method for a monocular visual odometer based on unsupervised deep learning, characterized in that: The following steps are involved: Step 1: Combine the depth consistency and image similarity loss functions to obtain a scale-consistent unsupervised deep learning network, and jointly train it with the RAFT optical flow network to obtain a more robust optical flow. Step 2: Based on the consistency error, sparse sampling is performed in the dense optical flow to obtain the corresponding relationship; Step 3: Select the optimal tracking method based on the correspondence, and use the deep network to perform depth alignment to obtain a visual odometer with consistent scale; In the step 1, the framework of the unsupervised deep learning network includes three parts: a deep network, a relative posture network and an optical flow network, the deep network receives a single RGB image as input and outputs an inverse depth map, the relative posture network and the optical flow network both receive two frames of images as input, the relative posture network outputs a six-degree-of-freedom relative posture between the two frames, and the optical flow network outputs a dual-channel optical flow between the two frames; In the step 1, during training, the depths of two adjacent frames of images are estimated simultaneously, and spatial consistency constraints are used to make their depth information consistent; The input of the attitude network and optical flow network are two adjacent RGB images. The relative pose estimation and depth estimation are combined to obtain a synthetic image. The photometric consistency loss function and image smoothness loss function are used to optimize the depth information and camera attitude. The RAFT network is jointly optimized by synthesizing the optical flow. In the absence of true depth information and optical flow information, the unsupervised deep learning network uses synthetic views and inter-frame similarity as a supervisory signal to train the network model; the deep network and the optical flow network are geometrically associated through a relative pose network, where the relative pose network is used to help constrain the deep network and the optical flow network and is only used during training; Consider two adjacent images I k and I k+1 , I k Represents the kth image, and the relative motion T between adjacent frames is obtained through the relative posture network and the deep network k→k+1 and single view depth D k ,D k+1 , D k represents the kth single view depth, according to the equation in, The pixel coordinates of the k+1th image are obtained by transforming the pixel coordinates of the kth image; is the predicted camera relative motion from sequence k to sequence k+1, is the predicted depth of the k-th image pixel; p is the image pixel coordinate, p k is the pixel coordinate of the kth image, K is the camera intrinsic parameter matrix; the input image I k Transform to get the composite image Since the image is a continuous discrete value, differentiable bilinear interpolation is used to obtain continuous pixel coordinate values, and thus a synthetic optical flow F is obtained. syn : Unsupervised training assumes that the surface appearance of the same object between frames is the same. On the basis of simple pixel-by-pixel difference, structural similarity loss is introduced to learn the structural information between adjacent frames, and L1 and SSIM loss are combined as the reconstructed image loss: Among them, L p : Loss function result; Structural similarity loss function; α = 0.85, SSIM is calculated using a 3×3 window, V is the effective common view area of ​​adjacent frames; In low-texture scenes or uniform areas, the assumed photometric invariance will lead to prediction void problems. In order to obtain smooth depth prediction, a first-order edge smoothing loss is introduced: Among them, L s : Smooth loss function result; Represents the image disparity gradient; Represents the gradient of the image edge probability map, where the subscripts i and j represent pixel coordinates; x and y represent pixel directions; Ii,j : Pixel point at position (i, j) of the image; are the first-order derivatives along the x and y directions respectively; For dynamic objects, the joint image segmentation network performs mask processing. According to Monodepth2, a binary mask is used to ignore objects that move synchronously with the camera. The mask is automatically calculated in the forward pass of the network: Where ω: binary mask; L p (I k ,I k+1 ): The image reconstruction loss function mentioned in formula (3) in the main text; They represent the k-th image, the composite image of the k-th image, and the k+1-th image respectively; they are fine-tuned by jointly training the synthetic optical flow and the error of the RAFT optical flow network: L f : Joint fine-tuning loss function results; flow : Common effective pixel area of ​​image optical flow field; F R (p): RAFT network optical flow prediction result; F syn (p): Optical flow prediction results of synthetic optical flow network; During training, in order to obtain structurally consistent depth prediction, D k+1 With D k Align and calculate the depth consistency loss: Among them, L dc : Depth consistency loss function result; D k (p): depth of the kth image; The depth of the kth image calculated by matching the optical flow network; S is the common effective area of ​​optical flow and depth, so as to obtain a consistent depth estimate; To sum up, the overall loss function L of the network is: L=L p +λ s L s +λ f L f +λ dc L dc (8) Among them, λ s , f and λ dc Represents the weight of each loss, all losses are applied to the deep network and the optical flow network; The specific steps of step 2 are: (2-1) Consistency of forward and backward optical flow: Get the forward optical flow F from the optical flow network f and backward optical flow F b And calculate the forward and backward consistency d F =F f (p)+F b (p+F f (p)) 2 ; (2-2) Sparse point sampling: Divide the image into 10×10 grid regions and take d points in each region. F The first 20 groups of sparse matching points less than the threshold δ; The specific steps of step three are: (3-1) Model selection: Calculate the essential matrix and homography matrix, and then calculate the model score R F , R F =S F / (S F +S H ), S F , S H are the F and R model scores respectively; if R F >0.5, then select 2D-2D tracking mode; otherwise, select 3D-3D tracking mode; (3-2) Scale recovery: For 2D-2D tracking mode, the normalized camera motion is obtained from the essential matrix decomposition R represents the rotation matrix, represents the displacement vector, whose modulus is unit 1; then the scale is aligned using the triangulation method to restore the scale factor s, and we get represents the camera motion, represents the actual displacement length obtained by scale recovery; for the 3D-3D tracking mode, the ICP method is used to solve and obtain T k→k+1 =[R|t], [R|t] represents the movement of the camera, and t represents the camera displacement vector.

2. The design method of a monocular visual odometer based on unsupervised deep learning according to claim 1, characterized in that: In the step (3-1), inspired by the ORB-SLAM initialization method, two tracking methods, 2D-2D and 3D-3D, are considered; ORB-SLAM uses the model score method to only select the initialization model, and the tracking process uses the constant speed motion model and the PnP method to solve the motion trajectory. Since there are both 2D-2D and 3D-3D correspondences, only the model score R is used. F Select the tracking method; first solve the homography matrix H cr and the essential matrix F cr : Among them, p c is the matching point of the previous frame in two adjacent frames, p r is the matching point of the latter frame in two adjacent frames, and then calculates S for the H model and F model respectively H and S F Score: Where M is H or F, ρ M : The intermediate calculation result of the model score S; d 2 : represents the symmetric transfer error; Γ: equivalent to T H , is the invalid data exclusion threshold; The i′th matching point in the current frame; The i′th matching point of the reference frame; They are the symmetrical transfer errors between the current frame and the reference frame, and between the reference frame and the current frame; the error T M is the distance threshold, referring to ORB-SLAM, let T H =5.99,T F =3.84, Γ and T H The definitions are the same; When the 3D point cloud structure is degraded, the depth information with consistent scale is obtained through the deep network to avoid decomposing the homography matrix, and the SVD method is used to solve [R|t]: n: represents the number of feature matching points between two adjacent images; i′: represents the matching point sequence number; R: represents the camera rotation matrix; Represents the i′th matching point in the kth image.

Images