A Visual Odometry Method and System Based on Image Depth Prediction and Monocular Geometry

Abstract

The present invention discloses a visual odometry method and system based on image depth prediction and monocular geometry. The method includes inputting two consecutive image frames, detecting and describing local features of the images using the Scale-Invariant Feature Transform (SIFT) algorithm, and then using the Fast Library for Approximate Nearest Neighbors (FLANN) algorithm to match corresponding feature point pairs between the two frames; solving for the essential matrix using epipolar geometry constraints to obtain the relative pose transformation of the camera; constructing and training a monocular depth prediction model to predict dense depth information for each input image frame; if the number of valid depth information pairs formed by two consecutive image frames is greater than a given threshold, using triangulation to estimate the scale factor to obtain the corrected relative pose transformation, otherwise using a combination of the Perspective-n-Point (PnP) projection algorithm, the Random Sample Consensus (RANSAC) algorithm, and local non-linear optimization to solve for the absolute pose transformation. The present invention effectively integrates the advantages of deep learning and traditional geometric methods, can adapt to dynamic environments, and improves the robustness and accuracy of monocular visual odometry.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and robot navigation, and particularly relates to a visual odometry method and system based on image depth prediction and monocular geometry. Background Art

[0002] Visual odometry is the process of estimating the motion trajectory of a robot based on an input image sequence and is one of the core modules of a simultaneous localization and mapping (SLAM) system. Although traditional geometric visual odometry methods based on feature point matching have high accuracy in controlled environments, they are prone to failure in complex scenarios with poor texture, dynamic objects, and lighting changes. In recent years, although end-to-end visual odometry methods based on deep learning have made some progress, they often lack geometric constraints, and their accuracy and robustness are not satisfactory. Existing hybrid visual odometry methods integrate classical geometric models and deep learning frameworks, but there is still room for improvement in dealing with complex scenarios such as high dynamics and low texture. Summary of the Invention

[0003] Object of the Invention: The present invention proposes a visual odometry method and system based on image depth prediction and monocular geometry to overcome the defects of the prior art and improve the robustness and accuracy of monocular visual odometry in dynamic environments.

[0004] Technical Solution: A visual odometry method based on image depth prediction and monocular geometry according to the present invention:

[0005] (1) Input two consecutive image frames, use the scale-invariant feature transform algorithm to detect and describe local features of the images, and then use the FLANN algorithm to match corresponding feature point pairs (F i , F j ) between the two frames of images;

[0006] (2) For the feature matching relationship obtained in step (1), use the epipolar geometry constraint to solve the essential matrix E to obtain the relative pose transformation [R, t] of the camera, where R is the rotation matrix and t is the translation vector;

[0007] (3) Construct and train a monocular depth prediction model to predict dense depth information for each input image frame;

[0008] (4) If the number of valid depth information pairs formed by two consecutive image frames is greater than a given threshold, use triangulation to estimate the scale factor s to obtain the corrected relative pose transformation [R, st]; otherwise, execute step (5);

[0009] (5) Use a combination of the perspective-n-point projection algorithm, the RANSAC algorithm, and local non-linear optimization to solve the absolute pose transformation [R, t].

[0010] Furthermore, the implementation process of step (1) is as follows:

[0011] Through Difference of Gaussian (DoG) kernel detection, compare pixel values in different scale spaces using the following formula, and find the extreme points as candidate key points:

[0012] D(x, y, σ) = (G(x, y, kσ) - G(x, y, σ)) * I(x, y)

[0013]

[0014] where x and y are the spatial coordinates of the image, I(x, y) represents the pixel value of the input image at point (x, y), k represents the multiple relationship between adjacent scales, and G(x, y, σ) is the Gaussian kernel at scale σ, which is used to generate different scale spaces of the input image I(x, y);

[0015] Use spline interpolation to perform a three-dimensional quadratic function fitting on the key points, and then find the true extreme points of the fitting function. Then, remove the points with low contrast using the following formula:

[0016]

[0017] where is the extreme point of the interpolation function, discard points less than a given threshold; when determining the main direction of each key point, obtain the main direction by statistically calculating the gradient histogram to make the feature rotation-invariant; finally, generate a key point descriptor. Taking each key point as the center, calculate the gradient direction histogram within its neighborhood to form a 128-dimensional vector as the feature vector representation of this key point; after obtaining the feature vectors of two images, use the FLANN algorithm to match the corresponding feature point pairs (F i , f j ).

[0018] Furthermore, step (2) is implemented through the following formula:

[0019]

[0020] where E is the essential matrix, K is the camera internal parameter which is a known parameter, and the relative pose transformation [R, t] of the camera is obtained by decomposing E.

[0021] Furthermore, the implementation process of step (3) is as follows:

[0022] Given the predicted depth D t and the camera pose matrix T t+1→t , reconstruct the image I t+1→t :

[0023] It+1→t = w(I t+1 , KT t+1→ tK -1 xD t [x])

[0024] where w(·) is a differentiable image warping function, and x represents the pixel value of a certain point in the image;

[0025] Using the generated I t+1→t and the reference image I t , construct the following objective function:

[0026] pe(I t+1→t , I t ) = αSSIM(I t+1→t , I t ) + (1 - α)||I t+1→t - I t ||1

[0027] where SSIM is the structural similarity index, and α is the weight for balancing the SSIM loss and the L1 loss;

[0028] For the reference view I t , according to its neighboring views I t-1 , I t+1 reconstruct the images I t+1→t , I t-1→t ; only calculate the photometric error between the smallest pair of reference pixels and synthesized pixels in the view:

[0029] L p = min(pe(I t , I t-1→t ), pe(I t , I t+1→t ))

[0030] And introduce an edge-aware depth smoothing term L s :

[0031]

[0032] where and calculate the gradient magnitudes of the depth map D t in the horizontal and vertical directions respectively, promoting the depth values of adjacent pixels to be close; and are weight terms based on the image gradient of the current view; in the edge region where the image gradient is large, this weight term is small, allowing discontinuities in the depth map; while in the flat region where the image gradient is small, this weight term is large, promoting the depth map to remain smooth; the final loss function is:

[0033] L = L p + λL s

[0034] The monocular depth prediction model is implemented and trained using PyTorch.

[0035] Furthermore, the implementation process of step (4) is as follows:

[0036] For a given pair of two consecutive image frames I i and I j of corresponding feature point pairs (F i , F j ), after normalization, we get x i and x j . Then, the three-dimensional point cloud X j corresponding to the second view is reconstructed by triangulation:

[0037]

[0038] where e3 is the vector [0, 0, 1] T ;

[0039] Project X j back onto the image plane to obtain depth information based on triangulation Then use the monocular depth prediction model to estimate the depth information D j of image I j . Subsequently, apply the outlier mask M d to the depth information D j to obtain the processed depth data Search for the depth information pairs in . If the number of depth information pairs is greater than a given threshold, calculate the depth ratio The RANSAC algorithm is introduced to perform robust fitting on the noisy data:

[0040]

[0041] |r i - s| ≤ ∈1

[0042] where ∈1 is the residual threshold and s is the scale factor.

[0043] Furthermore, the implementation process of step (5) is as follows:

[0044] Use the monocular depth prediction model to estimate the depth information D i of the current view I i , then apply the outlier mask M i to D d , and obtain the processed depth data Finally, using the camera internal parameters and Calculate the view I by back-projection i The corresponding 3D point cloud coordinates X i ; From all N pairs of correspondences between 3D space points and 2D pixels, use RANSAC to randomly sample the minimum internal point set and calculate the initial solution R0,t0 of the pose:

[0045]

[0046] Among them, ρ is the robust kernel function, F j Reference view I j Corresponding feature points; π is the projection function; starting from the initial solution R0, t0, the local nonlinear optimization method is used to optimize the pose solution {R * ,t *}:

[0047]

[0048] According to the optimized pose solution, the reprojection errors of all N pairs of points are calculated to determine the internal point set:

[0049] I={i||F j -π(R * ,t * ,X i )||<∈2} (3)

[0050] Among them, ∈2 is the inlier threshold; repeat the above process until the maximum number of iterations is reached or the number of inliers no longer increases, thus obtaining the optimal solution [R, t].

[0051] The present invention provides a visual odometer system based on image depth prediction and monocular geometry, comprising:

[0052] Feature detection and matching module, used to detect key points of input images and match corresponding feature points between consecutive image frames;

[0053] A 2D attitude solver, used for estimating a 2D attitude transformation of the camera based on the correspondence relationship between the feature point pairs;

[0054] Monocular depth prediction, used to predict a dense depth map for each input image frame;

[0055] A scale factor correction module, used to correct the scale factor of the 2D posture transformation using the depth information output by the depth prediction network;

[0056] A 3D attitude solver is used to directly solve the 3D attitude of the camera based on the corresponding relationship between the 3D coordinates of the feature point pairs and the 2D pixel coordinates when the scale factor correction module cannot operate.

[0057] Beneficial effects: The present invention combines the advantages of traditional geometric methods and the latest deep learning technologies, and not only theoretically solves the long-standing challenges in this field such as scale estimation and dynamic environment adaptation, but also demonstrates excellent performance in practical applications. First, the present invention combines classic algorithms for feature detection and geometric constraints, which can avoid the influence of complex environmental scenes such as lack of texture, dynamic objects, and illumination changes. At the same time, a 2D pose solving strategy based on the magnitude of optical flow is introduced to further improve the adaptability to dynamic scenes. Second, through the proposed novel scale factor correction method, the dense depth information output by the depth prediction network can be utilized, and the inherent scale uncertainty of the monocular visual odometer can be robustly estimated based on the RANSAC algorithm, avoiding long-term trajectory drift. Moreover, the monocular depth prediction network of the system adopts a self-supervised training method, which does not rely on additional ground truth annotation data. Even when trained only on a conventional dataset, it also demonstrates excellent generalization performance and can operate stably in extreme scenarios. Brief description of the drawings

[0058] Figure 1 is the overall framework schematic diagram of the visual odometer system of the present invention;

[0059] Figure 2 is the calculation schematic diagram of the flying-out mask of the present invention;

[0060] Figure 3 is the experimental result comparison diagram when different sequences in the KITTI dataset are used as verification data; among them, Figures (a) to (k) are the experimental result comparison diagrams when sequences 00 to 10 in the KITTI dataset are used as verification data respectively. Detailed implementation manners

[0061] The present invention will be further described in detail below with reference to the drawings.

[0062] As Figure 1 shown, the present invention proposes a visual odometer method based on image depth prediction and monocular geometry, including the following steps:

[0063] Step 1: Run the feature detection and matching module to match corresponding feature points between consecutive image frames. The specific steps are as follows:

[0064] Input two consecutive image frames, detect and describe the local features of the images using the scale-invariant feature transform algorithm, and then use the FLANN algorithm to match the corresponding feature point pairs (F i , F j ).

[0065] The purpose of scale - space extremum detection is to find key points in different scale spaces. It is detected through the Difference of Gaussians (DoG) kernel. By using the following formula to compare pixel values in different scale spaces, the extremum points are found as candidate key points:

[0066] D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)

[0067]

[0068] The cubic spline interpolation method is used to fit the key points with a three - dimensional quadratic function, and then the true extremum points (key points) of the fitting function are found. Then, the points with low contrast are removed, using the following formula:

[0069]

[0070] where, is the extremum point of the interpolation function, and the points less than the given threshold are discarded; when determining the main direction of each key point, the main direction is obtained by statistically calculating the gradient histogram, making the features rotation - invariant; finally, a key - point descriptor is generated. Centered on each key point, the gradient - direction histogram within its neighborhood is calculated to form a 128 - dimensional vector, which is used as the feature - vector representation of the key point; after obtaining the feature vectors of two images, the FLANN algorithm is used to match the corresponding feature - point pairs (F i ,F j ).

[0071] Step 2: Based on the 2D pose solver, the essential matrix E is solved for the epipolar - geometry constraint, thereby obtaining the relative pose transformation [R,t] of the cameras, which satisfies:

[0072]

[0073] where R is the rotation matrix, t is the translation vector, E is the essential matrix, K is the camera intrinsic parameter which is a known parameter, and F i ,F j are the feature - matching points obtained in step (1). Therefore, the relative pose transformation [R,t] of the cameras can be obtained by decomposing E. This solver monitors the optical - flow magnitude between consecutive frames in real - time and only decomposes the essential matrix to obtain the pose solution when the magnitude is large enough, which can avoid interference from small - scale dynamic points.

[0074] Step 3: Build a monocular depth - prediction network to predict a dense depth map for each input image frame. First, the monocular depth model needs to be trained. Given the predicted depth D t and the camera - pose matrix T t+1→t , the image I t+1→t can be reconstructed as follows:

[0075] I t+1→t = w(I t+1 , KT t+1→ tK -1 xD t [x])

[0076] where w(·) is a differentiable image warping function, and x represents the pixel value of a point in the image.

[0077] Using the generated I t+1→t and the reference image I t , the following objective function can be constructed:

[0078] pe(I t+1→t , I t ) = αSSIM(I t+1→t , I t ) + (1 - α)||I t+1→t - I t ||1

[0079] where SSIM is the structural similarity index, which is a metric for measuring the similarity between two images, and α = 0.85 is used to balance the weights of the SSIM loss and the L1 loss.

[0080] Specifically, as Figure 2 shown, for the reference view I t , according to its neighboring views I t-1 , I t+1 the images I t+1→t , I t-1→t are reconstructed. The present invention does not average the photometric errors between the reference pixels and the synthesized pixels in multiple views, but only calculates the photometric error between the pair of reference pixels and synthesized pixels with the smallest error:

[0081] L p = min(pe(I t , I t-1→t ), pe(I t , I t+1→t ))

[0082] Finally, an edge-aware depth smoothing term L s is introduced:

[0083]

[0084] where and calculate the gradient magnitudes of the depth map D t in the horizontal and vertical directions respectively, promoting the depth values of adjacent pixels to be close. and It is a weight term based on the gradient of the current view image. When the image gradient is large (such as in the edge region), this weight term is small, allowing discontinuities in the depth map; while in the flat region with a small image gradient, this weight term is large, promoting the smoothness of the depth map. The reason for introducing this depth smoothing term is that it is expected that the predicted depth map has segmentation in the object edge region and remains smooth in the object interior region, which conforms to the characteristics of the real scene. Through this loss term, the edge preservation and smoothness of depth prediction can be enhanced, thus generating a more reasonable depth estimation result.

[0085] The final loss function is: L = L p + λL s . The depth prediction model is implemented and trained using PyTorch. During the training process, the Adam optimization algorithm is used, with a total of 15 epochs of iteration, the learning rate is set to 0.0001, and the weight coefficient λ of the edge-aware depth smoothing regularization term is taken as 0.001. After the monocular depth model is trained, the monocular image is input into the model, and the output is the dense depth information of the current image.

[0086] Step 4: Based on the result of step (2), run the scale factor correction module. Using the dense depth information of the image, estimate the scale factor s using the triangulation method to obtain the corrected relative pose transformation [R, st]. The specific steps are as follows:

[0087] For a given pair of consecutive image frames I i and I j of corresponding feature point pairs (F i , F j ), after normalization, we get x i and x j . The three-dimensional point cloud X j corresponding to the second view can be reconstructed by triangulation:

[0088]

[0089] where e3 is the vector [0, 0, 1] T . Project X j back to the image plane to obtain the depth information based on triangulation Then use the monocular depth prediction model to estimate the depth information D j of image I j . Subsequently, apply the outlier mask M j to the depth information D d to obtain the processed depth data Search for and in the depth information pairs. If the number of depth information pairs is greater than a given threshold, calculate the depth ratio The RANSAC algorithm is introduced to perform robust fitting on the noisy data:

[0090]

[0091] |r i - s| ≤ ∈1

[0092] where ∈1 is an appropriate residual threshold. In this way, the scale factor estimate s of the visual odometer can be obtained. Based on step 2, the relative pose transformation [R, st] after scale correction can be obtained.

[0093] Step 5: Run the 3D pose solver, and use a combination of the perspective N - point projection algorithm, the RANSAC algorithm, and local non - linear optimization to accurately solve the absolute pose transformation [R, t].

[0094] First, use the monocular depth prediction model to estimate the depth information D i of the current view I i , then apply the outlier mask M i to D d to obtain the processed depth data Finally, use the camera intrinsic parameters and to calculate the three - dimensional point cloud coordinates X i corresponding to the view I i . Next, from all N pairs of three - dimensional space point and two - dimensional pixel point correspondences, use RANSAC to randomly sample the minimum inlier set and calculate the initial solution R0, t0

[0095]

[0096] where ρ is the robust kernel function, F j is the two - dimensional pixel coordinate of the reference view I j , X i is the three - dimensional space point of the current view I i , and π is the projection function. Starting from the initial solution R0, t0, use the local non - linear optimization method to optimize the pose solution {R * , t *}:

[0097]

[0098] Calculate the reprojection error of all N pairs of points according to the optimized pose solution to determine the inlier set:

[0099] I = {i | ||F j - π(R * , t * , X i )|| < ∈2}

[0100] Here, ∈2 is the inlier threshold. Repeat the above process until the maximum number of iterations is reached or the number of inliers no longer increases, thereby obtaining the optimal solution [R, t].

[0101] The present invention also proposes a visual odometry system based on image depth prediction and monocular geometry, including:

[0102] A feature detection and matching module, which is used to detect key points of the input image and match corresponding feature points between consecutive image frames;

[0103] A 2D pose solver, which is used to estimate the 2D pose transformation of the camera based on the corresponding relationship of the feature point pairs;

[0104] A monocular depth prediction network, which is used to predict a dense depth map for each frame of the input image;

[0105] A scale factor correction module, which is used to correct the scale factor of the 2D pose transformation by using the depth information output by the depth prediction network;

[0106] A 3D pose solver, which is used to directly solve the 3D pose of the camera based on the corresponding relationship between the 3D coordinates and 2D pixel coordinates of the feature point pairs when the scale factor correction module cannot operate.

[0107] In this embodiment, five evaluation metrics are used: average translation error (t_err), average rotation error (r_err), average translation error (ATE), relative pose error of translation (RPE(m)), and relative pose error of rotation (RPE(°)). The KITTI Odometry Dataset contains stereo camera and lidar data of 22 real urban scenes and provides accurate Ground Truth positions. As Figure 3 shown, where Figure 3 in (a) to (k) are respectively 11 sequences from sequence 00 to sequence 10 in the KITTI dataset as evaluation data. The visual odometry system proposed by the present invention is compared with various existing methods. The methods to be compared include geometry-based methods (VISO2 and ORB-SLAM2) and end-to-end deep learning methods, such as SC-SfM-Learner and Depth-VO-Feat, and DF-VO. The quantitative results are shown in Table 1. The best results are marked in bold underlined. It can be seen that the present invention is the best in four of the five metrics.

[0108] Table 1 Error comparison of the monocular visual odometry system and various methods in the KITTI dataset

[0109]

[0110]

[0111] In addition, from Figure 3 it can be seen that in the long-distance trajectory estimation without route loop (sequences 01, 02, 08), ORB-SLAM2 will have serious drift accumulation. Despite facing multiple challenges such as dynamic obstacle movement and drastic illumination changes, the present invention still demonstrates excellent trajectory estimation ability in these highly dynamic and complex scenarios, with the highest degree of coincidence with the ground truth trajectory.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual odometry method based on image depth prediction and monocular geometry, characterized in that, It includes the following steps: Step (1): Input two consecutive image frames, detect and describe local features of the images using the Scale-Invariant Feature Transform (SIFT) algorithm, and then use the Fast Library for Approximate Nearest Neighbors (FLANN) algorithm to match corresponding feature point pairs (F i , F j ) in the two frames of images; In step (2), for the feature matching relationship obtained in step (1), the essential matrix E is solved using the epipolar geometry constraint to obtain the relative pose transformation of the cameras. In step (3), a monocular depth prediction model is constructed and trained to predict dense depth information for each input image frame. In step (4), if the number of valid depth information pairs formed by two consecutive image frames is greater than a given threshold, the scale factor s is estimated using triangulation to obtain the corrected relative pose transformation [R, st], where R is the rotation matrix and t is the translation vector; otherwise, step (5) is executed. In step (5), the absolute pose transformation is solved by combining the perspective-n-point projection algorithm, the RANSAC algorithm, and local non-linear optimization. The said step (2) is implemented through the following formula: Among them, E is the essential matrix, K is the known camera intrinsic parameter, and the relative pose transformation [R, t] of the camera is obtained by decomposing E; (F i , F j ) is the feature point pair; The implementation process of the said step (3) is as follows: Given the predicted depth D t and the camera pose matrix T t+1→t , the image I is reconstructed t+1→t : I t+1→t = w(I t+1 , KT t+1→t K -1 xD t [x]) where w(·) is a differentiable image warping function, K is the camera internal parameter, and [x] represents the pixel value of a certain point in the image. Using the generated I t+1→t and the reference image I t , construct the following objective function: pe(I t+1→t ,I t ) = αSSIM(I t+1→t ,I t ) + (1 - α)||I t+1→t - I t ||1 where SSIM is the structural similarity index and α is the weight balancing the SSIM loss and the L1 loss. For reference view I t , according to its neighboring view I t-1 , I t+1 Image I is reconstructed t+1→t , I t-1→t ; Only calculate the photometric error between the smallest pair of reference pixels and synthetic pixels in the view: L p = min(pe(I t , I t-1→t ), pe(I t , I t+1→t )) and introduced an edge-aware depth smoothing term L s : Among them, and calculate the gradient magnitudes of the depth map D t in the horizontal and vertical directions respectively, to make the depth values of adjacent pixels close; and are weight terms based on the gradients of the current view image; in the edge regions with larger image gradients, this weight term is smaller, allowing discontinuities in the depth map; while in the flat regions with smaller image gradients, this weight term is larger, to make the depth map keep smooth; the final loss function is: L = L p + λL s The monocular depth prediction model is implemented and trained using PyTorch.

2. The visual odometry method based on image depth prediction and monocular geometry according to claim 1, wherein The implementation process of the said step (1) is as follows: Through Gaussian difference kernel detection, the pixel values are compared in different scale spaces using the following formula to find the extreme points as candidate key points: D(x, y, σ) = (G(x, y, kσ) - G(x, y, σ)) * I(x, y) where x and y are the spatial coordinates of the image, I(x, y) represents the pixel value of the input image at the point (x, y), k represents the multiple relationship between adjacent scales, and G(x, y, σ) is the Gaussian kernel at scale σ, which is used to generate different scale spaces of the input image I(x, y). Spline interpolation method is used to perform a three-dimensional quadratic function fitting on the key points, and then the true extreme points of the fitting function are found. Then, the points with low contrast are removed using the following formula: Among them, is the extreme point of the interpolation function and is discarded. Points less than the given threshold are discarded; when determining the main direction of each key point, the main direction is obtained by statistically calculating the gradient histogram so that the features have rotational invariance; finally, a key point descriptor is generated. Taking each key point as the center, the gradient direction histogram within its neighborhood is calculated to form a 128-dimensional vector as the feature vector representation of this key point; after obtaining the feature vectors of two images, the FLANN algorithm is used to match the corresponding feature point pairs (F i , F j ).

3. A visual odometry method based on image depth prediction and monocular geometry according to claim 1, characterized in that , The implementation process of the said step (4) is as follows: For two given consecutive image frames I i and I j of the corresponding feature point pairs (F i , F j ), after normalization, we obtain x i and x j . By triangulation, the three-dimensional point cloud x j corresponding to the reconstructed view is obtained as follows: Among them, e3 is the vector [0, 0, 1] T ; Project X j back onto the image plane to obtain depth information based on triangulation Then, use a monocular depth prediction model to estimate the depth information D j of the image I j . Subsequently, apply the outlier mask M j to the depth information D d to obtain the processed depth data Search for depth information pairs in . If the number of depth information pairs is greater than a given threshold, calculate the depth ratio The RANSAC algorithm is introduced for robust fitting of noisy data: |r i -s|≤∈1 where ∈1 is the residual threshold and s is the scale factor.

4. A visual odometry method based on image depth prediction and monocular geometry according to claim 1, characterized in that , The implementation process of the said step (5) is as follows: Use the monocular depth prediction model to estimate the current view I i The depth information D i , then D i Apply flyout mask M d , get the processed depth data Finally, using the camera internal parameters and Calculate the view I by back-projection i The corresponding 3D point cloud coordinates X i ; From all N pairs of correspondences between 3D space points and 2D pixels, use RANSAC to randomly sample the minimum internal point set and calculate the initial solution R0, t0 of the pose; Using the initial solution R0, t0 as the starting point, use the local nonlinear optimization method to optimize the pose solution {R * ,t * }; Calculate the reprojection errors of all N pairs of points based on the optimized pose solution and determine the internal point set: where ∈2 is the inlier threshold; repeat the above process until the maximum number of iterations is reached or the number of the inlier set no longer increases, thereby obtaining the final pose transformation.

5. A visual odometry system based on image depth prediction and monocular geometry using the method according to any one of claims 1 to 4, characterized in that, It includes: A feature detection and matching module, which is used to detect the key points of the input image and match the corresponding feature points between consecutive image frames; A 2D pose solver, which is used to estimate the 2D pose transformation of the camera based on the corresponding relationship of the feature point pairs; Monocular depth prediction, which is used to predict a dense depth map for each input image frame; A scale factor correction module, which is used to correct the scale factor of the 2D pose transformation using the depth information output by the depth prediction network; A 3D pose solver, which is used to directly solve the 3D pose of the camera based on the corresponding relationship between the 3D coordinates and 2D pixel coordinates of the feature point pairs when the scale factor correction module cannot run.

Images