Monocular vision depth estimation method based on dense estimation network and visual odometer

Abstract

The invention discloses a monocular vision depth estimation method based on a dense estimation network and a visual odometer. The monocular vision depth estimation method specifically comprises the following steps: step 1, detecting, tracking and screening feature points; 2, rough motion is calculated through epipolar geometry, and part of feature points are recovered; 3, calculating sparse absolute depth in a triangulation manner; 4, constructing and predicting a dense relative deep network; 5, outliers are introduced, and inaccurate depth estimators are eliminated and reduced through a kernel function; and step 6, based on a logarithmic function, establishing a depth estimation alignment mapping relation, and estimating dense depth estimation. According to the method, the limited triangulation depth and the dense relative depth estimated by the depth network are combined, and logarithmic function mapping between the two is established, so that high-precision dense depth estimation is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot vision positioning methods, and particularly relates to a monocular vision depth estimation method based on a dense estimation network and visual odometry. Background Art

[0002] A monocular simultaneous localization and mapping system (vSLAM) is a technology widely used in fields such as unmanned driving, robot navigation, and augmented reality. Its advantages lie in low hardware cost, wide availability, and easy calibration, which make it particularly attractive for many mobile applications focusing on outdoor and indoor scenarios. Although the monocular vSLAM system performs well in many aspects, there are still some challenges, especially related to scale unobservability and state initialization problems. Existing solutions usually rely on introducing additional sensors, such as stereo cameras, RGB-D cameras, or inertial measurement units (IMUs), to reduce the difficulty of state estimation. However, on the one hand, these methods increase the hardware cost, and on the other hand, it is still difficult to achieve robust estimation in the face of degenerate motion and sparse texture scenes. The main reason is that in these cases, the system is difficult to effectively estimate the depth of field and usually cannot obtain absolute depth information.

[0003] Currently, algorithms for dense depth estimation are generally divided into several categories. Stereo-view-based methods use two or more cameras to capture images from different perspectives and estimate depth through disparity calculation. Structure-from-motion methods attempt to recover the geometry of the scene based on a sequence of images captured by a moving camera. However, due to the unknown relative pose between cameras, depth recovery on an absolute scale is challenging. In addition, although vSLAM systems are used to track camera motion and build maps, they usually can only track hundreds to thousands of sparse feature points, which limits the density of depth information. In recent years, with the progress of deep learning technology, deep learning-based depth estimation methods have made significant progress. These methods perform well in providing relatively high-precision depth estimation and have been used to enhance the robustness of structured light measurement and other applications. However, most existing methods mainly focus on self-supervised monocular camera absolute depth estimation. Due to factors such as scene differences and hardware differences, these solutions still face difficulties in training, poor generalization ability, and insufficient accuracy. In addition, the presence of dynamic objects can also interfere with the depth estimation system and reduce performance. Summary of the Invention

[0004] The purpose of the present invention is to provide a monocular vision depth estimation method based on a dense estimation network and visual odometry, which solves the problem of difficult training of existing absolute depth estimation networks.

[0005] The technical solution adopted by the present invention is a monocular vision depth estimation method based on a dense estimation network and visual odometry, which specifically follows the following steps:

[0006] Step 1: Feature point detection, tracking, and screening;

[0007] Step 2: Calculate the rough motion using epipolar geometry and recover some feature points;

[0008] Step 3: Triangulate to calculate the sparse absolute depth;

[0009] Step 4: Construct and predict the dense relative depth network;

[0010] Step 5: Introduce outliers and kernel functions to reduce inaccurate depth estimates;

[0011] Step 6: Based on the logarithmic function, establish the depth estimation alignment mapping relationship and estimate the dense depth estimate.

[0012] The technical feature of the present invention also lies in that,

[0013] Specifically, Step 1 is as follows: First, use the corner detection algorithm to detect feature points in the image sequence input to the system; subsequently, effectively track the feature points in consecutive frames through the optical flow tracking method; calculate the disparity between the previous published frame and each frame in the consecutive frame sequence, and set a specific disparity threshold. When the disparity is greater than the set threshold, determine the current published frame.

[0014] Use the semantic segmentation algorithm to calculate the semantic information of the previous published frame and the current published frame, define the feature points within a specific semantic category range as potential dynamic feature points, and define other feature points as static feature points, that is, relatively stable feature points.

[0015] In Step 2, the feature point recovery means that first, use the stable feature points determined in Step 1 to calculate the preliminary pose change ψ, further calculate the essential matrix using the preliminary pose change, then calculate the first-order geometric error of the potential dynamic feature points using the essential matrix, and exclude the feature points with large geometric errors by setting a threshold, that is, exclude a certain number of feature points that do not conform to the model motion determined by the static feature points, so as to recover some feature points located on static vehicles, etc., thereby increasing the number of available feature points and improving the accuracy and robustness of the system pose estimation. Finally, input the static feature points and the recovered feature points into the epipolar geometry module to solve the pose change between two frames again.

[0016] In Step 2, use epipolar geometry to solve the optimization problem to estimate the motion of the system between two published frames:

[0017]

[0018] In Equation (1), ψ = (R, t) represents the attitude rotation matrix R and the three-dimensional position change t estimated by the system; and represents the published frame I 1and frame I 2 feature point pairs in; d represents the first-order geometric error, expressed as:

[0019]

[0020] In the formula, F represents the essential matrix, which is determined by the internal parameters K of the camera, the attitude rotation matrix R estimated by the system, and the three-dimensional position change t:

[0021] F(ψ) = K -T t ∧ RK -1 (3).

[0022] In step 4, dense prediction logically divides the network into two parts: an encoder and a decoder. The encoder is usually based on an image classification network and is pre-trained on a large dataset. The decoder is usually used to aggregate features from the encoder and transform them into the final dense prediction. In the study of dense prediction architectures, the focus usually lies on the decoder and the feature aggregation strategy.

[0023] In step 5, outlier detection is specifically as follows: According to the predicted semantic categories, perform connectivity detection to obtain the image segmentation result. Then, for image segmentation instances with regions larger than a certain predetermined threshold, extract the set of depth estimation values within the region. Next, by applying the "3σ" criterion, eliminate the outliers in these relative depth estimations;

[0024] For outliers in the absolute depth calculated by triangulation, a Cauchy robust kernel function is introduced to reduce the influence of these outlier depth points on system alignment:

[0025]

[0026] In the formula, where s is an adjustable parameter and res represents the residual. By applying a loss function to all residuals, the weight of the outlier terms can be reduced. Then, by applying a loss function to all residuals, large outliers will have their weights reduced and will not overly affect the optimization problem.

[0027] In step 6, for image I, based on the depth estimation network obtain the dense relative depth measure

[0028]

[0029] Use the logarithmic function to align the relative dense depth estimated in step 4 with the triangulated absolute sparse depth z 0 to achieve dense depth estimation:

[0030]

[0031] where z 0 and n 0 are parameters to be fitted; z 0 represents the translation component, and n 0 represents the attenuation coefficient; the LM algorithm is used to solve the above optimization problem. The maximum a posteriori estimation is calculated by minimizing the sum of the squares of the depth residuals, and the Cauchy robust kernel is used to handle these outliers. Let X=(z 0 , n 0 ), then we have:

[0032]

[0033] The second-order Taylor function is as follows:

[0034]

[0035] where J and μ represent the Jacobi matrix and the damping term respectively; E represents the identity matrix, and the k-th iteration can be expressed as:

[0036]

[0037] Finally, the quantity to be estimated X=(z 0 , n 0 ) is solved by iteration; then the dense depth estimated by the system can be calculated by the following formula:

[0038]

[0039] The beneficial effects of the present invention are that

[0040] the present invention proposes a relative depth estimation network based on the Transformer architecture to solve the problems of difficult training, poor generalization ability and insufficient accuracy of the absolute depth estimation network; in order to solve the scale ambiguity of monocular vSLAM, a logarithmic function mapping between these two depths is further established by using the limited triangulation depth and the dense relative depth estimated by the depth network to perform dense depth estimation; in order to reduce the influence of dynamic feature points or incorrect feature points on triangulation, the present invention combines semantic information to eliminate dynamic feature points, combines outlier detection and kernel functions to reduce the adverse effects of outliers on dense depth alignment, and realizes a dense absolute depth estimation method, introducing the depth of field information of deep learning into the traditional visual inertial odometer to make up for its weak links. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is the flow chart of the present invention;

[0042] Figure 2 is the result diagram of the dense relative depth prediction of the present invention;

[0043] Figure 3 It is a result graph of the present invention modeling the relationship between sparse absolute depth and dense relative depth through linear fitting and logarithmic fitting;

[0044] Figure 4 It is a result graph of the sparse absolute depth and the dense absolute depth after alignment of the present invention. Detailed implementation manners

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0046] The monocular vision depth estimation method of the present invention based on a dense estimation network and visual odometry is as Figure 1 shown, and specifically follows the following steps:

[0047] Step 1: Feature point detection, tracking and screening;

[0048] Specifically, Step 1 is as follows: First, use a corner detection algorithm to detect feature points in the input image sequence of the system; subsequently, effectively track the feature points in consecutive frames through an optical flow tracking method; calculate the disparity between the previous published frame and each frame in the consecutive frame sequence, and set a specific disparity threshold. When the disparity is greater than the set threshold, determine the current published frame;

[0049] Use a semantic segmentation algorithm to calculate the semantic information of the previous published frame and the current published frame, define the feature points within a specific semantic category (such as cars, cyclists, pedestrians, etc.) as potential dynamic feature points, and define other feature points as static feature points, that is, relatively stable feature points. The screened static feature points are used as the input for the rough motion estimation of the system to reduce the influence of dynamic features on the vSLAM system;

[0050] Step 2: Calculate the rough motion using epipolar geometry and recover some feature points;

[0051] In Step 2, the recovery of feature points means that first, use the stable feature points determined in Step 1 to calculate the preliminary pose change ψ, further use the preliminary pose change to calculate the essential matrix, then use the essential matrix to calculate the first-order geometric error of the potential dynamic feature points, and exclude the feature points with large geometric errors by setting a threshold, that is, exclude a certain number of feature points that do not conform to the model motion determined by the static feature points, so as to recover some feature points located on static vehicles, etc., thereby increasing the number of available feature points, thereby improving the accuracy and robustness of the system pose estimation. Finally, input the static feature points and the recovered feature points into the epipolar geometry module to solve the pose change between two frames again;

[0052] In Step 2, use epipolar geometry to solve the optimization problem to estimate the motion of the system between two published frames:

[0053]

[0054] In Equation (1), ψ=(R, t) represents the attitude rotation matrix R estimated by the system and the three-dimensional position change t; and represents the published frame I 1 and the feature point pair in frame I 2 ; d represents the first-order geometric error, expressed as:

[0055]

[0056] In the formula, F represents the essential matrix, which is determined by the internal parameter K of the camera, the attitude rotation matrix R estimated by the system, and the three-dimensional position change t:

[0057] F(ψ)=K -T t ∧ RK -1 (3);

[0058] Step 3: Triangulate to calculate the sparse absolute depth;

[0059] Step 4: Construct and predict the dense relative depth network;

[0060] In Step 4, the dense prediction logically divides the network into an encoder and a decoder. The encoder is usually based on an image classification network and is pre-trained on a large dataset. The decoder is usually used to aggregate the features from the encoder and transform them into the final dense prediction. In the research of the dense prediction architecture, the focus usually lies on the decoder and the feature aggregation strategy;

[0061] Step 5: Introduce outliers and kernel functions to reduce inaccurate depth estimates;

[0062] The outlier detection in Step 5 is specifically as follows: According to the predicted semantic category, perform connectivity detection to obtain the image segmentation result. Then, for the image segmentation instances in the regions with a value greater than a certain predetermined threshold, extract the set of depth estimates within the region. Then, by applying the "3σ" criterion, eliminate the outliers in these relative depth estimates;

[0063] For the outliers of the absolute depth calculated by the triangulation method, a Cauchy robust kernel function is introduced to reduce the influence of these outlier depth points on the system alignment:

[0064]

[0065] In the formula, where s is an adjustable parameter and res represents the residual. By applying the loss function to all residuals, the weight of the outlier term can be reduced. Then, by applying the loss function to all residuals, large outliers will have their weights reduced and will not overly affect the optimization problem;

[0066] Step 6: Based on the logarithmic function, establish a depth estimation alignment mapping relationship to estimate the dense depth estimation;

[0067] In step 6, for the image I, based on the depth estimation network Obtain the dense relative depth

[0068]

[0069] Figure 3 The results of linear fitting and logarithmic fitting for modeling the relationship between sparse absolute depth and dense relative depth are given. The results show that the logarithmic function fitting result is better than the linear function fitting result. Use the logarithmic function to transform the relative dense depth estimated in step 4 Align with the triangular absolute sparse depth z 0 To achieve dense depth estimation:

[0070]

[0071] In the formula, where z 0 and n 0 Are the parameters to be fitted; z 0 Represents the translation component, and n 0 Represents the attenuation coefficient; Use the LM algorithm to solve the above optimization problem. Minimize the sum of the squares of the depth residuals to calculate the maximum a posteriori estimate, and use the Cauchy robust kernel to handle these outliers. Let X = (z 0 , n 0 ), then there is:

[0072]

[0073] The second-order Taylor function is as follows:

[0074]

[0075] In the formula, J and μ represent the Jacobi matrix and the damping term respectively; E represents the identity matrix. Then the k-th iteration can be expressed as:

[0076]

[0077] Finally, iteratively solve the quantity to be estimated X = (z 0 , n 0 ); Then the dense depth estimated by the system Can be calculated by the following formula:

[0078]

[0079] The results of the sparse absolute depth and the aligned dense absolute depth are as Figure 4 Shown.

[0080] Example 1

[0081] In step 3 of the present invention, triangulation is a method for calculating the three-dimensional coordinates of feature points or object positions observed from multiple different perspectives. By matching feature points, determining the intersection points of lines of sight or light rays, and considering the baseline and distance measurements, the exact position of the target can be determined. This method is widely used in the fields of computer vision and measurement for three-dimensional reconstruction and position measurement. The present invention relates to the calculation of sparse absolute depth implemented by a triangulation algorithm. By utilizing the known camera parameters and solving for the pose changes, the absolute depth of points in a specific scene or object is estimated.

[0082] To overcome the problem of scale ambiguity, the present invention introduces Inertial Measurement Unit (IMU) data. Given a series of RGB images with synchronized IMU data, we run visual-inertial odometry to calculate the camera trajectory and generate the 3D world coordinates of a set of feature points that are continuously tracked throughout the image sequence.

[0083] In an environment with reasonable texture, each frame of the image usually contains multiple tracked features. By projecting the 3D coordinates of these feature points into the image space, we obtain a series of sparse feature points containing metric depth values. These sparse depth feature points will be used as input data for subsequent alignment tasks.

[0084] Example 2

[0085] In step 4 of the present invention, the backbone of the convolutional neural network usually downsamples the input image step by step in order to extract features at multiple scales. Downsampling helps to gradually expand the receptive field, convert low-level features into abstract high-level features, and at the same time ensure that the memory and computational requirements of the network remain within a controllable range. However, downsampling faces the prominent problem of loss of feature resolution and details in the deep part of the network, making it difficult to effectively recover in the decoder.

[0086] Therefore, the present invention uses a Transformer as the basic computational unit of the encoder and constructs a backbone structure that gradually combines feature representations using a convolutional decoder, ultimately achieving dense prediction. This dense depth estimation network based on the Transformer mechanism has significant advantages, especially in terms of feature retention. Figure 2 The results of the dense relative depth prediction proposed by the present invention are given.

[0087] The present invention relates to monocular relative depth estimation, which is generally regarded as a dense regression problem. Specifically, the present invention utilizes the image sequences and laser point cloud data available in the public dataset for the training of the network. This is achieved by mapping the point cloud data to the image coordinate system and converting the depth information into inverse depth as the training objective of the network.

[0088] One of the reasons for using inverse depth as the regression target is to reduce the influence of scale ambiguity. Under different datasets, sequences, and hardware settings, depth estimation may be affected by scale differences, resulting in difficulties in training and generalization. By representing depth as inverse depth, these scale differences can be circumvented to a certain extent, reducing the difficulty of training and improving the generalization ability of the depth estimation model.

[0089] However, using inverse depth also introduces the problem of lack of absolute scale in depth estimation. In subsequent steps, this problem will need to be solved in order to recover absolute scale information in depth estimation to meet the requirements of specific application fields.

[0090] Embodiment 3

[0091] In step 5 of the present invention, detecting outliers is of great significance because it can selectively remove those data points that may cause errors in depth estimation, thereby improving the accuracy of overall depth estimation and ultimately reducing the impact on the final system alignment.

[0092] The present invention adopts different outlier detection strategies, aiming to improve the accuracy of the network in predicting dense depth while reducing the potential impact of outliers on system alignment: (1) using "3σ" to eliminate the outlier relative depth estimation; (2) using "CauchLoss" for the sum of squared residuals to reduce the impact of outlier absolute depth on the results.

[0093] The monocular vision depth estimation method of the present invention based on a dense estimation network and visual odometry establishes a logarithmic function mapping between the two by combining the limited triangulation depth and the dense relative depth estimated by the depth network to achieve high-precision dense depth estimation. It also utilizes semantic information to assist in identifying and removing the feature points that may introduce interference to improve the accuracy of sparse absolute depth estimation, thereby further enhancing the accuracy of dense depth estimation. It also introduces a kernel function and an outlier depth elimination method to further improve the alignment accuracy of dense depth estimation. It improves the problem of system performance degradation caused by difficult depth estimation under conditions such as texture loss and occlusion in monocular visual odometry.

Claims

1. A monocular visual depth estimation method based on a dense estimation network and visual odometry, characterized in that, it is specifically implemented according to the following steps: Step 1: Feature point detection, tracking and screening; Step 2: Calculate the rough motion using epipolar geometry and recover some feature points; Step 3: Triangulate to calculate the sparse absolute depth; Step 4: Construct and predict a dense relative depth network; Step 5: Introduce outliers and kernel functions to reduce inaccurate depth estimation quantities; Step 6: Based on the logarithmic function, establish a depth estimation alignment mapping relationship to estimate the dense depth estimation.

2. The monocular visual depth estimation method based on a dense estimation network and visual odometry according to claim 1, characterized in that, the specific content of step 1 is: First, use a corner detection algorithm to detect feature points in the image sequence input to the system; Subsequently, effectively track the feature points in consecutive frames through an optical flow tracking method; Calculate the disparity between the previous published frame and each frame in the consecutive frame sequence, and set a specific disparity threshold. When the disparity is greater than the set threshold, determine the current published frame.

3. The monocular visual depth estimation method based on a dense estimation network and visual odometry according to claim 2, characterized in that, Use a semantic segmentation algorithm to calculate the semantic information of the previous published frame and the current published frame, define the feature points within a specific semantic category range as potential dynamic feature points, and define other feature points as static feature points, that is, relatively stable feature points.

4. The monocular visual depth estimation method based on a dense estimation network and visual odometry according to claim 1, characterized in that, The feature point recovery in step 2 means that first, use the stable feature points determined in step 1 to calculate the preliminary pose change ψ, further use the preliminary pose change to calculate the essential matrix, then use the essential matrix to calculate the first-order geometric error of the potential dynamic feature points, and exclude the feature points with larger geometric errors by setting a threshold, that is, exclude a certain number of feature points that do not match the model motion determined by the static feature points, so as to recover some feature points located on static vehicles, etc., thereby increasing the number of available feature points, thereby improving the accuracy and robustness of the system pose estimation. Finally, input the static feature points and the recovered feature points into the epipolar geometry module to solve the pose change between two frames again.

5. The monocular visual depth estimation method based on a dense estimation network and visual odometry according to claim 4, characterized in that, In step 2, use epipolar geometry to solve the optimization problem to estimate the motion of the system between two published frames: In Equation (1), ψ = (R, t) represents the attitude rotation matrix R estimated by the system and the three-dimensional position change t; and represents the published frame I 1 and the feature point pair in frame I 2 ; d represents the first-order geometric error, expressed as: In the formula, F represents the essential matrix, which is determined by the camera's internal parameter K and the estimated attitude rotation matrix R and the three-dimensional position change t of the system: F(ψ) = K -T t ∧ RK -1 (3).

6. The monocular visual depth estimation method based on a dense estimation network and visual odometry according to claim 1, characterized in that, In step 4, the dense prediction logically divides the network into two parts: an encoder and a decoder. The encoder is usually based on an image classification network and is pre-trained on a large dataset. The decoder is usually used to aggregate features from the encoder and transform them into the final dense prediction. In the research of dense prediction architectures, the focus usually lies on the decoder and the feature aggregation strategy.

7. The monocular visual depth estimation method based on a dense estimation network and visual odometry according to claim 1, characterized in that in step 5, the outlier detection is specifically as follows: according to the predicted semantic class, perform connectivity detection to obtain the image segmentation result, and then for the image segmentation instance of the region with an area greater than a certain predetermined threshold, extract the set of depth estimation values within this region. Then, by applying the "3σ" criterion, remove the outliers in these relative depth estimations; for the outliers of the absolute depth calculated by triangulation, a Cauchy robust kernel function is introduced to reduce the influence of these outlier depth points on the system alignment: where s is an adjustable parameter, res represents the residual. By applying a loss function to all residuals, the weight of the abnormal term can be reduced. Then, by performing a loss function on all residuals, large outliers will have their weights reduced and will not overly affect the optimization problem.

8. The monocular visual depth estimation method based on a dense estimation network and visual odometry according to claim 1, characterized in that In step 6, for image I, based on the depth estimation network obtain a dense relative depth measure Use the logarithmic function to align the relative dense depth estimated in step 4 with the triangular absolute sparse depth z 0 to achieve dense depth estimation: where z 0 and n 0 are parameters to be fitted; z 0 represents the translation component, and n 0 represents the attenuation coefficient; the LM algorithm is used to solve the above optimization problem, minimizing the sum of the squares of the depth residuals to calculate the maximum a posteriori estimate, and a Cauchy robust kernel is used to handle these outliers. Let X = (z 0 , n 0 ), then we have: the second-order Taylor function is as follows: where J and μ represent the Jacobi matrix and the damping term respectively; E represents the identity matrix, and the k-th iteration can be expressed as: Finally, iteratively solve for the quantity to be estimated X = (z 0 , n 0 ); then the dense depth estimated by the system can be calculated by the following formula:

Images