Method and system for eliminating mismatching of visual odometer based on optical flow method

Abstract

The invention discloses a visual odometer mismatching elimination method and system based on an optical flow method, and the system comprises a preliminary mismatching elimination module which is used for eliminating image feature points located at the edge position in a first frame image through employing an edge elimination strategy, predicting the feature points of the first frame image in a second frame image through employing the optical flow method, and determining the mismatching of the first frame image; reversely predicting feature points in the first frame image by using an optical flow method according to the predicted feature points of the second frame image, calculating the Euclidean distance of a corresponding feature point pair on a pixel level by combining the feature points extracted from the first frame image, constructing a threshold value, and preliminarily screening mismatching; and the mismatching elimination module is used for further screening mismatching by utilizing the preliminary screening result and combining a random sampling consistency algorithm and basic matrix consistency check. The method is used for improving the positioning precision and robustness of the unmanned system based on the optical flow method visual odometer.

Description

Technical Field

[0001] The present invention relates to the field of automatic control technology, and in particular to a method and system for eliminating mismatches of a visual odometer based on an optical flow method. Background Art

[0002] Synchronous positioning and mapping technology plays an important role in the field of automatic control of unmanned systems, especially in intelligent navigation and perception positioning. Synchronous positioning and mapping technology is mainly composed of odometer module, optimization module and loop detection module. In the odometer module, based on the type of sensor used, it is mainly divided into two types: visual odometer based on camera and radar odometer based on lidar. Grayscale invariance refers to the fact that the grayscale value of each pixel in two consecutive frames is constant, which is the basic assumption of the optical flow method. Based on grayscale invariance, the optical flow method can predict the position of the feature points in the previous frame image in the next frame image, and then establish feature matching between the feature points of the two frames of images, and then preliminarily calculate the motion posture of the camera between the two frames of images. In actual engineering applications, the positioning of unmanned systems requires the known motion posture of the camera between consecutive image frames, and the accuracy of the camera's motion posture is based on the quality of feature matching. With the development of unmanned systems, positioning equipment is facing more complex environments such as more dynamic obstacles and weak textures, so higher requirements are placed on the quality of feature matching and the robustness of odometers established by visual odometers based on optical flow.

[0003] With the advancement of information technology and artificial intelligence, unmanned systems are developing rapidly. When facing complex environments and performing complex tasks, unmanned systems require higher positioning accuracy. In recent years, how to eliminate false matches to improve the feature matching quality of visual odometers based on optical flow and the robustness of odometers has received widespread attention. For example, dynamic obstacles are detected based on multi-view geometry algorithms [A. Kundu, M. Krishna and J. Sivaswamy. Moving object detection by multi-view geometric techniques from a single camera mounted robot, 2009 IEEE / RSJ Int. Conf. Intell. Robots Syst (IROS) (IEEE, 2009), pp. 4306-4312.] This method uses different geometric perspectives to calculate the distance between all feature point pairs that form feature matching and the epipolar line to detect false matches. However, this method needs to obtain different geometric angles of the same environment and consumes a lot of computing resources, which makes the method inefficient and has poor real-time performance. For example, the mismatch elimination algorithm based on hierarchical quadtree [H.Xie, W.Chen, J.Wang and H.Wang, Hierarchical quadtree feature optical flow trackingbased sparse pose-graph visual-inertial SLAM, 2020IEEE Int.Conf.Robotics andAutomation(ICRA)(IEEE, 2020), pp.58-64.] uses the optical flow method to predict and track a very small number of valid feature points to improve the quality of feature matching. However, this also means that the algorithm ignores the valid feature information in some environments. Summary of the invention

[0004] Purpose of the invention: The present invention provides a method and system for eliminating mismatches of a visual odometer based on an optical flow method, which is used to improve the positioning accuracy and robustness of an unmanned system based on a visual odometer based on an optical flow method.

[0005] Technical solution: The present invention discloses a method for eliminating mismatches of a visual odometer based on an optical flow method, comprising the following steps:

[0006] Step 1: After removing the image feature points at the edge position in the first frame image using the edge removal strategy, the feature points of the first frame image are predicted in the second frame image using the optical flow method; based on the predicted feature points of the second frame image, the feature points are reversely predicted in the first frame image using the optical flow method, and the Euclidean distance of the corresponding feature point pairs is calculated at the pixel level in combination with the feature points extracted from the first frame image, and a threshold is constructed to preliminarily screen out false matches;

[0007] Step 2: Use the preliminary screening results and the random sampling consistency algorithm to fit the basic matrix, and then combine the basic matrix consistency test to further screen out mismatches.

[0008] Furthermore, in step 1, after removing the image feature points at the edge position in the first frame image using the edge removal strategy, the feature points of the first frame image are predicted in the second frame image using the optical flow method; relying on the predicted feature points of the second frame image, the feature points are reversely predicted in the first frame image using the optical flow method, and the Euclidean distance of the corresponding feature point pairs is calculated at the pixel level in combination with the feature points extracted from the first frame image, and a threshold is constructed. The preliminary screening of mismatches specifically includes the following steps:

[0009] Step 11: Use the edge removal strategy to remove the feature point group M at time k-1 k-1 The feature points located at the edge of the image are obtained

[0010] Step 12: Use the optical flow method to identify the feature point groups Forward prediction to obtain feature point groups Using optical flow method to identify feature point groups The prediction results are expressed as follows:

[0011]

[0012] in, Indicates the use of optical flow method to predict feature point groups forward Feature point group at time k Represents the camera coordinate system obtained using the forward optical flow at time k-1 The pixel coordinates corresponding to the i1th feature point in ;

[0013] Step 13: Use the optical flow method to identify the feature point groups Reverse prediction to obtain feature point groups Optical flow method for feature point groups The backward prediction has the following expression:

[0014]

[0015] in, Indicates the use of optical flow method to reversely predict feature point groups Feature point group at time k-1 Represents the camera coordinate system obtained using the forward optical flow at time k-1 The pixel coordinates corresponding to the i1th feature point in ;

[0016] Step 14: Calculate feature point groups and feature point groups Euclidean distance between corresponding feature point pairs at the pixel level where j∈{1,...,n};

[0017] Step 15: Calculate the mean D based on the Euclidean distance of each pair of feature matches init :

[0018]

[0019] Step 16: Calculate the result D based on the mean init , calculate the threshold λ, and combine the upper and lower bounds of the threshold, i.e., λ high ,λ low , ensuring that λ is always within the appropriate range;

[0020] Step 17: Compare feature point groups and feature point groups All features in D j and the size of the threshold λ, obtain the status of the feature points corresponding to each pair of feature matches j ;

[0021] Step 18: Eliminate feature point groups and feature point groups Status j = 0, and obtain the feature point group after preliminary screening of mismatches and feature point groups

[0022] Furthermore, in step 11, the feature point group M of the image at time k-1 k-1 It is a set of pixel coordinates corresponding to multiple feature points, which has the following form:

[0023]

[0024] in, Represents the camera coordinate system at time k-1 The mth k-1 The pixel coordinates corresponding to the feature points;

[0025] The feature point group after removing the edge feature points using the image edge removal strategy is expressed as:

[0026]

[0027] i1,...,i n Indicates the number of feature points after removing edge feature points of the image.

[0028] Furthermore, in step 16, the threshold λ has the following form:

[0029] λ=log β (D init +1)

[0030] Where β = 10;

[0031] Set upper and lower bounds for λ high =0.55,λ low = 0.2 to ensure that λ is always within the appropriate range, that is

[0032]

[0033] Furthermore, in step 17, the state of feature matching after preliminary screening is expressed as:

[0034]

[0035] Furthermore, in step 2, the preliminary screening results are used to fit the basic matrix in combination with the random sampling consistency algorithm, and then the basic matrix consistency test is combined to further screen out the mismatches, which specifically includes the following steps:

[0036] Step 21: In the feature point group and feature point groups Randomly select 8 pairs of matching feature points and use the eight-point method to obtain the analytical solution of the basic matrix F;

[0037] Step 22: Calculate the characteristic point group based on the principle of epipolar geometry The distance between all feature points and the epipolar line in the image is considered as an inner point if the distance is less than the set threshold, otherwise it is considered as an outer point;

[0038] Step 23: When the number of internal points is large enough, stop the iteration; otherwise, return to step 21 until all iterations are completed and the optimal basic matrix F is selected;

[0039] Step 24: Select from the feature point group and feature point groups A pair of matching feature points in , combined with the basic matrix F, determines whether the consistency of the basic matrix is ​​satisfied;

[0040] Step 25: When all the feature points are and feature point groups When the matched feature points are judged, the iteration stops, otherwise, it returns to step 24;

[0041] Step 26: Obtain the feature point group that satisfies the consistency of the basic matrix, that is, after further screening and feature point groups

[0042]

[0043] Furthermore, in step 24, the consistency check of the basic matrix is ​​expressed as

[0044]

[0045] in, Represents the camera coordinate system at time k The pixel coordinates corresponding to the lth feature point in , K represents the intrinsic parameter matrix of the camera, Represents the camera coordinate system at time k To the camera coordinate system at time k-1 The antisymmetric matrix of the translation vector, Represents the camera coordinate system at time k To the camera coordinate system at time k-1 The rotation matrix of .

[0046] Furthermore, in step 26, the consistency test of the basic matrix is ​​used to further screen the feature points, which is expressed as:

[0047]

[0048] in, ε is a small value, the characteristic point group and feature point groups It represents the feature point group after further screening at time k-1 and time k.

[0049] Accordingly, a false match elimination system for a visual odometer based on an optical flow method includes: a preliminary false match elimination module, which is used to use an edge elimination strategy to eliminate image feature points located at edge positions in a first frame image, and then use the optical flow method to predict feature points of the first frame image in a second frame image, and then use the optical flow method to reversely predict feature points in the first frame image based on the predicted feature points of the second frame image, and calculate the Euclidean distance of corresponding feature point pairs at the pixel level and construct a threshold value in combination with the feature points extracted from the first frame image to preliminarily screen false matches; and a false match elimination module, which uses the preliminary screening results, combined with a random sampling consistency algorithm and a basic matrix consistency check to further screen false matches.

[0050] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) For dynamic scenes and situations where the camera moves violently, the present invention exhibits stronger robustness. By optimizing feature extraction and matching strategies, it can more efficiently eliminate bad feature matches extracted from the scene, significantly reduce positioning errors caused by incorrect matching, and thus ensure the stability and reliability of the system; (2) For high-dynamic environments or weak-texture scenes, the present invention can accurately extract and match high-quality feature points in the scene, and can maintain excellent performance even under conditions where information is sparse or changes violently; and in a stable environment, the present invention further improves the selection efficiency of feature matching by optimizing the matching algorithm, and selects high-quality feature points. Whether in a complex or simple environment, the present invention can provide more reliable and accurate feature matching results, and its adaptability and adaptability are significantly improved; (3) By providing better feature matching, the present invention effectively enhances the positioning accuracy of the subsequent visual mileage calculation method, significantly improves the overall performance of the system, and the optimized feature matching scheme can reduce drift errors in complex dynamic scenes, and improve the robustness of data association in weak-texture scenes, ultimately ensuring that high-precision positioning can be achieved in a variety of application scenarios, meeting real-time and reliability requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0052] Figure 2 The figure is a schematic diagram of the process of eliminating mismatches of the visual odometer based on the optical flow method according to the present invention.

[0053] Figure 3 This is a schematic diagram of the comparison between several different algorithms of the present invention and the true value in the xyz direction. DETAILED DESCRIPTION

[0054] like Figure 1 and Figure 2 As shown, a method for eliminating mismatches of a visual odometer based on an optical flow method includes the following steps:

[0055] S1, using the edge removal strategy to remove feature points on the edge of the image, and constructing a threshold based on the forward optical flow and reverse optical flow, as well as the Euclidean distance of the corresponding feature points at the pixel level, to complete the preliminary removal of false matches;

[0056] S11, consider the feature point group M of the image at time k-1 k-1 , expressed as:

[0057]

[0058] in, Represents the camera coordinate system C at time k-1 k-1 The mthk-1 The pixel coordinates corresponding to the feature points.

[0059] Using the image edge removal strategy, the feature points within two pixel units of the image edge are removed, and the feature point group of the image at time k-1 is obtained after removing the edge feature points. It is expressed as:

[0060]

[0061] i1,...,i n Indicates the number of feature points after removing edge feature points of the image;

[0062] S12, the basic assumption of the optical flow method is grayscale invariance, which is specifically expressed as:

[0063]

[0064] Among them, k represents the kth moment, δk represents the smallest time change, Represents the i-th feature point The horizontal and vertical coordinates of the pixel coordinates, I(i) represents the gray value of the pixel;

[0065] The first-order Taylor expansion of the grayscale invariance of the optical flow method is specifically expressed as:

[0066]

[0067] Combined with the grayscale invariance of the optical flow method, the first-order Taylor expansion expression is simplified and specifically expressed as:

[0068]

[0069]

[0070] When considering discrete time k, we can determine Pixel gradient change Then the displacement vector of the pixel point is obtained;

[0071] Using optical flow method Make a prediction and get the feature point group of the image at time k It is expressed as:

[0072]

[0073] Among them, LK f (i) represents the forward optical flow;

[0074] S13, prediction results based on forward optical flow Using reverse optical flow Make a prediction and get the feature point group of the image at time k-1 It is expressed as:

[0075]

[0076] Among them, LK b (i) represents reverse optical flow;

[0077] S14, calculate the feature point group and feature point groups The Euclidean distance of the corresponding feature point pairs at the pixel level is expressed as:

[0078]

[0079] where j∈{1,...,n};

[0080] S15, calculate all the feature point groups and feature point groups The mean of the Euclidean distances of the corresponding feature point pairs at the pixel level is expressed as:

[0081]

[0082] S16, based on the calculation result of the mean value D init , calculate the threshold λ, expressed as:

[0083] λ=log β (D init +1);

[0084] Where β = 10;

[0085] Based on actual engineering experience, set upper and lower bounds for λ high =0.55,λ low =0.2 to ensure that λ is always within the appropriate range, expressed as:

[0086]

[0087] S17, compare feature point groups and feature point groups All features in D j and the size of the threshold λ, obtain the status of the feature points corresponding to each pair of feature matches j , status j It is expressed as:

[0088]

[0089] S18, remove feature point groups and feature point groups Status j = 0, and obtain the feature point group after preliminary screening of mismatches and feature point groups

[0090] S2, using the preliminary screening results, combined with the random sampling consistency algorithm to fit the basic matrix, and then combined with the basic matrix consistency test to further screen out mismatches;

[0091] S21, in the feature point group and feature point groups Randomly select 8 pairs of matching feature points and use the eight-point method to obtain the analytical solution of the basic matrix F;

[0092] S22, based on the principle of epipolar geometry, calculate the characteristic point group The distance between all feature points and the epipolar line in the image is considered as an inner point if the distance is less than the set threshold, otherwise it is considered as an outer point;

[0093] S23, when the number of internal points is large enough, stop the iteration, otherwise return to step 21 until all iterations are completed and the optimal basic matrix F is selected;

[0094] The consistency check of the basic matrix is ​​expressed as:

[0095]

[0096] in, Represents the camera coordinate system at time k The pixel coordinates corresponding to the lth feature point in , K represents the intrinsic parameter matrix of the camera, Represents the camera coordinate system at time k To the camera coordinate system at time k-1 The antisymmetric matrix of the translation vector, Represents the camera coordinate system at time k To the camera coordinate system at time k-1 The rotation matrix of .

[0097] S24, select from the feature point group and feature point groups A pair of matching feature points in and Combined with the basic matrix F, determine whether the consistency of the basic matrix is ​​satisfied, that is:

[0098]

[0099] in, ε is a small value, the characteristic point group and feature point groups It represents the feature point group that is mismatched after further elimination at time k-1 and time k.

[0100] S25, when all the feature point groups and feature point groups When the matched feature points are judged, the iteration stops, otherwise it returns to S24;

[0101] S26, obtain the feature point group that satisfies the consistency of the basic matrix, that is, after further screening and feature point groups

[0102] In this embodiment, simulation experiments were carried out on Jetson Orin NX using the Ubuntu 20.04 system and ROS environment, and compared with the traditional simultaneous positioning and mapping method VINS.

[0103] The mismatch elimination method and system parameter settings of the visual odometer based on the optical flow method are shown in Table 1:

[0104] Table 1 Mismatch elimination method and system parameters of visual odometer based on optical flow method

[0105]

[0106] Figure 3 It shows the comparison between several different algorithms and the true value in the xyz direction, where the gray dotted line marked as ship_GT represents the true value; Figure 3 The black solid line marked as ship_None in the upper left corner represents the VINS solution without any processing. Figure 3 The black solid line marked as ship_MASOR1 in the upper right corner represents the traditional method of filtering mismatches by only using the mean and covariance of feature points. Figure 3 The black solid line marked as ship_MASOR2 in the lower left corner represents the traditional method of filtering mismatches by only using the mean of feature points. Figure 3 The black solid line marked as ship_Ours in the lower right corner represents the method proposed by the present invention for screening out mismatches by using reverse optical flow and basic matrix consistency check. It can be seen that the mismatch elimination method of the visual odometer based on the optical flow method of the present invention can achieve better estimation effect.

[0107] Based on the same inventive concept, a false match elimination system for a visual odometer based on an optical flow method comprises: a preliminary false match elimination module, which is used to use an edge elimination strategy to eliminate image feature points located at edge positions in a first frame image, and then use the optical flow method to predict feature points of the first frame image in a second frame image, and then use the optical flow method to reversely predict feature points in the first frame image based on the predicted feature points of the second frame image, and calculate the Euclidean distance of corresponding feature point pairs at the pixel level and construct a threshold value in combination with the feature points extracted from the first frame image, so as to preliminarily screen false matches; and a further false match elimination module, which uses the preliminary screening results in combination with a random sampling consistency algorithm and a basic matrix consistency check to further screen false matches.

Claims

1. A method for eliminating mismatches of visual odometer based on optical flow method, characterized in that: The steps include: Step 1: After removing the image feature points at the edge position in the first frame image using the edge removal strategy, the feature points of the first frame image are predicted in the second frame image using the optical flow method; based on the predicted feature points of the second frame image, the feature points are reversely predicted in the first frame image using the optical flow method, and the Euclidean distance of the corresponding feature point pairs is calculated at the pixel level in combination with the feature points extracted from the first frame image, and a threshold is constructed to preliminarily screen out false matches; Step 2: Use the preliminary screening results and the random sampling consistency algorithm to fit the basic matrix, and then combine the basic matrix consistency test to further screen out mismatches.

2. The method for eliminating mismatches of visual odometer based on optical flow method as claimed in claim 1, characterized in that: In step 1, after removing the image feature points at the edge position in the first frame image using the edge removal strategy, the feature points of the first frame image are predicted in the second frame image using the optical flow method; based on the predicted feature points of the second frame image, the feature points are reversely predicted in the first frame image using the optical flow method, and the Euclidean distance of the corresponding feature point pairs is calculated at the pixel level in combination with the feature points extracted from the first frame image, and a threshold is constructed. The preliminary screening of mismatches specifically includes the following steps: Step 11: Use the edge removal strategy to remove the feature point group M at time k-1 k-1 The feature points located at the edge of the image are obtained Step 12: Use the optical flow method to identify the feature point groups Forward prediction to obtain feature point groups Step 13: Use the optical flow method to identify the feature point groups Reverse prediction to obtain feature point groups Optical flow method for feature point groups The backward prediction has the following expression: in, Indicates the use of optical flow method to reversely predict feature point groups Feature point group at time k-1 Step 14: Calculate feature point groups and feature point groups Euclidean distance between corresponding feature point pairs at the pixel level where j∈{1,...,n}; Step 15: Calculate the mean D based on the Euclidean distance of each pair of feature matches init : Step 16: Calculate the result D based on the mean init , calculate the threshold λ, and combine the upper and lower bounds of the threshold, i.e., λ high ,λ low , ensuring that λ is always within the appropriate range; Step 17: Compare feature point groups and feature point groups All features in D j and the size of the threshold λ, obtain the status of the feature points corresponding to each pair of feature matches j ; Step 18: Eliminate feature point groups and feature point groups Status j = 0, and obtain the feature point group after preliminary screening of mismatches and feature point groups 3. The method for eliminating mismatches of visual odometer based on optical flow method as claimed in claim 2, characterized in that: In step 11, the feature point group M of the image at time k-1 k-1 It is a collection of pixel coordinates corresponding to multiple feature points, and has the following form: in, Represents the camera coordinate system C at time k-1 k-1 The mth k-1 The pixel coordinates corresponding to the feature points; The feature point group after removing the edge feature points using the image edge removal strategy is expressed as: i1,...,i n Indicates the number of feature points after removing edge feature points of the image.

4. The method for eliminating mismatches of a visual odometer based on an optical flow method as claimed in claim 2, characterized in that: In step 12, the optical flow method is used to analyze the feature point group The prediction results are expressed as follows: in, Indicates the use of optical flow method to predict feature point groups forward Feature point group at time k 5. The method for eliminating mismatches of visual odometer based on optical flow method as claimed in claim 2, characterized in that: In step 16, the threshold λ has the following form: λ=log β (D init +1) Where β = 10; Set upper and lower bounds for λ high =0.55,λ low = 0.2 to ensure that λ is always within the appropriate range, that is 6. The method for eliminating mismatches of visual odometer based on optical flow method as claimed in claim 2, characterized in that: In step 17, the state of feature matching after preliminary screening is expressed as:

7. The method for eliminating mismatches of visual odometer based on optical flow method as claimed in claim 1, characterized in that: In step 2, the preliminary screening results are used to fit the basic matrix in combination with the random sampling consistency algorithm, and then the basic matrix consistency test is combined to further screen out the mismatches. The specific steps include the following: Step 21: In the feature point group and feature point groups Randomly select 8 pairs of matching feature points and use the eight-point method to obtain the analytical solution of the basic matrix F; Step 22: Calculate the feature point group based on the principle of epipolar geometry The distance between all feature points and the epipolar line in the image is considered as an inner point if the distance is less than the set threshold, otherwise it is considered as an outer point; Step 23: When the number of internal points is large enough, stop the iteration; otherwise, return to step 21 until all iterations are completed and the optimal basic matrix F is selected; Step 24: Select from the feature point group and feature point groups A pair of matching feature points in , combined with the basic matrix F, determines whether the consistency of the basic matrix is ​​satisfied; Step 25: When all the feature points are and feature point groups When the matched feature points are judged, the iteration stops, otherwise, it returns to step 24; Step 26: Obtain the feature point group that satisfies the consistency of the basic matrix, that is, after further screening and feature point groups 8. The method for eliminating mismatches of visual odometer based on optical flow method as claimed in claim 7, characterized in that: In step 24, the consistency check of the basic matrix is ​​expressed as in, Represents the camera coordinate system C at time k k The pixel coordinates corresponding to the lth feature point in , K represents the intrinsic parameter matrix of the camera, Represents the camera coordinate system C at time k k At time k-1, the camera coordinate system C k-1 The antisymmetric matrix of the translation vector, Represents the camera coordinate system C at time k k At time k-1, the camera coordinate system C k-1 The rotation matrix of .

9. The method for eliminating mismatches of visual odometer based on optical flow method as claimed in claim 7, characterized in that: In step 26, the consistency test of the basic matrix is ​​used to further screen the feature points, which is expressed as: in, ε is a small value, the characteristic point group and feature point groups It represents the feature point group after further screening at time k-1 and time k.

10. A system for implementing the method for eliminating mismatches of visual odometer based on optical flow method as claimed in claim 1, characterized in that: include: The module for initially eliminating false matches is used to use the edge elimination strategy to eliminate the image feature points located at the edge position in the first frame image, and then use the optical flow method to predict the feature points of the first frame image in the second frame image, and then use the optical flow method to reversely predict the feature points in the first frame image based on the predicted feature points of the second frame image, and calculate the Euclidean distance of the corresponding feature point pairs at the pixel level and construct a threshold value in combination with the feature points extracted from the first frame image to preliminarily screen out false matches; the module for eliminating false matches uses the preliminary screening results, combined with the random sampling consistency algorithm and the basic matrix consistency test to further screen out false matches.

Images