Multistage feature matching method based on SuperGlue neural network model

Abstract

The invention discloses a multistage feature matching method based on a SuperGlue neural network model, and the method comprises the following steps: S1, obtaining image matching data, and employing a ScanNet large-scale indoor data set as an experimental data set for verifying the multistage feature matching method; s2, calculating an initial transformation relation between the to-be-matched images through a SuperGlue neural network model, and determining an initial space relative position between homonymy points; s3, determining a root mean square error (RMSE), a matching score (Mscore) and an area (AUC) under a curve under different threshold conditions of the matching result of the SuperGlue neural network model on the two frames of images by using the matching result obtained in the S2; s4, introducing a geometric corresponding matching method, calculating and estimating a spatial correlation coefficient between homonymy points, and screening out corresponding homonymy points in a threshold range; s5, verifying the matching performance of the method for the matching result by adopting RMSE and Mscore by using the matching result obtained in the S4, estimating the camera pose precision by using AUC, and analyzing the performance of the improved method; the method is superior to traditional SIFT, ORB and original SuperGlue methods, and the new model has higher stability and superiority in the aspect of improving the camera pose estimation precision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a multi-level feature matching method based on a SuperGlue neural network model. Background Art

[0002] In visual odometry, accurately calculating the correspondence between feature points with the same name is crucial for accurately estimating the camera pose. Traditional matching methods such as Scale Invariant Feature Transform (SIFT), Oriented FAST and Rotated BRIEF (ORB), and Speed ​​Up Robust Features (SURF) are usually used to detect edges, extract features, and match feature points with the same name between images, and then estimate the correspondence between the same name points and the camera pose. However, when relying on traditional algorithms to handle image matching problems in complex scenes, it is often impossible to provide a camera pose estimation accuracy that meets the requirements due to possible noise, lighting changes, occlusion, and other factors.

[0003] To solve this problem, researchers introduced an end-to-end deep learning method. The end-to-end deep learning method can directly input the pose parameters into adjacent frame images to obtain the relative pose between two adjacent frames, thereby improving the robustness of feature point matching and the accuracy of camera pose estimation. This method usually involves using a neural network to learn the mapping that associates image correspondences to pose parameters, and directly obtains pose information from real data through mapping, without manually designing features or using traditional computer vision techniques.

[0004] Among them, Daniel et al. proposed the SuperPoint algorithm in 2018, which is based on MagicPoint and combines homography adaptation technology to realize self-supervised training of detectors, which is suitable for computer vision scenarios that solve multi-view and geometric problems. On this basis, Paul et al. and Daniel et al. proposed a SuperGlue algorithm in 2020 that simultaneously performs feature matching and removes mismatched points. The algorithm uses an end-to-end learning approach to integrate tasks such as feature extraction, matching, and pose estimation into a neural network, which can improve the performance of the overall system and better cope with matching tasks in complex scenes and changing conditions. By learning the feature representations and relationships in the scene, more accurate matching results are provided. Compared with traditional algorithms, the SuperGlue neural network model shows higher robustness and faster processing speed in image matching. Although the SuperGlue neural network model has largely overcome the limitations of traditional algorithms, further research and improvement are still needed in processing complex scenes.

[0005] In this regard, this patent proposes a multi-level feature matching method based on the SuperGlue neural network model. First, the key points are obtained through the SuperPoint algorithm. Secondly, the SuperGlue neural network model is used to match and filter out the mismatched points. Then, the essential matrix is ​​used to obtain the relative posture relationship between the two frames of images. Finally, the geometric correspondence matching algorithm is added to calculate the correlation coefficients between each point pair and sort the correlation coefficients from small to large. If there is a point with the largest correlation coefficient and it is greater than the threshold, the point is considered to be a matching point, thereby further improving the accuracy of camera posture estimation. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a multi-level feature matching method based on the SuperGlue neural network model.

[0007] Technical solution: The multi-level feature matching method based on the SuperGlue neural network model described in the present invention comprises the following steps:

[0008] S1: Obtain image matching data. Use ScanNet large-scale indoor dataset as the experimental dataset for verifying the multi-level feature matching method. The dataset consists of monocular sequences. The dataset contains images of the same scene with different perspectives, ground truth poses and depths. The present invention selects 2.3 million training sample sets and 1500 indoor scene test sample sets;

[0009] S2: The initial transformation relationship between the images to be matched is calculated through the SuperGlue neural network model to determine the rough spatial relative positions between the points with the same name;

[0010] S3: Using the matching results obtained in S2, determine the root mean square error (RMSE), matching score (Mscore) and area under the curve (AUC) of the SuperGlue neural network model for the matching results of the two frames of images under different threshold conditions;

[0011] S4: Introduce the geometric correspondence matching algorithm to calculate and estimate the spatial correlation coefficient between the same-name points, and filter out the corresponding same-name points within the threshold range;

[0012] S5: Using the matching results obtained in S4, RMSE and Mscore are used to verify the matching performance of the method, and AUC is used to estimate the camera pose accuracy to analyze the performance of the improved method;

[0013] The matching data in step S1 uses the ScanNet large-scale indoor dataset as an experimental dataset for verifying the multi-level feature matching method, which consists of monocular sequences. The dataset contains images of the same scene with different perspective changes, ground truth pose and depth. The present invention selects 230M training pairs and 1500 test pairs.

[0014] The transformation matrix in step S2 is shown in formula (1):

[0015]

[0016] in is the antisymmetric matrix of the translation matrix, and R is the rotation matrix.

[0017] The RMSE in step S3 is shown in formula (2):

[0018]

[0019] Where errs is the number of incorrect matches, pts11.shape[0] and pts11.shape[1] are the length and width of the image respectively;

[0020] Mscore is shown in formula (3):

[0021]

[0022] Where num_correct is the number of incorrect matches, and len(pts11) is the total number of matches;

[0023] AUC is shown in formula (4):

[0024]

[0025] In the formula is the serial number of the i-th class, M and N are the number of positive and negative classes;

[0026] In step S4, a geometric corresponding matching algorithm is introduced to calculate and estimate the spatial correlation coefficient between the points with the same name, and the corresponding points with the same name within the threshold range are screened out. The correlation coefficient p(c, r) in step S4 is shown in formula (5):

[0027]

[0028] Where m and n are the matching window sizes, g i，j is the grayscale value of (i, j).

[0029] In step S5, the performance of the method is evaluated from the perspective of the accuracy of estimating the camera pose, and the traditional SIFT algorithm, ORB algorithm, SuperGlue algorithm and SuperGlue+GC method are matched to analyze and compare the performance of the improved algorithm in this paper. At the same time, in order to verify the performance of the SuperGlue+GC method in camera pose estimation on images with changing perspectives and low texture areas compared to the original algorithm, the accuracy is evaluated using AUC.

[0030] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0031] 1. It is superior to the traditional SIFT algorithm and ORB algorithm. The accuracy of camera pose estimation of the new model is significantly improved compared with the above two methods.

[0032] 2. SuperGlue neural network model + geometric correspondence matching has high matching accuracy;

[0033] 3. In multi-viewpoint changes and low-texture areas, the SuperGlue neural network model + geometric correspondence matching shows stronger stability than traditional algorithms and the original SuperGlue neural network model, and the relative translation and rotation poses are closer to the true values. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the steps of the present invention;

[0035] Figure 2 This is a comparison chart for method matching;

[0036] Figure 3 RMSE and Mscore curves DETAILED DESCRIPTION

[0037] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0038] like Figure 1 As shown, this specific embodiment discloses a multi-level feature matching method based on the SuperGlue neural network model, comprising the following steps:

[0039] S1: Obtain image matching data, and use the ScanNet large-scale indoor dataset as an experimental dataset to verify the multi-level feature matching method. It consists of monocular sequences, and the dataset contains images of the same scene with different perspective changes, ground truth poses and depths. This paper selects 230M training pairs and 1500 test pairs;

[0040] S2: The initial transformation relationship between the images to be matched is calculated through the SuperGlue neural network model to determine the rough spatial relative positions between the points with the same name;

[0041] The transformation matrix in step S2 is shown in formula (1):

[0042]

[0043] in is the antisymmetric matrix of the translation matrix, and R is the rotation matrix;

[0044] S3: Using the matching results obtained in S2, determine the root mean square error (RMSE), matching score (Mscore) and area under the curve (AUC) of the SuperGlue neural network model for the matching results of the two frames of images under different threshold conditions;

[0045] The RMSE in step S3 is shown in formula (2):

[0046]

[0047] Where errs is the number of incorrect matches, pts11.shape[0] and pts11.shape[1] are the length and width of the image respectively;

[0048] Mscore is shown in formula (3):

[0049]

[0050] Where num_correct is the number of incorrect matches, and len(pts11) is the total number of matches;

[0051] AUC is shown in formula (4):

[0052]

[0053] In the formula is the serial number of the i-th class, M and N are the number of positive and negative classes;

[0054] S4: Introduce the geometric correspondence matching algorithm, calculate the estimated spatial correlation coefficient between the same-name points, and screen out the corresponding same-name points within the threshold range. The specific operation is to first set a window in the reference image, and then project the window onto the input image. Next, calculate the correlation coefficient between the feature point P in the left image window and the feature point within the range of n pixels from the point P′ in the right image window, and sort the correlation coefficients from small to large. If there is a maximum correlation coefficient that is greater than the threshold t, then calculate the inverse matrix F′ of F and the inverse matrix E′ of E in reverse. Similarly, calculate Q according to the formula j The corresponding point Q′ on the left imagej , if the distance between point P and Q′ on the left image is j If the maximum value of the correlation coefficient is greater than the threshold t within n pixels, the point is considered to be a matching point, thereby improving the accuracy of camera pose estimation. The expression is:

[0055]

[0056] Where m and n are the matching window sizes, g i，j is the grayscale value of (i, j).

[0057] S5: The performance of the method is evaluated in terms of the accuracy of camera pose estimation, and the traditional SIFT algorithm, ORB algorithm, SuperGlue algorithm and SuperGlue+GC (geometric correspondence matching) method are matched to analyze and compare the performance of the improved method in this paper. At the same time, in order to verify the performance of the SuperGlue+GC method in camera pose estimation on images with perspective changes and low texture areas compared to the original algorithm, the AUC is used to evaluate its accuracy.

[0058] In order to analyze the accuracy of the SuperGlue+GC method, the camera pose estimation accuracy is compared with the traditional SIFT algorithm, ORB algorithm and the original SuperGlue algorithm. The specific results are shown in Table 1.

[0059] Table 1 Comparison of camera pose estimation accuracy

[0060]

[0061] From Table 1, Figure 2 , Figure 3 It can be seen that:

[0062] (1) The number of matches in this algorithm is significantly better than that of the SIFT algorithm and the ORB algorithm. This is because this algorithm is based on deep learning technology and learns feature representation and matching through graph neural networks. Compared with traditional algorithms, deep learning features have stronger expressive power. Secondly, this algorithm not only matches local feature points, but also introduces global feature matching, and adopts an end-to-end training method, which is trained and optimized through large-scale data sets. Finally, the linear correlation between feature points is measured by combining geometric correspondence matching, providing a measure of the relationship between matching points, so that multiple accurate numbers of matching points can be obtained in sparse areas and multi-view changing images.

[0063] (2) By changing the scaling ratio of the internal function to change the internal parameter value, the RMSE and matching score are calculated. When the Scales value of the internal function increases to 2.5, the RMSE of the proposed algorithm is slightly higher than that of the ORB algorithm, and the Mscore value is slightly lower. This is because before the Scales value of the proposed algorithm is 2.5, the Scales value cannot adapt to the scale changes and sparse texture areas in the image, resulting in an increase in mismatches in the matching results, thereby affecting the Mscore value. However, after the Scales value is 2.5, the proposed algorithm can obtain features of more scales, further increasing the input information of the deep learning model, thereby obtaining more accurate matching results, thereby reducing the RMSE value and increasing the Mscore value.

[0064] (3) Compared with the traditional SIFT algorithm, ORB algorithm and SuperGlue algorithm, the proposed algorithm has higher relative pose estimation performance in multi-view changes and low-texture areas. When the threshold is 5°, the pose error AUC value of the proposed algorithm is 90.29% higher than that of other algorithms. This is mainly because the traditional algorithm and SuperGlue algorithm cannot accurately distinguish between correct and incorrect estimates when the threshold is 5°, resulting in all estimates being misclassified as errors. The proposed algorithm is based on deep learning technology and combined with geometric correspondence matching. It has higher discrimination and robustness for feature representation and can extract more accurate and informative feature points, thereby improving the pose estimation accuracy, so that the relative translation pose and rotation pose are closer to the true value. At the same time, the larger the threshold, the larger the pose estimation AUC value. The reason is that a larger threshold can filter out some noise interference or abnormal matching, thereby reducing the impact of interference factors on pose estimation. It helps to improve the accuracy and stability of pose estimation, thereby improving the AUC value.

[0065] (4) In rotation pose estimation, the method of this patent is 74.2°, 93.4° and 4.4° higher than the SIFT algorithm, ORB algorithm and the original SuperGlue algorithm, respectively; in relative translation pose estimation, the algorithm of this patent is 4.5°, 8.6° and 1.6° higher than other algorithms, respectively.

[0066] From the above conclusions, it can be seen that compared with traditional algorithms and deep learning algorithms, the method proposed in this patent can achieve higher-precision camera pose estimation. Through the multi-level feature matching method, it can effectively overcome problems such as low image texture, lighting and large viewing angle changes, thereby improving the camera pose.

Claims

1. A multi-level feature matching method based on SuperGlue neural network model, characterized in that: The following steps are involved: S1: Obtain image matching data and use the ScanNet large-scale indoor dataset as an experimental dataset to verify the multi-level feature matching method; S2: The initial transformation relationship between the images to be matched is calculated through the SuperGlue neural network model to determine the rough spatial relative positions between the points with the same name; The transformation matrix in step S2 is shown in formula (1): in is the antisymmetric matrix of the translation matrix, and R is the rotation matrix; S3: Using the matching results obtained in S2, determine the root mean square error (RMSE), matching score (Mscore) and area under the curve (AUC) of the SuperGlue neural network model for the matching results of the two frames of images under different threshold conditions; The RMSE in step S3 is shown in formula (2): Where errs is the number of incorrect matches, pts11.shape[0] and pts11.shape[1] are the length and width of the image respectively; Mscore is shown in formula (3): Where num_correct is the number of incorrect matches, and len(pts11) is the total number of matches; AUC is shown in formula (4): In the formula is the serial number of the i-th class, M and N are the number of positive and negative classes; S4: Introduce the geometry correspondence matching algorithm (GC), calculate and estimate the spatial correlation coefficient between the same-name points, and screen out the corresponding same-name points within the threshold range. The correlation coefficient p(c, r) in step S4 is shown in formula (5): Where m and n are the matching window sizes, g i，j is the gray value of (i, j); S5: Using the matching results obtained in S4, RMSE and Mscore are used to verify the matching performance of the method, and AUC is used to estimate the camera pose accuracy to analyze the performance of the improved method.

2. The prediction method according to claim 1, characterized in that: The two-frame image matching data in step S1 uses the ScanNet large-scale indoor dataset as an experimental dataset for verifying the multi-level feature matching method, which consists of a monocular sequence. The dataset contains images of the same scene with different perspective changes, ground truth pose and depth. The present invention selects 2.3 million training sample sets and 1,500 indoor scene test sample sets.

3. The prediction method according to claim 1, characterized in that: The SuperGlue neural network model in step S2 consists of two parts: an attention neural network and an optimal matching layer. The attention GNN method realizes feature point matching, in which the self-attention mechanism changes the nodes in the image, and the cross-attention mechanism updates the adjacent nodes between images. First, SuperPoint is used to obtain the position of the feature points and it is mapped to a high-dimensional space through 5 multilayer perceptrons (MLPs). Then these embedded vectors are directly added to the descriptors. The expression is: (0) x i =d i +MLP enc (p i ) (6) Image matching is a technique for matching the feature points of two images. The feature points of the two images can be regarded as nodes of a large image, where there are two types of edges between nodes: one is the edge between nodes in the same image, called self-edge (ε seif ), and the other is the edge between nodes in different images, called cross-edge (ε cross ). The weight sum of the previous layer node and its adjacent nodes is calculated through the message passing mechanism to update the node, and its expression is: in, is the intermediate expression of the i-th element in image A at layer l, and [·||·] is a concatenation operation. When the number of layers is even, ε=ε seif , when the number of layers is odd, ε=ε cross The attention mechanism is to repeatedly update the weights through the relationship between two nodes to obtain a descriptor f with better matching performance. i : Among them, W and b are weight and bias respectively. For the optimized matching layer, first construct a distribution matrix P∈[0,1] M×N And satisfy: Get the feature point descriptors f of two images through attention GNN i A and f i B , and do the inner product of the descriptor to calculate its score matrix: The allocation matrix P can be obtained by calculating the score matrix and maximizing the total score Then, the unmatched feature points are thrown into the dustbin channel to filter out the mismatched point pairs. Finally, the maximum total score is transformed into the optimal transmission problem. This method mainly uses the Sinkhorn algorithm to deal with the optimal transmission problem. The initial transformation relationship between the images to be matched is obtained through the epipolar constraint formula through epipolar geometry. Then, E and F are calculated according to the coordinates of the corresponding pixel points of the two frames of images. Finally, the rotation and translation matrices can be obtained by decomposing the singular values ​​to determine the rough spatial relative positions between the points with the same name.

4. The prediction method according to claim 1, characterized in that: In step S4, a geometric corresponding matching algorithm is introduced to calculate and estimate the spatial correlation coefficient between the points with the same name, and the corresponding points with the same name within the threshold range are screened out. The specific operation is to first set a window in the reference image, then project the window onto the input image, and then calculate the correlation coefficient between the feature point P in the left image window and the feature point within the range of n pixels from the point P' in the right image window, and sort the correlation coefficients from small to large. If there is a maximum correlation coefficient that is greater than the threshold t, then calculate the inverse matrix F' of F and the inverse matrix E' of E in reverse. Similarly, calculate Q according to the formula j The corresponding point Q′ on the left image j , if the distance between point P and Q′ on the left image is j If the maximum value of the correlation coefficient is greater than the threshold t within n pixels, the point is considered to be a matching point, thereby improving the accuracy of camera pose estimation. The expression is: Where m and n are the matching window sizes, g i，j is the grayscale value of (i, j).

5. The prediction method according to claim 1, characterized in that: In step S5, the performance of the method is mainly evaluated from the perspective of the accuracy of estimating the camera pose, and the traditional SIFT algorithm, ORB algorithm, SuperGlue algorithm and SuperGlue+GC method (the method of this paper) are matched to analyze and compare the performance of the improved method of this paper. At the same time, in order to verify the performance of the SuperGlue+GC method in camera pose estimation on images with perspective changes and low texture areas compared with the original algorithm, the accuracy is evaluated using AUC.

Images