Content-aware image feature matching method

Abstract

The application relates to a content-aware based image feature matching method. First, an improved two-stage feature matching method is proposed, in which a state-of-the-art model fitting method is used to pre-align the image pair in the first stage; the pre-aligned image is used as the input of the second stage; second, a block composed of a fully convolutional network and a mask predictor is used before the feature extractor to weight the features of the input image to enhance the extraction of local effective features; and the method improves the matching accuracy.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a content-aware image feature matching method. Background Technology

[0002] Feature matching refers to finding the correct correspondence between two images, such as... Figure 1 As shown. It is also the foundation for more advanced tasks in the field of computer vision (such as 3D reconstruction, image stitching, SLAM, and lane detection), and improving the probability of correct matching can lead to better development of these advanced tasks.

[0003] Classical feature matching methods typically consist of three steps: feature detection, feature description, and feature matching. Before the advent of deep learning methods, most methods were based on this process. These methods usually improve performance by modifying one step in the process. For example, the Sadder detector [1], the brightness comparison-based FAST detector [2] and its extended version FAST-ER [3] have improved the performance of the feature detection step. Other methods [4,5] focus on improving the feature description step. In addition, well-known traditional methods such as SIFT [6], SURF [7], KAZE [8], and AKAZE [9] have improved the first two steps at the same time. For the last step, classic model fitting methods such as RANSAC

[10] and its improved algorithm DSAC

[11] improve matching accuracy by estimating geometric transformations (such as polar geometry and homography).

[0004] With the emergence of deep learning methods, feature matching methods proposed in recent years have widely used neural networks to improve performance. Some of these learning-based methods [12,13] follow the classical process, while others [14,15] are end-to-end methods. SuperPoint

[12] jointly detects key points and computes relevant descriptors. SuperGlue

[13] uses graph neural networks to improve descriptors based on cross and self-attention. In different scenarios, improving the performance of only one or two steps of the method is not the best choice, so end-to-end methods have been proposed. D2-Net

[14] uses pre-trained VGG-16 as a feature extractor to obtain features. DFM

[15] uses pre-trained VGG-19 as a feature extractor to obtain deep features, and at the same time, pre-aligns the input image before matching to improve the performance of the algorithm. However, the number of feature points extracted by D2-Net and DFM is insufficient, resulting in fewer correct matches on non-planar images. In addition, the geometric estimation algorithm used for pre-alignment in DFM is not efficient enough, which also affects the final matching accuracy. Summary of the Invention

[0005] The purpose of this invention is to provide a content-aware image feature matching method that improves matching accuracy.

[0006] To achieve the above objectives, the technical solution of the present invention is: a content-aware image feature matching method. First, an improved two-stage feature matching method is proposed. In the first stage, a state-of-the-art model fitting method is used to pre-align image pairs, and the pre-aligned images are used as input for the second stage. Second, a block consisting of a fully convolutional network and a mask predictor is used before the feature extractor to weight the features of the input image.

[0007] Compared to existing technologies, this invention offers the following advantages: To improve the accuracy of end-to-end feature matching methods, especially when applied to challenging scenarios such as non-planar images, repeating images, or images with strong illumination variations, this invention proposes a content-aware image feature matching method. First, this method uses a state-of-the-art model fitting method to pre-align the input image, improving the quality of feature extraction. The aligned image is then used as input for the second stage. Second, this method incorporates a content-aware block into the feature extractor, predicting a probability map to highlight the effective parts of the image, thereby guiding feature extraction and extracting a greater number of effective features. Experiments show that our proposed method outperforms state-of-the-art traditional and deep learning methods on the HPatches dataset. Attached Figure Description

[0008] Figure 1 This is an example of image feature matching.

[0009] Figure 2 This is a flowchart of the method of the present invention.

[0010] Figure 3 A comparison of the image alignment results of two homography estimation algorithms is presented: (a) and (b) are the input images, where (a) is the target image, (c) is the result of alignment using MAGSAC++, and (d) is the result of alignment using RANSAC.

[0011] Figure 4 This is a diagram of the content-aware block structure.

[0012] Figure 5 The MMA evaluation results of nine feature matching methods on the HPatches dataset under different ratios are presented: including three scenarios: illumination change, viewpoint change, and overall. Detailed Implementation

[0013] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0014] This invention provides a content-aware image feature matching method. First, an improved two-stage feature matching method is proposed. In the first stage, a state-of-the-art model fitting method is used to pre-align image pairs, and the pre-aligned images are used as input for the second stage. Second, a block consisting of a fully convolutional network and a mask predictor is used before the feature extractor to weight the features of the input image.

[0015] The following is a detailed implementation process of the present invention.

[0016] This invention provides a content-aware image feature matching method, such as... Figure 2 As shown. The first stage consists of three steps. First, a pre-trained feature extractor (VGG-19) is used to extract features from the input image I. A and I B The first stage involves extracting features; secondly, initial matching is performed using Dense Nearest Neighbor Search (DNNS) on the feature map of the last layer; finally, these initial matches are used for homography matrix estimation for pre-alignment. The second stage comprises two steps. First, the pre-alignment results are fed into a feature extractor consisting of content-aware blocks and VGG-19 for feature extraction; secondly, DNNS is used for feature matching. The first highlight of this method is the use of a more robust model fitting algorithm to obtain a more accurate homography matrix for image alignment. The second highlight is the use of content-aware blocks to predict probability maps, guiding feature extraction. Compared to existing feature extraction methods, our method effectively enhances the extraction of locally effective features.

[0017] 1. Image pre-alignment based on model fitting algorithm

[0018] Unlike existing methods that use RANSAC to estimate the homography matrix for pre-alignment of the input image, the method of this invention uses the recently proposed robust model fitting method MAGSAC++ to estimate the homography matrix H. BA H BA Used to distort image I B Image I was obtained Bwarped (like Figure 2 (As shown). This method uses MAGSAC++ to more efficiently align two images, and then searches for a more accurate match. At this stage, as... Figure 3 As shown in (c), the MAGSAC++ algorithm achieves better results than RANSAC in the homography estimation task.

[0019] 2. Content-aware blocks

[0020] Feature-based matching methods typically achieve satisfactory performance on popular datasets, but they are highly dependent on the quantity and quality of features. When images contain challenging scenarios such as non-planar images, repeating images, or images with varying lighting, performance may deteriorate due to an insufficient number of effective features. Therefore, feature matching requires more effective features.

[0021] To address the aforementioned issues, this method incorporates a content-aware block prior to VGG-19. The content-aware block consists of a feature extractor and a mask predictor, which improves the quantity and quality of useful features. The feature extractor initially extracts feature maps from the input image. The mask predictor predicts probability maps, assigning higher probabilities to locations with more valid content. These probability maps are then used to weight the feature maps of the input image. The content-aware block is as follows: Figure 4 As shown.

[0022] Feature Extractor: To enable the network to autonomously learn deep features of image pairs, we use a fully convolutional network to form a feature extractor f(·), the details of which are shown in Table 1. It accepts an input of size H*W*1 and generates a feature map of size H*W*C. For the input image I... A and I B The feature extractors share weights and generate feature maps F. A and F B ,Right now

[0023] F i =f(I i ), i∈{A,B}

[0024] Mask Predictor: This method establishes a network to automatically learn the locations of effective features; the detailed structure is shown in Table 2. This network m(·) generates an inlier probability map, highlighting locations in the feature map that contribute more. The size of the probability map is related to the feature map F. A and F B Since the features extracted by f(·) are of the same size, the features are further weighted using the probability map, and then the two weighted feature maps G are used. A and G B Enter VGG-19, that is

[0025] M i =m(I i ),G i =F i M i , i∈{A,B}

[0026] Table 1 Feature Extractor Structure

[0027]

[0028] Table 2 Mask Predictor Structure

[0029]

[0030]

[0031] We typically evaluate feature matching tasks on image sequences based on illumination and viewpoint variations. We work on HPatches...

[16] The method of this invention was validated on a dataset containing 116 image sets, each consisting of 6 images of the same scene, taken from different viewpoints or under different lighting conditions, including planar and non-planar scenes. Each image set also includes a homography matrix as labels. In the experiments, we compared it with the classic algorithm SIFT. [6] SURF [7] ORB

[17] KAZE [8] AKAZE [9] And the deep learning-based algorithm SuperPoint

[12] Patch2Pix

[18] and DFM

[15] We also compared the results. Additionally, we removed the content-aware block (without CA) from our method to verify its impact. In our experiments, we used Mean Match Accuracy (MMA) to measure the performance of each method. MMA describes the average percentage of correctly matched features (i.e., inliers) across the entire dataset. Matches were considered inliers if the reprojection error (calculated from the label homography matrix and the match) was less than a given threshold.

[0032] We conducted two experiments to measure the effectiveness of the proposed method. (1) All comparison methods used mutual nearest neighbor search and bidirectional ratio test to find the correct match, and measured the MMA at different ratios from 0.1 to 1.0 with a step size of 0.1. (2) For each method, the ratio at which the best performance was obtained was fixed, and the MMA of all methods was compared at pixel thresholds of 1, 3, 5, and 10.

[0033] Figure 5 This demonstrates the MMA of nine feature matching methods on the HPatches dataset under different ratios. Figure 5 As can be observed, the curves for all comparison methods changed significantly with the change in ratio. This indicates that the change in ratio has a more significant impact on other methods, while the method of this invention is less affected.

[0034] Table 3 lists the MMA of each method under different pixel thresholds, demonstrating the strong competitiveness of the method presented in this invention. Compared to other methods (SIFT, SURF, ORB, KAZE, AKAZE, SuperPoint, Patch2Pix, and DFM), this method achieves the highest accuracy at any pixel threshold. At a pixel threshold of 1, the MMA of this method is significantly higher than that obtained by the suboptimal method. When the threshold is set to 5, the MMA of this method is equal to that obtained by Patch2Pix. When the thresholds are set to 1, 3, 5, and 10, the MMA of this method exceeds that of the end-to-end method DFM by 0.19, 0.06, 0.03, and 0.01, respectively. At a threshold of 1, SIFT achieves suboptimal performance (0.60). At a threshold of 3, the suboptimal method is Patch2Pix (0.88). At a threshold of 10, the suboptimal methods are Patch2Pix and DFM (0.96).

[0035] As shown in Table 3, this method demonstrates superiority even without content-aware blocks, achieving an MMA 0.07 higher than the suboptimal method (SIFT) when the pixel threshold is 1. When the pixel thresholds are set to 3 and 5, the MMA of this method is on par with the suboptimal method. The advantages of this method are further amplified by the introduction of content-aware blocks. Therefore, content-aware blocks can effectively improve the performance of this method.

[0036] Table 3. Optimal MMA for different image matching algorithms (optimal values ​​are indicated in bold).

[0037]

[0038] References:

[0039] [1]Aldana-Iuit J, D Mishkin, Chum O, et al. In the Saddle: Chasing Fast and Repeatable Features[C]. Proceedings of the IEEE International Conference on Pattern Recognition, 2016: 675-680.

[0040] [2]Miroslav, Trajkovi,and,et al.Fast corner detection[J].Image andVision Computing,1998,16(2):75-87.

[0041] [3]Rosten,Edward,Porter,et al.Faster and Better:A Machine LearningApproach to Corner Detection[J].IEEE Transactions on PatternAnalysis&MachineIntelligence,2008,32(1):105-119.

[0042] [4]Gong Y,Kumar S,Rowley H A,et al.Learning Binary Codes for High-Dimensional Data Using Bilinear Projections[C].Proceedings of the IEEEConference on Computer Vision and Pattern Recognition.,2013:484-491.

[0043] [5]Trzcinski T,Lepetit V.Efficient discriminative projections forcompact binary descriptors[C].European Conference on ComputerVision.Springer,Berlin,Heidelberg,2012:228-242.

[0044] [6]Lowe D G.Distinctive image features from scale-invariant keypoints[J].International Journal ofComputer Vision,2004,60(2):91-110.

[0045] [7]Bay H,Tuytelaars T,Gool L V.SURF:Speeded up robust features[C].European Conference on Computer Vision.Springer,Berlin,Heidelberg,2006:404-417.

[0046] [8]Alcantarilla P F,Bartoli A,Davison A J.KAZE features[C].EuropeanConference on Computer Vision.Springer,Berlin,Heidelberg,2012:214-227.

[0047] [9]Alcantarilla P F,Solutions T.Fast explicit diffusion foraccelerated features in nonlinear scale spaces[J].IEEE Transactions onPatternAnalysis&Machine Intelligence,2011,34(7):1281-1298.

[0048]

[10] Fischler M A,Bolles R C.Random Sample Consensus:A paradigm formodel fitting with applications to image analysis and automated cartography[J].Communications of the ACM,1981,24(6):381-395.

[0049]

[11] Brachmann E,Krull A,Nowozin S,et al.DSAC-Differentiable RANSACfor camera localization[C].Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition.2017:6684-6692.

[0050]

[12] DeTone D,Malisiewicz T,Rabinovich A.SuperPoint:Self-supervisedinterest point detection and description[C].Proceedings ofthe IEEE Conferenceon Computer Vision and Pattern Recognition Workshops.2018:224-236.

[0051]

[13] Sarlin P E,DeTone D,Malisiewicz T,et al.SuperGlue:Learningfeature matching with graph neural networks[C].Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2020:4938-4947.

[0052]

[14] Dusmanu M,Rocco I,Pajdla T,et al.D2-Net:A trainable CNN for jointdescription and detection of local features[C].Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2019:8092-8101.

[0053]

[15] Efe U,Ince K G,Alatan A.DFM:A performance baseline for deepfeature matching[C].Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition.2021:4284-4293.

[0054]

[16] Balntas V, Lenc K, Vedaldi A, et al. HPatches: A benchmark and evaluation of handcrafted and learned local descriptors [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 5173-5182.

[0055]

[17] Rublee E, Rabaud V, Konolige K, et al. ORB: An efficient alternative to SIFT or SURF [C]. International Conference on Computer Vision. IEEE, 2011: 2564-2571.

[0056]

[18] Zhou Q, Sattler T, Leal-Taixe L. Patch2Pix: Epipolar-guided pixel-level correspondences[C]. Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. 2021: 4669-4678.

[0057] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A content-aware image feature matching method, characterized in that, First, an improved two-stage feature matching method is proposed. In the first stage, a state-of-the-art model fitting method is used to pre-align image pairs, and the pre-aligned images are used as input for the second stage. Second, a block consisting of a fully convolutional network and a mask predictor is used before the feature extractor to weight the features of the input image. The specific implementation of the first stage is as follows: First, the pre-trained feature extractor VGG-19 is used to extract features from the input image I. A and I B Features are extracted from the feature map of the last layer; secondly, dense nearest neighbor search (DNNS) is used to perform initial matching on the feature map of the last layer; finally, the initial matching is used for homography matrix estimation for pre-alignment. The second stage is implemented as follows: First, the pre-alignment result is input into a feature extractor composed of a content-aware block and VGG-19 to extract features; second, dense nearest neighbor search (DNNS) is used for feature matching; the content-aware block consists of a second feature extractor and a mask predictor. The second feature extractor is used to initially extract the feature map of the input image, and the mask predictor is used to predict the probability map, that is, the location with more effective content has a higher probability. Then the probability map is used to weight the feature map of the input image.

2. The content-aware image feature matching method according to claim 1, characterized in that, The homography matrix H was estimated using the robust model fitting method MAGSAC++. BA H BA Used to distort image I B Image I was obtained Bwarped .

3. The content-aware image feature matching method according to claim 1, characterized in that, The second feature extractor is composed of a fully convolutional network that accepts an input of size H*W*1 and generates a feature map of size H*W*C; for the input image I A and I B The second feature extractor shares weights and generates a feature map F. A and F B ,Right now: F i =f(I i ),i∈{A,B} Where f(·) represents the second feature extractor.

4. The content-aware image feature matching method according to claim 3, characterized in that, The mask predictor automatically learns the locations of effective features by establishing a network m(·), which generates an inlier probability map that highlights locations in the feature map that contribute more. The size of the probability map is related to the feature map F. A and F B Since the features extracted by f(·) are of the same size, the features are further weighted using the probability map, and then the two weighted feature maps G are used. A and G B Enter VGG-19, which means: M i =m(I i ),G i =F i M i ,i∈{A,B}。

Images