A parallel reinforcement-based image feature matching method

Abstract

The present application relates to a kind of image feature matching methods based on parallel reinforcement, belong to computer vision technical field.First, an enhancement injection block is designed, not only utilize channel attention mechanism dynamic refinement feature weight, but also through enhancing network capability to retain important details, to alleviate the problem of excessive smoothing.Second, in order to alleviate the potential problem that CNN ignores weak local context, a parallel fusion block is proposed to integrate shallow and deep features, retain local details and enhance global context.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to an image feature matching method based on parallel reinforcement. BACKGROUND

[0002] Image feature matching, as a basic task in the field of computer vision, is to accurately match the corresponding relationship of feature points in two images. This step is the basis and an important step in various visual tasks, including camera pose estimation [1], image fusion [2] and image stitching [3]. The traditional feature matching process includes establishing initial correspondence and screening correct correspondence, and the outlier rejection is a key link to improve matching accuracy and robustness.

[0003] Traditional outlier rejection methods can be divided into three types: resampling-based methods, non-parametric model-based methods and relaxed methods [4]. In the resampling-based method, RANSAC [5] is the most representative method. It identifies the optimal model and inliers through the strategy of hypothesis and verification. In subsequent studies, various variants of RANSAC have appeared. For example, MAGSAC [6] replaces the pre-defined outlier threshold with an edge-based noise level strategy, providing greater flexibility in outlier rejection and noise handling in various data scenarios. In the non-parametric model-based method, VFC [7] introduces assumptions about the coherence and smoothness of the motion field, providing a new framework for non-rigid matching problems. CRC [8] models the motion field using a finite Fourier basis, further promoting the application of sparse ideas. The relaxed method (for example, [9],

[10] ) does not rely on strict model assumptions, but relies on local structures or statistical information in the data to identify outliers, and can manage more complex scenarios. LPM [9] rejects outliers by preserving the local neighborhood structure between feature points. GMS

[10] uses statistical information about the number of feature points in a grid region to determine true matches. Although these traditional methods have made progress, they are not up to the task when faced with highly contaminated data.

[0004] With the rise of deep learning, learning-based outlier rejection methods gradually show competitive advantages with the powerful learning and representation capabilities of neural networks. PointNet

[11] optimizes the classification and segmentation of point clouds using MLP, marking a major breakthrough in learning methods. PointCN

[12] adopts a similar architecture to PointNet, considering the outlier rejection task as a classification problem between inliers and outliers, as well as a basic matrix regression problem within the MLP framework. PointCN introduces a new context normalization technique that embeds global context. OANet

[13] introduces differential pooling and non-pooling layers to capture local context. CLNet

[14] gradually eliminates outliers through a local-to-global consensus learning process. PGFNet

[15] enhances the network's representation of inliers and reduces the impact of outliers through an iterative filtering mechanism and the design of GRA blocks.

[0005] These MLP-based methods have shown good results in capturing local and global context, however, MLP has limitations in handling complex interdependencies between feature points, making it difficult to fully capture local and global context. To overcome this bottleneck, ConvMatch

[16] attempts to introduce CNN to enhance feature extraction capabilities, but CNN is prone to over-smoothing when dealing with local discontinuity and global context, affecting matching accuracy.

[0006] References:

[0007] [1] X. Li, F. Zhuang, Y. Liu, R. Chen, L. Wei, C. Yang, PMA-Net: Progressive multistage adaptive feature learning for two-view correspondence, Knowledge-Based Systems (2024) 111927.

[0008] [2] H. Xu, J. Ma, J. Yuan, Z. Le, W. Liu, RFNet: Unsupervised network for mutually reinforcing multi-modal image registration and fusion, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 19679-19688.

[0009] [3] K. Heuristic-based image stitching algorithm with automation of parameters for smart solutions, Expert Systems with Applications 241 (2024) 122792.

[0010] [4] J. Ma, X. Jiang, A. Fan, J. Jiang, J. Yan, Image matching from handcrafted to deep features: A survey, International Journal of Computer Vision 129 (2021) 23-79.

[0011] [5] M. A. Fischler, R. C. Bolles, Random Sample Consensus: A paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM 24(6) (1981) 381-395.

[0012] [6] D. Barath, J. Matas, J. Noskova, MAGSAC: Marginalizing sample consensus, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern 525 Recognition, 2019, pp. 10197-10205.

[0013] [7] J. Ma, J. Zhao, J. Tian, A. L. Yuille, Z. Tu, Robust point matching via vector field consensus, IEEE Transactions on Image Processing 23(4) (2014) 1706-1721.

[0014] [8] A. Fan, X. Jiang, Y. Ma, X. Mei, J. Ma, Smoothness-driven consensus based on compact representation for robust feature matching, IEEE Transactions on Neural Networks and Learning Systems 34(8) (2023) 4460-4472.

[0015] [9] J. Ma, J. Zhao, J. Jiang, H. Zhou, X. Guo, Locality Preserving Matching, International Journal of Computer Vision 127 (2019) 512-531.

[0016]

[10] J. Bian, W.-Y. Lin, Y. Matsushita, S.-K. Yeung, T.-D. Nguyen, M.-M. Cheng, GMS: Grid-based motion statistics for fast, ultra-robust feature correspondence, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2017, pp. 4181-4190.

[0017]

[11] C. R. Qi, H. Su, K. Mo, L. J. Guibas, PointNet: Deep learning on point sets for 3d classification and segmentation, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2017, pp. 652-660.

[0018]

[12] K. M. Yi, E. Trulls, Y. Ono, V. Lepetit, M. Salzmann, P. Fua, Learning to find good correspondences, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2666-2674.

[0019]

[13] J. Zhang, D. Sun, Z. Luo, A. Yao, L. Zhou, T. Shen, Y. Chen, L. Quan, H. Liao, Learning two-view correspondences and geometry using order-aware network, in: Proceedings of the IEEE / CVF International Conference on Computer Vision, 2019, pp. 5845-5854.

[0020]

[14] C. Zhao, Y. Ge, F. Zhu, R. Zhao, H. Li, M. Salzmann, Progressive correspondence pruning by consensus learning, in: Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, pp. 6464-6473.

[0021]

[15] X. Liu, G. Xiao, R. Chen, J. Ma, PGFNet: Preference-guided filtering network for two-view correspondence learning, IEEE Transactions on Image Processing 32 (2023) 1367-1378.

[0022]

[16] S. Zhang, J. Ma, ConvMatch: Rethinking network design for two-view correspondence learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, 2023, pp. 3472–3479. Summary of the Invention

[0023] The purpose of this invention is to alleviate the problems of existing technologies and provide an image feature matching method based on parallel enhancement, which can effectively remove outliers and improve the accuracy and efficiency of dual-view image feature matching.

[0024] To achieve the above objectives, the technical solution of the present invention is: an image feature matching method based on parallel enhancement, comprising:

[0025] Design an enhanced injection block that dynamically refines feature weights using a channel attention mechanism, while preserving important details by enhancing network capabilities;

[0026] We propose a parallel fusion block to integrate shallow and deep features, preserving local details and enhancing the global context.

[0027] In one embodiment of the present invention, the method includes the following steps:

[0028] A. For a given image feature matching image pair, perform feature extraction and matching on the two images in the image feature matching image pair to generate an initial corresponding point set, and map the initial corresponding point set into a motion vector field.

[0029] B. Design an enhanced injection block that integrates a channel attention mechanism. The channel attention mechanism can selectively emphasize important contextual information in an image while effectively suppressing irrelevant or redundant information.

[0030] C. Design a parallel fusion block, which includes multiple parallel processing branches. Each branch focuses on processing a subset of image features at different scales. The results of each branch's independent processing are merged through an integration strategy to globally capture and utilize the rich contextual information in the image.

[0031] D. Using the initial set of corresponding points after processing by the enhanced injection block and the parallel fusion block, predict the interior point probability set of each pair of corresponding points;

[0032] E. Using the obtained probability set and initial corresponding point set as input, the weighted eight-point algorithm is used to estimate the essential matrix, and finally a best-performing image feature matching model is obtained.

[0033] In one embodiment of the present invention, in step A, the Scale Invariant Feature Transform (SIFT) algorithm is used to extract and match features between two images in an image feature matching image pair.

[0034] In one embodiment of the present invention, in step A, the initial set of corresponding points is mapped to a motion field vector field through a graph attention network.

[0035] In one embodiment of the present invention, in step B, the enhanced injection block integrating the channel attention mechanism is represented as follows:

[0036]

[0037] in, Let SE(·) represent the feature map processed by the channel attention mechanism, σ(·) represent the SE layer, σ(·) represent the sigmoid activation function, σ(θ) represent the global average pooling on the input feature map, W1 and b1 represent the weights and biases of the first linear transformation, and W2 and b2 represent the weights and biases of the second linear transformation; CNR(·) represents the sequential combination of convolution, normalization, and ReLU activation functions. This indicates the output of the enhanced injection block.

[0038] In one embodiment of the present invention, in step C, the parallel fusion block includes three convolutional branches and one SSE branch. The convolutional branches use 1×1 convolution Conv1, 3×3 convolution Conv3 and 5×5 convolution Conv5. These convolutional branches are constructed as convolutional normalization branches, and each branch focuses on processing a subset of image features at different scales.

[0039] In one embodiment of the present invention, in step C, the parallel fusion block is represented as follows:

[0040]

[0041] in, This is represented as the convolutional normalization branch, which fuses three feature maps along the channel dimension, combining high-resolution low-semantic context from shallow features and low-resolution high-semantic context from deep features, respectively. ε(·) represents the SiLU nonlinear activation function. This indicates direct addition.

[0042] In one embodiment of the present invention, in step E, the weighting strategy of the weighted eight-point algorithm is based on the interior point probability.

[0043] The present invention also provides an image feature matching system based on parallel enhancement, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps of the method described above.

[0044] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.

[0045] Compared with existing technologies, this invention has the following advantages: The method proposes two modules: an enhanced injection block, which introduces a channel attention mechanism to adaptively adjust feature channel weights, focus on key features, reduce oversmoothing, and improve model learning and analysis capabilities; and a parallel fusion block, which enhances the model's ability to capture edge information and weak context by learning multi-scale and multi-level features in parallel. This invention can effectively remove outliers, improving the accuracy and efficiency of feature matching for dual-view images. This method can be applied to multiple fields such as camera pose estimation, image fusion, and image stitching. Attached Figure Description

[0046] Figure 1 This is a flowchart of an embodiment of the method of the present invention.

[0047] Figure 2 This is a diagram showing the effect of an embodiment of the present invention on the YFCC100M dataset.

[0048] Figure 3 This is a diagram showing the effect of an embodiment of the present invention on the SUN3D dataset. Detailed Implementation

[0049] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0050] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0051] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0052] This invention provides an image feature matching method based on parallel enhancement, comprising:

[0053] Design an enhanced injection block that dynamically refines feature weights using a channel attention mechanism, while preserving important details by enhancing network capabilities;

[0054] We propose a parallel fusion block to integrate shallow and deep features, preserving local details and enhancing the global context.

[0055] The following is a detailed implementation process of the present invention.

[0056] like Figure 1 As shown, this invention provides an image feature matching method based on parallel enhancement, specifically including the following steps:

[0057] A. For a given image feature matching image pair (i.e., composed of two related images), the SIFT (Scale Invariant Feature Transform) algorithm is used to extract and match features from the two images, generating an initial set of corresponding points (NCorrespondences). This initial set of corresponding points is then mapped to a motion field vector field using a Graph Attention Network.

[0058] B. A reinforcement injection block was designed, which integrates a channel attention mechanism. This mechanism can selectively emphasize important contextual information in the image while effectively suppressing irrelevant or redundant information, thereby significantly reducing over-smoothing and improving the recognition and stability of corresponding points.

[0059]

[0060] in Let represent the feature map processed by channel attention, SE(·) represent the SE layer, σ(·) represent the sigmoid activation function, σ(θ) represent the global average pooling on the input feature map, W1 and b1 represent the weights and biases of the first linear transformation, and similarly W2 and b2 represent the weights and biases of the second linear transformation.

[0061]

[0062] Where CNR(·) represents the sequential combination of convolution, normalization, and ReLU activation functions. This indicates the output of the enhanced injection block.

[0063] C. A Parallel Fusion Block was designed. This module contains multiple parallel processing branches, including three convolutional branches and one SSE branch. The convolutional branches use 1×1 convolutions (denoted as Conv1), 3×3 convolutions (denoted as Conv3), and 5×5 convolutions (denoted as Conv5), which are constructed as convolution normalization branches, each focusing on processing a subset of image features at different scales.

[0064]

[0065] in, This is represented as the convolutional normalization branch, which fuses three feature maps along the channel dimension, combining high-resolution low-semantic context from shallow features with low-resolution high-semantic context from deep features. ε(·) represents the SiLU nonlinear activation function. This indicates direct addition.

[0066] D. Using the initial set of corresponding points processed by the reinforcement injection block and the parallel fusion block, the probability set of interior points for each pair of corresponding points is further predicted through a graph attention network.

[0067] E. Using the obtained probability set and correspondence set as input, a weighted eight-point algorithm is employed to estimate the essential matrix. The weighting strategy is based on inlier probabilities, allowing high-confidence correspondence pairs to play a greater role in the estimation process, thereby improving the accuracy and robustness of the essential matrix and ultimately yielding a best-performing image feature matching model.

[0068] Figure 2 , Figure 3 The images show the results of the method in this embodiment on the YFCC100M dataset and the SUN3D dataset, respectively.

[0069] The present invention also provides an image feature matching system based on parallel enhancement, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps of the method described above.

[0070] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.

[0071] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An image feature matching method based on parallel enhancement, characterized in that, include: Design an enhanced injection block that dynamically refines feature weights using a channel attention mechanism, while preserving important details by enhancing network capabilities; We propose a parallel fusion block to integrate shallow and deep features, preserving local details and enhancing global context. The method includes the following steps: A. For a given image feature matching image pair, perform feature extraction and matching on the two images in the image feature matching image pair to generate an initial corresponding point set, and map the initial corresponding point set into a motion vector field. B. Design an enhanced injection block that integrates a channel attention mechanism. The channel attention mechanism can selectively emphasize important contextual information in an image while effectively suppressing irrelevant or redundant information. C. Design a parallel fusion block, which includes multiple parallel processing branches. Each branch focuses on processing a subset of image features at different scales. The results of each branch's independent processing are merged through an integration strategy to globally capture and utilize the rich contextual information in the image. D. Using the initial set of corresponding points after processing by the enhanced injection block and the parallel fusion block, predict the interior point probability set of each pair of corresponding points; E. Using the obtained probability set and initial corresponding point set as input, the weighted eight-point algorithm is used to estimate the essential matrix, and finally a best-performing image feature matching model is obtained.

2. The image feature matching method based on parallel enhancement according to claim 1, characterized in that, In step A, the Scale Invariant Feature Transform (SIFT) algorithm is used to extract and match features between two images in the image feature matching image pair.

3. The image feature matching method based on parallel enhancement according to claim 1, characterized in that, In step A, the initial set of corresponding points is mapped to a motion field vector field through a graph attention network.

4. The image feature matching method based on parallel enhancement according to claim 1, characterized in that, In step B, the enhanced injection block with integrated channel attention mechanism is represented as follows: in, Let SE(·) represent the feature map processed by the channel attention mechanism, σ(·) represent the SE layer, σ(·) represent the sigmoid activation function, σ(θ) represent the global average pooling on the input feature map, W1 and b1 represent the weights and biases of the first linear transformation, and W2 and b2 represent the weights and biases of the second linear transformation; CNR(·) represents the sequential combination of convolution, normalization, and ReLU activation functions. This indicates the output of the enhanced injection block.

5. The image feature matching method based on parallel enhancement according to claim 4, characterized in that, In step C, the parallel fusion block includes three convolutional branches and one SSE branch. The convolutional branches use 1×1 convolution Conv1, 3×3 convolution Conv3, and 5×5 convolution Conv5. These convolutional branches are constructed as convolutional normalization branches, and each branch focuses on processing a subset of image features at different scales.

6. The image feature matching method based on parallel enhancement according to claim 5, characterized in that, In step C, the parallel fusion block is represented as follows: in, This is represented as the convolutional normalization branch, which fuses three feature maps along the channel dimension, combining high-resolution low-semantic context from shallow features with low-resolution high-semantic context from deep features, respectively. ε(·) represents the SiLU nonlinear activation function. This indicates direct addition.

7. The image feature matching method based on parallel enhancement according to claim 1, characterized in that, In step E, the weighting strategy of the weighted eight-point algorithm is based on the inlier probability.

8. An image feature matching system based on parallel enhancement, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the steps of the method as described in any one of claims 1-7.

9. A computer-readable storage medium having stored thereon computer program instructions executable by a processor, wherein when the processor executes the computer program instructions, it is able to implement the steps of the method as described in any one of claims 1-7.

Images