Image feature matching method based on parallel enhancement

Abstract

The invention relates to an image feature matching method based on parallel enhancement, and belongs to the technical field of computer vision. Firstly, an enhanced injection block is designed, feature weights are dynamically refined by using a channel attention mechanism, and important details are reserved by enhancing network capability so as to relieve the problem of excessive smoothness. Secondly, in order to alleviate the potential problem that the CNN neglects the weak local context, a parallel fusion block is proposed to integrate shallow and deep features, local details are retained, and the global context is enhanced.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to an image feature matching method based on parallel enhancement. Background Art

[0002] Image feature matching is a basic task in the field of computer vision. Its core lies in accurately matching the correspondence between feature points in two images. This step is the basis and important step in various visual tasks, including camera pose estimation[1], image fusion[2] and image stitching[3]. The traditional feature matching process includes establishing initial correspondence and screening correct correspondence, among which outlier removal is the key link to improve matching accuracy and robustness.

[0003] Traditional outlier removal methods can be divided into three types: resampling-based methods, nonparametric model-based methods, and relaxed methods [4]. Among the resampling-based methods, RANSAC [5] is the most representative method. It identifies the optimal model and inliers through a hypothesis and verification strategy. In subsequent studies, various variants of RANSAC have emerged. For example, MAGSAC [6] replaces the predefined outlier threshold with a strategy based on marginalized noise level, providing greater flexibility in outlier rejection and noise processing in various data scenarios. Among the nonparametric model-based methods, VFC [7] introduces assumptions about the coherence and smoothness of the motion field, providing a new framework for non-rigid matching problems. CRC [8] uses finite Fourier basis to model the motion field, further advancing the application of sparse ideas. Relaxed methods (e.g., [9],

[10] ) do not rely on strict model assumptions, but rely on local structures or statistical information in the data to identify outliers, and can manage more complex scenarios. LPM [9] performs outlier rejection by preserving the local neighboring structure between feature points. GMS

[10] uses the statistical information of the number of feature points in a grid region to determine the true match. Although these traditional methods have made progress, they are unable to cope with highly contaminated data.

[0004] With the rise of deep learning, learning-based outlier removal methods have gradually shown their competitive advantages with the powerful learning and representation capabilities of neural networks. PointNet

[11] uses MLP to optimize the classification and segmentation of point clouds, marking a major breakthrough in learning methods. PointCN

[12] adopts a similar architecture to PointNet and regards the outlier rejection task as a classification problem between inliers and outliers, as well as a basic matrix regression problem within the MLP framework. PonitCN introduces a new context normalization technique that embeds global context. OANet

[13] introduces differential pooling and unpooling layers to capture local context. CLNet

[14] gradually eliminates outliers through a local-to-global consensus learning process. PGFNet

[15] enhances the network's representation ability for inliers and reduces the impact of outliers through an iterative filtering mechanism and the design of GRA blocks.

[0005] These MLP-based methods have shown good results in capturing local and global contexts. However, MLP has limitations in dealing with complex interdependencies between feature points and is difficult to fully capture local and global contexts. To overcome this bottleneck, ConvMatch

[16] attempts to introduce CNN to enhance feature extraction capabilities, but CNN is prone to oversmoothing when dealing with local discontinuities and global contexts, which affects matching accuracy.

[0006] References:

[0007] [1]X.Li,F.Zhuang,Y.Liu,R.Chen,L.Wei,C.Yang,PMA-Net: Progressivemultistage adaptive feature learning for two-view correspondence,Knowledge-Based Systems(2024)111927.

[0008] [2]H.Xu,J.Ma,J.Yuan,Z.Le,W.Liu,RFNet: Unsupervised network formutually reinforcing multi-modal image registration and fusion,in:Proceedingsofthe IEEE / CVF Conference on Computer Vision and Pattern Recognition,2022,pp.19679–19688.

[0009] [3] K. Prokop,D. Heuristic-based image stitching algorithm withautomation of parameters for smart solutions,Expert Systems withApplications241(2024)122792.

[0010] [4]J.Ma,X.Jiang,A.Fan,J.Jiang,J.Yan,Image matching from handcraftedto deep features:A survey,International Journal ofComputer Vision 129(2021)23–79.

[0011] [5]M.A.Fischler,R.C.Bolles,Random Sample Consensus:A paradigm formodel fitting with applications to image analysis and automated cartography,Communications ofthe ACM 24(6)(1981)381–395.

[0012] [6]D.Barath,J.Matas,J.Noskova,MAGSAC:Marginalizing sample consensus,in:Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern525Recognition,2019,pp.10197–10205.

[0013] [7]J.Ma,J.Zhao,J.Tian,A.L.Yuille,Z.Tu,Robust point matching viavector field consensus,IEEE Transactions on Image Processing 23(4)(2014)1706–1721.

[0014] [8]A.Fan,X.Jiang,Y.Ma,X.Mei,J.Ma,Smoothness-driven consensus based oncompact representation for robust feature matching,IEEE Transactions onNeural Networks and Learning Systems 34(8)(2023)4460–4472.

[0015] [9]J.Ma,J.Zhao,J.Jiang,H.Zhou,X.Guo,Locality Preserving Matching,International Journal ofComputer Vision 127(2019)512–531.

[0016]

[10] J.Bian,W.-Y.Lin,Y.Matsushita,S.-K.Yeung,T.-D.Nguyen,M.-M.Cheng,GMS:Grid-based motion statistics for fast,ultra-robust featurecorrespondence,in:Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition,2017,pp.4181–4190.

[0017]

[11] C.R.Qi,H.Su,K.Mo,L.J.Guibas,PointNet:Deep learning on point setsfor 3d classification and segmentation,in:Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2017,pp.652–660.

[0018]

[12] K.M.Yi,E.Trulls,Y.Ono,V.Lepetit,M.Salzmann,P.Fua,Learning to findgood correspondences,in:Proceedings of the IEEE Conference on Computer Cisionand Pattern Recognition,2018,pp.2666–2674.

[0019]

[13] J.Zhang,D.Sun,Z.Luo,A.Yao,L.Zhou,T.Shen,Y.Chen,L.Quan,H.Liao,Learning two-view correspondences and geometry using order-aware network,in:Proceedings of the IEEE / CVF International Conference on Computer Vision,2019,pp.5845–5854.

[0020]

[14] C.Zhao,Y.Ge,F.Zhu,R.Zhao,H.Li,M.Salzmann,Progressivecorrespondence pruning by consensus learning,in:Proceedings of the IEEE / CVFInternational Conference on Computer Vision,2021,pp.6464–6473.

[0021]

[15] X.Liu,G.Xiao,R.Chen,J.Ma,PGFNet:Preference-guided filteringnetwork for two-view correspondence learning,IEEE Transactions on ImageProcessing 32(2023)1367–1378.

[0022]

[16] S. Zhang, J. Ma, ConvMatch: Rethinking network design for two-view correspondence learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, 2023, pp. 3472–3479.. Summary of the invention

[0023] The purpose of the present invention is to alleviate the problems existing in the prior art and provide an image feature matching method based on parallel enhancement, which can effectively remove outliers and improve the accuracy and efficiency of dual-view image feature matching.

[0024] To achieve the above object, the technical solution of the present invention is: an image feature matching method based on parallel enhancement, comprising:

[0025] Design a reinforcement injection block that dynamically refines feature weights using a channel attention mechanism while preserving important details by enhancing network capabilities;

[0026] A parallel fusion block is proposed to integrate shallow and deep features, preserving local details and enhancing global context.

[0027] In one embodiment of the present invention, the method comprises the following steps:

[0028] A. For a given image feature matching image pair, extract and match the two images in the image feature matching image pair to generate an initial corresponding point set, and map the initial corresponding point set into a motion vector field.

[0029] B. Design a reinforcement injection block that integrates a channel attention mechanism, which can selectively emphasize important contextual information in the image while effectively suppressing irrelevant or redundant information;

[0030] C. Design a parallel fusion block. The parallel fusion block includes multiple parallel processing branches. Each branch focuses on processing a subset of image features at different scales. The results of independent processing by each branch are merged through an integration strategy to globally capture and utilize the rich contextual information in the image.

[0031] D. Using the initial corresponding point set processed by the enhanced injection block and the parallel fusion block, predict the inlier probability set of each pair of corresponding points;

[0032] E. Taking the obtained probability set and the initial corresponding point set as input, the weighted eight-point algorithm is used to estimate the essential matrix, and finally an image feature matching model with the best performance is obtained.

[0033] In one embodiment of the present invention, in step A, a scale-invariant feature transform (SIFT) algorithm is used to extract and match features of two images in an image feature matching image pair.

[0034] In one embodiment of the present invention, in step A, the initial corresponding point set is mapped into a motion field vector field through a graph attention network.

[0035] In one embodiment of the present invention, in step B, the enhanced injection block integrating the channel attention mechanism is expressed as follows:

[0036]

[0037] in, represents the feature map processed by the channel attention mechanism, SE(·) represents the SE layer, σ(·) represents the sigmoid activation function, σ(θ) represents the global average pooling on the input feature map, W1 and b1 represent the weight and bias of the first linear transformation, W2 and b2 represent the weight and bias of the second linear transformation; CNR(·) represents the sequential combination of convolution, normalization and ReLU activation functions, Represents the output of the reinforcement injection block.

[0038] In one embodiment of the present invention, in step C, the parallel fusion block includes three convolution branches and one SSE branch, the convolution branches use 1×1 convolution Conv1, 3×3 convolution Conv3 and 5×5 convolution Conv5, these convolution branches are constructed as convolution normalization branches, and each branch focuses on processing image feature subsets at different scales.

[0039] In one embodiment of the present invention, in step C, the parallel fusion block is represented as follows:

[0040]

[0041] in, It is represented as a convolutional normalization branch, which fuses three feature maps in the channel dimension, combining high-resolution low-semantic context from shallow features and deep features with low-resolution high-semantic context, respectively. ε(·) represents the SiLU nonlinear activation function. Indicates direct addition.

[0042] In one embodiment of the present invention, in step E, the weighting strategy of the weighted eight-point algorithm is performed based on the interior point probability.

[0043] The present invention also provides an image feature matching system based on parallel enhancement, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0044] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects: the method of the present invention proposes two modules: an enhanced injection block, which introduces a channel attention mechanism to adaptively adjust feature channel weights, focus on key features, reduce over-smoothing, and improve model learning and analysis capabilities; a parallel fusion block, which enhances the model's ability to capture edge information and weak context by parallel learning of multi-scale and multi-level features; the method of the present invention can effectively remove outliers and improve the accuracy and efficiency of dual-view image feature matching; the method of the present invention can be applied to multiple fields such as camera posture estimation, image fusion and image stitching. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The figure is a flow chart of an embodiment of the method of the present invention.

[0047] Figure 2 This is a diagram showing the effect of an embodiment of the present invention on the YFCC100M dataset.

[0048] Figure 3 This is a diagram showing the effect of an embodiment of the present invention on the SUN3D dataset. DETAILED DESCRIPTION

[0049] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0050] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0051] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0052] The present invention provides an image feature matching method based on parallel enhancement, comprising:

[0053] Design a reinforcement injection block that dynamically refines feature weights using a channel attention mechanism while preserving important details by enhancing network capabilities;

[0054] A parallel fusion block is proposed to integrate shallow and deep features, preserving local details and enhancing global context.

[0055] The following is the specific implementation process of the present invention.

[0056] like Figure 1 As shown, the present invention provides an image feature matching method based on parallel enhancement, which specifically includes the following steps:

[0057] A. For a given image feature matching image pair (i.e., consisting of two interrelated images), the SIFT (Scale Invariant Feature Transform) algorithm is used to extract and match the features of the two images to generate an initial set of corresponding points (NCorrespondences), which is then mapped to a motion field vector field through a graph attention network (GraphAttentionNetworks).

[0058] B. A reinforcement injection block is designed, which integrates the channel attention mechanism. This mechanism can selectively emphasize important contextual information in the image while effectively suppressing irrelevant or redundant information, thereby significantly reducing the over-smoothing phenomenon and improving the recognition and stability of corresponding points.

[0059]

[0060] in represents the feature map processed by channel attention, SE(·) represents the SE layer, σ(·) represents the sigmoid activation function, σ(θ) represents the global average pooling on the input feature map, W1 and b1 represent the weight and bias of the first linear transformation, and similarly W2 and b2 represent the weight and bias of the second linear transformation.

[0061]

[0062] where CNR(·) represents the sequential combination of convolution, normalization, and ReLU activation functions, Represents the output of the reinforcement injection block.

[0063] C. A Parallel Fusion Block is designed. This module contains multiple parallel processing branches, including three convolution branches and one SSE branch. The convolution branches use 1×1 convolution (denoted as Conv1), 3×3 convolution (denoted as Conv3), and 5×5 convolution (denoted as Conv5), which are constructed as convolution normalization branches, and each branch focuses on processing a subset of image features at different scales.

[0064]

[0065] in, It is represented as a convolutional normalization branch, which fuses three feature maps in the channel dimension, combining high-resolution low-semantic context from shallow features and deep features with low-resolution high-semantic context. ε(·) represents the SiLU nonlinear activation function, Indicates direct addition.

[0066] D. Using the initial corresponding point set processed by the enhanced injection block and the parallel fusion block, the graph attention network is further used to predict the inlier probability set of each pair of corresponding points.

[0067] E. The obtained probability set and correspondence set are used as input, and the weighted eight-point algorithm is used to estimate the essential matrix. The weighted strategy is based on the probability of the inner point, so that the corresponding point pairs with high confidence play a greater role in the estimation process, thereby improving the accuracy and robustness of the essential matrix, and finally obtaining an image feature matching model with the best performance.

[0068] Figure 2 , Figure 3 The effect diagrams of the method of this embodiment on the YFCC100M dataset and the SUN3D dataset are respectively shown.

[0069] The present invention also provides an image feature matching system based on parallel enhancement, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0070] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0071] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0072] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0073] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0075] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. An image feature matching method based on parallel enhancement, characterized in that: include: Design a reinforcement injection block that uses the channel attention mechanism to dynamically refine feature weights while preserving important details by enhancing network capabilities; A parallel fusion block is proposed to integrate shallow and deep features, preserving local details and enhancing global context.

2. The image feature matching method based on parallel enhancement according to claim 1, characterized in that: The method comprises the following steps: A. For a given image feature matching image pair, extract and match the two images in the image feature matching image pair to generate an initial corresponding point set, and map the initial corresponding point set into a motion vector field. B. Design a reinforcement injection block that integrates a channel attention mechanism, which can selectively emphasize important contextual information in the image while effectively suppressing irrelevant or redundant information; C. Design a parallel fusion block. The parallel fusion block includes multiple parallel processing branches. Each branch focuses on processing a subset of image features at different scales. The results of independent processing by each branch are merged through an integration strategy to globally capture and utilize the rich contextual information in the image. D. Using the initial corresponding point set processed by the enhanced injection block and the parallel fusion block, predict the inlier probability set of each pair of corresponding points; E. Taking the obtained probability set and the initial corresponding point set as input, the weighted eight-point algorithm is used to estimate the essential matrix, and finally an image feature matching model with the best performance is obtained.

3. The image feature matching method based on parallel enhancement according to claim 2 is characterized in that: In step A, the scale-invariant feature transform (SIFT) algorithm is used to extract and match the features of the two images in the image feature matching image pair.

4. The image feature matching method based on parallel enhancement according to claim 2, characterized in that: In step A, the initial corresponding point set is mapped into a motion field vector field through the graph attention network 5. The image feature matching method based on parallel enhancement according to claim 2, characterized in that: In step B, the reinforcement injection block integrating the channel attention mechanism is expressed as follows: in, represents the feature map processed by the channel attention mechanism, SE(·) represents the SE layer, σ(·) represents the sigmoid activation function, σ(θ) represents the global average pooling on the input feature map, W1 and b1 represent the weight and bias of the first linear transformation, W2 and b2 represent the weight and bias of the second linear transformation; CNR(·) represents the sequential combination of convolution, normalization and ReLU activation functions, Represents the output of the reinforcement injection block.

6. The image feature matching method based on parallel enhancement according to claim 5, characterized in that: In step C, the parallel fusion block includes three convolution branches and one SSE branch. The convolution branches use 1×1 convolution Conv1, 3×3 convolution Conv3, and 5×5 convolution Conv5. These convolution branches are constructed as convolution normalization branches, and each branch focuses on processing image feature subsets at different scales.

7. The image feature matching method based on parallel enhancement according to claim 6, characterized in that: In step C, the parallel fusion block is represented as follows: Among them, ConvNorm(·)=BN(Conv1)⊕BN(Conv3)⊕BN(Conv5), which represents the convolution normalization branch, fusing three feature maps in the channel dimension, combining the high-resolution low-semantic context from shallow features and deep features with the low-resolution high-semantic context, ε(·) represents the SiLU nonlinear activation function, and ⊕ represents direct addition.

8. The image feature matching method based on parallel enhancement according to claim 1, characterized in that: In step E, the weighting strategy of the weighted eight-point algorithm is performed based on the interior point probability.

9. An image feature matching system based on parallel enhancement, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

Images