Feature matching method based on local and global interactive transformer

Abstract

The application discloses a feature matching method based on local and global interactive transformer, and belongs to the technical field of computer vision. The application aims at the problem that the existing feature matching method does not support interaction between global representation and local representation, and influences the matching result. The application comprises the following steps: a basic network module is used to extract fine-grained features and coarse-grained features from a pair of images to be processed, and a position coding module is used to obtain a feature sequence of the coarse-grained features; a global feature extraction submodule is used to obtain global features based on the feature sequence; a local feature extraction submodule is used to obtain local features based on the feature sequence; a feature interaction submodule is used to obtain an interactive feature sequence based on the global features, the local features and the feature sequence; a filtering regression head is used to fuse to obtain an updated feature sequence; a coarse matching module is used to obtain a coarse matching result based on the updated feature sequence; and a fine matching module is used to obtain a fine matching result. The application is used for feature matching of two images.

Description

Technical Field

[0001] This invention relates to a feature matching method based on local and global interactive transformers, belonging to the field of computer vision technology. Background Technology

[0002] Local feature matching, a key technology in computer vision, aims to extract feature points from different images and establish correspondences between them. It is widely used in tasks such as image registration, 3D reconstruction, and object recognition. Regarding feature detection and matching strategies, detector-based methods can be categorized into post-detection descriptor, post-descriptor detection, joint detection and descriptor, and graph-based methods. Classic detector-based methods such as SIFT, SURF, and ORB successfully detect unique and robust keypoints in images using feature detectors and add descriptors for pairing. While feature detectors help focus on key regions, they limit the matching search space and cannot extract duplicate keypoints. Over-reliance on handcrafted features makes it difficult for these methods to balance high matching robustness and computational efficiency. When dealing with image pairs with significant viewpoint changes, it is difficult to establish reliable correspondences.

[0003] In recent years, the success of deep learning has provided new solutions for local feature matching. Convolutional Neural Networks (CNNs), through large-scale training data, can learn more robust features and optimize feature detection and extraction modules. To overcome the limitations of traditional nearest neighbor search matching methods, end-to-end learnable matching networks have been proposed, which can optimize the matching process through global contextual information. For example, SuperGlue uses Graph Neural Networks (GNNs) and self-attention mechanisms to learn the matching process, not just features. SuperGlue combines self-attention and cross-attention through a context aggregation strategy, allowing image features to influence each other, thereby improving the global consistency of matching. This research shows that learning the matching process has a significant impact on improving matching stability compared to simply learning features.

[0004] Thanks to its superior model capabilities and the excellent potential of multi-head attention mechanisms to capture global dependencies, transformers are widely used in detector-free matching methods. Transformer-based detector-free methods overcome the limitations of traditional CNN methods in capturing global contextual information and long-distance feature relationships due to their fixed receptive fields, which is crucial for handling complex scenes and low-texture regions. As a representative work, LoFTR avoids traditional feature detection and description steps by computing initial matching on low-resolution feature maps and then performing local optimization on high-resolution feature maps. Alternating feature updates using self-attention and cross-attention, along with the use of a linear transformer to achieve controllable computational costs, gives LoFTR powerful model capabilities. The global attention mechanism enables it to generate dense matching points in low-texture regions and maintain high matching accuracy under large viewpoint changes. However, the global matching method, the quadratic complexity of the self-attention mechanism, and the ever-increasing model complexity to achieve better performance introduce additional computational overhead to detector-free methods. Restricting the tokens of the self-attention mechanism to specific regions and introducing inductive bias, as well as designing a lightweight backbone with fewer parameters, have become two mainstream approaches. Furthermore, among the current state-of-the-art detector-free methods, some models focus on local feature extraction or use efficient global information aggregation, while others combine the two.

[0005] In terms of local-global fusion, advanced local feature matching methods rely on self-attention for global feature propagation and cross-attention for cross-image information interaction. These methods utilize transformers combined with multi-scale feature extraction and hierarchical fusion to integrate and interact with local-global information. Most studies employ simple local-global sequential structures or use linear operations to combine local and global information through local-global parallel structures, with insufficient research on the bidirectional interaction between them. This leads to existing models potentially neglecting local feature consistency and struggling to integrate effective global information, significantly impacting matching accuracy.

[0006] Although detector-free methods exhibit superior performance in feature matching tasks, some problems still exist:

[0007] (1) Existing detector-free feature matching methods mainly focus on optimizing global features while neglecting the extraction of local adjustments. Local features contain rich geometric information and are key to feature matching. In recent years, several methods have successfully extracted local features; for example, FMRT effectively captures local features at two scales, and ASPaanformer effectively extracts local features through transformer operations within a local window. However, although these methods extract local features, they do not support the interaction between global and local representations. Local features can enhance global features, and conversely, local features can obtain a clearer representation through the global context.

[0008] (2) Current location encoding methods for feature matching mainly emphasize representing the spatial location of feature points within an image. However, in feature matching tasks, location information changes significantly when there are differences in viewpoint and perspective between two images. Therefore, developing a viewpoint-robust location encoding method is of great significance for improving feature matching performance.

[0009] Therefore, a local feature matching network is needed that can achieve high-quality matching performance with multiple feature extraction and interaction modes. Summary of the Invention

[0010] To address the problem that existing feature matching methods do not support the interaction between global and local representations, which affects the matching results, this invention provides a feature matching method based on a local and global interactive transformer.

[0011] The present invention provides a feature matching method based on a local and global interactive transformer, comprising:

[0012] A feature matching network is constructed, including a basic network module, a position encoding module, a local and global interaction module, a coarse matching module, and a fine matching module; the local and global interaction module includes a global feature extraction submodule, a local feature extraction submodule, a feature interaction submodule, and a filtering regression head;

[0013] Image I A and Image I B As a pair of images to be matched; first, image I... A With Image I A As a basic network module for inputting the image to be processed, fine-grained features are extracted. and fine-grained features and coarse-grained characteristics and coarse-grained characteristics coarse-grained characteristics and coarse-grained characteristics The feature sequence is then obtained by position encoding module. and characteristic sequences

[0014] Feature sequence and characteristic sequences The global feature F is obtained by feature extraction through the global feature extraction submodule. g ;

[0015] Feature sequence The local feature F is obtained by feature extraction through the local feature extraction submodule. l ;

[0016] Global feature F g Local features F l and characteristic sequences The interaction feature sequence M is obtained by feature extraction through the feature interaction submodule;

[0017] Interactive feature sequence M and feature sequence The updated feature sequence is obtained by feature fusion after filtering and regression head.

[0018] Similarly, image I B and Image I B As a way to obtain fine-grained features from the image to be processed and fine-grained features and coarse-grained characteristics and coarse-grained characteristics And obtain the feature sequence and characteristic sequences This leads to the updated feature sequence.

[0019] Updated feature sequence and the updated feature sequence As input to the local and global interaction modules, the same processing is performed to obtain the updated feature sequence. Then update the feature sequence and the updated feature sequence As input to the local and global interaction modules, the same processing is performed to obtain the updated feature sequence. Iterate the above process n times to obtain the updated feature sequence. and the updated feature sequence

[0020] The coarse matching module is based on the updated feature sequence. and the updated feature sequence The inner product is calculated to obtain the score matrix S, which in turn yields the soft allocation matrix D; then, combined with the image I... A and Image I B Key point descriptions yield image I A and Image IB coarse matching result p c ;

[0021] Fine-grained features and fine-grained features For the same feature, it is represented as a fine-grained feature. Fine-grained features and fine-grained features For the same feature, it is represented as a fine-grained feature. The fine matching module is based on the coarse matching result p c Fine-grained characteristics and fine-grained features Feature matching is performed to obtain image I. A and Image I B Feature fine matching result P f ;

[0022] During the n iterations, after each iteration, the network parameters of the local and global interaction module, the coarse matching module, and the fine matching module are adjusted according to the loss function.

[0023] According to the feature matching method based on local and global interactive transformers of the present invention, the position encoding module performs coarse-grained feature matching. Position encoding is performed to obtain the feature sequence. The method is as follows:

[0024] Use W pos As a coarse-grained feature The spatial position index in the X direction, H pos As a coarse-grained feature The Y-direction spatial position index is used; 1×1, 3×1, and 5×1 one-dimensional convolutions are performed on the pixels determined by the X-direction and Y-direction spatial position index points respectively to obtain multi-scale positional codes (x1,y1), (x2,y2), and (x3,y3):

[0025]

[0026] In the formula, ReLU(·) represents the ReLU activation function, BNorm(·) represents the batch normalization operation, and Conv(·) represents the convolutional layer;

[0027] Learnable adaptive fusion weights based on convolution results are constructed through linear transformation. Then, the multi-scale location codes are weighted and fused:

[0028]

[0029] In the formula, Softmax(·) represents the softmax normalization process, 1B×3 B represents a fixed vector initialized to a constant of 1, and B represents the size of the patch. Represents learnable linear transformation weights. x represents the bias term. pe Weighted encoding in the X direction, y pe Weighted encoding in the Y direction;

[0030] Weighted encoding of the X direction x pe Weighted encoding in the Y direction y pe Mapping to a higher-dimensional space yields the weighted encoding x′ in the X direction of the higher-dimensional space. pe Weighted encoding y′ in the Y direction of high-dimensional space pe :

[0031] x′ pe =Conv 1×1 (x pe ),y′ pe =Conv 1×1 (y pe (3);

[0032] Then, horizontal and vertical embeddings are superimposed through a broadcast mechanism to obtain the embedded encoding x″. pe and post-embedding encoding y″ pe Then, the 2D coordinates Epos are constructed through fusion. A :

[0033] Epos A =x″ pe +y″ pe (4);

[0034] Then the characteristic sequence for:

[0035]

[0036] In the formula, rearrange(·) represents size adjustment, and ImgSeq(·) represents converting the image into a sequence.

[0037] According to the feature matching method based on local and global interactive transformers of the present invention, the feature sequence and characteristic sequences The global feature F is obtained by feature extraction through the global feature extraction submodule. g The method is as follows:

[0038] Use fully connected layers to process feature sequences and characteristic sequences Perform linear transformations to obtain the query vector Q′, key vector K′, and value vector V′:

[0039] Q′=UWQ ,K′=RW K V′=RW V (6),

[0040] Where U represents R represents W Q W is a learnable query weight matrix used for linear feature transformation. K W is the learnable key weight matrix used for linear feature transformation. V This is the learnable weight matrix used for linear feature transformation;

[0041] Rearranging (Q′,K′,V′) yields Q″,K″,V″:

[0042] Q″,K″,V″=rearrange(Q′,K′,V′)(7);

[0043] The binary mask m corresponding to U U The binary mask m corresponding to R R Convert to floating-point type to obtain floating-point mask and floating-point mask Further fine-tuning and element-wise expansion of Q″, K″, and V″ yields Q″′, K″′, and V″′:

[0044]

[0045] m U ,m R ∈{0,1} N , where N is the image I A The number of grid cells;

[0046] In the formula, ⊙ represents element-wise multiplication, ∈ represents the minimum value, and ∈ = 1 × 10 -6 1 {·} For indicator functions;

[0047] By permuting Q″′, K″′, and V″′ along the channel dimension, we obtain the query vector Q, key vector K, and value vector V for attention calculation. We then calculate KV and normalize Q and KV using the L2 norm in the last dimension, adjusting the result with parameter γ to obtain the adjusted probability distribution plot. and after adjustment

[0048]

[0049] In the formula, ||·||2 represents the normalization process of the L2 norm;

[0050] Layer normalization is performed on the probability distribution graph to obtain the integrated tensor A:

[0051]

[0052] In the formula, LNorm(·) represents the layer normalization operation, Mer(·) represents the multi-head merging operation, and W A This is the learnable weight matrix used for linear feature transformation;

[0053] Adjusting the size of the integrated tensor A yields the global feature F. g :

[0054] F g =SeqImg(rearrange(A))(11),

[0055] In the formula, SeqImg(·) represents the sequence to image conversion operation.

[0056] According to the feature matching method based on local and global interactive transformers of the present invention, the feature sequence The local feature F is obtained by feature extraction through the local feature extraction submodule. l The method is as follows:

[0057] After rearranging the dimensions of the feature sequence U, it is transformed into a feature map U. img Using 3×3 depthwise separable convolution pairs on feature map U img Local neighborhood information is aggregated and the activation values ​​of each channel are normalized; then, depthwise separable convolution is used to further fuse the features in the spatial dimension to obtain the focused feature F. u :

[0058] U img =SeqImg(rearrange(U))

[0059] F u =DW 3×3 (BNorm(DW 3×3 (U img ))) ( 12),

[0060] In the formula, DW(·) represents a depth-separable convolutional layer;

[0061] Then the local feature F is obtained. l :

[0062] F l =F u ⊙Sig(F u (13),

[0063] In the formula, Sig(·) represents the Sigmoid activation function.

[0064] According to the feature matching method based on local and global interactive transformers of the present invention, the global feature F g Local features F l and characteristic sequences The method for obtaining the interactive feature sequence M through feature extraction by the feature interaction submodule is as follows:

[0065] From global feature F g and local features F l Obtain interaction features and interaction features

[0066]

[0067] Then with feature map U img Adding them together, we obtain the equilibrium characteristic F. i :

[0068]

[0069] In the formula This represents element-wise addition.

[0070] Then the interaction feature sequence M is:

[0071] M = rearrange(ImgSeq(F) i ))(16).

[0072] According to the feature matching method based on local and global interactive transformers of the present invention, the interactive feature sequence M and the feature sequence The updated feature sequence is obtained by feature fusion after filtering and regression head. The method is as follows:

[0073] A multilayer perceptron (MLP) is used to perform coarse matching on the interaction feature sequences M and U: the interaction feature sequences M and U are matched along...

[0074] The splicing features are obtained by splicing along the spatial dimension, and then the splicing features are split into F1 and F2 along the channel dimension:

[0075] F1,F2=Chunk(Concat(U,M)W 24 (17),

[0076] In the formula, Concat represents the splicing operation along the spatial dimension, Chunk represents the splitting along the channel dimension, and W 24 This represents the learnable weight matrix used for linear feature transformation. For the set of real numbers, Number of channels;

[0077] After performing layer normalization on F2, the dimension order is adjusted and the result is converted into a feature map. Then use 3×3 depthwise separable convolution to extract feature maps. Spatial relationships and local structures, and convert them back to sequences.

[0078]

[0079] F1 and sequence The fused result is then subjected to GELU activation and linear transformation with a fully connected layer to obtain a coarse-matching feature MLP. C :

[0080]

[0081] In the formula W 21 This represents the learnable weight matrix used for linear feature transformation.

[0082] coarse matching feature MLP C Each feature dimension is normalized and then connected using residuals to obtain the updated feature sequence F:

[0083]

[0084] The updated feature sequence F is used as the updated feature sequence.

[0085] According to the feature matching method based on local and global interactive transformers of the present invention, the updated feature sequence and the updated feature sequence The method to obtain it is as follows:

[0086]

[0087] According to the feature matching method of the local and global interactive transformer of the present invention, the coarse matching module obtains the coarse matching result P. c The process is as follows:

[0088] For the updated feature sequence and the updated feature sequence The inner product is calculated to obtain the score matrix S, which in turn yields the soft allocation matrix D.

[0089]

[0090] In the formula, p represents the number of rows, q represents the number of columns; <·,·> represent the inner product, and Softmax(S) c This indicates that a softmax operation is performed on each column, Softmax(S). rThis indicates that a softmax operation is performed on each row;

[0091] From the soft assignment matrix D, obtain coarse matching point pairs with high confidence to obtain the coarse matching index set M. c :

[0092]

[0093] In the formula, τ represents the matching threshold, and MNN represents the nearest neighbor criterion;

[0094] Then we get the coarse matching result P. c :

[0095]

[0096] In the formula P A For image I A The key point, P B For image I B The key point.

[0097] According to the feature matching method based on local and global interactive transformers of the present invention, the fine matching module obtains the feature fine matching result P. f The method is as follows:

[0098] The fine matching module matches each matching point (P) A (p),P B (q)) in fine-grained features and fine-grained features The position in the middle, the local window feature with a cropping size of ω. and local window features ω represents the width of the patch; it represents the width of each pair of local window features. and local window features Flattened into a sequence, combined into and and and as well as and A local and global interaction module is used to perform interaction with feature sequences. and characteristic sequences The same operation is performed, and this is repeated n times, to obtain the corresponding aggregated features. and aggregation features In the corresponding fine-matching module, the filtering regression head in the local and global interaction module uses a multilayer perceptron (MLP) to perform fine-matching processing on the interaction feature sequences M and U, resulting in fine-matching features from the MLP. F for:

[0099] MLP F =GELU(Concat(U,M)·W 24 )·W 41 (25),

[0100] In the formula W 41 The learnable weight matrix is ​​used for linear feature transformation.

[0101] In the processing of the corresponding fine-matching module, fine-matching feature MLP is adopted. F Replace coarse matching feature MLP C ;

[0102] The fine-matching module is based on aggregated features and aggregation features Construct normalized coordinates E (x,y) :

[0103]

[0104] In the formula, (x,y) are the coordinates of the points in the normalized coordinate system;

[0105] Map the normalized coordinates to image I A and Image I B The original image coordinates are used to obtain the feature matching result P. f :

[0106]

[0107] In the formula For the t-th coarse matching point pair, the corresponding image I A The key point, For the t-th coarse matching point pair, the corresponding image I B The key point is that E(t) is the corresponding E (x,y) The original image coordinates are given, s is the downsampling ratio, and κ is the number of coarse matching point pairs.

[0108] According to the feature matching method based on local and global interactive transformers of the present invention, the loss function L = L m +L r ;

[0109] Where the matching loss L m for:

[0110]

[0111] In the formula |D gt | represents the actual number of matches;

[0112] Regression loss L r for:

[0113]

[0114] In the formula δ gt δ is the actual offset, and δ is the calculated offset value.

[0115] The beneficial effects of this invention are as follows: The method of this invention is applied to the field of deep learning, constructing a feature matching network based on a multi-scale Transformer. This method uses an adaptive selection mechanism to adjust the cross-scale feature representation between image information and the scene, thereby establishing a reliable correspondence. This method effectively achieves multi-level feature adaptive negative feedback adjustment and constructs adaptive local and global information interaction, thus effectively enhancing key local feature information and multi-scale fusion, regressing the six-degree-of-freedom pose of the image, predicting uncertainties, and completing visual localization. Attached Figure Description

[0116] Figure 1 This is the overall network framework diagram of the feature matching method based on local and global interactive transformers described in this invention;

[0117] Figure 2 This is a schematic diagram illustrating the principle of the local and global interaction module;

[0118] Figure 3 This is a schematic diagram of the SAPE (Position Encoding Module).

[0119] Figure 4 This is a schematic diagram of the feature processing performed by the multilayer perceptron (MLP) in the filter regression head for the coarse matching module and the fine matching module, respectively. Detailed Implementation

[0120] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0121] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0122] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0123] Specific Implementation Method 1: Combination Figures 1 to 4 As shown, this invention provides a feature matching method based on a local and global interactive transformer, comprising:

[0124] A feature matching network is constructed, including a basic network module, a position encoding module, a local and global interaction module, a coarse matching module, and a fine matching module; the local and global interaction module includes a global feature extraction submodule, a local feature extraction submodule, a feature interaction submodule, and a filtering regression head;

[0125] Image I A and Image I B As a pair of images to be matched; first, image I... A With Image I A As a basic network module for inputting the image to be processed, fine-grained features are extracted. and fine-grained features and coarse-grained characteristics and coarse-grained characteristics coarse-grained characteristics and coarse-grained characteristics The feature sequence is then obtained by position encoding module. and characteristic sequences

[0126] Feature sequence and characteristic sequences The global feature F is obtained by feature extraction through the global feature extraction submodule. g ;

[0127] Feature sequence The local feature F is obtained by feature extraction through the local feature extraction submodule. l ;

[0128] Global feature F g Local features F l and characteristic sequences The interaction feature sequence M is obtained by feature extraction through the feature interaction submodule;

[0129] Interactive feature sequence M and feature sequence The updated feature sequence is obtained by feature fusion after filtering and regression head.

[0130] Similarly, image I B and Image I B As a way to obtain fine-grained features from the image to be processed and fine-grained features and coarse-grained characteristics and coarse-grained characteristics And obtain the feature sequence and characteristic sequences This leads to the updated feature sequence.

[0131] Updated feature sequence and the updated feature sequence As input to the local and global interaction modules, the same processing is performed to obtain the updated feature sequence. Then update the feature sequence and the updated feature sequence As input to the local and global interaction modules, the same processing is performed to obtain the updated feature sequence. Iterate the above process n times to obtain the updated feature sequence. and the updated feature sequence

[0132] The coarse matching module is based on the updated feature sequence. and the updated feature sequence The inner product is calculated to obtain the score matrix S, which in turn yields the soft allocation matrix D; then, combined with the image I... A and Image I B Key point descriptions yield image I A and Image I B coarse matching result P c ;

[0133] Fine-grained features and fine-grained features For the same feature, it is represented as a fine-grained feature. Fine-grained features and fine-grained features For the same feature, it is represented as a fine-grained feature. The fine matching module is based on the coarse matching result P c Fine-grained characteristics and fine-grained features Feature matching is performed to obtain image I. A and Image I B Feature fine matching result P f ;

[0134] During the n iterations, after each iteration, the network parameters of the local and global interaction module, the coarse matching module, and the fine matching module are adjusted according to the loss function.

[0135] This implementation proposes a feature matching network (MLGMatcher), a local feature matching network that achieves high-quality matching performance through multi-feature extraction and interactive modes.

[0136] When humans observe the details of an object and perform image matching, they typically scan the image repeatedly and establish key correspondences between various features. The former indicates that network depth is beneficial to improving matching ability. The latter reflects the important role of bidirectional interaction between local and global features in visual matching projects. Based on this fact, this implementation proposes MLGMatcher, a novel detector-free method based on a local-global interaction transformer, for efficient local feature matching. MLGMatcher consists of five parts: a base network, a multi-scale perceptual positional encoding (SAPE), a local-global interaction transformer (LBIFormer), a coarse matching module (CMM), and a fine matching module (FMM). Given two images, MLGMatcher extracts rich feature representations through a CNN-based backbone and adds positional encodings to them based on scale using SAPE. Then, LBIFormer is used to achieve adaptive modeling of all local, global, and local-global bidirectional interactions in an efficient and stable manner. Specifically, global features extracted by GFE and local features extracted by LFE are cross-modulated point-by-point to achieve bidirectional feature enhancement. Next, the FW-FFN embeds an inter-channel dynamic control module to further mix feature information through non-linear transformations. After n rounds of self-attention and cross-attention processing, the enhanced features are used by the CMM to generate fine-matching candidate regions. The fine-matching method further refines the matching, aligns the attention information, and generates sub-pixel-level matching results.

[0137] Further, basic network architecture design (backbone):

[0138] Given image pairs Using a known ResNet18 as the backbone, downsampling is extracted.

[0139] coarse-grained characteristics and fine-grained features C represents the number of channels, and H and W represent the height and width of the input image, respectively. Use a grid of N = H / 8 × W / 8 to pair the image pair I. A ,I B Divide the area and define its center position as the key point. For MLGMatcher, use As the initial visual descriptor for key points.

[0140] Multiscale Perceptual Position Coding (SAPE):

[0141] Unlike traditional methods that encode fixed sine / cosine positions for only one-dimensional sequences, SAPE uses coarse-grained features. This method takes 2D structured data as input and leverages multi-scale receptive fields to extract rich geometric information, enabling efficient encoding of spatial location information. It allows models to encode spatial priors in a learnable, multi-scale, and orientation-aware manner, which is crucial for tasks requiring dense correspondences or geometric reasoning (e.g., visual matching, 2D alignment, or pose estimation).

[0142] for Explicitly distinguish spatial position indexes in the X and Y directions This involves using pixel coordinate tensors to achieve two-dimensional spatial coupling. For each direction, different sized one-dimensional convolutional kernels (1×1, 3×1, 5×1) are used to obtain feature representations of multiple receptive fields, followed by channel dimension normalization and ReLU nonlinear transformation.

[0143] Combination Figure 1 and Figure 3 As shown, the location encoding module handles coarse-grained features. Position encoding is performed to obtain the feature sequence. The method is as follows:

[0144] Use W pos As a coarse-grained feature The spatial position index in the X direction, H pos As a coarse-grained feature The Y-direction spatial position index is used; 1×1, 3×1, and 5×1 one-dimensional convolutions are performed on the pixels determined by the X-direction and Y-direction spatial position index points respectively to obtain multi-scale positional codes (x1,y1), (x2,y2), and (x3,y3):

[0145]

[0146] In the formula, ReLU(·) represents the ReLU activation function, BNorm(·) represents the batch normalization operation, and Conv(·) represents the convolutional layer;

[0147] Learnable adaptive fusion weights based on convolution results are constructed through linear transformation. Then, the multi-scale location codes are weighted and fused:

[0148]

[0149] In the formula, Softmax(·) represents the softmax normalization process, 1 B×3 B represents a fixed vector initialized to a constant of 1, and B represents the size of the patch. Represents learnable linear transformation weights. Indicates the bias term. xpe Weighted encoding in the X direction, y pe Weighted encoding in the Y direction;

[0150] Weighted encoding of the X direction x pe Weighted encoding in the Y direction y pe Mapping to a higher-dimensional space yields the weighted encoding x′ in the X direction of the higher-dimensional space. pe Weighted encoding y′ in the Y direction of high-dimensional space pe :

[0151] x′ pe =Conv 1×1 (x pe ), y′ pe =Conv 1×1 (y pe (3);

[0152] Then, horizontal and vertical embeddings are superimposed through a broadcast mechanism to obtain the embedded encoding. and post-embedding encoding Then perform fusion to construct 2D coordinates Epos A :

[0153] Epos A =x″ pe +y″ pe (4);

[0154]

[0155] Finally, positional encoding is added to the original features, and the size is adjusted to a sequence. Then the characteristic sequence for:

[0156]

[0157] In the formula, rearrange(·) represents size adjustment, and ImgSeq(·) represents converting the image into a sequence.

[0158] Similarly

[0159] Figure 1 In this context, "Linear" represents a linear transformation.

[0160] Local-to-Global Interaction Transformer (LBIFormer):

[0161] Define based on the position-encoded sequence. As input to LBIFormer, it enables deep feature aggregation. LBIFormer consists of four main parts: Global Feature Extraction (GFM), Local Feature Extraction (LFM), Local-Global Interaction (FW-FFN) module, and Filtered Regression Head (FW-FFN). They are designed to model local and global information and their bidirectional interactions in a context-aware manner, effectively improving feature fusion capabilities and matching accuracy.

[0162] Combination Figure 2 As shown, the global feature extraction module transforms the query vector into a global query context, using element-wise multiplication to model the correlation between all keypoints. LBIFormer alternates between self-sequence and cross-sequence attention based on the label type information contained in the sub-layer list, allowing the two sets of features to continuously reinforce each other and interact. For input features, U and R are the same for self-attention (…). or However, for cross-attention, U and R are different. or ).

[0163] Feature sequence and characteristic sequences The global feature F is obtained by feature extraction through the global feature extraction submodule. g The method is as follows:

[0164] First, MAA uses three fully connected layers to process the feature sequence. and characteristic sequences Perform linear transformations to obtain the query vector Q′, key vector K′, and value vector V′:

[0165] Q′=UW Q ,K′=RW K V′=RW V (6),

[0166] Where U represents R represents W Q W is a learnable query weight matrix used for linear feature transformation. K W is the learnable key weight matrix used for linear feature transformation. V This is the learnable weight matrix used for linear feature transformation;

[0167] Rearrange (Q′,K′,V′) to obtain To calculate the attention matrix, where h represents the number of attention heads, and

[0168] Q″,K″,V″=rearrange(Q′,K′,V′) (7);

[0169] To mask out information from invalid regions such as padded or irrelevant areas, element-wise multiplication using a mask is performed to minimize the features at the corresponding locations. If no mask was set in the input dataset, this step is unnecessary.

[0170] The binary mask m corresponding to U U The binary mask m corresponding to R R Convert to floating-point type to obtain floating-point mask and floating-point mask Further fine-tuning and element-wise expansion of Q″, K″, and V″ yields Q″′, K″′, and V″′:

[0171]

[0172] m U ,m R ∈{0,1} N , where N is the image I A The number of grid cells;

[0173] In the formula, ⊙ represents element-wise multiplication, ∈ represents the minimum value, and ∈ = 1 × 10 -6 1 {·} For indicator functions;

[0174] By permuting Q″′, K″′, and V″′ along the channel dimension, we obtain the query vector Q, key vector K, and value vector V for attention calculation. and These are used as the query vector Q, key vector K, and value vector V in attention computation. To ensure that each element has the same scale, the computation... To describe the relevance of each position, Q and KV are normalized using the L2 norm in the last dimension, and then adjusted using the parameter γ. The matrix obtained by the dot product represents the attention weight at each position, and normalization transforms it into a probability distribution; the adjusted probability distribution plot is then obtained. and after adjustment

[0175]

[0176] In the formula, ||·||2 represents the normalization process of the L2 norm;

[0177] Next, the attention score between the query vector and the dot product is calculated to measure their similarity. Based on this, the outputs of the multi-head attention are merged dimensionally, and a linear projection is performed to further learn the cross-head feature relationships, enhancing information interaction. This helps adjust the way information is combined from different attention heads, making it more expressive. To reduce the impact of scale instability on gradient propagation, each token of the feature tensor is normalized layer by layer to have zero mean and unit variance, resulting in the integrated tensor A. This operation accelerates convergence and improves robustness.

[0178] Layer normalization is performed on the probability distribution graph to obtain the integrated tensor A:

[0179]

[0180] In the formula, LNorm(·) represents the layer normalization operation, and Mer(·) represents the multi-head merging operation, i.e., the output dimension is... This is the learnable weight matrix used for linear feature transformation;

[0181] Adjusting the size of the integrated tensor A yields the global features.

[0182] F g =SeqImg(rearrange(A))(11),

[0183] In the formula, SeqImg(·) represents the sequence to image conversion operation.

[0184] Local feature extraction submodule:

[0185] Feature sequence The local feature F is obtained by feature extraction through the local feature extraction submodule. l The method is as follows:

[0186] After rearranging the dimensions of the feature sequence U, it is converted into a feature map. Use a 3×3 depthwise separable convolution pair on feature map U img Local neighborhood information is aggregated, and the activation values ​​of each channel are normalized to prevent gradient vanishing; then, depthwise separable convolution is used to further fuse the features in the spatial dimension to obtain focused features.

[0187] U img =SeqImg(rearrange(U))

[0188] F u =DW 3×3 (BNorm(DW 3×3 (U img ))) (12),

[0189] In the formula, DW(·) represents a depth-separable convolutional layer;

[0190] Next, we introduce F u The activated weights are adaptively adjusted to obtain local features.

[0191] F l =F u ⊙Sig(F u (13),

[0192] In the formula, Sig(·) represents the Sigmoid activation function.

[0193] Local-Global Interaction Submodule:

[0194] Unlike previous methods that relied entirely on the transformer's ability to capture contextual information for fusing local and global features, this implementation additionally designs a bidirectional adaptive interaction module: this module uses global features F g and local features F l As input, modulation based on gating weights is performed to achieve complementary fusion of bidirectional information flows.

[0195] For feature maps F with rich local and global contextual information g ,F l By using their activated weights to adaptively adjust each other, information fusion is enhanced and a local-global adaptive interaction is established.

[0196] Global feature F g Local features F l and characteristic sequences The method for obtaining the interactive feature sequence M through feature extraction by the feature interaction submodule is as follows:

[0197] From global feature F g and local features F l Obtain interaction features and interaction features

[0198]

[0199] Element-wise multiplication is used to blend local-global features, and a 1×1 convolution is used to blend the channels, enhancing feature representation. To further enhance local feature representation, the results are compared with those in the above text. img By aggregating neighborhood information and adding enhanced features that have been normalized, the excessive influence of global interaction information on the local structure is avoided. This feature mixture and balancing yields the balanced feature F. i :

[0200]

[0201] In the formula, ⊕ represents element-wise addition;

[0202] Finally, the interaction feature map Reorganized into sequences for subsequent processing Then the interaction feature sequence M is:

[0203] M = rearrange(ImgSeq(F) i ))(16).

[0204] Furthermore, the Filtered Regression Head (FW-FFN):

[0205] FW-FFN defines a spatial gating module to enhance local information during coarse matching, while using existing methods during fine matching.

[0206] Combination Figure 4 As shown, the interactive feature sequence M and the feature sequence The updated feature sequence is obtained by feature fusion after filtering and regression head. The method is as follows:

[0207] A multilayer perceptron (MLP) is used to perform coarse matching on the interaction feature sequences M and U: the interaction feature sequences M and U are concatenated along the spatial dimension to obtain concatenated features. During the coarse matching process, the concatenated features after expanding the dimensions are split into F1 values ​​along the channel dimension. It is represented as:

[0208] F1,F2=Chunk(Concat(U,M)·W 24 (17),

[0209] In the formula, Concat represents the splicing operation along the spatial dimension, Chunk represents the splitting along the channel dimension, and W 24 This represents the learnable weight matrix used for linear feature transformation. For the set of real numbers, Number of channels;

[0210] After performing layer normalization on F2, the dimension order is adjusted and the result is converted into a feature map. Then use 3×3 depthwise separable convolution to extract feature maps. Spatial relationships and local structures are analyzed to enhance local information and convert it back into a sequence. Adjusted to the same shape as F1 for information fusion:

[0211]

[0212] F1 and sequence The fused result is then subjected to GELU activation and linear transformation with a fully connected layer to obtain a coarse-matching feature MLP. C :

[0213]

[0214] In the formula W 21 This represents the learnable weight matrix used for linear feature transformation.

[0215] coarse matching feature MLP C Each feature dimension is normalized, and residual connections are used to ensure gradient flow, resulting in the updated feature sequence F:

[0216]

[0217] The updated feature sequence F is used as the updated feature sequence.

[0218] This concludes the FW-FFN series.

[0219] all in all:

[0220]

[0221] In the formula, Itr(·) represents the feature interaction submodule.

[0222] Given input The LBIFormer process is executed n times to obtain an enhanced feature sequence containing relative position and rich contextual information. During each execution, two self-attention mechanisms and two cross-attention mechanisms are employed to integrate intra-image / inter-image information. In this implementation, the updated feature sequence after the nth process... and the updated feature sequence The method to obtain it is as follows:

[0223]

[0224] In the formula, LBI represents the operation of the local and global interaction modules.

[0225] Coarse Matching Module (CMM):

[0226] Combination Figure 1 As shown, the coarse matching module obtains the coarse matching result P. c The process is as follows:

[0227] Referring to the method of Loftr, the updated feature sequence is... and the updated feature sequence The score matrix is ​​obtained by performing inner product calculation. Then, the soft allocation matrix is ​​obtained using the dual-softmax method.

[0228]

[0229] In the formula, p represents the number of rows, q represents the number of columns; <·,·> represent the inner product, and Softmax(S) c This indicates that a softmax operation is performed on each column, Softmax(S). r This indicates that a softmax operation is performed on each row;

[0230] Under the condition of satisfying the mutual nearest neighbor criterion (MNN), the soft assignment scores between point pairs are evaluated, and coarse matching point pairs with high confidence are obtained from the soft assignment matrix D, resulting in the coarse matching index set M. c :

[0231]

[0232] In the formula, τ represents the matching threshold, and MNN represents the nearest neighbor criterion;

[0233] Define coarse matching based on its key point coordinates. Define its center position as the key point.

[0234] Then we get the coarse matching result P. c :

[0235]

[0236] In the formula P A For image I A The key point, P B For image I B The key point.

[0237] Fine Matching Module (FMM):

[0238] Combination Figure 1 and Figure 4 As shown, the fine matching module obtains the feature fine matching result P. f The method is as follows:

[0239] Given a coarse match P c MLGMatcher locates each matching point (P) A (p),P B (q)) in fine-grained features The fine matching module determines the position of each matching point (P) based on its position within the range. A (p),P B (q)) in fine-grained features and fine-grained features The position in the middle, the local window feature with a cropping size of ω. and local window features κ represents the number of coarse matching point pairs, and ω represents the width of the patch; each pair of local window features and local window features Flattened into a sequence, combined into and and and as well as and A local and global interaction module is used to perform interaction with feature sequences. and characteristic sequences The same operation is performed, and the LBIFormer is iterated n times to obtain the corresponding aggregated features. and aggregation features In the corresponding fine-matching module, the filtering regression head in the local and global interaction module uses a multilayer perceptron (MLP) to perform fine-matching processing on the interaction feature sequences M and U, obtaining the fine-matching feature MLP. F .

[0240] Based on the traditional transformer method, U and M are concatenated along the channel dimension as input, and a filter-based feedforward network (FW-FFN) is used to extract discriminative features to achieve powerful deep feature aggregation. The multilayer perceptron (MLP) consists of two fully connected layers and a GELU activation function, obtaining richer features by expanding the hidden dimensions between the fully connected layers.

[0241] MLP F =GELU(Concat(U,M)·W 24 )·W 41 (25),

[0242] Concat(·) represents a concatenation operation along spatial dimensions. In the formula W 41 The learnable weight matrix is ​​used for linear feature transformation. GELU(·) represents the GELU activation function.

[0243] In the processing of the corresponding fine-matching module, fine-matching feature MLP is adopted. F Replace coarse matching feature MLP C ;

[0244] Similar to the method in CMM, a similarity matrix is ​​calculated to obtain the similarity distribution for each patch. Then, a softmax attention distribution is constructed for regressing normalized coordinates.

[0245] The fine-matching module is based on aggregated features and aggregation features Construct normalized coordinates E (x,y) :

[0246]

[0247] In the formula, (x,y) are the coordinates of the points in the normalized coordinate system;

[0248] Map the normalized coordinates to image I e and Image I B The original image coordinates are used to obtain the feature matching result P. f :

[0249]

[0250] In the formula For the t-th coarse matching point pair, the corresponding image I A The key point, For the t-th coarse matching point pair, the corresponding image I B The key point is that E(t) is the corresponding E (x,y) The original image coordinates are given, s is the downsampling ratio, and κ is the number of coarse matching point pairs.

[0251] In this embodiment of MLGMatcher, the loss function L = L m +L r ;

[0252] Where the matching loss L m Based on the soft allocation matrix D, we have:

[0253]

[0254] In the formula |D gt | represents the actual number of matches;

[0255] Regression loss L r The value δ is calculated based on the offset:

[0256]

[0257] In the formula δ gt δ is the actual offset, and δ is the calculated offset value.

[0258] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A feature matching method based on a local and global interactive transformer, characterized in that, include: Construct a feature matching network, including a basic network module, a position encoding module, a local and global interaction module, a coarse matching module, and a fine matching module; The local and global interaction module includes a global feature extraction submodule, a local feature extraction submodule, a feature interaction submodule, and a filtering regression head; Image I A and Image I B As a pair of images to be matched; first, image I... A With Image I A As a basic network module for inputting the image to be processed, fine-grained features are extracted. and fine-grained features and coarse-grained characteristics and coarse-grained characteristics coarse-grained characteristics and coarse-grained characteristics The feature sequence is then obtained by position encoding module. and characteristic sequences Feature sequence and characteristic sequences The global feature F is obtained by feature extraction through the global feature extraction submodule. g ; Feature sequence The local feature F is obtained by feature extraction through the local feature extraction submodule. l ; Global feature F g Local features F l and characteristic sequences The interaction feature sequence M is obtained by feature extraction through the feature interaction submodule; Interactive feature sequence M and feature sequence The updated feature sequence is obtained by feature fusion after filtering and regression head. Similarly, image I B and Image I B As a way to obtain fine-grained features from the image to be processed and fine-grained features and coarse-grained characteristics and coarse-grained characteristics And obtain the feature sequence and characteristic sequences This leads to the updated feature sequence. Updated feature sequence and the updated feature sequence As input to the local and global interaction modules, the same processing is performed to obtain the updated feature sequence. Then update the feature sequence and the updated feature sequence As input to the local and global interaction modules, the same processing is performed to obtain the updated feature sequence. Iterate the above process n times to obtain the updated feature sequence. and the updated feature sequence The coarse matching module is based on the updated feature sequence. and the updated feature sequence The inner product is calculated to obtain the score matrix S, which in turn yields the soft allocation matrix D; then, combined with the image I... A and Image I B Key point descriptions yield image I A and Image I B coarse matching result P c ; Fine-grained features and fine-grained features For the same feature, it is represented as a fine-grained feature. Fine-grained features and fine-grained features For the same feature, it is represented as a fine-grained feature. The fine matching module is based on the coarse matching result P c Fine-grained characteristics and fine-grained features Feature matching is performed to obtain image I A and Image I B Feature fine matching result P f ; During the n iterations, after each iteration, the network parameters of the local and global interaction module, the coarse matching module, and the fine matching module are adjusted according to the loss function.

2. The feature matching method based on local and global interactive transformers according to claim 1, characterized in that, The location encoding module provides coarse-grained features. Position encoding is performed to obtain the feature sequence. The method is as follows: Use W pos As a coarse-grained feature The spatial position index in the X direction, H pos As a coarse-grained feature Y-direction spatial position index; One-dimensional convolutions of 1×1, 3×1, and 5×1 are performed on the pixels determined by the spatial position index points in the X and Y directions, respectively, to obtain multi-scale positional codes (x1,y1), (x2,y2), and (x3,y3): In the formula, ReLU(·) represents the ReLU activation function, BNorm(·) represents the batch normalization operation, and Conv(·) represents the convolutional layer; Learnable adaptive fusion weights based on convolution results are constructed through linear transformation. Then, the multi-scale location codes are weighted and fused: In the formula, Softmax(·) represents the softmax normalization process, 1 B×3 B represents a fixed vector initialized to a constant of 1, and B represents the size of the patch. Represents learnable linear transformation weights. x represents the bias term. pe Weighted encoding in the X direction, y pe Weighted encoding in the Y direction; X-direction weighted encoding x pe and Y-direction weighted encoding y pe to a high-dimensional space, obtaining high-dimensional space X-direction weighted encoding x′ pe and high-dimensional space Y-direction weighted encoding y′ pe : x′ pe =Conv 1×1 (x pe ),y′ pe =Conv 1×1 (y pe ) (3); Then, horizontal and vertical embeddings are superimposed through a broadcast mechanism to obtain the embedded encoding x″. pe and post-embedding encoding y″ pe Then, the 2D coordinates Epos are constructed through fusion. A : Epos A =x″ pe +y″ pe (4); Then the characteristic sequence for: In the formula, rearrange(·) represents size adjustment, and ImgSeq(·) represents converting the image into a sequence.

3. The feature matching method based on a local and global interactive transformer according to claim 2, characterized in that, Feature sequence and characteristic sequences The global feature F is obtained by feature extraction through the global feature extraction submodule. g The method is as follows: Use fully connected layers to process feature sequences and characteristic sequences Perform linear transformations to obtain the query vector Q′, key vector K′, and value vector V′: Q′=UW Q ,K′=RW K ,V′=RW V , (6), Where U represents R represents W Q W is a learnable query weight matrix used for linear feature transformation. K W is the learnable key weight matrix used for linear feature transformation. V This is the learnable weight matrix used for linear feature transformation; Rearranging (Q′,K′,V′) yields Q″,K″,V″: Q″,K″,V″=rearrange(Q′,K′,V′) (7); The binary mask m corresponding to U U The binary mask m corresponding to E R Convert to floating-point type to obtain floating-point mask and floating-point mask Further fine-tuning and element-wise expansion of Q″, K″, and V″ yields Q″′, K″′, and V″′: m U ,m R ∈{0,1} N , where N is the image I A The number of grid cells; In the formula, ⊙ represents element-wise multiplication, ∈ represents the minimum value, and ∈ = 1 × 10 -6 1 {·} For indicator functions; By permuting Q″′, K″′, and V″′ along the channel dimension, we obtain the query vector Q, key vector K, and value vector V for attention calculation. We then calculate KV and normalize Q and KV using the L2 norm in the last dimension, adjusting the result with parameter γ to obtain the adjusted probability distribution plot. and after adjustment In the formula, ||·||2 represents the normalization process of the L2 norm; Layer normalization is performed on the probability distribution graph to obtain the integrated tensor A: In the formula, LNorm(·) represents the layer normalization operation, Mer(·) represents the multi-head merging operation, and W A This is the learnable weight matrix used for linear feature transformation; Adjusting the size of the integrated tensor A yields the global feature F. g : F g =SeqImg(rearrange(A)) (11), In the formula, SeqImg(·) represents the sequence to image conversion operation.

4. The feature matching method based on a local and global interactive transformer according to claim 3, characterized in that, Feature sequence The local feature F is obtained by feature extraction through the local feature extraction submodule. l The method is as follows: After rearranging the dimensions of the feature sequence U, it is transformed into a feature map U. img Using 3×3 depthwise separable convolution pairs on feature map U img Local neighborhood information is aggregated and the activation values ​​of each channel are normalized; then, depthwise separable convolution is used to further fuse the features in the spatial dimension to obtain the focused feature F. u : In the formula, DW(·) represents a depth-separable convolutional layer; Then the local feature F is obtained. l : F l =F u ⊙Sig(F u ) (13), In the formula, Sig(·) represents the Sigmoid activation function.

5. The feature matching method based on local and global interactive transformers according to claim 4, Its features are, Global feature F g Local features F l and characteristic sequences The method for obtaining the interactive feature sequence M through feature extraction by the feature interaction submodule is as follows: From global feature F g and local features F l Obtain interaction features and interaction features Then with feature map U img Adding them together, we obtain the equilibrium characteristic F. i : In the formula, ⊕ represents element-wise addition; Then the interaction feature sequence M is: M=rearrange(imgSeq(F i )) (16)。 6. The feature matching method based on local and global interactive transformers according to claim 5, characterized in that, Interactive feature sequence M and feature sequence The updated feature sequence is obtained by feature fusion after filtering and regression head. The method is as follows: A multilayer perceptron (MLP) is used to perform coarse matching on the interaction feature sequences M and U: the interaction feature sequences M and U are concatenated along the spatial dimension to obtain concatenated features, and then the concatenated features are split into F1 and F2 along the channel dimension. F1,F2=Chunk(Concat(U,M)·W 24 ) (17), In the formula, Concat represents the splicing operation along the spatial dimension, Chunk represents the splitting along the channel dimension, and W 24 This represents the learnable weight matrix used for linear feature transformation. For the set of real numbers, Number of channels; After performing layer normalization on F2, the dimension order is adjusted and the result is converted into a feature map. Then use 3×3 depthwise separable convolution to extract feature maps. Spatial relationships and local structures, and convert them back to sequences. F1 and sequence The fused result is then subjected to GELU activation and linear transformation with a fully connected layer to obtain a coarse-matching feature MLP. C : In the formula W 21 This represents the learnable weight matrix used for linear feature transformation. coarse matching feature MLP C Each feature dimension is normalized and then connected using residuals to obtain the updated feature sequence F: F=U⊕LNorm(MLP C (U,M)) (20), The updated feature sequence F is used as the updated feature sequence.

7. The feature matching method based on a local and global interactive transformer according to claim 6, characterized in that, Updated feature sequence and the updated feature sequence The method to obtain it is as follows:

8. The feature matching method based on local and global interactive transformers according to claim 7, characterized in that, The coarse matching module obtains the coarse matching result P. c The process is as follows: For the updated feature sequence and the updated feature sequence The inner product is calculated to obtain the score matrix S, which in turn yields the soft allocation matrix D. In the formula, p represents the number of rows, q represents the number of columns; <·,·> represent the inner product, and Softmax(S) c This indicates that a softmax operation is performed on each column, Softmax(S). r This indicates that a softmax operation is performed on each row; From the soft assignment matrix D, obtain coarse matching point pairs with high confidence to obtain the coarse matching index set M. c : In the formula, τ represents the matching threshold, and MNN represents the nearest neighbor criterion; Then we get the coarse matching result P. c : In the formula P A For image I A The key point, P B For image I B The key point.

9. The feature matching method based on a local and global interactive transformer according to claim 8, characterized in that, The fine matching module obtains the feature fine matching result P. f The method is as follows: The fine matching module matches each matching point (P) A (p),P B (q)) in fine-grained features and fine-grained features The position in the middle, the local window feature with a cropping size of ω. and local window features ω represents the width of the patch; it represents the width of each pair of local window features. and local window features Flattened into a sequence, combined into and and and as well as and A local and global interaction module is used to perform interaction with feature sequences. and characteristic sequences The same operation is performed, and this is repeated n times, to obtain the corresponding aggregated features. and aggregation features In the corresponding fine-matching module, the filtering regression head in the local and global interaction module uses a multilayer perceptron (MLP) to perform fine-matching processing on the interaction feature sequences M and U, resulting in fine-matching features from the MLP. F for: MLP F =GELU(Concat(U,M)·W 24 )·IN 41 (25), In the formula W 41 The learnable weight matrix is ​​used for linear feature transformation. In the processing of the corresponding fine-matching module, fine-matching feature MLP is adopted. F Replace coarse matching feature MLP C ; The fine-matching module is based on aggregated features and aggregation features Construct normalized coordinates E (x,y) : In the formula, (x,y) are the coordinates of the points in the normalized coordinate system; Map the normalized coordinates to image I A and Image I B The original image coordinates are used to obtain the feature matching result P. f : In the formula For the t-th coarse matching point pair, the corresponding image I A The key point, For the t-th coarse matching point pair, the corresponding image I B The key point is that E(t) is the corresponding E (x,y) The original image coordinates are given, s is the downsampling ratio, and κ is the number of coarse matching point pairs.

10. The feature matching method based on a local and global interactive transformer according to claim 9, characterized in that, Loss function L = L m +L r ; Where the matching loss L m for: In the formula |D gt | represents the actual number of matches; Regression loss L r for: In the formula δ gt δ is the actual offset, and δ is the calculated offset value.

Images