A feature matching method based on channel guidance and frequency domain perception

Abstract

This invention belongs to the field of computer vision technology, specifically, it relates to a feature matching method based on channel guidance and frequency domain awareness. The steps are as follows: S1: Enhance the descriptors of two images respectively; S2: Encode the feature points of the two images respectively using a keypoint encoder; S3: Aggregate the enhanced descriptors and encoded feature points of the two images using a self-attention layer to obtain the contextual information of the image's internal features; S4: Interact and fuse the contextual information of the image's internal features from the other image with the contextual information of the image's internal features from the current image using a cross-attention layer; S5: Repeat steps S3 and S4 L times to obtain the final feature matrix sum after L fusions of the two images; S6: Solve for the optimal allocation matrix from the final feature matrix using the Sinkhorn algorithm; S7: Obtain the matching relationship of feature points between the two images based on the optimal allocation matrix.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology. Specifically, this invention relates to a feature matching method based on channel guidance and frequency domain awareness. Background Technology

[0002] Feature matching, as a fundamental component of computer vision, has been widely applied in fields such as scene matching navigation, autonomous localization of UAVs / robots, 3D reconstruction, and map updating. In these tasks, the system first needs to establish pixel-level or feature-level correspondences between two or more images, and then calculate camera pose or scene structure based on geometric models. Therefore, the accuracy and robustness of feature matching directly determine the reliability of subsequent localization or reconstruction results.

[0003] Existing technologies mainly fall into two categories: traditional methods and deep learning-based methods. Traditional methods typically use nearest neighbor or nearest neighbor distance ratio to measure distance within the description subspace, and then use RANSAC and its derivative algorithms to eliminate mismatches. These methods perform well in scenes with rich textures and stable lighting, but in complex scenes such as low lighting and similar structures, the initial matching quality deteriorates, leading to a predicament of scarce interior points and limited geometric accuracy in the geometric verification stage.

[0004] In recent years, deep learning-based methods such as SuperGlue and LightGlue have incorporated graph neural networks or Transformer structures to aggregate global information about feature point locations and their descriptors, thereby improving matching robustness. However, these methods still have significant shortcomings in complex scenarios: firstly, all dimensions of the descriptor are treated equally, and the discrimination signal is easily overwhelmed by redundant dimensions; secondly, they mainly focus on spatial domain feature transformations, neglecting the frequency domain structural information implicit in the descriptors; furthermore, the training phase does not pay sufficient attention to high-confidence mismatches. These shortcomings lead to a decrease in the matching accuracy of existing technologies in complex scenarios.

[0005] In summary, existing technologies suffer from decreased matching accuracy and insufficient robustness in complex scenarios such as low light and similar structures. There is an urgent need to introduce a new matching optimization method to overcome these shortcomings in order to meet the needs of tasks such as high-precision visual positioning and 3D perception. Summary of the Invention

[0006] Objective: In computer vision tasks, achieving accurate geometric matching between two images relies on establishing precise correspondences between feature points. Image matching typically follows a "feature extraction – feature matching" paradigm. First, feature points and descriptors are extracted using traditional methods (SIFT, ORB) or convolutional neural networks. Then, correspondences between feature points are established based on the similarity of the descriptors. However, the initial descriptors have limited discriminative power in complex scenes, hindering matching performance. Therefore, this invention provides a feature matching method based on channel guidance and frequency domain awareness to address the problems of decreased accuracy and insufficient robustness in existing technologies under complex environments such as low light and structural similarity.

[0007] Technical solution: A feature matching method based on channel guidance and frequency domain awareness includes: S1: Enhance the descriptors of the two images respectively; S2: Encode the feature points of the two images separately using a keypoint encoder; S3: The enhanced descriptors and encoded feature points of the two images are aggregated through a self-attention layer to gather the contextual information of the internal features of the images; S4: The image internal feature context information of another image is interacted with and fused with the image internal feature context information of this image through a cross-attention layer; S5: Repeat steps S3 and S4 L times to obtain the final feature matrix after fusing the two images L times respectively. and ; S6: Solve for the optimal allocation matrix from the final feature matrix using the Sinkhorn algorithm; S7: Obtain the matching relationship of feature points between two images based on the optimal allocation matrix.

[0008] Furthermore, regarding the image In The set of descriptors corresponding to each feature point The enhancement process is as follows: Let's take image A as an example: right The feature matrix consists of descriptors corresponding to each feature point. Perform spatial feature compression Each channel All the information below is aggregated into a single value, resulting in a channel-level statistical vector. Each element in the vector represents the global response intensity of the original feature in the corresponding channel, as shown in the formula below:

[0009] C The number of channels for the descriptor; Feature activation Channel statistics output during the compression stage As input, This is the parameter set for the feature activation module; firstly, through a dimension... The fully connected layer has the following parameter matrix: ,in The preset channel compression ratio is then applied; subsequently, it undergoes the ReLU function. After nonlinear activation, a dimension of The second fully connected layer has the following parameter matrix: This restores the feature dimension to its original number of channels, and finally passes the sigmoid function. Normalization yields the channel weight vector Each element represents the importance of the corresponding feature channel, and the calculation formula is as follows:

[0010] The generated channel weight vector and the original feature matrix Perform channel-by-channel multiplication to achieve adaptive weighting of channels, as shown below:

[0011] The channel-weighted descriptor is obtained by segmenting and transforming it according to the channel. For each channel's data, a Discrete Cosine Transform (DCT) is applied to map the features from the spatial domain to the frequency domain, resulting in a frequency domain representation. As shown in the following formula:

[0012] Among them, the normalization factor The expression is:

[0013] The descriptor matrix of the image after frequency domain transformation It is obtained by concatenating the components using stack operations, as shown in the following formula: ; After completing the frequency domain transformation, a frequency domain attention-aware module (FC) was designed to uncover the importance of different frequency components. This module consists of a two-layer fully connected neural network that models the dependencies in the frequency domain representation through nonlinear mapping. The channel weights output by the FC are... The expression is as follows:

[0014] in, , These are the parameters of the weight matrix. The preset channel compression ratio, The ReLU activation function is used first, and finally, the sigmoid activation function is used. Perform normalization; Enhanced feature matrix The expression is as follows, where · represents element-wise multiplication:

[0015] From the enhanced feature matrix The extracted descriptor is .

[0016] Furthermore, the encoder receives the position coordinates of the feature points. and confidence level The triplet Then, through the stacking of multiple convolutional layers and activation functions in the MLP, the input features are gradually mapped into a higher-dimensional feature space. The expression is as follows: .

[0017] Furthermore, in S3, the specific process is as follows: Let's illustrate with image A; Enhanced descriptor With the output of the keypoint encoder The initial feature vector is obtained by concatenation: :

[0018] in, This is the weight matrix. For splicing operations; The self-attention layer first inputs the feature matrix. The query matrix is ​​generated through three independent linear transformation layers. Key matrix Sum matrix : , ,

[0019] in This indicates the attention layer number, including self-attention and cross-attention. Start from 0; , , Represents an independent parameter matrix; The following calculations are then performed to output a context-aware feature representation:

[0020] The input feature matrix is ​​updated by connecting the residuals to the information after self-attention, resulting in the updated feature matrix. The expression is as follows: .

[0021] Furthermore, in S4, the cross-attention mechanism achieves cross-image attention guidance by using features of one image as the query and features of another image as the key and value. For example, the expression is shown below: , ,

[0022]

[0023] in, , Images and images The input feature matrix after passing through the self-attention layer , , Represents an independent parameter matrix, Given the feature dimension of the key vector, the result calculated by this formula is the image. The feature points generate a new feature vector that incorporates contextual information from image B. The above operation is equivalent to making the image To image "Ask a question," focus on the image. Which areas are most valuable for reference, thus assisting in image processing? Each node in the graph can make a more accurate judgment on the correspondence between different parts of the graph; The input feature matrix is ​​updated to obtain :

[0024] Similarly, for images Execute with image Cross-attention to context forms a bidirectional interactive structure.

[0025] Furthermore, in S6, the process is as follows: By calculating the final feature matrix and The inner product of vectors in the matrix constructs the initial matching score matrix. ,matrix elements in Representation diagram No. Features and diagrams The Middle The similarity of each feature The calculation formula is as follows:

[0026] in, For the updated eigenvector matrix The extracted first Descriptor of a feature point For the updated eigenvector matrix The extracted first Descriptor of a feature point Represents the dot product operation of vectors; In the original score matrix Under the defined similarity metric, we seek an ideal binary matching matrix. Its elements Representing feature points Match successful; A "trash can" mechanism was introduced to convert the original score matrix into a trash can. Expand to The extended matrix The newly added last row and last column represent the matching scores of each feature point with the "trash can," and the values ​​at these positions are all set as learnable parameters. ,in, Its numerical values ​​are automatically optimized through end-to-end training, and correspondingly, the matching matrix to be solved is also expanded to... ; The core objective of the optimal matching layer is to compute the optimal allocation matrix that satisfies the constraints. By optimizing the objective function Maximize the solution of the optimal allocation matrix , The matrix satisfies the following constraints:

[0027] Wherein, constraint vector and The expression is shown in the following formula:

[0028] matrix The sum of each row of elements in the image must be less than or equal to 1 to ensure the image... Any feature point in the image The image must contain no more than one matching candidate point; correspondingly, the sum of the elements in each column must be less than or equal to 1 to ensure the image... Each feature point in the target image There is at most one best matching point in the image, where N represents the image. The trash can can hold up to The number of unmatched points corresponds to the set Total points Similarly; In the scoring matrix Under the defined similarity metric, the solution to the optimal transport problem is transformed into satisfying distribution constraints. and The optimal matching scheme is obtained by introducing an entropy regularization term, transforming the discrete allocation problem into a differentiable optimization problem, and then efficiently solving it using the Sinkhorn algorithm to obtain the optimal allocation matrix. .

[0029] Furthermore, in S7, the process is as follows: Optimal allocation matrix The matrix represents the predicted output, with its top-left corner... elements in the submatrix This indicates that the obtained graph Feature points in and picture Middle feature points The matching probability is used to filter the most likely matching pairs by mutual maximum value. A probability threshold is set, and only matching pairs with a probability value higher than the threshold are retained.

[0030] An electronic device includes: a processor and a memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device implements a feature matching method based on channel guidance and frequency domain awareness.

[0031] A readable storage medium comprising: software instructions; When the software instructions are executed in the electronic device, the electronic device implements a feature matching method based on channel guidance and frequency domain awareness.

[0032] Beneficial effects: This invention proposes a feature matching method based on channel guidance and frequency domain awareness, which enhances the sensitivity and discriminativeness of descriptors to local structural changes, strengthens the network's ability to discriminate easily confused regions, and improves the accuracy and robustness of the network in complex scenarios such as low light and structural similarity. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is the overall framework of the network described in Part IV of this specification.

[0035] Figure 2 This is a structural diagram of the descriptor enhancement module in Part S1 of this specification. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] The features and illustrative embodiments of various aspects of the present invention will now be described in detail. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention. The invention is by no means limited to any specific setups and methods set forth below, but covers any improvements, substitutions, and modifications to structures, methods, and devices without departing from the spirit of the invention. Well-known structures and techniques are not shown in the drawings and the following description to avoid unnecessarily obscuring the invention.

[0038] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing and simplifying the invention, and should not be construed as limiting the invention. Furthermore, the use of ordinal numbers (e.g., "first and second," etc.) is for distinguishing objects and is not limited to this order, and should not be construed as indicating or implying relative importance.

[0039] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly, encompassing both direct connection and indirect connection via an intermediate medium. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0040] It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other, and the various embodiments can be referenced and cited in each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0041] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0042] To address the issue of insufficient discriminative power of descriptors in complex scenarios, a descriptor enhancement framework based on spatial-frequency domain co-optimization is established. This framework combines channel attention mechanisms with frequency domain analysis, employing a channel selection strategy in the spatial domain to enhance the discriminative dimension, and leveraging spatial frequency response characteristics in the frequency domain to increase sensitivity to local structural changes, thereby improving the overall discriminative power and robustness of descriptor matching.

[0043] The overall framework of the method is as follows Figure 1 As shown, the specific process is as follows: S1: The descriptors of the two images are enhanced using a descriptor enhancement module. Addressing the issue that current descriptors primarily rely on spatial domain information, making it difficult to fully capture the essential features of local geometric structures, a descriptor enhancement module is proposed. Through an adaptive weighting mechanism and frequency domain response extraction, frequency domain information provides additional discriminative capabilities, enhancing the descriptor's ability to perceive local geometric structures, better representing the image's geometric structure, and improving the accuracy and robustness of matching in complex scenes.

[0044] The structure diagram of the descriptor enhancement module is as follows: Figure 2 As shown. (For the image) In The set of descriptors corresponding to each feature point Taking image enhancement as an example, let's take the image enhancement process as an example. Similarly (image) The number of feature points in the middle is ).

[0045] First of all, The feature matrix consists of descriptors corresponding to each feature point. ( C Spatial feature compression is performed on the number of channels of the descriptor. This achieves average pooling in the spatial dimension, for each channel. All the information below is aggregated into a single value, resulting in a channel-level statistical vector. Each element in the vector represents the global response intensity of the original feature in the corresponding channel, as shown in the formula below:

[0046] Feature stimulation ( Channel statistics output during the compression stage As input, This is the parameter set for the feature activation module. First, it is passed through a dimension... Fully connected layer (parameter matrix is) ),in This is the preset channel compression ratio. Dimensionality reduction effectively reduces feature redundancy, followed by ReLU (…). After nonlinear activation, a dimension of is used. The second fully connected layer (parameter matrix is) This process restores the feature dimension to its original number of channels. This dimensionality-upgrading process reconstructs and filters the feature channels, ultimately using the sigmoid function ( Normalization is performed to obtain the channel weight vector. Each element represents the importance of the corresponding feature channel, and the calculation formula is as follows:

[0047] In the final processing stage, the generated channel weight vector and the original feature matrix Performing channel-by-channel multiplication to achieve adaptive weighting of channels can be represented as follows:

[0048] Then, the channel-weighted descriptor is segmented and transformed according to the channel to obtain... The Discrete Cosine Transform (DCT) is applied to the data of each channel to map the features from the spatial domain to the frequency domain, thus obtaining the frequency domain representation. As shown in the following formula:

[0049] Among them, the normalization factor The expression is:

[0050] The descriptor matrix of the image after frequency domain transformation It is obtained by concatenating the components using stack operations, as shown in the following formula:

[0051] After completing the frequency domain transformation, a frequency domain attention-aware module (FC) was designed to uncover the importance of different frequency components. This module consists of a two-layer fully connected neural network that models the dependencies in the frequency domain representation through nonlinear mapping. The channel weights output by the FC are... The expression is as follows:

[0052] in, , These are the parameters of the weight matrix. The preset channel compression ratio, The ReLU activation function is used first, and finally, the sigmoid activation function is used. Normalize.

[0053] Enhanced feature matrix The expression is as follows (· represents element-wise multiplication):

[0054] From the enhanced feature matrix The extracted descriptor is .

[0055] S2: Feature points in both images are encoded using a keypoint encoder. The goal of the keypoint encoder is to improve the representation of low-dimensional coordinates and confidence information in high-dimensional space. Considering the nonlinear mapping capability of feature representation and the preservation of keypoint geometry, a multi-layer perceptron (MLP) is introduced as the basic feature transformation component. The encoder receives the position coordinates of the feature points. and confidence level The triplet Then, through the stacking of multiple convolutional layers and activation functions in the MLP, the input features are gradually mapped into a higher-dimensional feature space. The expression is shown below:

[0056] S3: The enhanced descriptors and encoded feature points from the two images are then aggregated using a self-attention layer to incorporate the contextual information of the image's internal features. This layer receives the output from the descriptor enhancement module. With the output of the keypoint encoder The initial feature vector is obtained after concatenation. The calculation formula is as follows:

[0057] in, This is the weight matrix. This is for splicing operations.

[0058] Next, with diagram Let's take an example to introduce it (Figure) Similarly), the self-attention layer first inputs the feature matrix. ,in Indicates the nth attention layer (including self-attention and cross-attention). Starting from 0). The query matrix is ​​generated through three independent linear transformation layers. ), key matrix (Key, ) and value matrix (Value, ):

[0059]

[0060]

[0061] In the above formula, For the input feature matrix, , , Represents an independent parameter matrix.

[0062] The subsequent calculation process is as follows:

[0063] Calculate the query matrix using matrix multiplication. Bond matrix The similarity score reflects the strength of the correlation between different positions. To avoid gradient instability caused by excessively large inner product values, the similarity score is scaled by dividing it by the feature dimension of the key vector. The square root of the value is normalized, and then the Softmax function is applied to probabilistically process the scaled score to obtain the normalized attention weight distribution. The weight matrix and the value matrix are then compared. Perform weighted summation to output a context-aware feature representation.

[0064] Finally, the input feature matrix is ​​updated by connecting the residuals to the information after self-attention, resulting in the updated feature matrix. The expression is as follows:

[0065] S4: The updated feature matrices of the two images, processed by the self-attention layer, are input into the cross-attention layer to achieve interaction and information fusion between the features of the two images. Unlike the self-attention mechanism where the query, key, and value all come from the same set of features, the cross-attention mechanism uses the features of one image as the query and the features of the other image as the key and value, thereby achieving cross-image attention guidance. Let's take an example to introduce it (Figure) Similarly, the expression is as follows:

[0066]

[0067]

[0068]

[0069] in, , Images and images The input feature matrix after passing through the self-attention layer , , Represents an independent parameter matrix, Given the feature dimension of the key vector, the result calculated by this formula is the image. The feature points generate a new feature vector that incorporates contextual information from image B. The above operation is equivalent to making the image To image "Ask a question," focus on the image. Which areas are most valuable for reference, thus assisting in image processing? Each node in the graph can make a more accurate judgment on the correspondence between different nodes.

[0070] Then, the input feature matrix is ​​updated to obtain... :

[0071] Similarly, this can also be applied to images. Execute with image Cross-attention to context forms a bidirectional interactive structure.

[0072] S5: Repeat steps S3 and S4. Second-rate S4 and S5 are combined into a "self-attention-cross-attention unit" and stacked consecutively. For example, the first time you achieve self-attention and Then cross-attention is performed to obtain and Repeat L times until you get and .

[0073] S6: Solve for the optimal allocation matrix from the final feature matrix updated in step S5 using the Sinkhorn algorithm.

[0074] By calculating the updated feature vector matrix and The inner product of vectors in the matrix constructs the initial matching score matrix. ,matrix elements in Representation diagram No. Features and diagrams The Middle The similarity of each feature The calculation formula is as follows:

[0075] in, For the updated eigenvector matrix The extracted first Descriptor of a feature point For the updated eigenvector matrix The extracted first Descriptor of a feature point This represents the inner product operation of vectors.

[0076] In the original score matrix Under the defined similarity metric, we seek an ideal binary matching matrix. Its elements Representing feature points Match successful. To handle unmatched points caused by factors such as occlusion and detection failure in real-world scenarios, an innovative "trash can" mechanism is introduced into the model, which stores the original score matrix... Expand to The extended matrix The newly added last row and last column represent the matching scores of each feature point with the "trash can," and the values ​​at these positions are all set as learnable parameters. ,in, Its numerical values ​​are automatically optimized through end-to-end training, and correspondingly, the matching matrix to be solved is also expanded to... .

[0077] The core objective of the optimal matching layer is to compute the optimal allocation matrix that satisfies the constraints. By optimizing the objective function Maximize the solution of the optimal allocation matrix , The matrix satisfies the following constraints:

[0078] Wherein, constraint vector and The expression is shown in the following formula:

[0079] matrix The sum of each row of elements in the image must be less than or equal to 1 to ensure the image... Any feature point in the image The image contains no more than one matching candidate point; correspondingly, the sum of each column element must be less than or equal to 1 to ensure the image... Each feature point in the target image There is at most one best matching point in the image, where N represents the image. The trash can can hold up to 1 unmatched point (corresponding to the set) (Total points) Similarly.

[0080] In the scoring matrix Under the defined similarity metric, the solution to the optimal transport problem is one that satisfies the distribution constraints. and The optimal matching scheme is found. By introducing an entropy regularization term, the discrete allocation problem is transformed into a differentiable optimization problem, which is then efficiently solved using the Sinkhorn algorithm to obtain the optimal allocation matrix. .

[0081] S7: Obtain the matching relationship of feature points between image pairs based on the solved optimal allocation matrix.

[0082] Optimal allocation matrix The matrix represents the network's predicted output, with its top-left corner... elements in the submatrix This indicates that the obtained graph Feature points in and picture Middle feature points The matching probability is used to filter the most likely matching pairs by mutual maximum value. To further improve accuracy, a probability threshold is set, and only matching pairs with a probability value higher than the threshold are retained.

[0083] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A feature matching method based on channel guidance and frequency domain awareness, characterized in that... The method includes: S1: Enhance the descriptors of the two images separately; enhance the images... In The set of descriptors corresponding to each feature point The enhancement process is as follows: right The feature matrix consists of descriptors corresponding to each feature point. Perform spatial feature compression Each channel All the information below is aggregated into a single value, resulting in a channel-level statistical vector. Each element in the vector represents the global response intensity of the original feature in the corresponding channel, as shown in the following formula: C The number of channels for the descriptor; Feature activation Channel statistics output during the compression stage As input, This is the parameter set for the feature activation module; firstly, through a dimension... The fully connected layer has the following parameter matrix: ,in The preset channel compression ratio is then applied; subsequently, it undergoes the ReLU function. After nonlinear activation, a dimension of The second fully connected layer has the following parameter matrix: This restores the feature dimensions to the original number of channels, and finally passes the sigmoid function. Normalization yields the channel weight vector Each element represents the importance of the corresponding feature channel, as shown in the following formula: The generated channel weight vector and the original feature matrix Channel-by-channel multiplication is represented as: The channel-weighted descriptor is obtained by segmenting and transforming it according to the channel. The discrete cosine transform (DCT) is applied to the data of each channel to obtain the frequency domain representation. As shown in the following formula: Normalization factor The expression is: Descriptor matrix of the image after frequency domain transformation It is obtained by concatenating the data through stack operations, as shown below: ; After frequency domain transformation, a frequency domain attention-aware module (FC) is designed to mine the importance of different frequency components. This module consists of a two-layer fully connected neural network that models the dependencies in the frequency domain representation through nonlinear mapping. The channel weights output by the FC are... The expression is as follows: in, , For the parameters of the weight matrix, The preset channel compression ratio, The ReLU activation function is used first, and finally, the sigmoid activation function is used. Perform normalization; Enhanced feature matrix The expression is as follows, where · represents element-wise multiplication: From the enhanced feature matrix The extracted descriptors are ; S2: Encode the feature points of the two images separately using a keypoint encoder; S3: The enhanced descriptors and encoded feature points of the two images are respectively aggregated through a self-attention layer to gather the contextual information of the internal features of the images; S4: The image internal feature context information of another image is interacted with and fused with the image internal feature context information of this image through a cross-attention layer; S5: Repeat steps S3 and S4 L times to obtain the final feature matrix after fusing the two images L times respectively. and ; S6: Solve for the optimal allocation matrix from the final feature matrix using the Sinkhorn algorithm; S7: Obtain the matching relationship of feature points between two images based on the optimal allocation matrix.

2. The method according to claim 1, characterized in that... The encoder receives the position coordinates of the feature points. and confidence level The triplet Then, through the stacking of multiple convolutional layers and activation functions in the MLP, the input features are gradually mapped into a higher-dimensional feature space. The expression is as follows: 。 3. The method according to claim 2, characterized in that... In S3, the specific process is as follows: To illustrate, see image A; Enhanced descriptor With the output of the keypoint encoder The initial feature vector is obtained by concatenation: : in, This is the weight matrix. For splicing operations; The self-attention layer first takes the input feature matrix The query matrix is ​​generated through three independent linear transformation layers. Key matrix Sum matrix : ， ， in This indicates the attention layer number, including self-attention and cross-attention. Start from 0; , , Represents an independent parameter matrix; The following calculations are then performed to output a context-aware feature representation: The input feature matrix is ​​updated by connecting the residuals to the information after self-attention, resulting in the updated feature matrix. The expression is as follows: 。 4. The method according to claim 3, characterized in that... In S4, the cross-attention mechanism guides attention across images by using features from one image as the query and features from another image as the key and value. For example, the expression is shown below: ， ， in, , Images and images The input feature matrix after passing through the self-attention layer , , Represents an independent parameter matrix, Given the feature dimension of the key vector, the result calculated by this formula is the image. The feature points generated a new feature vector that incorporates contextual information from image B. The above operation is equivalent to making the image To image "Ask a question," focus on the image. Which areas are most valuable for reference, thus assisting in image processing? Each node in the graph can make a more accurate judgment on the correspondence between different parts of the graph; The input feature matrix is ​​updated to obtain : Similarly, for images Execute with image Cross-attention to context forms a bidirectional interactive structure.

5. The method according to claim 4, characterized in that... In S6, the process is as follows: By calculating the final feature matrix and The inner product of vectors in the matrix constructs the initial matching score matrix. ,matrix elements in Representation diagram No. Features and diagrams Middle The similarity of each feature The calculation formula is as follows: in, For the updated eigenvector matrix The extracted first Descriptor of a feature point For the updated eigenvector matrix The extracted first Descriptor of a feature point Represents the dot product operation of vectors; In the original score matrix Under the defined similarity metric, we seek an ideal binary matching matrix. Its elements Representing feature points Match successful; A "trash can" mechanism was introduced to process the original score matrix. Expand to The extended matrix The newly added last row and last column represent the matching scores of each feature point with the "trash can," and the values ​​at these positions are all set as learnable parameters. ,in, Its numerical values ​​are automatically optimized through end-to-end training, and correspondingly, the matching matrix to be solved is also expanded to... ; The core objective of the optimal matching layer is to compute the optimal allocation matrix that satisfies the constraints. By optimizing the objective function Maximize the solution of the optimal allocation matrix , The matrix satisfies the following constraints: Wherein, constraint vector and The expression is shown in the following formula: matrix The sum of each row of elements in the image must be less than or equal to 1 to ensure the image... Any feature point in the image The image must contain no more than one matching candidate point; correspondingly, the sum of the elements in each column must be less than or equal to 1 to ensure the image... Each feature point in the target image There is at most one best matching point in the image, where N represents the image. The trash can can hold up to The number of unmatched points corresponds to the set Total points Similarly; In the scoring matrix Under the defined similarity metric, the solution to the optimal transport problem is transformed into satisfying distribution constraints. and The optimal matching scheme is obtained by introducing an entropy regularization term, transforming the discrete allocation problem into a differentiable optimization problem, and then efficiently solving it using the Sinkhorn algorithm to obtain the optimal allocation matrix. .

6. The method according to claim 5, characterized in that... In S7, the process is as follows: Optimal allocation matrix The matrix represents the predicted output, with its top-left corner... elements in the submatrix This indicates that the obtained graph Feature points in and picture Middle feature points The matching probability is used to filter the most likely matching pairs by mutual maximum value. A probability threshold is set, and only matching pairs with a probability value higher than the threshold are retained.

7. An electronic device, characterized in that, include: Processor and memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device performs the method as described in any one of claims 1-6.

8. A readable storage medium, characterized in that, The readable storage medium includes: software instructions; When the software instructions are executed in an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-6.

Images