Feature matching method and system based on local feature fusion and self-attention mechanism

Abstract

The application discloses a feature matching method and system based on local feature fusion and self-attention mechanism; an image processing module is used to extract a feature descriptor of an input image and add position information to obtain features of image points; self-attention context feature fusion is performed on each feature in a source image and a target image and randomly sampled neighbor points, information in the images is efficiently combined, and therefore the representation ability of a feature map is enhanced; the Euclidean distance is calculated between each point in an image and a point on another image, the first k points are selected, information transmission is performed through a cross-attention model, a feature representation with cross-image information is obtained, the similarity of features in two images is calculated through final feature information, and the final matching result is obtained through iterative solution using a Sinkhorn algorithm.

Description

Technical Field

[0001] This invention relates to image matching technology, specifically to a feature matching method and system based on local feature fusion and self-attention mechanism. Background Technology

[0002] Feature matching is a fundamental problem in computer vision and pattern recognition, aiming to establish correspondences between features (e.g., points, regions) in a source and target image. To perform feature matching, it first requires representing each feature in both images with a discriminative descriptor. Then, a metric learning model is used to obtain the correlation / similarity between pairs of feature points in the two images. Finally, a linear assignment algorithm is performed to find the correspondences between the features. In recent years, deep learning models have been explored for feature matching tasks. These methods can integrate feature representation, metric learning, and final correspondence estimation in an end-to-end manner.

[0003] The core problem of deep learning feature matching is how to learn (1) to distinguish the representation of feature points in each intra-frame image, and (2) to consistently represent feature points across two inter-frame images.

[0004] Recently, people have used self-attention and cross-attention models to solve this problem and achieve feature matching; specifically, self-attention is used to capture the relationships between features, so that context-aware representations of feature units within each image can be learned, while cross-attention is used to model the interaction between features between two images, so that information can be communicated across images.

[0005] In summary, existing feature matching methods have the following problems:

[0006] (1) Previous self / cross attention models typically mix information from all original features, resulting in redundant learning and high computational costs.

[0007] (2) Feature embedding representations generated by fusing too many redundant and irrelevant features during feature aggregation, which interfere with the transmission of information. Summary of the Invention

[0008] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies and provide a feature matching method and system based on local feature fusion and self-attention mechanism. From the perspective of information fusion, this invention uses neighbors to aggregate contextual information when calculating self-attention for each feature point in the graph, which greatly improves the efficiency of information fusion and can combine more useful information while eliminating interfering information.

[0009] Technical solution: The present invention provides a feature matching method based on local feature fusion, comprising the following steps:

[0010] Step (1): Input the source image and target image to be matched, select the network model as the feature extraction module to obtain the feature descriptors of the source image and target image, and combine the location information to generate the initial point features;

[0011] Step (2): Select neighbors for each point in the source image and target image separately and perform information fusion. Obtain effective information and enhanced image feature representation through the neighbors. The method for selecting neighbors is: for each point, first randomly sample and select its k points as neighbors, and then perform information fusion between the point itself and these k neighbors through self-attention.

[0012] Step (3): For each feature point on the two enhanced images obtained in step (2), use Euclidean distance to calculate the k closest points across the image, and use a cross-attention network to fuse cross-image information between the source image and the target image to obtain the point features of the image with cross-image information.

[0013] Step (4): Calculate the similarity of the point feature representations of the source image and the target image, and use the sinkorn algorithm to iterate and train multiple times to obtain the final matching result.

[0014] Furthermore, the method used in step (1) to extract feature descriptors is as follows: extract feature descriptors with dimensions of 128 or 256 using SIFT or Superpoint methods, and use a multilayer perceptron to map the position information (x, y) to the same dimension; the combination of the two is the initial point feature.

[0015] Furthermore, the detailed method of step (2) is as follows:

[0016] For each point in the source image k neighbors are obtained through random sampling. For example, formula (1) for point Calculate k neighbors

[0017] Then F is processed through three linear mapping layers. s Convert to A s Convert to and The information fusion formula (1) is as follows:

[0018]

[0019] and These are all intermediate computational quantities, through and Calculate the weights, and then use the weights and... Aggregate image information, where c represents the feature dimension and LN(·) represents layer normalization; each node updates its representation by aggregating messages from its neighbors;

[0020] For each point in the target image The k nearest neighbors of each point to the current point are obtained using cosine similarity. Then, f is processed through three linear mapping layers. i t Convert to A t Convert to and The information fusion formula (2) is as follows:

[0021]

[0022] and These are all intermediate computational quantities, through and Calculate the weights, and then use the weights and... Aggregate image information.

[0023] Furthermore, the specific method for step (3) of using a cross-attention module to enhance the interactive learning between the anchor features of the source image and the target image is as follows:

[0024] Calculate each point in the source graph. With each point in the target map The Euclidean distance is shown in formula (3):

[0025]

[0026] in This represents the Euclidean distance between the first node in the source image and the i-th node in the target image, and selects the k points in the target image with the highest distances. To perform information fusion;

[0027] Cross-attention module enhances feature points f of the source image i s Feature points selected from the target image The specific methods for interactive learning between them are as follows:

[0028] F is achieved through three linear mapping layers. s Convert to B t Convert to and The information transfer across graphs is as shown in formula (4):

[0029]

[0030] The target image uses the same method to select the first k points from the source image. By f incorporating information from the source graph, three linear mapping layers are used to transform f i t Convert to B s Convert to and As shown in formula (5);

[0031]

[0032] Furthermore, the specific process of step (4) is as follows:

[0033] First, the similarity matrix S between the source and target images is calculated using the point features with cross-image information obtained in step (3). Then, based on the similarity matrix S (metric S), a linear allocation algorithm is used to predict the matching of feature points between the two images. The linear allocation algorithm Sinkhorn is iteratively calculated as shown in formula (6).

[0034] S = Sinkhorn(S) (6)

[0035] Sinkhorm is differentiable and can be integrated with the learning modules mentioned above in an end-to-end manner.

[0036] Let T denote the set of basic truth correspondences between source and target features after crossing the graph;

[0037] Then, the matching prediction loss function used in the training phase is expressed as Equation (7):

[0038] L1=-∑ (i,j)∈T logS i,j (7)

[0039] Furthermore, if their epipolar distance is less than a threshold, the feature pairs are marked as matching pairs;

[0040] Calculate the cross-entropy loss for matching and non-matching binary classification in z-person processing units, and express it as L. z , where z∈{1,2…Z}, and Z is the number of running units of the cross-graph information fusion module;

[0041] Therefore, the final total loss is expressed as formula (8):

[0042]

[0043] Where 'a' is the equilibrium parameter.

[0044] Furthermore, in step (4), the learning rate of the model is 0.0001, and after 300,000 iterations, the learning rate decays at a rate of 0.999996 until 900,000 iterations.

[0045] This invention also discloses a system for implementing a feature matching method based on local feature fusion and self-attention mechanism, including a feature extraction module, a local neighbor feature fusion module, a cross-graph information fusion module, and a matching solution module. The feature extraction module is used to extract initial point features from the source image and the target image. The local neighbor feature fusion module fuses the image information of the source image and the target image respectively to obtain their respective enhanced image features. The cross-graph information fusion module performs cross-graph fusion on the enhanced image features of the source image and the target image respectively. The matching solution module calculates the similarity value based on the feature information and solves the matching result.

[0046] Beneficial effects: This invention can efficiently achieve feature matching; it simultaneously models the distinguishable representation of feature points in each frame and the consistent representation of feature points across frames. Instead of calculating self-attention and cross-attention for all feature points, this invention only randomly selects k nodes, thus effectively integrating information. Compared with existing technologies, this invention has the following advantages:

[0047] (1) The present invention can capture the correlation of features within an image and explore the interaction between images to learn a consistent representation of two images.

[0048] (2) The present invention provides a novel efficient feature matching network that combines local feature representation, metric learning and correspondence prediction in an end-to-end manner.

[0049] (3) The local feature fusion module of the present invention (selecting k points for feature fusion) can realize the complementary effect between features, highlight useful information, suppress redundant information, and further enhance the representation ability of the feature map. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the overall classification process of the present invention;

[0051] Figure 2 This is a schematic diagram of neighbor information aggregation according to the present invention. Detailed Implementation

[0052] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0053] like Figure 1 As shown, a feature matching method based on local feature fusion and self-attention mechanism according to the present invention includes the following steps:

[0054] Step (1): Input the source image and target image to be matched, extract feature descriptors of dimension 128 or 256 using SIFT or Superpoint methods, and use a multilayer perceptron to map the position information (x, y) to the same dimension; the combination of the two is the initial point feature.

[0055] Step (2): Select neighbors for each point in the source image and the target image respectively and perform information fusion to obtain effective information through the neighbors and obtain the enhanced image feature representation;

[0056] For each point in the source image k neighbors are obtained through random sampling. For example, formula (1) for point Calculate k neighbors

[0057] Then F is processed through three linear mapping layers. s Convert to A s Convert to and The information fusion formula (1) is as follows:

[0058]

[0059] and These are all intermediate computational quantities, through and Calculate the weights, and then use the weights and... Aggregate image information, where c represents the feature dimension and LN(·) represents layer normalization; each node updates its representation by aggregating messages from its neighbors;

[0060] For each point in the target image The k nearest neighbors of each point to the current point are obtained using cosine similarity. Then, f is processed through three linear mapping layers. i t Convert to A t Convert to and The information fusion formula (2) is as follows:

[0061]

[0062] and These are all intermediate computational quantities, through and Calculate the weights, and then use the weights and... Aggregate image information.

[0063] Step (3): For the enhanced image feature representation obtained in step (2), select the first k points across the graph and use a cross-attention network to fuse cross-graph information between the source image and the target image to obtain point features of the image with cross-graph information.

[0064] The specific method for enhancing the interactive learning between anchor features of the source and target images using the cross-attention module is as follows:

[0065] Calculate each point in the source graph. With each point in the target map The Euclidean distance is shown in formula (3):

[0066]

[0067] in This represents the Euclidean distance between the first node in the source image and the i-th node in the target image, and selects the k points in the target image with the highest distances. To perform information fusion;

[0068] Cross-attention module enhances feature points f of the source image i s Feature points selected from the target image The specific methods for interactive learning between them are as follows:

[0069] F is achieved through three linear mapping layers. s Convert to B t Convert to and The information transfer across graphs is as shown in formula (4):

[0070]

[0071] The target image uses the same method to select the first k points from the source image. By f incorporating information from the source graph, three linear mapping layers are used to transform f i t Convert to B s Convert to and As shown in formula (5);

[0072]

[0073] Step (4): Calculate the similarity of the point feature representations of the source image and the target image, and use the sinkorn algorithm for multiple iterations to obtain the final matching result:

[0074] First, the similarity matrix S between the source and target images is calculated using the point features with cross-image information obtained in step (3). Then, based on the similarity matrix S (metric S), a linear allocation algorithm is used to predict the matching of feature points between the two images. The linear allocation algorithm Sinkhorn is iteratively calculated as shown in formula (6).

[0075] S = Sinkhorn(S) (6)

[0076] Sinkhorm is differentiable and can be integrated with the learning modules mentioned above in an end-to-end manner.

[0077] Let T denote the set of basic truth correspondences between source features and target features.

[0078] Then, the matching prediction loss function used in the training phase is expressed as Equation (7):

[0079] L1=-∑ (i,j)∈T logS i,j (7)

[0080] Furthermore, if their epipolar distance is less than a threshold, the feature pairs are marked as matching pairs;

[0081] Calculate the cross-entropy loss for matching and non-matching binary classifications in r processing units, and express it as L. z , where z∈{1,2…Z}, and Z is the number of running units of the cross-graph information fusion module;

[0082] Therefore, the final total loss is expressed as formula (8):

[0083]

[0084] Where 'a' is the balancing parameter. In this embodiment, the Sinkhorn method is used for iterative training, with the model's learning rate set to 0.0001. After 300,000 iterations, a learning rate decay is applied at a rate of 0.999996, continuing until 900,000 iterations.

[0085] This embodiment trains the model on GL3D and tests the matching results on the yfcc, fm-bench, and scannet datasets.

[0086] The system for implementing a feature matching method based on local feature fusion and self-attention mechanism in this embodiment includes a feature extraction module, a local neighbor feature fusion module, a cross-graph information fusion module, and a matching solution module. The feature extraction module extracts initial point features from the source image and the target image. The local neighbor feature fusion module fuses the image information of the source image and the target image respectively to obtain their respective enhanced image features. The cross-graph information fusion module performs cross-graph fusion on the enhanced image features of the source image and the target image. The matching solution module calculates the similarity value based on the feature information and solves the matching result.

Claims

1. A feature matching method based on local feature fusion and self-attention mechanism, characterized in that: Includes the following steps: Step (1): Input the source image and target image to be matched, select the network model as the feature extraction module to obtain the feature descriptors of the source image and target image, and combine the location information to generate the initial point features; Step (2): For each point in the source image and the target image, select neighbors individually for information fusion. The method for selecting neighbors is as follows: for each point, first select k points as neighbors through random sampling, and then fuse the information of the point itself with these k neighbors through self-attention. The detailed method is as follows: For each point F in the source image s ={ , , … }, obtain k neighbors A through random sampling. s ={ , , … }, for point Calculate the interaction information of k neighbors, and then pass it through three linear mapping layers. Convert to , will A s Convert to and Then the information fusion formula (1) is as follows: (1) , and These are all intermediate computational quantities, through and Calculate the weights, and then use the weights and... Aggregate image information, where c represents the feature dimension and LN(·) represents layer normalization; each node updates its representation by aggregating messages from its neighbors; For each point F in the target image t ={ , , … }, obtain the k nearest neighbors A to the current point through cosine similarity. t ={ , , … Then, through three linear mapping layers, Convert to , will A t Convert to and Then the information fusion formula (2) is as follows: (2) , and These are all intermediate computational quantities, through and Calculate the weights, and then use the weights and... Aggregate image information; Step (3): For each feature point in the two enhanced images obtained in step (2), the k nearest points across the image are calculated using Euclidean distance. Cross-image information is fused between the source and target images using a cross-attention network to obtain point features of the image with cross-image information. The detailed method is as follows: Calculate F for each point in the source graph s ={ , , … } and each point F in the target graph t ={ , , … The Euclidean distance of} is shown in formula (3): (3) in Let B represent the Euclidean distance between the first node in the source image and the i-th node in the target image, and select the k points B in the target image that have the highest distances. t ={ , , … Information fusion is performed. Cross-attention module enhances feature points of source image Feature points B selected from the target image t ={ , , … The specific methods for interactive learning between them are as follows: Through three linear mapping layers Convert to B t Convert to and Then, the information transfer across graphs is as shown in formula (4): (4) The target image uses the same method to select the first k points B from the source image. s ={ , , … By fusing information from the source graph, three linear mapping layers are used to... Convert to B s Convert to and As shown in formula (5); (5)； Step (4): Calculate the similarity of the point feature representations of the source image and the target image, and use the sinkorn algorithm to iterate and train multiple times to obtain the final matching result.

2. The feature matching method based on local feature fusion and self-attention mechanism according to claim 1, characterized in that: The method used in step (1) to extract feature descriptors is as follows: extract feature descriptors with dimensions of 128 or 256 using SIFT or Superpoint, and use a multilayer perceptron to map the position information (x, y) to the same dimension; the combination of the two is the initial point feature.

3. The feature matching method based on local feature fusion and self-attention mechanism according to claim 1, characterized in that: The specific process of step (4) is as follows: First, use the point features with cross-image information from step (3) to calculate the similarity matrix S between the source image and the target image; then, use the linear allocation algorithm based on the similarity matrix S to achieve matching prediction between the feature points of the two images. The linear allocation algorithm Sinkhorn is iteratively calculated as shown in equation (6): S = Sinkhorn(S) (6) Sinkhorm is differentiable and can be integrated with learning modules in an end-to-end manner; Let T denote the set of basic truth correspondences between source and target features after crossing the graph; Then, the matching prediction loss function used in the training phase is expressed as Equation (7): (7) Furthermore, if their epipolar distance is less than a threshold, the feature pairs are marked as matching pairs; Calculate the cross-entropy loss for matching and non-matching binary classifications in z processing units, and express it as... , where z {1, 2…Z}, where Z is the number of running units of the cross-graph information fusion module; Therefore, the final total loss is expressed as formula (8): (8) Where 'a' is the equilibrium parameter.

4. The feature matching method based on local feature fusion and self-attention mechanism according to claim 1, characterized in that: In step (4), the learning rate of the model is 0.0001. After 300,000 iterations, the learning rate decays at a rate of 0.999996 until 900,000 iterations.

5. A system for implementing the feature matching method based on local feature fusion and self-attention mechanism as described in any one of claims 1 to 4, characterized in that: It includes a feature extraction module, a local neighbor feature fusion module, a cross-graph information fusion module, and a matching solution module. The feature extraction module is used to extract the source image and the target image to obtain the initial point features. The local neighbor feature fusion module fuses the image information of the source image and the target image respectively to obtain their respective enhanced image features. The cross-image information fusion module fuses the enhanced image features of the source and target images across the image; the matching solution module calculates the similarity value based on the feature information and solves the matching result.

Images