Image matching method combining hybrid scale feature descriptor and neighbor consistency

Abstract

The application discloses an image matching method combining a hybrid scale feature descriptor and neighbor consistency, which sequentially passes through different branch feature description networks, splices single scale and multi-scale feature descriptors in dimensions to generate a hybrid scale feature descriptor, inputs the hybrid scale descriptor into an optimal transport matching layer to obtain an initial allocation matrix, and refines the initial allocation matrix through a shared weight graph neural network again to obtain a final matching. The application fuses single scale and multi-scale descriptors, so that the hybrid descriptor can maintain robustness to various geometric deformations and high saliency, and meanwhile, the application removes incorrect matching pairs by using geometric prior, so that the final matching effect is high in accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology based on deep learning, and mainly relates to an image matching method combining mixed-scale feature descriptors and neighbor consistency. Background Art

[0002] Image feature matching refers to establishing point-to-point correspondences between two 2D views of the same 3D scene. Image matching is the cornerstone of many downstream 3D computer vision tasks, including 3D reconstruction, visual localization, structure from motion (SfM), simultaneous localization and mapping (SLAM), etc. Given a pair of images, the traditional feature matching method is: (1) feature detection (2) feature description (3) feature matching (4) outlier removal.

[0003] Early feature matching methods tended to manually design feature point extractors and descriptors, achieving some success. In recent years, deep learning methods have adopted data-driven strategies, enabling the generation of descriptors that are more robust to changes in illumination and viewpoint. Convolutional neural networks were initially adopted as tools for feature point detection and description. To expand the receptive field and aggregate greater context, Transformers have been widely used in feature matching in recent years. Due to the abundance of sparse textures in some scenes, feature point repeatability deteriorates and extraction becomes more difficult. Methods without feature point detectors tend to first establish dense matching between views and then refine reliable matches. However, features extracted by convolutional neural networks lose fine-grained detail through multiple layers of downsampling, making it impossible to accurately match small objects in the scene. How to ensure that learned descriptors are both rich in fine-grained detail and robust to various geometric deformations has become a pressing challenge for those skilled in the art. Summary of the Invention

[0004] The present invention addresses the shortcomings of existing methods without feature point detectors and provides an image matching method that combines hybrid-scale feature descriptors with neighbor consistency. The method, after sequentially applying a convolution-attention hybrid and an enhanced self-attention network, concatenates feature descriptors of different scales in the feature dimension to obtain an initial allocation matrix. After the initial matching point pairs are passed through a graph neural network based on a shared weight, the allocation matrix is ​​modified to achieve image matching. By fusing single-scale and multi-scale feature descriptors, the present invention avoids the loss of detail caused by downsampling operations in convolutional neural networks, while taking neighbor consistency into account to ensure geometric consistency of the matches, ultimately achieving highly accurate matching results.

[0005] In order to achieve the above object, the technical scheme adopted by the present application is: a method for image matching combining mixed scale feature descriptors and neighbor consistency, which sequentially passes through a network based on convolution and attention mixing and a network based on enhanced self-attention, splices the feature descriptors of different scales in the feature dimension to obtain an initial assignment matrix, and modifies the assignment matrix after the initial matching point pair passes through a graph neural network based on a shared weight to realize image matching.

[0006] As an improvement of the present application, the method comprises the following steps:

[0007] S1, feature extraction: different resolution feature extraction is performed on the input original picture of the same picture taken from different angles through the FPN network, the feature map obtained through feature extraction has different spatial resolution and semantic information, and the 1 / 2 resolution size feature map and the 1 / 8 size feature map of the original picture are used as the feature description in the next step;

[0008] S2, single scale feature description; the 1 / 8 size feature map obtained in step S1 is input into the neural network based on convolution and attention mixing after position coding, and a single scale feature descriptor is obtained; the mixed self-attention layer in the neural network based on convolution and attention mixing additionally increases a convolution branch, and the cross-attention layer remains unchanged, the convolution branch of the mixed self-attention layer restores the local geometric structure of the original picture, and the attention branch interacts the information inside the features; the cross-attention layer realizes the information interaction of different features, and updates the features of each layer;

[0009] S3, multi-scale feature description; the original picture taken from different angles obtained in step S1 is input into the network based on enhanced self-attention as input, and a multi-scale feature descriptor is output; the key matrix (K) and the value matrix (V) in the enhanced self-attention in the network based on enhanced self-attention are down-sampled in different proportions in different self-attention heads, each self-attention head performs information transmission of different scale features to generate a multi-scale feature descriptor;

[0010] S4, fusion of different scale features: the single scale feature descriptor obtained in step S2 and the multi-scale feature descriptor obtained in step S3 are spliced in the feature dimension;

[0011] S5, input the mixed scale descriptor obtained in step S4 into the optimal matching layer to obtain an initial assignment matrix; based on the set threshold, the initial matching point pair is selected;

[0012] S6, neighbor consistency filtering outliers: the initial matching point pair obtained in step S4 is modeled into a graph structure and input into a graph neural network with a shared weight, and the output of the graph neural network is used to modify the initial assignment matrix to obtain a new matching point pair.

[0013] S7, matching refinement: input the 1 / 2 size feature map obtained in step S1 and the hybrid descriptor obtained in step S4 into a fully connected neural network to obtain an enhanced 1 / 2 size feature map; input the obtained feature map and the new matching point pair of pixel-level precision obtained in step S6 into a matching refinement network to output a final matching of sub-pixel-level precision, thereby constructing a complete image matching model and realizing image matching.

[0014] As an improvement of the present application, the 1 / 8 size feature map is positionally encoded and rearranged into a one-dimensional tensor in step S2; a hybrid self-attention layer and a cross-attention layer are obtained through convolution and self-attention fusion

[0015] As an improvement of the present application, the sparse attention map-based network training process in step S2 is specifically as follows:

[0016] The hybrid self-attention mechanism and the cross-attention mechanism are alternately used in different layers of the network. When the hybrid self-attention mechanism is used, the similarity between each pixel in the feature map is learned; when the cross-attention mechanism is used, the similarity of each pixel between the feature maps is learned, and finally the information transmitted between the network layers is obtained through a layer of fully connected neural network.

[0017] As another improvement of the present application, step S3 further comprises:

[0018] S31: downsample the key matrix (K) and the value matrix (V) in different self-attention heads at different scales,

[0019]

[0020]

[0021] V i = V i + LE(V i ),

[0022] In the formula, X represents the input feature, represents a linear mapping matrix, r i represents the downsample ratio of the i-th feature head, MTA(·) represents a multi-scale aggregation operation, and LE(·) is a convolutional neural network;

[0023] S32: information transmission is performed on the query matrix (Q), the key matrix (K) and the value matrix (V) obtained in step S31,

[0024]

[0025] In the formula, dh represent the feature dimension of each feature head.

[0026] As another improvement of the present application, the operation of the same dimension in step S4 specifically includes: splicing the 256-dimensional single-scale feature descriptor and the 128 multi-scale feature descriptor in the feature dimension to obtain a 384-dimensional feature descriptor.

[0027] As yet another improvement of the present application, the step S5 specifically includes: first calculating the similarity matrix between the two hybrid descriptors ,

[0028]

[0029] In the formula, τ is a constant, and <·> represents inner product. The similarity matrix serves as the cost matrix of a partial assignment problem, and solving the partial assignment problem can obtain an optimal solution confidence assignment matrix, thereby obtaining the initial matching.

[0030] As a further improvement of the present application, in step S6, the sparse descriptors of the corresponding point pairs are extracted, a sparse similarity matrix P is calculated through inner product, the correspondence relationship of the point sets between images can be regarded as the correspondence relationship of nodes in a graph structure, thereby constructing a node matrix R A , a node matrix R B and an edge matrix E A , E B Here, each node only retains the edge between the two most similar nodes, and a graph neural network with a shared parameter is used,

[0031] d A = Ψ (R A , E A ),

[0032] d B = Ψ (R A , E B ),

[0033] In the formula, Ψ is a graph neural network, and the difference between d A , d B can be used to correct the initial assignment matrix to obtain a new matching that meets the neighbor consistency and has pixel-level accuracy.

[0034] As a further improvement of the present application, in step S7, in the enhanced feature map with a size of 1 / 2 of the original map, a local window with a size of 5x5 is intercepted around each matching point, and after the windows are serialized, a local fine-grained descriptor is obtained through the single-scale feature description network in step S2, the peak response of the descriptor at each matching point on the local fine-grained descriptor of the other map is calculated, and a final matching result with sub-pixel accuracy is obtained.

[0035] Compared with the prior art, the present application has the beneficial effects:

[0036] 1、 The present application improves the feature matching method of the feature point detector, and supplements the local geometric structure information for the single-scale feature descriptor.

[0037] 2、 The present application combines the hybrid-scale feature descriptor, which enhances the saliency of the feature descriptor and maintains the robustness of the descriptor to illumination and view angle transformation.

[0038] 3、 The present application designs a novel outlier filtering method, which verifies whether the obtained initial matching has neighbor consistency, enhances the reliability of the matching result, and has a wide application prospect in the fields of three-dimensional reconstruction, visual positioning and navigation. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is the step flow chart of the method of the present application;

[0040] Figure 2 is the picture matching schematic diagram after the method of the present application is used in embodiment 2 of the present application. DETAILED DESCRIPTION

[0041] The present application will be further illustrated in combination with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not used to limit the scope of the present application.

[0042] Embodiment 1

[0043] The image matching method combining the hybrid-scale feature descriptor and the neighbor consistency, as shown in the figure, comprises the following steps: Figure 1

[0044] S1, feature extraction: the pictures input from different angles and shooting the same picture are subjected to feature extraction at different resolutions through the FPN network. The feature map obtained by feature extraction has different spatial resolutions and semantic information, and the feature map of the original image 1 / 2 resolution size and the feature map of 1 / 8 size are used as the feature description in the next step;

[0045] S2, single-scale feature description: the 1 / 8 size feature map obtained in step S1 is encoded and input into the neural network based on the convolution and attention hybrid, to obtain a single-scale feature descriptor; the hybrid self-attention layer in the neural network based on the convolution and attention hybrid additionally increases a convolution branch, and the cross-attention layer remains unchanged, the convolution branch of the hybrid self-attention layer restores part of the geometric structure of the original image, and the attention branch performs information interaction between features; the cross-attention layer realizes information interaction between different features, and updates the features of each layer;

[0046] ​S3, multi-scale feature description; taking the original picture taken from different angles obtained in step S1 as input, inputting into the network based on enhanced self-attention, and outputting multi-scale feature description; in the network based on enhanced self-attention, the key matrix (K) and the value matrix (V) in the enhanced self-attention are down-sampled in different proportions in different self-attention heads, each self-attention head performs information transmission of different scale features, and a multi-scale feature description is generated;

[0047] S4, different scale feature fusion: splicing the single scale feature description obtained in step S2 and the multi-scale feature description obtained in step S3 in the feature dimension;

[0048] S5, inputting the mixed scale description obtained in step S4 into an optimal matching layer to obtain an initial assignment matrix; based on the set threshold, selecting an initial matching point pair;

[0049] S6, in the step S6, sparse descriptors of the corresponding point pairs are extracted, a sparse similarity matrix P is calculated by inner product, and the corresponding relationship of the point sets between images can be regarded as the corresponding relationship of the nodes in the graph structure, so that a node matrix E A , E B and an edge matrix E A , E B are constructed, here each node only retains the edge between the most similar two nodes, and a graph neural network with a shared parameter is used,

[0050] d A =Ψ(R A , E A ),

[0051] d B =Ψ(R A , E B ),

[0052] In the formula, Ψ is a graph neural network, d A , and the difference between d B can be used to correct the initial assignment matrix to obtain a new matching with pixel-level accuracy that meets the neighbor consistency.

[0053] S7, inputting the 1 / 2 size feature map obtained in step S1 and the mixed description obtained in step S4 into a fully connected neural network to obtain an enhanced 1 / 2 size feature map; inputting the obtained feature map and the pixel-level accuracy new matching point pair obtained in step S6 into a matching fine-tuning network, and outputting a final matching with sub-pixel-level accuracy.

[0054] Embodiment 2

[0055] The joint mixed scale feature description and neighbor consistency image matching method comprises the following steps:

[0056] S1: Feature extraction of input image pairs, input image pairs are input into the feature extraction network to obtain feature maps of different resolutions.

[0057] The experimental data set is MegaDepth, which is composed of 1 million Internet images of 196 different outdoor scenes.

[0058] Each image is initialized to be cropped to 840*840 size and converted to a gray image form as input.

[0059] S2: A neural network based on a hybrid of convolution and attention, using the feature map of 1 / 8 of the original image size obtained by step S1 after position encoding as input.

[0060] Input into the neural network based on a hybrid of convolution and attention, output single-scale feature descriptors, and the dimension of the descriptor is 256.

[0061] S3: Train the network based on enhanced self-attention, use the original image of step S1 as input, and the dimension of the descriptor is 128.

[0062] S4: Concatenate the single-scale feature descriptors obtained in step S2 and the multi-scale feature descriptors obtained in step S3 in the feature dimension.

[0063] S5: Input the mixed-scale descriptor obtained in step S4 into the optimal matching layer to obtain the initial assignment matrix; based on the set threshold, select the initial matching point pair.

[0064] First, calculate the similarity matrix between the two mixed descriptors,

[0065]

[0066] In the formula, τ is a constant, and <·> represents the inner product. The similarity matrix is used as the cost matrix of the partial assignment problem, and the optimal solution confidence assignment matrix can be obtained by solving this partial assignment problem, thereby obtaining the initial matching.

[0067] S6: Extract the sparse descriptors of the corresponding point pairs, calculate the sparse similarity matrix P by inner product, and the corresponding relationship of the point set between images can be regarded as the corresponding relationship of the nodes in the graph structure, thereby constructing the node matrix R A , R B and edge matrix E A , E B , here each node only retains the edge between the most similar two nodes, and via a graph neural network with a shared parameter,

[0068] d​A = Ψ(R A , E A ),

[0069] d B = Ψ(R A , E B ),

[0070] where Ψ is a graph neural network, d A , the difference between d B can be used to correct the initial assignment matrix to obtain new pixel-level accuracy matching points that meet the neighbor consistency.

[0071] S7: input the 1 / 2 size feature map obtained in step S1 and the hybrid descriptor obtained in step S4 into a fully connected neural network to obtain an enhanced 1 / 2 size feature map; input the obtained feature map and the new pixel-level accuracy matching points obtained in step S6 into a matching fine-tuning network to output the final sub-pixel level accuracy matching. Figure 2 After the method, all matching results are displayed. Figure 2 As can be seen from the above, the present application can accurately match small scale objects in images, and has strong robustness to scale and view angle transformation.

[0072] In summary, the present application is an image matching method combining hybrid scale feature descriptors and neighbor consistency, and the image matching model trained using the method can be run on a computer or other device, and by inputting an original image pair, a sub-pixel accuracy corresponding point set can be output, and the method has wide application in the fields of three-dimensional reconstruction, visual positioning and navigation, multi-target tracking, etc.

[0073] It should be noted that the above content only illustrates the technical idea of the present application, and cannot limit the protection scope of the present application, and for ordinary skilled persons in the art, a number of improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements all fall within the protection scope of the claims of the present application.

Claims

1. An image matching method combining mixed-scale feature descriptors and neighbor consistency, characterized in that: The following steps are involved: S1, Feature Extraction: The original image of the same scene taken from different perspectives is input and subjected to feature extraction at different resolutions through the FPN network. The feature maps obtained through feature extraction have different spatial resolutions and semantic information. The feature maps with 1 / 2 resolution and 1 / 8 resolution of the original image are used as feature descriptions in the next step; S2, single-scale feature description: The 1 / 8 size feature map position obtained in step S1 is encoded and input into a neural network based on a hybrid of convolution and attention to obtain a single-scale feature descriptor; the hybrid self-attention layer in the neural network based on a hybrid of convolution and attention adds an additional convolution branch while the cross-attention layer remains unchanged. The convolution branch of the hybrid self-attention layer restores the local geometric structure of the original image while the attention branch performs information exchange within the features; the cross-attention layer realizes information exchange between different features and updates the features of each layer; S3, multi-scale feature description: The original image obtained in step S1 and taken from different perspectives is input into a network based on enhanced self-attention, and a multi-scale feature descriptor is output; the key matrix K and the value matrix V in the enhanced self-attention in the enhanced self-attention network are downsampled at different ratios in different self-attention heads, and each self-attention head transmits information of features at different scales to generate a multi-scale feature descriptor; S4, different scale feature fusion: the single-scale feature descriptor obtained in step S2 and the multi-scale feature descriptor obtained in step S3 are spliced ​​in the feature dimension; S5, inputting the mixed-scale descriptor obtained in step S4 into the optimal matching layer to obtain an initial allocation matrix; based on the set threshold, selecting initial matching point pairs; S6, filtering outliers using neighbor consistency: The initial matching point pairs obtained in step S4 are modeled into a graph structure and input into a weighted graph neural network. The output of the graph neural network is used to modify the initial assignment matrix to obtain new matching point pairs. S7, matching refinement: The 1 / 2 size feature map obtained in step S1 and the hybrid descriptor obtained in step S4 are input into a fully connected neural network to obtain an enhanced 1 / 2 size feature map; the obtained feature map and the new matching point pairs with pixel-level accuracy obtained in step S6 are input into a matching refinement network, which outputs the final matching with sub-pixel accuracy, thereby constructing a complete image matching model and realizing image matching.

2. The image matching method combining mixed-scale feature descriptors and neighbor consistency as claimed in claim 1, characterized in that: In step S2, the 1 / 8 size feature map is positionally encoded and rearranged into a one-dimensional tensor; a single-scale feature descriptor is obtained through a hybrid self-attention layer and a cross-attention layer that fuses convolution and self-attention.

3. The image matching method combining mixed-scale feature descriptors and neighbor consistency as claimed in claim 2, characterized in that: The neural network training process based on the mixture of convolution and attention in step S2 is specifically as follows: the hybrid self-attention mechanism and the cross-attention mechanism are used alternately in different layers of the network. When the hybrid self-attention mechanism is used, the similarity between each pixel in the feature map is learned; when the cross-attention mechanism is used, the similarity between each pixel between feature maps is learned, and finally the information transmitted between network layers is obtained through a layer of fully connected neural network.

4. The image matching method combining mixed-scale feature descriptors and neighbor consistency as claimed in claim 2, characterized in that: The step S3 further comprises: S31: The key matrix (K) and value matrix (V) are downsampled at different scales in different self-attention heads. V i =V i +LE(V i ), In the formula, X represents the input feature, Represents the linear mapping matrix, r i represents the downsampling ratio of the i-th feature head, MTA(·) represents the multi-scale aggregation operation, and LE(·) is a convolutional neural network; S32: The query matrix (Q), key matrix (K) and value matrix (V) obtained in step S31 are used for information transmission. Where d h Represents the feature dimension of each feature head.

5. The image matching method combining mixed-scale feature descriptors and neighbor consistency according to claim 3 or 4, characterized in that: The splicing in the feature dimension in step S4 specifically includes: splicing the 256-dimensional single-scale feature descriptor and the 128-dimensional multi-scale feature descriptor in the feature dimension to obtain a 384-dimensional feature descriptor.

6. The image matching method combining mixed-scale feature descriptors and neighbor consistency as claimed in claim 5, characterized in that: The step S5 is specifically as follows: first calculate two mixed descriptors The similarity matrix between Where τ is a constant, <·> represents the inner product, and the similarity matrix serves as the cost matrix of the partial assignment problem. Solving this partial assignment problem can obtain the optimal solution confidence assignment matrix, thereby obtaining the initial matching.

7. The image matching method combining mixed-scale feature descriptors and neighbor consistency as claimed in claim 6, characterized in that: In step S6, the sparse descriptors of the corresponding point pairs are extracted, and the sparse similarity matrix P is calculated by the inner product. The correspondence between the point sets between images can be regarded as the correspondence between the nodes in the graph structure, thereby constructing the node matrix R A , R B and edge matrix E A , E B , where each node only retains the edges between the two most similar nodes and the rest, through a graph neural network with shared parameters, d A =Ψ(R A ,AND A ), d B =Ψ(R A ,AND B ), Where Ψ is the graph neural network, d A , d B The difference can be used to modify the initial assignment matrix to obtain a new match with pixel-level accuracy that meets the neighborhood consistency.

8. The image matching method combining mixed-scale feature descriptors and neighbor consistency as claimed in claim 7, characterized in that: In step S7, a 5×5 local window is captured with each matching point as the center in the enhanced feature map of half the size of the original image. These windows are serialized and passed through the single-scale feature description network in step S2 to obtain local fine-grained descriptors. The peak response of the descriptor at each matching point on the local fine-grained descriptor of the other image is calculated to obtain the final matching result with sub-pixel accuracy.

Images