Local feature matching system based on key point detection

Abstract

The invention discloses a local feature matching system based on key point detection. The local feature matching system comprises an encoder, a deep Transform model and a matching module, inputting a to-be-matched image pair (IA, IB) into an encoder, and extracting a fine feature # imgabs0 # coarse feature # imgabs1 # and key points PA and PB of the image pair (IA, IB); a rough feature # imgabs2 # and an image pair (IA, IB) are input into a deep Transform model, the deep Transform model performs feature aggregation on the image IA and the image IB to obtain a key point feature # imgabs3 #, and a matching module converts the key point feature # imgabs4 # into a credibility matrix C, performs matching of a key point PA and a key point PB based on the credibility matrix C, and performs matching enhancement based on a fine feature # imgabs5 # to complete matching of the image pair (IA, IB). A weight (parameter) multiplexing technology is adopted to share task parameters between continuous Transform layers according to task requirements, so that the model can maintain the expression ability of features, the model performance is improved, and the model scale can be effectively reduced; in addition, due to the use of the multi-scale key point detector, propagation of redundant information is reduced, feature specificity is enhanced, and model efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image matching, and more specifically, the present invention relates to a local feature matching system based on key point detection. Background Art

[0002] Local feature matching is a key step in many computer vision tasks such as SLAM, pose estimation, and visual localization, which aims to establish accurate correspondences between a pair of images. However, it is challenging to obtain robust and accurate correspondences in the face of challenges such as texture-poor areas, motion blur, illumination and viewpoint variations, and repetitive patterns.

[0003] Make full use of the global receptive field provided by Transformer to aggregate long-range visual and geometric information across key points. LoFTR is an outstanding representative of them. It uses linear Transformer with self-attention and cross-attention layers to simultaneously encode the key point features of two images and calculate the soft assignment matrix to achieve coarse-grained to fine-grained matching. The design and architecture of the transformer block in these methods play a key role in determining the matching performance. For example, Quadtree innovatively proposed a Transformer layer that can build a token pyramid and calculate hierarchical attention, which significantly improves the performance. ASpanFormer introduces a new attention mechanism that can adaptively adjust the span, thereby achieving state-of-the-art performance on multiple evaluation benchmarks.

[0004] However, the above methods mainly focus on designing complex Transformer architectures, which increases the model size and further increases the storage and computational costs. Summary of the invention

[0005] The present invention provides a local feature matching method based on weight reuse, aiming to improve the above problems.

[0006] The present invention is implemented in this way: a local feature matching system based on key point detection, the system comprising:

[0007] It consists of an encoder, a deep Transformer model, and a matching module;

[0008] The image pair to be matched (I A ,I B ) is input to the encoder, which extracts the image pair (I A ,I B ) Coarse Features And the key point P A , P B ;

[0009] The coarse features and image pair (I A ,I B ) Input deep Transformer model, deep Transformer model for image I A 、Image I B Perform feature aggregation to obtain key point features

[0010] The matching module converts key point features Convert to the credibility matrix C, and perform key point P based on the credibility matrix C A With key point P B Matching, then based on fine features Perform matching enhancement to complete the image pair (I A ,I B ) is matched.

[0011] Furthermore, the deep Transformer model is composed of s Transformer layers connected in sequence, and the k-1th Transformer layer will share the parameter θ s Passed to the kth Transformer layer, if the model parameter θ of the kth Transformer layer k The corresponding loss is lower than the shared parameter θ s , then update the shared parameter to the model parameter θ k , if the model parameter θ k The corresponding loss is higher than the shared parameter θ s , then the shared parameter θ is not updated s , and then share the parameter θ s Passed to the k+1th Transformer layer.

[0012] Furthermore, the Transformer layer is composed of a dense feature aggregation module, a sparse feature aggregation module, and a sparse to dense feature aggregation module, the dense feature aggregation module is connected to the sparse feature aggregation module, and the dense feature aggregation module, the sparse feature aggregation module and the sparse to dense feature aggregation module are connected;

[0013] Among them, the dense feature aggregation module is used to aggregate the dense coarse features of the feature points output by the encoder. Aggregate them separately to obtain feature F AD 、F BD ; The sparse feature aggregation module is used to extract feature F AD 、F BD The representative key points in form the feature F AK 、F BK , for feature F AK 、FBK After polymerization, the enhanced feature F is formed AE 、F BE ; The sparse to dense feature aggregation module will enhance the feature F AE 、F BE The representative key point features in are passed to the feature F AD 、F BD , thus generating features and

[0014] Furthermore, the visual description F A and F B Input the dense feature aggregation module, which performs T1 recursive feature aggregation, expressed as:

[0015]

[0016] in, They represent the initial visual description F A 、F B The enhanced features formed after i recursions, They represent the initial visual description F A 、F B The enhanced features formed after i-1 recursions, θ i represents the model parameters of the current Transformer layer after i recursions, θ s Represents the shared parameters passed by the previous Transformer layer.

[0017] Furthermore, image I A The representative key point extraction process is as follows:

[0018] Will Convert to a 2D feature map and calculate the sum along the channel dimension to generate

[0019] For image I A For the pixel with coordinates (i, j), a local window of size w×w is cut with the corresponding pixel as the center to generate the confidence score of the pixel.

[0020]

[0021] in, They are image I A The confidence score and features of the pixel at coordinate (i,j), i∈{0,1,2,...,H / 8-1}, j∈{0,1,2,...,w / 8-1};

[0022] Use w=3, 5, 7 to crop the multi-scale local window and generate the confidence score of pixel (i, j) The confidence score

[0023] Keep S A The first K key points with the highest confidence scores in image I A The representative key point P AK .

[0024] Furthermore, the sparse feature aggregation module performs AK , Feature F BK Perform T2 recursive updates, and the aggregation process is expressed as:

[0025]

[0026] Among them, F AK 、F BK As initial value Represents the feature F AK 、F BK The enhanced features formed after i recursions, Represents the feature F AK 、F BK The enhanced features formed after i-1 recursions, θ i represents the model parameters of the current Transformer layer after i recursions, θ s Represents the shared parameters passed by the previous Transformer layer.

[0027] Furthermore, the feature F output by the sparse feature aggregation module AE 、F BE The representative key point features in are passed to the feature F output by the dense feature aggregation module AD 、F BD , the sparse to dense feature aggregation module performs T3 recursive updates, and the update process is expressed as:

[0028]

[0029] Among them, the enhanced feature F output by the dense feature aggregation module AD 、F BD As initial value The representative key point features F output by the sparse feature aggregation module AE 、F BE As initial value

[0030] Furthermore, the formation process of the credibility matrix C is as follows:

[0031] Determine the characteristics The score matrix S between the representative key points in,

[0032] The score matrix S is converted into the credibility matrix C, C(i,j) = Softmax(S(i,·))·Softmax(S(·,j)), where S(i,·) and S(·,j) represent all elements in the i-th row and j-th column of the score matrix S, respectively, and C(i,j) represents the credibility of the match between the i-th key point and the j-th key point.

[0033] Furthermore, the image I is divided into 8×8 grids A ,I B , and the central pixels of all grids are taken as the key points P of the corresponding images A , P B .

[0034] The present invention adopts weight (parameter) reuse technology to share task parameters between continuous Transformer layers according to task requirements, so that the model can not only maintain the expressive ability of features and improve model performance, but also effectively reduce the model scale; in addition, the use of multi-scale key point detectors reduces the propagation of redundant information, enhances the specificity of features, and improves model efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the structure of a local feature matching system based on key point detection provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The specific implementation modes of the present invention are further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0037] The image pair (I A ,I B) is input into the encoder, and the CNN-based encoder is first used to generate low-resolution coarse features for coarse-level matching and high-resolution fine features for refining coarse matches into fine matches; the coarse features are flattened into a one-dimensional sequence and input into the deep Transformer, and the model performs long-range feature aggregation across images and within images. In particular, the model adopts the proposed weight reuse technology, which can continuously share weight parameters between Transformer layers, effectively aggregating features while reducing the size of the model; in addition, a multi-scale key point detection module is introduced to identify representative key points and replace all key points for feature aggregation, thereby reducing the transfer of unimportant information to a certain extent while keeping the computational overhead controllable. After the deep Transformer model, the coarse matching module is used to generate dense matches at the coarse level. Finally, the matching refinement module is used to refine the matching to the sub-pixel level to establish accurate and reliable matching between images.

[0038] Figure 1 A schematic diagram of the structure of a local feature matching system based on key point detection provided by an embodiment of the present invention. For ease of explanation, only the part related to the embodiment of the present invention is shown. The system includes:

[0039] The deep Transformer model consists of an encoder, a deep Transformer model, and a matching module. The deep Transformer model is composed of s Transformer layers connected in sequence.

[0040] The image pair to be matched (I A ,I B ) is input to the encoder, which extracts the image pair (I A ,I B ) Coarse Features And the key point P A , P B ; The coarse features and image pair (I A ,I B ) Input deep Transformer model, deep Transformer model for image I A 、Image I B Perform feature aggregation to obtain key point features The matching module converts key point features Convert to the credibility matrix C, and perform key point P based on the credibility matrix C A With key point P B Matching, then based on fine features Perform matching enhancement to complete the image pair (I A ,I B ) is matched.

[0041] (1) The encoder extracts fine features, coarse features, and key points;

[0042] A convolutional neural network (CNN) with a feature pyramid (FPN) is used to map the image pairs to be matched (I A ,I B ) is input into the encoder, and the encoder extracts the image pair to be matched (I A ,I B ) with 1 / 2 downsampling of fine features and 1 / 8 downsampled coarse features Divide the image I into an 8×8 grid A ,I B , and the central pixels of all grids are taken as the key points P of the corresponding images A ,P B ∈R N×2 , the coarse features As an initial visual description Where N = H / 8×W / 8.

[0043] (2) Deep Transformer model aggregation features;

[0044] The deep Transformer model is composed of s Transformer layers connected in sequence. The k-1th Transformer layer will share the parameter θ s Passed to the kth Transformer layer, if the model parameter θ of the kth Transformer layer k The corresponding loss is lower than the shared parameter θ s , then update the shared parameter to the model parameter θ k , if the model parameter θ k The corresponding loss is higher than the shared parameter θ s , then the shared parameter θ is not updated s , and then share the parameter θ s Pass it to the k+1th Transformer layer and loop through the above process.

[0045] The effect of the number of Transformer layers on matching performance has been examined from different angles. Stacking more Transformer layers can extract deep aggregate features, thereby improving matching accuracy. On the other hand, when people match images, they will repeatedly scan back and forth. The more times they scan, the easier it is to remember easy-to-match features. However, building a deep Transformer architecture will lead to the problem that the model size grows linearly with the number of Transformer layers. In order to solve this problem, the present invention introduces weight reuse technology, which recycles the weights of continuous Transformer layers and can effectively control the scale of the model.

[0046] In this embodiment of the present invention, the Transformer layer is composed of a linear self-attention layer and a cross-attention layer. First, the linear attention technology used to construct the Transformer layer is introduced, which is the basis of the weight reuse technology. The input features of the Transformer layer are represented as U and R. Given U and R, firstly, the query, key and value are generated by linear projection, which are represented as Where N is the number of key points, Represents the dimensions of query, key, and value. The above process can be expressed as:

[0047] Q=UW Q ,K=RW K ,V=RW V (1)

[0048] Among them, W Q ,W K ,W V is a learnable parameter implemented by three multi-layer perceptrons (MLPs). Linear attention is applied to model the global dependencies between key points as follows:

[0049] M=φ(Q)(φ(K) T V),φ(·)=elu(·)+1 (2)

[0050] Where M represents the retrieved global information and elu(·) represents the nonlinear activation function. Then, another MLP is used to process the retrieved global information: M′=MW O , are the parameters of the MLP. Finally, a feed-forward network (FFN) is applied after the linear attention layer to extract deep aggregate features. The FFN consists of two linear transformation layers and a nonlinear activation function between them, which can be expressed as:

[0051]

[0052] Among them, W1 and W2 are the parameter matrices of the two linear transformation layers, [·||·] represents the connection operation along the channel dimension, and σ represents a nonlinear activation function such as GELU.

[0053] Transformer layers with self-attention layers are used to aggregate long-range features within an image, and Transformer blocks with cross-attention layers are used to aggregate long-range features between images. Specifically, for the Transformer block with self-attention layers, the input features U and R are the same, such as (U = F A ,R=F A ), for the Transformer block with crisscross attention layer, the input features U and R come from two images (U = F A ,R=F B Or U=F B ,R=F A )The complete Transformer layer is represented as:

[0054] U′,R′=Trans(U,R;θ) (4)

[0055] Among them, U′, R′ represents the features after the Transformer layer aggregation, θ is the parameter of the Transformer layer with self / cross attention, that is, θ = {W Q ,W K ,W V ,W1,W2,W O ,}, the main parameters of the model are composed of θ.

[0056] Based on the above attention mechanism, a deep Transformer layer with self-attention and cross-attention is constructed. In order to further reduce the computational overhead caused by an overly deep Transformer network, the present invention decomposes the construction of the deep Transformer architecture into: (1) dense feature aggregation module, (2) sparse feature aggregation block, and (3) sparse to dense feature aggregation block.

[0057] (21) Dense feature aggregation module:

[0058] The visual description F A and F B Input the Transformer layer, and then apply the weight recycling technique on the Transformer layer to perform T1 recursive feature aggregation, which can be expressed as:

[0059]

[0060] in, They represent the initial visual description F A 、FB The enhanced features formed after i recursions, θ i represents the model parameters of the current Transformer layer after i recursions, θ s Represents the shared parameters passed by the previous Transformer layer; the enhanced features formed after T1 is passed Use F AD 、F BD express,

[0061] (22) Sparse feature aggregation module: In order to reduce the computational cost caused by too deep Transformer layers, the present invention proposes a multi-scale key point detection module for identifying representative key points to replace all key points for further feature aggregation. At the same time, using representative key points for feature aggregation also has the advantage of reducing the propagation of redundant information. The inspiration of the multi-scale key point detection module comes from the fact that key points with rich visual information can be distinguished from their neighboring key points at different scales. More specifically, after updating the feature vector T1 times, For example, Convert to a 2D feature map and calculate the sum along the channel dimension to generate Then, for image I A For the pixel with coordinates (i, j), a local window of size w×w is cut with the pixel as the center, and then the confidence score of the pixel is generated.

[0062]

[0063] in, They are image I A The confidence score and features of the pixel at the coordinate (i, j) in the center, i∈{0,1,2,...,H / 8-1}, j∈{0,1,2,...,w / 8-1}, therefore, formula (6) compares the relationship between the central element and its adjacent elements through subtraction operation. For a pixel with a high value in the feature map and a large gap with the surrounding pixels, it can have a higher confidence score and can be regarded as a representative key point. In order to improve the robustness of representative key point detection, multi-scale local windows are cropped using w=3, 5, 7 to generate the confidence score of pixel (i, j) The confidence score It is expressed as:

[0064]

[0065] S A ∈R H / 8×w / 8 express The confidence score mask of SA , retain the key points with the top K confidence scores as image I A The representative key point P AK ∈R K×2 , with the descriptor For subsequent feature aggregation. The same process is used to B Get the F BK The representative key point P BK The proposed weight recycling technique is used on the Transformer layer to perform T2 recursive updates to promote deep feature aggregation between representative key points. Each complete Transformer layer is represented as:

[0066]

[0067] Among them, F AK 、F BK As initial value After the sparse feature aggregation module, updated representative key point features are further obtained and They are defined as F AE 、F BE .

[0068] (23) Sparse to dense feature aggregation module: Once the updated representative key point features F are obtained AE and F BE , introduces a sparse to dense feature aggregation module to pass the features of representative key points to all key points in the feature map, thereby generating features and It is used to generate rough matching and simulate T3 recursive updates using weight recycling technology, which can be expressed as:

[0069]

[0070]

[0071] Among them, the enhanced feature F output by the dense feature aggregation module AD 、F BD As initial value The representative key point features F output by the sparse feature aggregation module AE 、F BE As initial value The enhanced features formed after T3 recursion Expressed as

[0072] (3) The matching module performs key point matching between images.

[0073] Rough matching extraction: given Define the score matrix S∈R between the selected representative key points N ×N for:

[0074]

[0075] Then, double softmax is applied to transform the score matrix S into the credibility matrix C:

[0076] C(i,j)=Softmax(S(i,·))·Softmax(S(·,j)) (11)

[0077] Among them, S(i,·) and S(·,j) represent all elements in the i-th row and j-th column of the score matrix S, that is, the similarity score between the i-th key point and all key points, and the similarity score between all key points and the j-th key point. C(i,j) represents the credibility of the match between the i-th key point and the j-th key point. Based on the credibility matrix C, the threshold λ and the mutual nearest neighbor (MNN) criterion are used to select the key point P from the key point P. A ,P B Extracting coarse matching points in, n represents the number of predicted matches.

[0078] Fine match extraction: After extracting the coarse match, generate the fine match At the coarse matching point Several local windows of size w×w are cropped around the image, and the fine features extracted by CNN encoding are extracted. to enhance the cropped features. Then, the inner product of the transformed features is calculated and the softmax function is applied to generate a probability distribution map. Finally, the expectation of the probability distribution is calculated as the offset δ∈R n×2 Finally, fine matching It can be expressed as:

[0079]

[0080] The matching of the two images is completed based on the final fine matching result.

[0081] The local feature matching system based on key point detection provided by the present invention has the following beneficial technical effects:

[0082] (1) Reducing model parameters and improving model efficiency: The present invention adopts weight (parameter) reuse technology to share task parameters between continuous Transformer layers according to task requirements, so that the model can not only maintain the expressive ability of features and improve model performance, but also effectively reduce the model scale; in addition, the use of multi-scale key point detectors reduces the propagation of redundant information, enhances the specificity of features, and improves model efficiency.

[0083] (2) Enhanced feature expression capability: This invention builds a deeper Transformer network and combines multiple feature aggregation forms to establish a closer association between features at different levels and scales, thereby enhancing the expressiveness of features. Especially in complex scenarios, it can capture more detailed information, effectively improving the distinguishability and expressiveness of features, thereby improving the model's ability to recognize and classify fine-grained features.

[0084] (3) Improving feature matching effect: The present invention strengthens the local and global dependencies of features through a deep Transformer network, effectively improving the accuracy and robustness of feature matching, especially when the image content is complex, the target is occluded or deformed, and can still maintain a good matching effect.

[0085] (4) Stronger robustness: The present invention significantly enhances the robustness of the model in complex environments through deep Transformer networks and multi-scale key point detection. Regardless of harsh conditions such as low light, noise interference or target occlusion, the model can effectively maintain stable performance. The feature transformation module enhances the diversity of features, allowing the model to better cope with various changes in different scenarios and reduce the negative impact caused by external environmental fluctuations.

[0086] The present invention has been described exemplarily. Obviously, the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A local feature matching system based on key point detection, characterized in that: The system comprises: It consists of an encoder, a deep Transformer model, and a matching module; The image pair to be matched (I A ,I B ) is input to the encoder, which extracts the image pair (I A ,I B ) Coarse Features And the key point P A , P B ; The coarse features and image pair (I A ,I B ) Input deep Transformer model, deep Transformer model for image I A 、Image I B Perform feature aggregation to obtain key point features The matching module converts key point features Convert to the credibility matrix C, and perform key point P based on the credibility matrix C A With key point P B Matching, then based on fine features Perform matching enhancement to complete the image pair (I A ,I B ) is matched.

2. The local feature matching system based on key point detection as claimed in claim 1, characterized in that: The deep Transformer model is composed of s Transformer layers connected in sequence. The k-1th Transformer layer will share the parameter θ s Passed to the kth Transformer layer, if the model parameter θ of the kth Transformer layer k The corresponding loss is lower than the shared parameter θ s , then update the shared parameter to the model parameter θ k , if the model parameter θ k The corresponding loss is higher than the shared parameter θ s , then the shared parameter θ is not updated s , and then share the parameter θ s Passed to the k+1th Transformer layer.

3. The local feature matching system based on key point detection as claimed in claim 2, characterized in that: The Transformer layer consists of a dense feature aggregation module, a sparse feature aggregation module, and a sparse to dense feature aggregation module. The dense feature aggregation module is connected to the sparse feature aggregation module, and the dense feature aggregation module, the sparse feature aggregation module and the sparse to dense feature aggregation module are connected. Among them, the dense feature aggregation module is used to aggregate the dense coarse features of the feature points output by the encoder. Aggregate them separately to obtain feature F AD 、F BD ; The sparse feature aggregation module is used to extract feature F AD 、F BD The representative key points in form the feature F AK 、F BK , for feature F AK 、F BK After polymerization, the enhanced feature F is formed AE 、F BE ; The sparse to dense feature aggregation module will enhance the feature F AE 、F BE The representative key point features in are passed to the feature F AD 、F BD , thus generating features and 4. The local feature matching system based on key point detection as claimed in claim 3, characterized in that: The visual description F A and F B Input the dense feature aggregation module, which performs T1 recursive feature aggregation, expressed as: in, They represent the initial visual description F A 、F B The enhanced features formed after i recursions, They represent the initial visual description F A 、F B The enhanced features formed after i-1 recursions, θ i represents the model parameters of the current Transformer layer after i recursions, θ s Represents the shared parameters passed by the previous Transformer layer.

5. The local feature matching system based on key point detection as claimed in claim 3, characterized in that: Image I A The representative key point extraction process is as follows: Will Convert to a 2D feature map and calculate the sum along the channel dimension to generate For image I A For the pixel with coordinates (i, j), a local window of size w×w is cut with the corresponding pixel as the center to generate the confidence score of the pixel. in, They are image I A The confidence score and features of the pixel at coordinate (i,j), i∈{0,1,2,...,H / 8-1}, j∈{0,1,2,...,w / 8-1}; Use w=3, 5, 7 to crop the multi-scale local window and generate the confidence score of pixel (i, j) The confidence score Keep S A The first K key points with the highest confidence scores in image I A The representative key point P AK .

6. The local feature matching system based on key point detection as claimed in claim 5, characterized in that: The sparse feature aggregation module performs AK , Feature F BK Perform T2 recursive updates, and the aggregation process is expressed as: Among them, F AK 、F BK As initial value Represents the feature F AK 、F BK The enhanced features formed after i recursions, Represents the feature F AK 、F BK The enhanced features formed after i-1 recursions, θ i represents the model parameters of the current Transformer layer after i recursions, θ s Represents the shared parameters passed by the previous Transformer layer.

7. The local feature matching system based on key point detection as claimed in claim 3, characterized in that: The feature F output by the sparse feature aggregation module AE 、F BE The representative key point features in are passed to the feature F output by the dense feature aggregation module AD 、F BD , the sparse to dense feature aggregation module performs T3 recursive updates, and the update process is expressed as: Among them, the enhanced feature F output by the dense feature aggregation module AD 、F BD As initial value The representative key point features F output by the sparse feature aggregation module AE 、F BE As initial value 8. The local feature matching system based on key point detection as claimed in claim 1, characterized in that: The formation process of the credibility matrix C is as follows: Determine the characteristics The score matrix S between the representative key points in, The score matrix S is converted into the credibility matrix C, C(i,j) = Softmax(S(i,·))·Softmax(S(·,j)), where S(i,·) and S(·,j) represent all elements in the i-th row and j-th column of the score matrix S, respectively, and C(i,j) represents the credibility of the match between the i-th key point and the j-th key point.

9. The local feature matching system based on key point detection as claimed in claim 1, characterized in that: Divide the image I into an 8×8 grid A ,I B , and the central pixels of all grids are taken as the key points P of the corresponding images A , P B .

Images