Multimodal image feature matching method based on spatial and channel two-dimensional attention

Abstract

The application discloses a kind of multi-modal image feature matching methods based on space and channel two-dimensional attention, the method designs a kind of containing space and channel two-dimensional attention attention mechanism, feature extraction can be carried out to space and channel two-dimensional simultaneously, to improve the information interaction ability of model between channel, to help model to locate on more important feature, to improve matching efficiency.Based on the way of space and channel two-dimensional attention, the model not only solves the feature matching problem under multi-modal image, and has excellent matching ability and matching accuracy, has good matching result and very good generalization under a variety of different modal data sets, has very high practical application value.In addition, when using the model for feature matching under multi-modal image, only need to input the data set to be matched into the trained, based on space and channel two-dimensional matching network, can be automatically carried out feature matching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a multi-modal image feature matching method based on spatial and channel two-dimensional attention. BACKGROUND

[0002] Feature matching technology between multi-modal image data has a wide range of application scenarios in many aspects, including visual positioning, motion structure (SfM), global mapping, etc. The previous mainstream research is a two-stage local feature matching method based on a detector, and the common steps include: feature detection, descriptor extraction and feature matching. However, the method based on the detector depends on the results of the feature point detector, which to some extent leads to the lack of effectiveness, resulting in low model matching efficiency. With the excellent performance of Transformer in the field of natural language processing, more and more researches have focused on applying Transformer in the field of computer vision. ViT benefits from its excellent global modeling capability and is widely used in various image processing tasks.

[0003] Although ViT can help the model to interact long-range information in space, it lacks the interaction of inter-channel information, which leads to its lack of ability to locate important features in the channel, which is not conducive to feature matching under multi-modal images. Secondly, a large number of studies have shown that inter-channel information interaction is of great benefit to feature extraction. Therefore, it is urgently needed to build a model that can simultaneously interact information in spatial and channel two dimensions, has excellent performance and high efficiency. SUMMARY

[0004] The main purpose of the present application is to provide a multi-modal image feature matching method based on spatial and channel two-dimensional attention, which extends the spatial attention Transformer, enhances the information interaction between channels of the model, and improves the ability of the model to locate important features, in order to solve the problem of multi-modal image feature matching.

[0005] To achieve the above purpose, the present application provides a multi-modal image feature matching method based on spatial and channel two-dimensional attention, which comprises the following steps:

[0006] S1: using an improved FPN architecture to preliminarily extract the features of the image;

[0007] S2: using spatial attention to interact features in the spatial dimension;

[0008] S3: using position encoding to consolidate image position information;

[0009] S4: attention interaction in the channel dimension of the image to complete the final feature extraction;

[0010] S5: using bidirectional softmax to process the features after attention interaction, training the model, and realizing feature matching under multi-modal images.

[0011] Optionally, the improved FPN architecture specifically comprises four BasicBlocks, four UpConv blocks, four Connect blocks, and four 3x3 convolution layers for dimension reduction.

[0012] Optionally, the BasicBlock is composed of two 3x3 convolution layers, two BatchNorm normalization functions, and one ReLU activation; the UpConv block is composed of a bilinear interpolation function, a 3x3 convolution layer, and a BatchNorm function; and the Connect block is composed of a Concat function.

[0013] Optionally, in step S2, the expression of the feature interaction step in the spatial dimension using spatial attention is specifically as follows:

[0014] x s =Attention(x)

[0015]

[0016] Q=xW Q

[0017] K=xW K

[0018] V=xW V

[0019] wherein the matrix W is a learnable matrix for generating vectors Q, K, and V according to the input, d is a scaling factor, x s is a feature map after spatial information interaction, and x is a feature map.

[0020] Optionally, in step S3, the position encoding adopts convolution-based position encoding, which is composed of a 3x3 convolution layer and a GLEU() activation function.

[0021] Optionally, the expression of the position encoding is specifically as follows:

[0022] x p =GLEU(Conv 3×3 (x s )+x s

[0023] Optionally, step S4 specifically comprises:

[0024] based on x p and three learnable matrices W c, obtain a query vector Q c , obtain a key vector K c , obtain a value vector V c ; transpose K c ; multiply the transpose of the key vector K c and the value vector V c , and perform softmax normalization on the obtained result to obtain an attention score;

[0025] transpose Q c , multiply the transposed result with the attention score, transpose the multiplication result, and pass it through a projection layer to obtain an output x c that has undergone channel information interaction.

[0026] x c perform a LayerNorm operation, an Mlp operation, a Dropout operation, and a residual operation with the initial x c to obtain a final output x F .

[0027] Optionally, the expression of the step S4 is specifically:

[0028]

[0029]

[0030]

[0031]

[0032]

[0033] x F = Dropout(Mlp(LayerNorm(x c )))+x c

[0034] wherein, @ represents matrix multiplication, attn represents the attention score, and Proj is a Linear layer with the same input and output dimensions.

[0035] Optionally, in the step S5, the expression of processing the feature after attention interaction using bidirectional softmax is specifically:

[0036] P(i,j) = softmax(S(i,·)) j · softmax(S(·,j)) i

[0037] Wherein, softmax represents a normalization operation, softmax(S(i,·)) j Indicates the operation of doing softmax on all elements of the ith row, softmax(S(·,j)) i Indicates the operation of doing softmax on all elements of the jth column, and the two results are multiplied to obtain the final matching probability matrix P.

[0038] Optionally, in step S5, the expression for training the model is specifically:

[0039]

[0040] Wherein, |M conf |Indicates the total number of matching samples, ∑ k (·)Indicates the sum of k samples, GT i,j Indicates the label, and P(i,j) indicates the correct matching probability.

[0041] The beneficial effects of the present application are that a new multi-modal image feature matching method is proposed, and an attention mechanism containing spatial and channel double-dimensional attention is designed. The improved model can simultaneously extract features in spatial and channel double dimensions to improve the information interaction ability between channels of the model, so as to help the model to locate the matching on more important features, thereby improving the matching efficiency. Based on the spatial and channel double-dimensional attention, the model not only solves the feature matching problem under multi-modal images, but also has excellent matching ability and matching accuracy, and has good matching results and very good generalization under various different modal datasets, and has high practical application value. In addition, when using the present model for feature matching under multi-modal images, only the to-be-matched dataset needs to be input into the trained matching network based on spatial and channel double dimensions, and the feature matching can be automatically performed. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flowchart of the multi-modal image feature matching method based on spatial and channel double dimensions of the embodiment of the present application;

[0043] Figure 2 is a structure diagram of the improved FPN for feature extraction of the embodiment of the present application;

[0044] Figure 3 is a general architecture diagram of the multi-modal image feature matching method based on spatial and channel double dimensions of the embodiment of the present application.

[0045] Figure 4 is a structure diagram of the Attention Block of the embodiment of the present application;

[0046] Figure 5 is a specific flow chart of the Attention Block of the embodiment of the present application;

[0047] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0050] The embodiment of the present application provides a multi-modal image feature matching method based on spatial and channel two-dimensional attention, which refers to Figure 1 .

[0051] As shown in Figure 1 , the embodiment discloses a multi-modal image feature matching method based on spatial and channel two-dimensional attention, which realizes feature matching under various image data, including the following steps:

[0052] S1: using the improved FPN module to preliminarily extract the features of a group of images.

[0053] As shown in Figure 2 , the FPN module of the present application contains multiple downsampling and upsampling operations. The upsampling and downsampling operations are mainly composed of 3x3 convolution, ReLU activation function and BatchNorm normalization function.

[0054] Among them, the input of the FPN module is the original feature map, represented by x1 and x2. Both feature maps will pass through this improved FPN module and output the corresponding 1 / 2 feature map and 1 / 8 feature map of the original size.

[0055] For the 1 / 8 feature map, it is generated by 1 / 32, 1 / 16 and 1 / 4 through fusion. The specific steps are: the 1 / 32 feature map is subjected to upsampling and dimension reduction operation, and is subjected to concat operation on the channel with the 1 / 16 feature map to generate a new 1 / 16 feature map x 1 / 16 . x 1 / 16 will first be subjected to dimension reduction operation so that the dimension is the same as that of the 1 / 8 feature map, and then will be upsampled to be mapped to 1 / 8 size, and subjected to concat operation with the original 1 / 8 feature map to generate a new 1 / 8 feature map x 1 / 8 . Further, the 1 / 4 feature map will be subjected to downsampling and dimension increasing operation, and subjected to concat operation with x1 / 8 fusion, to get the final 1 / 8 feature map x c , as the input of the spatial and channel dual-dimensional attention interaction.

[0056] For 1 / 2 feature maps, 1 / 4 and 1 / 8 will be fused with 1 / 2 feature maps in the same way as the above operation process, to get the final 1 / 2 feature map x f , for subsequent refinement of the matching results;

[0057] x i = Concat(x i , Up(x i+1 ))

[0058] x c = Concat(x 1 / 8 , Down(x 1 / 4 ))

[0059] x j+1 = Concat(x j Up(x j+1 ))

[0060] x f = x 1 / 2

[0061] Where Up and Down represent up-sampling and down-sampling operations respectively. i∈{1 / 8, 1 / 16, 1 / 32}, j∈{1 / 2, 1 / 4, 1 / 8}.

[0062] The traditional FPN architecture will have some problems. For example, for 1 / 2 feature maps, too rough small feature maps (1 / 32, 1 / 16) may have an adverse effect on the 1 / 2 size refinement requirement. The improved FPN generated 1 / 2 feature Figure 1 to a certain extent, avoid the fusion of these adverse information, will be more able to meet the matching needs of images of different scales.

[0063] S2: using spatial attention to interact in the spatial dimension.

[0064] As shown in Figure 3 , based on the interaction of spatial and channel dual-dimensional attention, it is mainly composed of Attention Block. The input x c will first add position information through absolute position encoding, and complete information enhancement through a series of Attention Block. Each Attention Block contains spatial and channel attention.

[0065] The spatial attention is divided into self-attention and cross-attention. In the case of self-attention, the query vector Q and the key vector K and the value vector V come from the same feature map; in the case of cross-attention, the query vector Q and the key vector K and the value vector V come from two feature maps. The vectors Q, K, V and the spatial attention can be represented by the following formula:

[0066] Q = x c W Q

[0067] K = x c W K

[0068] V = x c W V

[0069]

[0070] x s = Attention(x c )

[0071] wherein the matrix W is a learnable matrix for generating vectors Q, K and V according to the input, d is a scaling factor, x s is a feature map after spatial information interaction.

[0072] Specifically, as shown in Figure 5 , the spatial attention module includes two Linear, self-attention or cross-attention layers (Self-Attn or Cross-Attn), and an Mlp layer. The first Linear corresponds to generating three learnable matrices W, and then x s is obtained through the spatial attention layer. Then, x s will pass through a Linear layer and an Mlp layer to obtain the final output:

[0073] x s = Mlp(Linear(x s ))

[0074] S3: Consolidate position information using position encoding.

[0075] As shown in Figure 4 , a position encoding is additionally added between the spatial attention and the channel attention to consolidate the position information, so as to avoid the case that the position information is gradually blurred or lost in multiple iterations. The position encoding is realized based on convolution, mainly composed of a 3x3 convolution and a GLEU() activation function, and can be represented as:

[0076] x p = GLEU(Conv 3×3 (xs )+x s

[0077] where Conv 3×3 represents a 3x3 convolution, x p represents a feature map after adding position information.

[0078] As shown in Figure 5 , the role of position encoding is specifically shown. The convolution-based position encoding (ConvPostion) is placed before the Mlp layer of spatial attention and the Linear layer of channel attention. To consolidate position information, which is very important for channel attention.

[0079] S4: Attention interaction is performed in the channel dimension of the image to complete the final feature extraction.

[0080] As shown in Figure 4 , after passing through each spatial attention (Self-Attn or Cross-Attn) and adding position encoding, channel attention (Channel-Attn) is performed to complete the information interaction between channels, improve the ability of the model to focus on important features, and better concentrate matching on important features to improve matching efficiency.

[0081] Like the spatial attention layer, the channel attention layer is based on x p and three learnable matrices W c , to obtain query vector Q c , key vector K c and value vector V c . Then transpose K c from NxC to CxN. Further, multiply the transpose of key vector K c and value vector V c , and perform softmax normalization operation on the obtained result to obtain the attention score. This process can be represented as:

[0082]

[0083]

[0084]

[0085]

[0086] where @ represents matrix multiplication, and attn represents the attention score.

[0087] Then Q cTransposed, the result of the transpose is multiplied by the attention score. The result of the multiplication will be transposed from CxN to NxC, and finally pass through the projection layer to get the output x that has passed through the channel information interaction c The specific generation method is as follows:

[0088]

[0089] Where Proj is a Linear layer with the same input and output dimensions.

[0090] Finally, x c After a LayerNorm, an Mlp, a Dropout, and an initial x c Residual, get the final output x F , which is specifically represented as:

[0091] x F = Dropout (Mlp (LayerNorm (x c )) ) + x c

[0092] As Figure 5 shown, the channel attention flow is shown. After passing through ConvPostion, x c After channel attention (Channel-Attn), the result is normalized (LayerNorm) through a projection layer. Finally, an Mlp layer and a Dropout operation are used to get the final output.

[0093] S5: Use bidirectional Softmax to process the features after attention interaction, train the model, and realize feature matching.

[0094] As Figure 1 shown, the feature map x F after a series of Attention-Block information enhancement will pass through a bidirectional softmax operation (Dual-Sfotmax). To get the nearest neighbor matching probability between the two images, the probability matrix P can be represented as:

[0095] P(i,j) = softmax(S(i,·)) j · softmax(S(·,j)) i

[0096] Where softmax represents the normalization operation, softmax(S(i,·)) jdenotes the operation of softmax on all elements of the i-th row, softmax(S(·,j)) i denotes the operation of softmax on all elements of the j-th column, and the two results are multiplied to obtain the final matching probability matrix P.

[0097] The loss function used for supervised training of the model is as follows:

[0098]

[0099] where |M conf | denotes the total number of matched samples, ∑ k (·) denotes the sum of k samples, GT i,j denotes the label, and P(i,j) denotes the probability of correct matching.

[0100] When using the network for feature matching under multi-modal images, only the data set to be matched is input into the trained matching network based on spatial and channel two-dimensional attention, and the feature matching can be automatically performed.

[0101] Embodiment 2

[0102] Homography estimation experiment

[0103] (1) Data set: three kinds of multi-modal data sets are used for pose estimation effectiveness verification, including Optical-SAR, NYU-Depth V2, and RGB-NIR Scene.

[0104] The Optical-SAR data set includes six main scenes such as islands, plains, and rivers. The present application selects 16000 pairs of images for training and 500 pairs of images for testing.

[0105] The NYU-Depth V2 data set is composed of video sequences of various indoor scenes, which are captured by the RGB and depth related cameras of Microsoft Kinect. It contains 1449 images from 464 scenes in 3 cities, and the present application selects 1049 pairs of images for training and 400 pairs for testing.

[0106] The RGB-NIR Scene data set is composed of 4477 images of 9 categories, which are captured in RGB and near infrared (NIR). The scenes include: countryside, field, forest, indoor, mountain, ancient building, street, city and water. The present application selects 400 pairs of images for training and 48 pairs of images for testing.

[0107] (2) Evaluation index: In each test sequence, a reference image is paired with a source image. This example uses Pydegensac to perform the calculation of homography estimation test, and further uses DEGENSAC as robust estimation. The accuracy under the area of the cumulative curve where the angular error reaches the threshold of 3, 5 and 10 pixels is reported respectively.

[0108] Table 1 Homography estimation on Optical-SAR dataset

[0109] Method @3px @5px @10px MatchosNet 20.87 42.36 65.38 HardNet 24.19 45.27 67.89 Tfeat 12.74 31.06 56.39 MatchNet 3.78 14.24 38.04 FeMIT 28.89 49.40 70.07 The invention 29.94 50.85 71.70

[0110] Table 2 Homography estimation on NYU-Depth V2 dataset

[0111] Method @3px @5px @10px MatchosNet 39.55 55.90 74.63 HardNet 39.03 54.59 73.38 Tfeat 38.01 53.13 71.64 MatchNet 29.51 46.00 65.14 FeMIT 37.56 58.74 78.09 The invention 38.48 59.43 79.07

[0112] Table 3 Homography estimation on RGB-NIR Scene dataset

[0113] Method @3px @5px @10px MatchosNet 12.54 28.80 52.09 HardNet 18.35 39.02 63.37 Tfeat 0.42 2.39 12.33 MatchNet 0.61 3.51 15.35 FeMIT 69.08 80.36 89.72 The invention 38.48 59.43 79.07

[0114] (3) Result analysis: As shown in Table 1, for homography estimation under three pixel thresholds, the present application is superior to other methods on the Optical-SAR dataset; as shown in Table 2, on the NYU-Depth V2 dataset, the present application is superior to other methods for homography estimation under two pixel thresholds of 5 and 10; as shown in Table 3, on the RGB-NIR Scene dataset, the present application is superior to other methods for homography estimation under three pixel thresholds. It is proved that the present application has effectiveness, robustness and generalization for feature matching under multi-modal image dataset.

[0115] It can be understood that, in the description of the present specification, the description of the terms "an embodiment", "another embodiment", "other embodiments", or "first embodiment to Nth embodiment" and the like means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0116] It should be noted that, in this document, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0117] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0118] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and the necessary general hardware platform, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) as described above, and includes a number of instructions for making a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in the various embodiments of the present application.

[0119] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made by using the content of the specification and drawings of the present application, is also included in the patent protection scope of the present application.

Claims

1.A multi-modal image feature matching method based on spatial and channel two-dimensional attention, characterized in that, The method comprises the following steps: S1: using an improved FPN architecture to preliminarily extract features of the image; The improved FPN architecture specifically comprises four BasicBlocks, four UpConv blocks, four Connect blocks and four 3*3 convolutional layers for dimension reduction; The BasicBlock is composed of two 3*3 convolutional layers, two BatchNorm normalization functions and one ReLU activation; the UpConv block is composed of a bilinear interpolation function, a 3*3 convolutional layer and a BatchNorm function; and the Connect block is composed of a Concat function; S2: using spatial attention to interact features in the spatial dimension; S3: using position encoding to consolidate image position information; S4: performing attention interaction in the channel dimension of the image to complete the final feature extraction; S5: using bidirectional softmax to process the features after attention interaction, training the model and realizing feature matching under the multi-modal image. 2.The method of claim 1, wherein, In the step S2, the expression of the step of using spatial attention to interact features in the spatial dimension is specifically as follows: ； ； ； ； ； wherein the matrix is a learnable matrix according to the input to generate a vector , is a scaling factor, is a feature map after spatial information interaction, is a feature map. 3.The method of claim 1, wherein, In the step S3, the position encoding adopts the convolution-based position encoding composed of a 3x3 convolution, a activation function. 4.The method of claim 3, wherein, The expression of the position encoding is specifically as follows: ； wherein denotes a 3 x 3 convolution, denotes the feature map with added positional information. 5.The method of claim 1, wherein, The step S4 specifically comprises: Based on and three learnable matrices , obtain the query vector , the key vector , and the value vector ; transpose ; multiply the transpose of the key vector and the value vector , and perform softmax normalization on the obtained result to obtain the attention score; Will transpose, the result of the transpose is multiplied by the attention score, and the result of the multiplication will be transposed; after the projection layer, the output after the channel information interaction is obtained ; passing one , one , one , and the initial residual, to obtain the final output . 6.The method of claim 1, wherein, The expression of the step S4 is specifically as follows: ； ； ； ； ； ； wherein, denotes matrix multiplication, and attn denotes an attention score, is a layer, the input and output dimensions are the same. 7.The method of claim 1, wherein, In the step S5, the expression of using bidirectional softmax to process the features after attention interaction is specifically as follows: ； wherein, denotes a normalization operation, denotes an operation on all elements of the row, denotes an operation on all elements of the column, denotes an operation on all elements of the column, multiplying both results to obtain the final matching probability matrix . 8.The method of claim 1, wherein, In the step S5, the expression of training the model is specifically as follows: ； wherein, represents the total number of matched samples, represents the sum over k samples, represents the label, represents the probability of a correct match.

Images