Image feature matching model considering geometric prior information attention calculation

Abstract

The present application belongs to the field of computer vision and remote sensing image processing, and relates to a method for constructing an image feature matching model considering attention calculation of geometric prior information and the model, which comprises the following steps: 1) constructing a data set; 2) constructing an image feature matching model in the mode of attention calculation of geometric prior information; 3) constructing a loss function of the image feature matching model, training the image feature matching model by using the constructed data set, and obtaining the weight of the image feature matching model. The present application provides a method for constructing an image feature matching model considering attention calculation of geometric prior information, which can gradually improve the matching precision and robustness in the scene of complex geometric deformation and large visual angle change, and can cope with diversified matching scenes.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and image processing, and relates to a method for constructing an image feature matching model and the model, in particular to a method for constructing an image feature matching model considering attention calculation of geometric prior information and the model. BACKGROUND

[0002] Image matching is one of the core tasks in the field of computer vision, which is widely used in three-dimensional reconstruction, visual positioning, and simultaneous localization and mapping of various geometric vision tasks. These tasks need to establish reliable feature correspondence between image pairs to complete the recovery of spatial structure, pose estimation, etc. However, due to various challenges existing in real scenes, such as large changes in viewing angle, different lighting conditions, sparse texture regions, repetitive texture patterns, and motion blur, the accuracy of image matching is often significantly limited. Therefore, how to achieve high robustness and high precision of image matching in complex scenes has become a key problem in current research.

[0003] Traditional image matching methods usually adopt a detector-based method, including three stages of feature point detection, feature description, and feature matching. In this mode, the algorithm first extracts sparse key points in the image through the detector, then characterizes the key points through the descriptor, and finally finds the optimal match in the feature space. However, the performance of this kind of method is highly dependent on the quality of key point detection and description. When facing scenes with scarce texture or image pairs with large geometric deformation and lighting interference, the key point detector often fails to extract enough interest points with strong repeatability, which in turn leads to matching failure.

[0004] To address the limitations of traditional methods, a series of detector-free matching methods have emerged in recent years. These methods generate global and local feature correspondences directly from the original image, avoiding the dependence on key point detection. This not only expands the application scenarios of matching, but also significantly improves the adaptability to extreme scenarios. For example, the matching method based on Transformer can achieve more robust feature alignment through long-distance dependency modeling and local detail attention, especially in areas with sparse or repetitive patterns. As a representative research, LoFTR updates cross-view features through self-attention blocks and cross-attention blocks, and uses linear transformers instead of global attention to achieve more controllable computational cost while achieving very good matching results. Although LoFTR performs well in practical applications, its main problem is the lack of detailed local interaction between pixel labels, which may limit its ability to extract highly accurate and well-positioned correspondences. New research results reveal that the cross-attention map of the linear transformer generated by LoFTR tends to spread between larger areas, rather than effectively focusing on the actual corresponding areas. And in the actual image matching process, not all areas of the image to be matched are overlapping and matchable. In fact, only a small part of the area has an effective matching relationship, while the vast majority of the area is irrelevant or unrelated. This means that when performing feature matching, more attention should be paid to areas with potential matching relationships, rather than blindly comparing the entire image. By identifying and filtering out irrelevant areas, the accuracy and efficiency of matching can be significantly improved, resulting in more accurate feature matching and better application results. SUMMARY

[0005] To solve the above technical problems in the background art, the present application provides a method for constructing an image feature matching model that can gradually improve the accuracy and robustness of matching in complex geometric deformation and large view angle change scenarios, and can cope with diversified matching scenarios considering geometric prior information attention calculation.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] A method for constructing an image feature matching model considering geometric prior information attention calculation, characterized in that the method comprises the following steps:

[0008] 1) Construct a data set;

[0009] 2) Construct an image feature matching model using geometric prior information attention;

[0010] 3) constructing a loss function of the image feature matching model, training the image feature matching model by using the data set constructed in step 1), and obtaining the image feature matching model weight.

[0011] Preferably, step 1) is specifically: selecting images of multiple scenes in the Megadepth data set as a training data set for training real landmark information; each scene includes at least 100 pairs of images.

[0012] Preferably, the specific implementation mode of step 2) is:

[0013] 2.1) performing local image feature extraction on the images in the data set obtained in step 1);

[0014] 2.2) performing coarse feature matching on the result obtained in step 2.1) to obtain a coarse feature matching result;

[0015] 2.3) performing medium-granularity feature matching on the coarse feature matching result obtained in step 2.2) to obtain a medium-granularity feature matching result;

[0016] 2.4) performing fine feature matching on the medium-granularity feature matching result obtained in step 2.3) to obtain a fine feature matching result;

[0017] 2.5) constructing an architecture of the image feature matching model according to the coarse feature matching, the medium-granularity feature matching, and the fine feature matching.

[0018] Preferably, step 2.1) is to extract multi-scale local image features based on a ResNet backbone network and combined with a feature pyramid network in the data set obtained in step 1); preferably, the specific implementation mode of step 2.1) is:

[0019] 2.1.1) extracting features from the images in the data set obtained in step 1) through four layers of the ResNet backbone network to obtain intermediate feature maps of 1 / 2 scale, 1 / 4 scale, 1 / 8 scale, and 1 / 16 scale, respectively;

[0020] 2.1.2) performing fusion processing on the intermediate feature maps by using the feature pyramid network, starting from the deepest 1 / 16 scale intermediate feature map, adjusting the channel number by using 1x1 convolution, and then performing up-sampling on the 1 / 16 scale intermediate feature map by using a bilinear interpolation method to obtain the same spatial resolution as the 1 / 8 scale intermediate feature map, adding the feature map obtained by up-sampling the 1 / 16 scale intermediate feature map to the 1 / 8 scale intermediate feature map to form a 1 / 8 scale medium-granularity feature map;

[0021] 2.1.3) Further extract features from the medium-grained feature map of 1 / 8 scale obtained in step 2.1.2) through a 3x3 convolution, and then use Batch Normalization and LeakyReLU activation function to enhance the nonlinear expression ability, adjust the output channel number through 1x1 convolution, upsample to 1 / 4 scale using bilinear interpolation, and add the 1 / 4 scale intermediate feature map to form a 1 / 4 scale feature map;

[0022] 2.1.4) Repeat step 2.1.3) based on the 1 / 4 scale feature map obtained in step 2.1.3) to obtain a fine-grained feature map of 1 / 2 scale;

[0023] 2.1.5) The intermediate feature map of 1 / 16 scale obtained in step 2.1.1) is directly taken as a coarse-grained feature map of 1 / 16 scale; the coarse-grained feature map of 1 / 16 scale, the medium-grained feature map of 1 / 8 scale obtained in step 2.1.2), and the fine-grained feature map of 1 / 2 scale obtained in step 2.1.4) are taken as the results of local image feature extraction, and the coarse-grained feature map of 1 / 16 scale, the medium-grained feature map of 1 / 8 scale, and the fine-grained feature map of 1 / 2 scale are denoted as and、

[0024] Preferably, the specific implementation of step 2.2) is as follows:

[0025] 2.2.1) Convert the coarse-grained feature map of 1 / 16 scale obtained in step 2.1.5) to obtain a converted coarse-grained feature; preferably, the specific implementation of step 2.2.1) is as follows:

[0026] First, position encoding is performed on the coarse-grained feature map of 1 / 16 scale obtained in step 2.1.5):

[0027]

[0028] The position encoding and the coarse-grained feature map of 1 / 16 scale are subjected to vector addition operation to obtain a position-encoded coarse-grained feature; then linear self-attention and cross-attention calculation are performed on the position-encoded coarse-grained feature, and the calculation method of the linear self-attention and cross-attention is as follows:

[0029]

[0030] Attention(Q, K, V) = D -1 ·(φ(Q)·(φ(K T V))

[0031] Wherein:

[0032] R is the numerical type of the tensor;

[0033] K is the key in the attention mechanism;

[0034] V is the value matrix;

[0035] n and dk represent the dimensions of Q and K after being processed by the function φ(x) respectively;

[0036] T is the transpose of the matrix;

[0037] D is the normalization factor;

[0038] φ(x) is a kernel function that converts Q and K into a non-negative mapping, and is specifically defined as:

[0039] φ(x) = elu(x) + 1

[0040] After calculating the attention, the calculated attention is multiplied element-wise with the 1 / 16 scale rough feature map to obtain the converted rough feature and The and are the converted features of and respectively;

[0041] 2.2.2) using a double softmax method to perform rough feature matching on the converted rough features and obtained in step 2.2.1), to obtain a rough feature matching result; preferably, the specific implementation of the step 2.2.2) is:

[0042] First, flatten the converted rough features and feature maps to obtain feature vectors feat0 and feat1, and further perform L2 norm normalization processing, and the expressions of the feature vectors feat0 and feat1 are:

[0043] feat0 = Normalize(feat0);

[0044] feat1 = Normalize(feat1)

[0045] Wherein:

[0046] feat0 and feat1 are the feature vectors obtained after flattening the 1 / 16 scale rough converted feature maps;

[0047] feat0 corresponds to the flattened feature vector;

[0048] feat1 corresponds flattened eigenvectors;

[0049] Normalize refers to L2 norm normalization operation;

[0050] Secondly, the flattened feature matrix is obtained according to the normalized eigenvectors:

[0051]

[0052] wherein:

[0053] R is the numerical type of the tensor;

[0054] N0 and N1 respectively represent the length of the flattened sequence H0xW0 and H1xW1;

[0055] H0 and W0 respectively represent the height and width of the rough feature ;

[0056] H1 and W1 respectively represent the height and width of the rough feature ;

[0057] C represents the number of channels, and the number of channels of the rough feature and the rough feature is equal;

[0058] The similarity matrix S between the feature matrix F0 and F1 is:

[0059] S i，j = F0[i, : ]·F1[j, :] T

[0060] wherein:

[0061] S i，j represents the similarity between the i-th row of F0 corresponding feature and the j-th row of F1 corresponding feature;

[0062] T is the transpose of the matrix;

[0063] The softmax normalization is applied to the similarity matrix S in the row and column directions respectively, and the formula is as follows:

[0064]

[0065] wherein:

[0066] S i，j represents the similarity between the i-th row of F0 corresponding feature and the j-th row of F1 corresponding feature;

[0067] t is a temperature parameter for controlling the smoothness of softmax; ​

[0068] The softmax in the row direction and the column direction are respectively:

[0069]

[0070] The final bidirectional normalized similarity is: sim i，j = row i，j · col i，j , thus obtaining the normalized similarity matrix:

[0071]

[0072] Apply a threshold τ to the normalized similarity matrix, only keep the point pairs higher than the threshold, and eliminate the low confidence matching points:

[0073]

[0074] Finally, apply mutual nearest neighbor constraint to the retained point pairs: if the nearest neighbor of feature point i is j, and the nearest neighbor of j is also i, then (i, j) is considered as a valid match, otherwise it is considered as a false match and deleted, ensuring the uniqueness of the matching result; based on the obtained matching result, the affine transformation matrix between the images to be matched is robustly estimated using the RANSAC algorithm.

[0075] Preferably, the specific implementation of step 2.3) is:

[0076] 2.3.1) Perform medium-grained feature conversion on the medium-grained features using the converted coarse features obtained in step 2.2.1); preferably, the specific implementation of step 2.3.1) is: and are up-sampled and fused with the medium-grained features and , and then medium-grained feature conversion is performed;

[0077] Preferably, the self-attention calculation in the medium-grained scale feature conversion is completed using a linear attention method; the specific implementation of the self-attention calculation in the medium-grained feature conversion is:

[0078] a.1) Window division and affine transformation guidance:

[0079] First, the query feature map needs to be divided, which decomposes the query feature map into fixed-size query windows; each fixed-size query window is represented by its four vertex coordinates, which are the upper left corner, the upper right corner, the lower right corner and the lower left corner;

[0080] To obtain the affine transformation matrix guided key-value window vertex coordinates, a key-value window is opened at the center position of the query window, and the four vertex coordinates of the key-value window are input to the affine transformation matrix. The formula of affine transformation is as follows:

[0081] x' = A x + b

[0082] Wherein:

[0083] x represents the vertex coordinates of the query window;

[0084] A and b are the linear part and translation part of the affine transformation matrix, respectively;

[0085] x' is the coordinate of each vertex, used for locking the key-value window;

[0086] Through affine transformation, the vertex coordinates of the key-value window are mapped into the space of the key-value feature map, generating four vertices of the key-value window.

[0087] a.2) Key-value window validity check:

[0088] After the window transformation is completed, it is necessary to verify whether the key-value window is located within the valid area of the key-value feature map. For each key-value window, it is checked whether its vertex falls within the boundary of the feature map, and the number of valid vertices is counted. If the number of valid vertices meets a certain proportion, the window is considered valid. Otherwise, it is excluded. At the same time, the query window is also filtered according to the valid mask.

[0089] a.3) Feature sampling and attention calculation:

[0090] For the query window, feature extraction is directly performed from the query feature matrix. For each valid key-value window, feature sampling is performed from the key-value feature map using the bilinear interpolation method. For each valid key-value window, the extracted query feature, key feature and value feature are recombined into the form of multi-head attention, and the attention is calculated in the standard attention manner.

[0091]

[0092] Wherein:

[0093] Q, K and V are the query, key and value feature matrices, respectively;

[0094] d is a scaling factor to alleviate the gradient vanishing problem caused by large values. After the self-attention and cross-attention are calculated, the converted medium-grained features are further processed and

[0095] 2.3.2) performing medium-granularity feature matching on the result obtained in step 2.3.1) by using a double-softmax method to obtain a medium-granularity feature matching result; preferably, the specific implementation of step 2.3.2) is exactly the same as that of step 2.2.2).

[0096] Preferably, the specific implementation of step 2.4) is:

[0097] First, the medium-granularity feature matching result is used to locate and open a window of a certain size in the fine-granularity feature map, and the features in the opened window are used to calculate the heat map of the corresponding region, and the heat map is processed by the spatial expectation method to finally obtain the accurate coordinates of the matching points, thereby refining the matching result. and

[0098] Preferably, the specific implementation of step 3) is:

[0099] 3.1) constructing a total loss function of the model, and the expression of the total loss function of the model is: L = L c + L m + L f ;

[0100] Wherein:

[0101] L c , L m and L f represent the loss function of the coarse feature, the loss function of the medium-granularity feature, and the loss function of the fine-granularity feature, respectively;

[0102] Wherein: L c and L m use the negative log-likelihood loss to measure the difference between the predicted feature matching similarity and the true matching label, and the calculation formula is:

[0103]

[0104] Wherein:

[0105] y i，j represents the true label of feature i and feature j;

[0106] ρ i，j is the matching similarity of feature i and feature j, i.e. sim i，j ;

[0107] w ij is the matching degree weight, which is calculated according to the proportion of positive and negative samples;

[0108] For L f ​, the L2 loss function is used to optimize the accurate coordinates of the matching points, and the specific formula is:

[0109]

[0110] Wherein:

[0111] The accurate coordinates of the feature points predicted by the matching model are represented as:

[0112] o i The real feature point coordinates are represented as:

[0113] w i The matching accuracy weight is used to measure the uncertainty of each feature point matching, and is represented by the variance of the feature point matching heat map.

[0114] 3.2) The architecture of the image feature matching model obtained in step 2) is trained using the total loss function of the model obtained in step 3.1), and the weights of the model are iteratively updated. After the image feature matching model finally converges, the image feature matching model weight is obtained.

[0115] The image feature matching model considering geometric prior information attention calculation is constructed based on the construction method of the image feature matching model considering geometric prior information attention calculation as described above.

[0116] The application of the image feature matching model considering geometric prior information attention calculation in remote sensing image processing, automatic driving and / or three-dimensional reconstruction.

[0117] The advantages of the present application are:

[0118] The application provides an image feature matching model considering geometric prior information attention calculation, and a construction method of the model. BRIEF DESCRIPTION OF DRAWINGS

[0119] Figure 1 The overall technical framework diagram of the model provided by the application.

[0120] Figure 2 The actual effect comparison diagram of the model (upper) and the LoFTR model (lower) under 0.95 high confidence provided by the application. DETAILED DESCRIPTION

[0121] Example 1

[0122] Reference Figure 1 The application provides a construction method of an image feature matching model considering geometric prior information attention calculation, and the implementation steps are described in detail as follows.

[0123] First step, construction of a data set

[0124] The main work of this step is to establish the dataset. 368 scenes under the Megadepth dataset are selected, and 100 pairs of images under each scene are selected as training data. The information composed of the depth map and the camera pose of the images is used as the training real landmark information.

[0125] Secondly, the construction of the image feature matching model considering attention calculation of geometric prior information

[0126] The overall architecture diagram of the image feature matching model considering attention calculation of geometric prior information is shown in Figure 1 The model is mainly composed of four parts: local image feature extraction, rough feature matching, medium-granularity feature matching, and fine feature matching. Through the coarse-to-fine staged processing of extracting low, medium, and high resolution features, the final matching result is obtained. The specific implementation steps are as follows:

[0127] 2.1 Local image feature extraction (Feature Encoder)

[0128] The images in the dataset obtained in the first step are extracted using a standard convolutional neural network architecture combined with a feature pyramid network (FPN) to extract multi-scale local image features. Specifically, three different scale features are extracted from the input image, including a 1 / 16 scale coarse-granularity feature map, a 1 / 8 scale medium-granularity feature map, and a 1 / 2 scale fine-granularity feature map, respectively denoted as , This module is composed of multiple convolutional layers and basic blocks, which gradually extract image features and perform down-sampling. During processing, the features are integrated through 1x1 convolution and further feature extraction and non-linear mapping through 3x3 convolution. In order to realize feature fusion, a bilinear interpolation method is used to integrate the up-sampled features, and finally a multi-scale feature output with rich semantic information is generated.

[0129] For example, the process of local image feature extraction can specifically include:

[0130] The image in the data set obtained in the first step is gradually extracted features by four layers of ResNet backbone network, respectively obtaining 1 / 2 scale, 1 / 4 scale, 1 / 8 scale and 1 / 16 scale intermediate feature maps; the intermediate feature maps are fused by using the feature pyramid network, starting from the deepest 1 / 16 scale intermediate feature map, adjusting the channel number by 1*1 convolution, then using the bilinear interpolation method to upsample the 1 / 16 scale intermediate feature map to obtain the same spatial resolution as the 1 / 8 scale intermediate feature map, adding the feature map obtained after up-sampling the 1 / 16 scale intermediate feature map to the 1 / 8 scale intermediate feature map to form a 1 / 8 scale medium-grained feature map; the 1 / 8 scale medium-grained feature map is further extracted features by a 3*3 convolution, and the Batch Normalization and LeakyReLU activation function are used to enhance the non-linear expression ability after convolution, the output channel number is adjusted by 1*1 convolution, and the bilinear interpolation is used to upsample to 1 / 4 scale, and added to the 1 / 4 scale intermediate feature map to form a 1 / 4 scale feature map; based on the 1 / 4 scale feature map, the previous steps are repeated to finally obtain a 1 / 2 scale fine feature map; the 1 / 16 scale intermediate feature map is directly taken as a 1 / 16 scale coarse feature map; the 1 / 16 scale coarse feature map, the 1 / 8 scale medium-grained feature map and the 1 / 2 scale fine feature map are taken as the results of local image feature extraction, and the 1 / 16 scale coarse feature map, the 1 / 8 scale medium-grained feature map and the 1 / 2 scale fine feature map are respectively denoted as and,

[0131] 2.2 Coarse feature matching

[0132] The main purpose of coarse feature matching is to obtain a relatively reliable affine transformation matrix to obtain important geometric prior information between the to-be-matched image pair, and to provide guidance for subsequent medium feature matching. A feature enhancement and matching method similar to LoFTR is adopted, which mainly includes two steps: coarse feature conversion and coarse feature matching.

[0133] 2.2.1 Coarse feature conversion

[0134] Firstly, the 1 / 16 scale coarse feature map is position encoded in the following form:

[0135]

[0136] In this way, each pixel element is given unique position information, which is crucial for the ability to generate matches in visually insignificant areas. Then, linear self-attention and cross-attention calculation is performed on the position-encoded features, and the calculation formula is:

[0137]

[0138] Attention(Q, K, V) = D -1 ·(φ(Q)·(φ(K) T V))

[0139] where D is a normalization factor. φ(x) is a kernel function defined as:

[0140] φ(x) = elu(x) + 1

[0141] Q and K are transformed into non-negative mappings using φ(x):

[0142]

[0143] After calculating the attention, the calculated attention is multiplied element-wise with the 1 / 16 scale rough feature map to obtain the converted rough feature and and are the converted features of and respectively.

[0144] 2.2.2 Rough feature matching

[0145] After obtaining the attention-enhanced converted rough feature, the present application adopts a double-softmax method for feature matching.

[0146] In order to facilitate matching, the converted rough features and are first flattened to obtain feature vectors feat0 and feat1, and are normalized to L2 norm:

[0147] feat0 = Normalize(feat0), feat1 = Normalize(feat1)

[0148] The flattened feature matrix is obtained:

[0149]

[0150] where:

[0151] N0 = H0 x W0, N1 = H1 x W1. Then, the feature points of the two images are calculated by point multiplication to obtain the similarity matrix

[0152] S i，j = F0[i, :] · F1[j, :] T

[0153] S i，j represents the similarity between the ith feature and the jth feature.

[0154] To ensure the matching has bidirectionality, i.e. each feature has a normalized probability in both directions, the softmax normalization is applied to the similarity matrix S in both row and column directions, called bidirectional softmax normalization. The formula is as follows:

[0155]

[0156] where S i，j is the similarity value, t is the temperature parameter, used to control the smoothness of softmax. The softmax in row direction and column direction are:

[0157]

[0158] The final bidirectional normalized similarity is: sim i，j = row i，j · col i，j , thus the normalized similarity matrix

[0159]

[0160] Further thresholding is applied to the normalized similarity matrix, only keeping the point pairs that are higher than the threshold (0.2 in this embodiment), and removing the low confidence matching points:

[0161]

[0162] Finally, mutual nearest neighbor constraint is applied to the remaining point pairs. If the nearest neighbor of feature point i is j, and the nearest neighbor of j is also i, then (i, j) is considered as a valid match, otherwise it is considered as a false match and deleted, ensuring the uniqueness of the matching result.

[0163] Based on the obtained matching result, the RANSAC algorithm can be used to robustly estimate the affine transformation matrix between images.

[0164] 2.3 Medium feature matching

[0165] Medium feature matching mainly includes two steps: medium feature conversion and medium feature matching.

[0166] 2.3.1 Medium feature conversion

[0167] The coarse features and are upsampled and fused with the medium features and , and then the medium feature conversion is performed.

[0168] Self-attention calculation in medium-scale feature conversion is completed using a linear attention method. Cross-attention calculation in medium-scale feature conversion, the present application proposes a cross-attention method based on affine transformation window compensation, in order to improve the accuracy of cross-attention under large rotation angle, scale zoom deformation. The main steps include: window division and affine transformation guide, window effectiveness check, feature sampling and attention calculation.

[0169] 1) Window division and affine transformation guide:

[0170] In the calculation of cross-attention, the query feature map needs to be divided first, which is decomposed into fixed size (4x4 in this example) query window. Each query window is represented by its four vertex coordinates, which are the top left corner, the top right corner, the bottom right corner and the bottom left corner. These vertex coordinates define the spatial range of the query window in the feature map and are used for subsequent attention calculation.

[0171] In order to obtain the vertex coordinates of the key-value window guided by affine transformation, an 8x8 key-value window is opened at the center position of the query window, and the four vertex coordinates of the key-value window are input into the affine transformation module. The formula of affine transformation is as follows:

[0172] x' = A x + b

[0173] Where x represents the vertex coordinates of the query window, A and b are the linear part and translation part of the affine transformation matrix respectively. Through affine transformation, the vertex coordinates are mapped to the space of the key-value feature map, generating the four vertices of the key-value window.

[0174] 2) Key-value window effectiveness check:

[0175] After window transformation, it is necessary to verify whether the key-value window is located in the effective area of the key-value feature map. For each key-value window, check whether its vertex falls within the boundary of the feature map, and count the number of valid vertices. If the number of valid vertices meets a certain proportion (such as more than 50%), it is considered that the window is valid; otherwise, it is excluded. At the same time, it will also be screened according to the effective mask of the query window. The purpose of this step is to ensure that the subsequent attention calculation is carried out in the effective area.

[0176] 3) Feature sampling and attention calculation:

[0177] The query feature is directly extracted from the query feature matrix, and for each valid key-value window, the feature is sampled from the key-value feature map using the bilinear interpolation method.

[0178] Then, for each valid window, the extracted query feature, key feature and value feature are recombined into the form of multi-head attention. Then the attention is calculated in the standard attention way:

[0179]

[0180] where, are query, key, value matrix respectively. d is a scaling factor to alleviate the gradient vanishing problem caused by large values. After calculating the self-attention and cross-attention, further processing can obtain the conversion feature and

[0181] 2.3.2 Medium feature matching

[0182] The medium feature matching method adopts the same method as "2.2.2 Rough feature matching", and matches and to obtain the medium feature matching result.

[0183] 2.4 Fine feature matching:

[0184] Fine feature matching adopts the same method as LoFTR algorithm fine matching. Specifically, it first locates a certain size window (5x5 in this embodiment) in the feature map using the medium feature matching result, and calculates the corresponding region heat map combined with the feature map to represent the potential distribution of the matching points. Through the spatial expectation method, the heat map is deduced to calculate the accurate coordinates of the matching points, so as to realize the refinement of the matching result.

[0185] Step 3, image feature matching model training

[0186] The total loss function L of the constructed model is L = L c + L m + L f . L c , L m and L f represent the loss functions at the rough scale, the medium scale and the fine scale respectively. L c , L m use negative log-likelihood loss to measure the difference between the predicted matching confidence and the true matching label, and the calculation formula is:

[0187]

[0188] where, y i，j represents the true label of feature i and feature j, ρ i，j is the matching similarity sim i，j of feature i and feature j, w ij is the matching degree weight, which is calculated according to the proportion of positive and negative samples. For L f , L2 loss function is used to optimize the accurate coordinate offset of the matching point, and the specific formula is:

[0189]

[0190] wherein, represents the accurate coordinate of the matching model predicted feature point, o i represents the real feature point coordinate, w i is the matching accuracy weight, which measures the uncertainty of each feature point matching, and is expressed by the feature point matching heat map variance. In the embodiment, the Adam optimizer is used for training, the initial learning rate is set to 1x10 3 , the batch size is 1, and the model converges after 10 days of training on a 4090 graphics card. Attached Figure 2 The comparison result graph of the method of the application and the LoFTR algorithm is given, and in the case of high confidence (0.95), the number of successfully matched points of LoFTR is very small, but the method of the application can still obtain a large number of matched points, and satisfactory results can still be obtained in the case of large geometric deformation of the image.

[0191] Meanwhile, the application also provides an image feature matching model constructed based on the method as described above, which can be well applied in remote sensing image processing, automatic driving and / or three-dimensional reconstruction.

Claims

1. A method for constructing an image feature matching model that takes into account geometric prior information and attention calculation, characterized in that: The method for constructing the image feature matching model that takes into account geometric prior information and attention calculation includes the following steps: 1) Construct the dataset; 2) An image feature matching model is constructed using geometric prior information attention computation, specifically: 2.1) Perform local image feature extraction on the images in the dataset obtained in step 1), obtaining a coarse feature map at a scale of 1 / 16, a medium-grained feature map at a scale of 1 / 8, and a fine feature map at a scale of 1 / 2 as the results of local image feature extraction, and label them as follows: , , , and, ; 2.2) Perform coarse feature matching on the results obtained in step 2.1) to obtain coarse feature matching results, specifically: 2.2.1) The result obtained in step 2.1) is transformed to obtain the coarse features after transformation; the specific implementation method is as follows: First, the obtained coarse feature map at a 1 / 16 scale is positionally encoded: The location-encoded coarse feature map at a 1 / 16 scale is vector-added to obtain the location-encoded coarse feature map. Then, linear self-attention and cross-attention are calculated on the location-encoded coarse feature map. The calculation methods for linear self-attention and cross-attention are as follows: in: R is a numeric type of tensor; K is the key feature matrix in the attention mechanism; V is the eigenvalue matrix; n q and d k They represent and After function Processed dimensions; T is the transpose of the matrix; Normalization factor; For kernel functions and Transform into a nonnegative mapping, specifically defined as: After calculating the attention, the calculated attention is multiplied element-wise with a coarse feature map at a 1 / 16 scale to obtain the transformed coarse feature map. and The and They are and The transformation characteristics; 2.2.2) The coarse transformation features obtained in step 2.2.1) are processed using a double softmax method. and Perform coarse feature matching to obtain coarse feature matching results; the specific implementation method is as follows: First, convert the coarse features. and The feature map is flattened to obtain feature vectors feat0 and feat1, which are then further normalized using the L2 norm. The expressions for the feature vectors feat0 and feat1 are: in: feat0 and feat1 are the feature vectors obtained after flattening the coarse transformed feature map at a scale of 1 / 16. feat0 corresponds to Flattened feature vectors; feat1 corresponds to Flattened feature vectors; Normalize refers to the L2 norm normalization operation; Secondly, the flattened feature matrix is ​​obtained from the normalized feature vectors: in: R is a numeric type of tensor; N0 and N1 represent the lengths of the flattened sequence as H0×W0 and H1×W1, respectively. and These represent rough features. Height and width; and These represent rough features. Height and width; C represents the number of channels, the coarse feature and rough features The number of channels is equal; Feature matrix and Similarity matrix between for: in: express No. Row correspondence features and No. Similarity between corresponding features in rows; T is the transpose of the matrix; For similarity matrix Apply softmax normalization in both the row and column directions, as shown in the following formula: in: express No. Row correspondence features and No. Similarity between corresponding features in rows; Used to control the smoothness of softmax; The softmax values ​​in the row and column directions are as follows: The final bidirectional normalized similarity is: Thus, the normalized similarity matrix is ​​obtained: Apply threshold to the normalized similarity matrix Only retain point pairs with a confidence level above the threshold, and discard low-confidence matching points: Finally, apply the mutual nearest neighbor constraint to the retained point pairs: if the feature points The nearest neighbor is ,and The nearest neighbor is also Then it is believed If a match is valid, it is considered a false match and deleted to ensure the uniqueness of the matching result. Based on the obtained matching result, the RANSAC algorithm is used to robustly estimate the affine transformation matrix between the images to be matched. 2.3) Perform medium-granularity feature matching on the coarse feature matching results obtained in step 2.2) to obtain medium-granularity feature matching results, specifically: 2.3.1) Using the coarse-grained features obtained in step 2.2.1), perform medium-grained feature transformation on the medium-grained features. Specifically, this involves transforming the coarse-grained features... and Perform upsampling and compare with medium-granularity features and The features are then fused and then subjected to medium-granularity feature transformation. The self-attention calculation in the medium-granularity feature transformation is performed using a linear attention method; the specific implementation of the self-attention calculation in the medium-granularity feature transformation is as follows: a.1) Window partitioning and affine transformation guidance: First, the query feature map needs to be divided into fixed-size query windows. Each fixed-size query window is represented by the coordinates of its four vertices, namely the top left, top right, bottom right, and bottom left corners. To obtain the vertex coordinates of the key-value window guided by the affine transformation matrix, open a key-value window at the center of the query window and input the coordinates of its four vertices into the affine transformation matrix; the formula for the affine transformation is as follows: in: Represents the vertex coordinates of the query window; and These are the linear and translation parts of the affine transformation matrix, respectively. These are the coordinates of each vertex, used to lock the key-value window; Through affine transformation, the vertex coordinates of the key value window are mapped to the space of the key value feature map, generating the four vertices of the key value window; a.2) Key-value window validity check: After the window transformation is completed, it is necessary to verify whether the key-value window is located within the valid region of the key-value feature map. For each key-value window, check whether its vertices fall within the boundary of the feature map and count the number of valid vertices. If the number of valid vertices meets a certain proportion, the window is considered valid; otherwise, it is discarded. At the same time, it will also be filtered based on the valid mask of the query window. a.3) Feature sampling and attention calculation: For the query window, features are extracted directly from the query feature matrix; for each valid key-value window, features are sampled from the key-value feature map using bilinear interpolation; for each valid key-value window, the extracted query features, key features, and value features are recombined into a multi-head attention structure, and attention is calculated using standard attention methods. in: , , These are the query, key, and value matrices, respectively. The scaling factor is used; after calculating self-attention and cross-attention, further processing is performed to obtain medium-granularity features in the transformation. and ; 2.3.2) The results obtained in step 2.3.1) are subjected to medium-granularity feature matching using the double softmax method to obtain medium-granularity feature matching results; the specific implementation of step 2.3.2) is exactly the same as that of step 2.2.2). 2.4) Perform fine-grained feature matching on the medium-grained feature matching results obtained in step 2.3) to obtain fine-grained feature matching results; 2.5) Construct an image feature matching model based on coarse feature matching, medium-granularity feature matching, and fine feature matching; 3) Construct the loss function of the image feature matching model, and use the dataset obtained in step 1) to train the image feature matching model to obtain the image feature matching model weights.

2. The method for constructing an image feature matching model that considers geometric prior information and attention calculation according to claim 1, characterized in that: Step 1) specifically involves selecting images from multiple scenes in the Megadepth dataset as a training dataset for training real landmark information; each scene includes at least 100 pairs of images.

3. The method for constructing an image feature matching model that considers geometric prior information and attention calculation according to claim 2, characterized in that: Step 2.1) involves extracting multi-scale local image features from the dataset obtained in step 1) based on the ResNet backbone network and combined with the feature pyramid network; the specific implementation of step 2.1) is as follows: 2.1.1) The images in the dataset obtained in step 1) are processed stepwise through four layers of the ResNet backbone network to extract features, resulting in intermediate feature maps at scales of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 respectively. 2.1.2) The intermediate feature maps are fused using a feature pyramid network. Starting from the deepest intermediate feature map at a scale of 1 / 16, the number of channels is adjusted by 1×1 convolution. Then, the intermediate feature map at a scale of 1 / 16 is upsampled using bilinear interpolation to obtain the same spatial resolution as the intermediate feature map at a scale of 1 / 8. The feature map obtained by upsampling the intermediate feature map at a scale of 1 / 16 is added to the intermediate feature map at a scale of 1 / 8 to form a medium-grained feature map at a scale of 1 / 8. 2.1.3) The medium-grained feature map at the 1 / 8 scale obtained in step 2.1.2) is further processed by a 3×3 convolution to extract features. After convolution, Batch Normalization and LeakyReLU activation functions are used to enhance the non-linear expressive power. The number of output channels is adjusted by 1×1 convolution, and bilinear interpolation is used to upsample to the 1 / 4 scale. This is then added to the intermediate feature map at the 1 / 4 scale to form the feature map at the 1 / 4 scale. 2.1.4) Based on the 1 / 4 scale feature map obtained in step 2.1.3), repeat step 2.1.3) to obtain a 1 / 2 scale fine feature map; 2.1.5) The intermediate feature map at a scale of 1 / 16 obtained in step 2.1.1) is directly used as the coarse feature map at a scale of 1 / 16; the coarse feature map at a scale of 1 / 16, the medium-grained feature map at a scale of 1 / 8 obtained in step 2.1.2), and the fine feature map at a scale of 1 / 2 obtained in step 2.1.4) are used as the results of local image feature extraction. The coarse feature map at a scale of 1 / 16, the medium-grained feature map at a scale of 1 / 8, and the fine feature map at a scale of 1 / 2 are respectively denoted as... , , , and, .

4. The method for constructing an image feature matching model that considers geometric prior information and attention calculation according to claim 1, characterized in that: The specific implementation method of step 2.4) is as follows: First, the results of medium-granularity feature matching are used to refine the fine-grained features. and The system locates and opens a window of a certain size, and uses the features in the opened window to calculate the heat map of the corresponding area. The heat map is then processed using the spatial expectation method to obtain the precise coordinates of the matching point, thus refining the matching results.

5. The method for constructing an image feature matching model that considers geometric prior information and attention calculation according to claim 1, characterized in that: The specific implementation method of step 3) is as follows: 3.1) Construct the overall loss function of the model, the expression of which is: in: and These represent the loss functions for coarse-grained features, medium-grained features, and fine-grained features, respectively. in: The negative log-likelihood loss is used to measure the difference between the predicted feature matching similarity and the true matching label. The formula for its calculation is as follows: in: Indicate the true labels of features i and j; It is the matching similarity between feature i and feature j; It is the matching degree weight; for The L2 loss function is used to optimize the precise coordinates of the matching points. The specific formula is as follows: in: This represents the precise coordinates of the feature points predicted by the matching model. Represents the actual coordinates of the feature points; To match precision weights; 3.2) Based on the total loss function of the model obtained in step 3.1), train the image feature matching model obtained in step 2), iteratively update the weights of the model, and obtain the image feature matching model weights after the architecture of the image feature matching model finally converges.

Images