Multi-temporal Remote Sensing Image Registration Method Based on Cross Attention and Deformable Convolution

Abstract

The present invention relates to a multi-temporal remote sensing image registration method based on cross-attention and deformable convolution. First, based on the backbone network VGG16, deformable convolution is introduced to construct a deformable convolution block DeVgg to adapt to remote sensing images with obvious geometric deformations such as different shapes and scales, enhancing the robustness of the algorithm. Secondly, the features extracted by the backbone network are input into the cross-attention module to better capture the spatial correspondence of images under different background changes. While retaining the core information of the feature map, it can efficiently identify the feature information that needs attention, thereby improving the overall matching accuracy. In the feature matching stage, to solve the problem of poor feature point matching quality, the brute-force matching BFMatcher algorithm is used for rough registration, and then combined with an adaptive constraint threshold to screen out high-quality matching points.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and particularly to a multi-temporal remote sensing image registration method based on cross-attention and deformable convolution. Background Art

[0002] Traditional image registration methods can be divided into two categories: (1) feature-based methods; (2) region-based methods. For feature-based methods, they are generally divided into three separate stages: feature detection, feature description, and feature matching. In the feature detection stage, significant points such as corner points need to be detected from each image as interest points, and then local descriptors are extracted around these corner points. The corresponding relationship between the two can be found through nearest neighbor search or more complex matching algorithms to complete registration. Scale-invariant feature transform is a popular and widely used feature-based registration method, but it has a long matching time and is prone to detail loss. Based on the superiority of SIFT, many improved versions have been proposed to enhance the performance of its feature extraction, descriptor, and matching, such as RootSIFT, PCA-SIFT, RSIFT, etc. Common point feature extraction methods also include Harris operator, SURF, ORB, and FAST algorithms. However, their manually designed descriptors cannot accurately handle the impacts brought by non-linear illumination, shadows, and occlusions, and have poor matching effects on remote sensing images with complex changes. To solve the problem of difficult matching of images with complex background changes in registration, some scholars have proposed a Siamese convolutional network based on feature extraction.

[0003] Region-based image registration is also known as template matching, that is, finding the most similar part between the input image and the template image. Different parts in an image have different correlation region indicators. First, a template window of appropriate size is selected in the input image, and the template image is slid within the input image starting from a specified position, and the maximum correlation index is used to determine the similar region in the image to be registered. Ye et al. constructed a similarity measurement using the geometric features between images. This algorithm can effectively resist the differences brought by non-linear radiation between images, break through the limitation that traditional similarity measures rely on gray information, and can obtain a high correct matching rate. Liu designed a multi-modal registration method using a Siamese network, further improving the registration accuracy of multi-modal images.

[0004] After the rough matching of features is completed, the RANSAC algorithm is commonly used to eliminate the wrong points in registration. However, the screening of inliers depends on the self-set parameter threshold, and different types of images have different differences, making the setting of the threshold particularly important. MSAC is a modified version of RANSAC. It uses the median instead of the mean as the elimination threshold, improving the robustness of the algorithm to outliers and other noise sources in the data. MLESAC is also modified based on RANSAC. It uses maximum likelihood to estimate the parameters of the model, providing a more robust and accurate solution than RANSAC, especially when there is a large amount of noise in the processed data. To better solve this problem, this paper proposes an algorithm with an adaptive threshold, which can better screen out high-quality matching pairs.

[0005] In the aspect of image registration, the method of extracting key points and making feature descriptors by neural networks has become increasingly mature. Methods such as LoFTR, D2-net, SuperGlue, and Patch2Pix have been successively proposed. The LoFTR method uses self-attention layers and mutual information layers inspired by Transformer to obtain the feature descriptors of two images, and can produce high-quality matching results in areas with low texture, motion blur, or repeated picture patterns. Compared with the traditional method of first extracting key points and then making descriptors, the D2-net method designs a strategy to extract key points and descriptors simultaneously, making the feature module and the descriptor module highly coupled, and can extract highly robust descriptors. The SuperGlue method combines a feature detector and a matcher as a complete pipeline, merges detection and matching into a single network, and uses the attention mechanism of Transformer to adaptively enhance the global information of features, improving the matching accuracy. However, the attention mechanism also brings an increase in computational requirements. The Patch2Pix method proposes a new perspective to learn the correspondence relationship. It directly optimizes the features from the matching network without explicitly defining feature points. However, due to its principle of relying on patches, it may lead to the loss of global context information, and Patch2Pix is sensitive to the selected specific hyperparameters and requires careful adjustment. The training data used by D2-net is image pairs with pixel-level correspondence obtained from more than 300,000 depth information. It can achieve good extraction effects in images with large illumination differences and angle changes, but it also brings a greater computational load and is difficult to achieve the unity of registration accuracy and registration real-time performance.

[0006] However, the existing technologies have the following deficiencies:

[0007] 1. It is insufficient in extracting common features from remote sensing images with complex background changes

[0008] Traditional CNNs have a local receptive field, which means that they only focus on local regions of the input image when extracting features. This may lead to the inability of these networks to capture a wider range of context information when looking for global correspondences.

[0009] 2. Insufficient ability to extract features from deformed images

[0010] In multi-temporal remote sensing images, due to the influence of illumination and viewing angle differences, the same ground object may exhibit different change information. Existing convolutional neural network methods often use fixed geometric structure modules and are difficult to be effectively applied to deformed images.

[0011] 3. When using the RANSAC algorithm, a fixed threshold needs to be set manually, and it is difficult to effectively eliminate mismatched points

[0012] After the rough feature matching is completed, the RANSAC algorithm is commonly used to eliminate the wrong points in registration. However, the screening of inliers depends on the self-set parameter threshold, and different types of images have different differences, making the setting of the threshold particularly important. The threshold setting of traditional RANSAC often relies on the author's experience and is difficult to be adaptively applied to images with complex background changes. Summary of the Invention

[0013] Aiming at the deficiencies of the existing technology, the present invention proposes a multi-temporal remote sensing image registration method based on cross-attention and deformable convolution. The registration method first constructs a multi-temporal remote sensing image registration model, and the registration model includes a feature extraction module and a feature matching module. The spatial correspondence between local features and global features is strengthened through a cross-attention mechanism, and the expression ability of deformed features is improved by using a deformable convolution block. An adaptive threshold constraint is adopted in the feature extraction stage to obtain high-quality matching points. The registration method specifically includes:

[0014] Step 1: Prepare a pair of remotely sensed registration images to be registered, and the image pair includes an image to be registered and a reference image;

[0015] Step 2: Construct an end-to-end multi-temporal remote sensing registration model, obtain multi-temporal remote sensing image features through a feature extraction module, and then complete rough matching, wrong point elimination and fine matching operations through a feature matching module. Specifically:

[0016] Step 21: The feature extraction module includes a cross-attention module and a deformable convolution block. The reference image and the image to be registered are respectively input into the first three stacked deformable convolution blocks to obtain the first feature map F1 of the reference image and the second feature map F2 of the image to be registered;

[0017] Step 22: Add a cross-attention module after the third deformable convolution block. Input the first feature map F1 of the reference image and the second feature map F2 of the image to be registered into the cross-attention module for feature fusion, so as to strengthen the spatial correspondence between the global features and the local features, and output the third feature map F3 of the reference image and the fourth feature map F4 of the image to be registered after feature fusion. The operations of the cross-attention module are specifically as follows:

[0018] Step 221: When extracting the features of the reference image, the cross-attention module takes the first feature map F1 of the reference image as the main input and the second feature map F2 of the image to be registered as the cross-modal input;

[0019] Step 222: Convert the first feature map and the second feature map into feature vectors through linear transformation. Use the dot product operation to multiply the two feature vectors element by element to calculate the similarity between them, and then calculate the attention weight distribution according to the similarity calculation result. After normalization through the Softmax function, obtain the feature matrix Y. Then, perform weighted summation of the feature matrix Y and the feature vector converted from the main input according to the attention weight distribution, and finally perform feature fusion to output the third feature map F3 of the reference image;

[0020] Step 223: When extracting the features of the image to be registered, the cross-attention module takes the second feature map F2 of the image to be registered as the main input and the first feature map of the reference image as the cross-modal input. After the same operations as in Step 222, output the fourth feature map F4 of the image to be registered;

[0021] Step 23: In the fourth deformable convolution module, take the fused feature third feature map F3 obtained in Step 22 and the fourth feature map F4 of the image to be registered as inputs respectively. Through deformable convolution operations, perform iterative sampling on the input fused features, and use average pooling to adjust the size of the feature map to 1 / 4 of the input image, obtaining the final feature map F out1 of the reference image and the final feature map F out2 ;

[0022] Step 3: Take the final feature map F out1 of the reference image and the final feature map F out2 of the image to be registered as inputs. The feature matching module performs feature point screening and descriptor generation on the input feature maps. The feature matching module includes a coarse matching module and a fine matching module;

[0023] Step 31: The coarse matching module is completed by two stages: preliminary feature point screening and brute force matching algorithm. The first preliminary matching point set I1 of the reference image and the second preliminary matching point set I2 of the image to be registered and the feature descriptors are obtained by adopting the maximum screening strategy in the channel direction and local plane. The Euclidean distance between the feature descriptors of the reference image and the image to be registered is then calculated using the brute force matching algorithm to complete the coarse matching.

[0024] Step 32: The precise matching module filters and screens the first matching point set I1 and the second matching point set I2 through the adaptive threshold module to achieve precise matching, specifically:

[0025] Step 321: Calculate a set of difference average values for the first matching point set I1 and the second matching point set I2;

[0026] Step 322: Select 10 pairs of corresponding feature points from the first matching point set and the second matching point set filtered in step 31, respectively, to establish a constraint equation that can accurately describe the perspective transformation relationship between the two views, thereby constraining the parameter estimation of the homography matrix;

[0027] Step 323: Sort the obtained difference average value set in descending order, remove the first 5% and the last 10% of the data, and sum and average the remaining data to use as the inlier screening constraint threshold of the adaptive threshold module. When the feature distance of each feature point to be matched is less than the constraint threshold, it is regarded as an inlier point, otherwise it is removed;

[0028] Step 324: When the number of internal points no longer changes, update the parameter model and end the iteration;

[0029] Step 325: After eliminating the above feature point errors, the final image matching result is output.

[0030] According to a preferred embodiment, the calculation of the deformable convolution block can be divided into two stages:

[0031] The first stage is to introduce an offset on the input feature map so that the convolution kernel can be flexibly sampled near the current position;

[0032] In the second stage, bilinear interpolation is used to perform sampling based on the predicted offset, and deformable convolution is used to dynamically adjust the sampling position of the receptive field. The final output feature map has the same dimension as the input feature image.

[0033] The beneficial effects of the present invention are:

[0034] 1. The cross-attention module designed in the present invention can more effectively capture the local and global correspondence between the to-be-registered image and the reference image. The model can more effectively focus on and extract the key features in the image, while enhancing the feature representation. By realizing feature interaction and fusion, while retaining the core information of the feature map, it can efficiently identify the feature information that needs to be focused on, thereby improving the overall matching accuracy.

[0035] 2. The deformable convolutional network of the present invention can perform deformable convolution on the deformed image, and can more accurately extract the features with geometric deformation in the multi-temporal remote sensing image, enhancing the robustness of the algorithm.

[0036] 3. Aiming at the deficiencies of the fixed threshold in the existing RANSAC algorithm, the present invention designs an adaptive threshold screening method. The adaptive threshold can better adapt to the differences between different images, effectively retain high-quality matching points, and can more specifically eliminate false matches according to the image itself, enhancing the stability of the RANSAC algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic structural diagram of the multi-temporal remote sensing image registration model of the present invention;

[0038] Figure 2 is a comparison diagram of the effects of the deformable convolution and the traditional convolution of the present invention;

[0039] Figure 3 is a comparison diagram of the experimental effects on the Google Earth dataset;

[0040] Figure 4 is a comparison diagram of the experimental effects on the Hpatches dataset. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] To make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the specific embodiments and referring to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessarily confusing the concepts of the present invention.

[0042] The following is a detailed description in conjunction with the accompanying drawings.

[0043] The multi-temporal remote sensing image registration model CMRM of the present invention: CNN Multitemporal Remote Sensing Matching.

[0044] The deformable convolution block DeVgg of the present invention: Deformable VGG16.

[0045] The RANSAC of the present invention refers to the Random Sample Consensus algorithm.

[0046] Aiming at the deficiencies of the existing technologies, the present invention proposes a multi-temporal remote sensing image registration method based on cross-attention and deformable convolution. The registration method first constructs a multi-temporal remote sensing image registration model. Figure 1 It is a schematic structural diagram of the multi-temporal remote sensing image registration network CMRM of the present invention. As Figure 1 shown, the registration model includes a feature extraction module and a feature matching module, strengthens the spatial correspondence between local features and global features through the cross-attention mechanism, and improves the expression ability of deformed features by using the deformable convolution block. An adaptive threshold constraint is adopted in the feature extraction stage to obtain high-quality matching points. The registration method specifically includes:

[0047] Step 1: Prepare the remote sensing registration image pair to be registered, and the image pair includes the image to be registered and the reference image;

[0048] Step 2: Construct an end-to-end multi-temporal remote sensing registration model, obtain the multi-temporal remote sensing image features through the feature extraction module, and then complete the operations of rough matching, wrong point elimination and fine matching through the feature matching module. Specifically:

[0049] Step 21: The feature extraction module includes a cross-attention module and a deformable convolution block. The reference image and the image to be registered are respectively input into the first three stacked deformable convolution blocks to obtain the first feature map F1 of the reference image and the second feature map F2 of the image to be registered;

[0050] The backbone network is VGG16, and deformable convolution is introduced behind the pooling layers of the first, second, third and fourth layers of VGG16 respectively to construct four deformable convolution blocks DeVgg.

[0051] The deformable convolution block is specifically: Deformable convolution is an improved convolution operation. By introducing an offset, the traditional convolution kernel can perform adaptive position sampling at each sampling point.

[0052] The calculation of the deformable convolution block can be divided into two stages:

[0053] The first stage is to introduce an offset on the input feature map so that the convolution kernel can perform flexible sampling near the current position. The second stage then uses the bilinear interpolation method for sampling according to the predicted offset, dynamically adjusts the sampling position of the receptive field by using the deformable convolution, and the finally output feature map has the same dimension as the input feature image.

[0054] In multi-temporal remote sensing images, the photographed objects often have large deformation differences. Therefore, the deformable convolution block can extract more effective feature information than ordinary convolution.

[0055] Figure 2 This is a comparison diagram of the effects of deformable convolution and traditional convolution in the present invention. From left to right are the original image, the input image, and the visualization diagrams of feature extraction of the first to fifth convolutional layers, Figure 2 (a) is the effect diagram of feature extraction by deformable convolution, Figure 2 (b) is the effect diagram of feature extraction by ordinary convolution. The comparison effect with ordinary convolution is as Figure 2 shown. It can be seen from Figure 2 that deformable convolution can better reflect the basic contour features of the image and also better capture the details and local features of the target.

[0056] Step 22: Add a cross-attention module after the third deformable convolution block, and input the first feature map F1 of the reference image and the second feature map F2 of the image to be registered into the cross-attention module for feature fusion, so as to strengthen the spatial correspondence between the global features and local features, and output the third feature map F3 of the reference image and the fourth feature map F4 of the image to be registered after feature fusion;

[0057] In order to avoid the loss of feature map information and enhance the invariant expression ability of image features, change the stride of the pooling layer in the third deformable convolution block from 2 to 1, perform iterative sampling on the input feature map, and change the max pooling to average pooling. At the same time, the size of the feature map can be adjusted to 1 / 4 of the input image, and the positioning accuracy can also be doubled.

[0058] The operation of the cross-attention module is specifically as follows:

[0059] Step 221: When extracting the features of the reference image, the cross-attention module takes the first feature map F1 of the reference image as the main input and the second feature map F2 of the image to be registered as the cross-modal input;

[0060] Step 222: Convert the first feature map and the second feature map into feature vectors through linear transformation, use the dot product operation to multiply the two feature vectors element by element to calculate the similarity between them, then calculate the attention weight distribution according to the similarity calculation result, obtain the feature matrix Y after normalization through the Softmax function, and then perform weighted summation of the feature matrix Y and the feature vector converted from the main input according to the attention weight distribution, and finally perform feature fusion to output the third feature map F3 of the reference image;

[0061] Step 223: When extracting the features of the image to be registered, the cross-attention module takes the second feature map F2 of the image to be registered as the main input and the first feature map of the reference image as the cross-modal input. After the same operation as in Step 222, output the fourth feature map F4 of the image to be registered;

[0062] This can comprehensively integrate the non-local correlation between the entire main feature map and the positions of the cross-modal feature maps, and retain the original information of the main feature map at the feature positions to be calculated;

[0063] Step 23: In the fourth deformable convolution module, take the fused feature third feature map F3 obtained in Step 22 and the fourth feature map F4 of the image to be registered as inputs respectively. Through deformable convolution operations, iteratively sample the input fused features, and use average pooling to adjust the size of the feature map to 1 / 4 of the input image, obtaining the final feature map F of the reference image out1 and the final feature map F of the image to be registered out2 ;

[0064] In this invention, the fifth layer (Conv5_1 + Conv5_2 + Conv5_3 + Pool5) of VGG16 and the fully connected layer at the end of VGG16 are removed. Considering that a large number of convolution operations in the convolutional layer of the network will produce a certain number of negative output values, and after passing through the ReLU activation function, these negative values will be replaced by 0, which will lead to a large loss of feature information and thus affect the feature detector. Therefore, in the registration network proposed in this invention, the ReLU activation function in all convolutional layers is changed to the Mish activation function;

[0065] Step 3: The final feature map F of the reference image out1 and the final feature map F of the image to be registered out2 are used as inputs. The feature matching module screens feature points and creates descriptors for the input feature maps. The feature matching module includes a rough matching module and a fine matching module;

[0066] Step 31: The rough matching module is completed in two stages: preliminary screening of feature points and brute-force matching algorithm. By adopting a maximum screening strategy in the channel direction and local plane, obtain the preliminary first matching point set I1 of the reference image and the preliminary second matching point set I2 of the image to be registered and the feature descriptors, and then use the brute-force matching algorithm BFMatcher to calculate the Euclidean distance between the feature descriptors of the reference image and the image to be registered to complete the rough matching;

[0067] If directly using all pixel points in the final feature maps F out1 、F out2 as features for rough matching, the features will be too dense and not significant enough. Therefore, it is first necessary to perform feature screening by the rough matching module.

[0068] Step 32: The fine matching module filters and screens the first matching point set I1 and the second matching point set I2 through an adaptive threshold module to achieve fine matching. Specifically:

[0069] Step 321: Calculate a set of difference average values for the first matching point set I1 and the second matching point set I2; the difference average value set is used for subsequent calculation of the inlier screening constraint threshold to retain high-quality matching points.

[0070] After the rough matching, some false matches are inevitable. Therefore, it is necessary to filter and screen the first matching point set I1 and the second matching point set I2 in combination with the adaptive threshold method to achieve precise matching.

[0071] In a specific implementation, it is assumed that each feature point in the first matching point set I1 is , select the two points with the smallest Euclidean distance in the feature point set I2 and record them as and , the total number of feature points extracted from the image to be registered is n, then the average value set of their difference values G is obtained D , the calculation method is as follows:

[0072]

[0073] Step 322: Select 10 pairs of corresponding feature points from the first matching point set and the second matching point set filtered in step 31, respectively, to establish a constraint equation that can accurately describe the perspective transformation relationship between the two views, thereby constraining the parameter estimation of the homography matrix;

[0074] The homography matrix can align images at different scales and perspectives to facilitate feature matching.

[0075] Step 323: Sort the obtained difference average value set in descending order, remove the first 5% and the last 10% of the data, and sum and average the remaining data to use as the inlier screening constraint threshold of the adaptive threshold module. When the feature distance of each feature point to be matched is less than the constraint threshold, it is regarded as an inlier point, otherwise it is removed;

[0076] Step 324: When the number of internal points no longer changes, the parameter model is updated and the iteration ends. The adaptive threshold extracted in step 323 can well adapt to the differences between different images, which effectively retains high-quality matching points and enhances the stability of the RANSAC algorithm.

[0077] Step 325: After eliminating the above feature point errors, the final image matching result is output.

[0078] To further illustrate the effectiveness of the method proposed in this invention, the objective evaluation criterion of the number of correctly matched points (NCM) is used for comparison. It is widely used in other matching tasks, and the number of feature points, the root mean square error (RMSE) of the matched points, the matching time consumption (MT), and the mean matching accuracy MMA (Mean Matching Accuracy, MMA) are used as evaluation indicators. Among them, the smaller the RMSE, the higher the measurement accuracy.

[0079] This invention conducts experiments on three groups of remote sensing images and compares them with several existing registration methods. It is respectively compared with some classic image registration algorithms on the MRSI dataset, Google Earth, and HPatches dataset. The specific results are as follows. On the MRSI dataset, it is compared with the method 1 RIFT method and the method 2 CoFSM method. The experimental results are shown in Table 1.

[0080] Table 1 Comparison of the average number of correctly matched points on the MRSI dataset

[0081]

[0082] According to the analysis of Table 1, it can be seen that in the method 1 RIFT method, the average NCM of all image pairs is 412.52, and the average standard deviation is 5.294. The RIFT method has high matching accuracy and robustness, but has a high computational complexity and requires a large amount of computing resources and time. In the method 2 CoFSM method, the average NCM of all image pairs is 498.52, and the average standard deviation is 2.956. It has excellent matching effects in these six groups of experimental data. This is because the CoFSM method transforms the image registration problem into a combinatorial optimization problem, making it have high registration accuracy and computational efficiency, but it is more sensitive to interference such as illumination and noise of the input image. The average NCM of all image pairs of CMRM is 486.83, and the average standard deviation is 3.118. The method of this invention performs excellently on multi-temporal images and map optical images, but performs unsatisfactorily on depth optical and SAR optical datasets because of the huge modality differences between them.

[0083] On the Google Earth dataset, it is compared with the method 1 RIFT, the method 3 CFOG, the method 4 CMM-Net, and the method 5 DELF algorithms. The five selected methods are all excellent algorithms for local feature-based image matching. Among them, the RIFT method and the CFOG method are representative algorithms in the multi-modal registration algorithms in the past two years. The DELF method and the CMM-Net method are based on convolutional neural networks and perform excellently in registration tasks with cluttered backgrounds and partial occlusions. The results are shown in Table 2.

[0084] Table 2 Comparison of Detection Results of Different Methods on Google Earth Dataset

[0085]

[0086] By analyzing Table 2, it can be seen that in these 6 groups of experiments, the CMRM algorithm of the present invention can match a certain number of correct matching points, and the feature points extracted by the CMRM of the present invention are evenly distributed and accurately positioned. The RMSE value of CMRM is on average smaller than the other 4 registration methods, which indicates that the registration effect in this experiment is good and the information utilization rate between images is high; both the RIFT method and the DELF method can obtain more matching points, but in images with large scale transformations, the matching performance of the RIFT method is significantly reduced; the CMM-Net method can also perform well in images with drastic background changes, and the CFOG method only obtains a very small number of matches. This shows that among the above 6 matching algorithms, the CMRM algorithm has the best performance in multi-temporal registration.

[0087] Figure 3 Figure is the comparison diagram of the experimental results of the present invention on the Google Earth dataset. As Figure 3 shown, it demonstrates the feature matching ability of the registration model of the present invention. Among them, Figure 3 the first row is the matching result of the RIFT method, Figure 3 the second row is the matching result of the CFOG method, Figure 3 the third row is the matching result of the method of the present invention, Figure 3 the fourth row is the matching result of the DELF method, Figure 3 the fifth row is the matching result of the CMM-Net method. It can be intuitively seen from <X Figure 3 that the present invention can match a certain number of correct matching points, and the feature points extracted by the method of the present invention are evenly distributed and accurately positioned. It can be seen from Figure 3 that both the RIFT method and the DELF method can obtain more matching points, but in images with large scale transformations, the matching performance of the RIFT method is significantly reduced; the CMM-Net method can also perform well in images with drastic background changes, and CFOG only obtains a very small number of matches.

[0088] To prove the benefits of the deformable convolution and cross-attention of the present invention, ablation experiments were carried out on the HPatches dataset for verification, and the results are shown in Table 3 and Figure 4 as shown.

[0089] Table 3 Ablation Experiment Results

[0090]

[0091] In the HPatches dataset, the mean matching accuracy (MMA) with an error threshold of 3 - 5 is used as the evaluation metric. The mean matching accuracy (MMA) is the average of the correct matches in image pairs considering multiple pixel error thresholds.

[0092] As can be seen from Table 3, when only the cross - attention module is added, the mean matching accuracy (MMA) on the dataset is increased by 0.24, 0.248, and 0.147 respectively compared to the baseline network, effectively verifying that when the attention module is added, the network's ability to express the spatial correspondence between features of multi - temporal remote - sensing images is improved; when only the deformable convolutional block DeVgg is added, the feature extraction ability and registration ability are significantly higher than those without adding the deformable convolutional block DeVgg, which verifies the effectiveness of the deformable convolutional block DeVgg. The main reason is that during the training process, the training data has obvious background changes, making the features extracted by the network have a robust expressiveness and strengthening the feature expression ability of deformation. By combining the cross - attention and the deformable convolutional block, the mean matching accuracy (MMA) values on the public dataset are increased by 0.251, 0.254, and 0.214 respectively compared to the baseline network, verifying the effectiveness of the proposed algorithm.

[0093] Figure 4 It is a comparison graph of the ablation experiment results. Figure 4 (a) is the matching result after adding the cross - attention module and the deformable convolution. Figure 4 (b) is the matching result without adding the cross - attention module and the deformable convolution. It can be seen that in the same scene, after adding the deformable convolution module and the cross - attention module, the number of feature matching points significantly increases. The reason is that through the deformable convolutional block, the network's ability to extract deformed features is strengthened. The cross - attention module realizes the information interaction and association between the reference image and the image to be registered. The multi - temporal remote - sensing image registration method of the present invention not only performs excellently in the multi - temporal remote - sensing image registration task, but also can achieve good performance on images such as map optics and day - night images.

[0094] It should be noted that the above - mentioned specific embodiments are exemplary. Those skilled in the art can come up with various solutions inspired by the disclosed content of the present invention, and these solutions also belong to the disclosed scope of the present invention and fall within the protection scope of the present invention. Those skilled in the art should understand that the description and drawings of the present invention are illustrative and do not constitute a limitation to the claims. The protection scope of the present invention is defined by the claims and their equivalents.

Claims

1. A multi-temporal remote sensing image registration method based on cross-attention and deformable convolution, characterized by: The registration method first constructs a multi-temporal remote sensing image registration model, which includes a feature extraction module and a feature matching module. The cross-attention mechanism is used to strengthen the spatial correspondence between local features and global features, and the deformable convolution block is used to improve the expression ability of deformable features. In the feature extraction stage, an adaptive threshold constraint is used to obtain high-quality matching points. The registration method specifically includes: Step 1: Prepare a remote sensing registration image pair to be registered, wherein the image pair includes an image to be registered and a reference image; Step 2: Build an end-to-end multi-temporal remote sensing registration model. Use the feature extraction module to obtain multi-temporal remote sensing image features. Then use the feature matching module to complete the coarse matching, error point removal, and fine matching operations. Specifically: Step 21: The feature extraction module includes a cross-attention module and a deformable convolution block. The reference image and the image to be registered are input into the first three stacked deformable convolution blocks respectively to obtain the first feature map F1 of the reference image and the second feature map F2 of the image to be registered; Step 22: Add a cross-attention module after the third deformable convolution block, input the first feature map F1 of the reference image and the second feature map F2 of the image to be registered into the cross-attention module for feature fusion, thereby strengthening the spatial correspondence between global features and local features, and output the third feature map F3 of the reference image and the fourth feature map F4 of the image to be registered after feature fusion. The specific operation of the cross-attention module is as follows: Step 221: When extracting features of the reference image, the cross-attention module uses the first feature map F1 of the reference image as the main input and the second feature map F2 of the image to be registered as the cross-modal input; Step 222: Convert the first feature map and the second feature map into feature vectors through linear transformation, multiply the two feature vectors element-by-element using a dot product operation to calculate the similarity between the two, and then calculate the attention weight distribution based on the similarity calculation result. After normalization through the Softmax function, obtain the feature matrix Y, and then perform weighted summation of the feature matrix Y and the feature vector converted by the main input according to the attention weight distribution. Finally, perform feature fusion to output the third feature map F3 of the reference image; Step 223: When extracting features of the image to be registered, the cross-attention module uses the second feature map F2 of the image to be registered as the main input and the first feature map of the reference image as the cross-modal input. After the same operation as step 222, the fourth feature map F4 of the image to be registered is output. Step 23: In the fourth deformable convolution module, the third feature map F3 of the fusion feature obtained in step 22 and the fourth feature map F4 of the image to be registered are respectively used as inputs. Through the deformable convolution operation, the input fusion feature is iteratively sampled, and the average pooling is used to adjust the size of the feature map to 1 / 4 of the input image to obtain the final feature map F of the reference image. out1 And the final feature map F of the image to be registered out2 ; Step 3: Final feature map F of the reference image out1 And the final feature map F of the image to be registered out2 As input, the feature matching module screens feature points of the input feature map and creates descriptors. The feature matching module includes a coarse matching module and a fine matching module; Step 31: The coarse matching module is completed by two stages: preliminary feature point screening and brute force matching algorithm. The maximum screening strategy is adopted in the channel direction and local plane to obtain the preliminary first matching point set I1 of the reference image, the preliminary second matching point set I2 of the image to be registered, and the feature descriptor. The brute force matching algorithm is then used to calculate the Euclidean distance between the feature descriptors of the reference image and the image to be registered to complete the coarse matching. Step 32: The precise matching module filters and screens the first matching point set I1 and the second matching point set I2 through the adaptive threshold module to achieve precise matching, specifically: Step 321: Calculate a set of difference average values for the first matching point set I1 and the second matching point set I2; Step 322: Select 10 pairs of corresponding feature points from the first matching point set and the second matching point set filtered in step 31, respectively, to establish a constraint equation that can accurately describe the perspective transformation relationship between the two views, thereby constraining the parameter estimation of the homography matrix; Step 323: Sort the obtained difference average value set in descending order, remove the first 5% and the last 10% of the data, and sum and average the remaining data to use as the inlier screening constraint threshold of the adaptive threshold module. When the feature distance of each feature point to be matched is less than the constraint threshold, it is regarded as an inlier point, otherwise it is removed; Step 324: When the number of internal points no longer changes, update the parameter model and end the iteration; Step 325: After eliminating the above feature point errors, the final image matching result is output.

2. The remote sensing image registration method according to claim 1, wherein: The computation of the deformable convolution block can be divided into two stages: The first stage is to introduce an offset on the input feature map so that the convolution kernel can be flexibly sampled near the current position; In the second stage, bilinear interpolation is used to perform sampling based on the predicted offset, and deformable convolution is used to dynamically adjust the sampling position of the receptive field. The final output feature map has the same dimension as the input feature image.

Images