Optical and SAR image robust matching method and system

Abstract

The invention discloses an optical and SAR image robust matching method and system, and belongs to the technical field of image processing. A feature extraction module of a constructed feature matching network model is used for obtaining multi-scale pixel-by-pixel depth dense feature maps corresponding to optical and SAR images; the feature measurement learning module is used for converting a three-dimensional feature map of the multi-scale pixel-by-pixel depth dense feature map to a frequency domain, and obtaining feature similarity thermodynamic diagrams corresponding to the optical and SAR images; the image reconstruction module is used for generating a reconstructed SAR image of which the size is the same as that of the input original SAR image; training the constructed feature matching network model by constructing a joint loss function; and inputting the grey-scale maps of the optical and SAR images into the trained feature matching network model, and outputting a feature similarity thermodynamic diagram so as to match the optical and SAR images. According to the method, on the basis of the constructed feature matching network model, multi-scale feature fusion and common feature expression of optical and SAR images are realized, and the robustness of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more particularly to a robust matching method and system for optical and SAR images. Background Art

[0002] Multimodal remote sensing image registration aims to align images from different sensors, different viewpoints, or captured at different times. Accurately matching images is key to achieving this goal and forms the foundation for subsequent joint applications. Optical and Synthetic Aperture Radar (SAR) images, the two most commonly used types of remote sensing images, are widely used in image fusion, change detection, target recognition, and aircraft visual navigation.

[0003] Optical images are easy to interpret due to their rich texture and color information, but their image quality is easily affected by environmental factors such as cloud cover and lighting conditions. In contrast, SAR (SAR) effectively compensates for the shortcomings of optical imaging systems, offering all-weather observation capabilities. However, SAR images are inevitably subject to speckle noise due to their inherent imaging characteristics. Because optical and SAR images complement each other in terms of their imaging characteristics and information, their combined application can provide valuable complementary information about the imaged area.

[0004] Modern remote sensing systems utilize sophisticated orbital sensor technology to acquire remote sensing images with high geolocation accuracy. Preprocessing can eliminate large global geometric transformations such as rotation and scale changes, reducing the difference between images to a translation of just a few dozen pixels. Therefore, research on matching optical and SAR images focuses on solving the limited translation estimation problem. However, due to differences in image acquisition characteristics, significant nonlinear radiometric distortion exists between optical and SAR images of the same scene, greatly increasing the difficulty of matching.

[0005] In order to establish reliable feature descriptions or effective similarity measurements from highly heterogeneous image pairs, researchers have mainly adopted two types of methods: one is the matching method based on manual features, and the other is the matching method using deep learning.

[0006] Multimodal image matching methods based on handcrafted features can be broadly categorized into two types: feature-based and region-based. Feature-based matching methods typically involve three steps: keypoint detection, feature description, and feature matching. In heterogeneous image matching between optical and SAR images, due to significant nonlinear radiometric differences, researchers have proposed various solutions. However, given the complexity and diversity of ground object scenes, the potential of handcrafted features to further improve matching performance is limited.

[0007] With the rapid development of deep learning technology, deep learning-based matching methods have demonstrated excellent performance and have gradually become a research hotspot in the field of image matching. However, most deep learning-based matching methods rely solely on distance metrics between features to guide the network to learn common features, while ignoring the importance of multi-scale feature fusion and feature translation reconstruction. This limits the model's ability to learn high-quality, widely applicable common feature descriptions.

[0008] In summary, due to the significant geometric distortion and nonlinear radiation differences between optical and SAR images, the remote sensing image features of different types of land objects are significantly different. The existing remote sensing image matching methods and the constructed network models ignore the importance of multi-scale feature fusion and feature translation reconstruction, resulting in the limited ability of the model to extract common features, which in turn affects the universality and accuracy of the network model's description of common features. Summary of the Invention

[0009] In response to the problems existing in the above-mentioned fields, the present invention proposes a robust matching method and system for optical and SAR images, constructs a feature matching network model, and improves the matching accuracy of optical images and SAR images.

[0010] To solve the above technical problems, the present invention discloses a robust matching method for optical and SAR images, comprising the following steps: Acquire the optical image and SAR image to be matched; A feature matching network model is constructed, which includes a feature extraction module, a feature metric learning module, and an image reconstruction module. The backbone network of the feature extraction module includes two pseudo-twin network architectures that do not share network weight parameters, which are used to extract multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively. The feature metric learning module introduces fast Fourier transform on the basis of the traditional pixel-by-pixel sliding strategy to convert the three-dimensional feature map of the multi-scale pixel-by-pixel depth dense feature map into the frequency domain to obtain the feature similarity heat map corresponding to the optical image and SAR image. The image reconstruction module introduces a feature reconstruction branch and connects the convolutional layer of the reconstruction network branch to the last layer of the decoder of the pseudo-twin network architecture of the optical image to generate a reconstructed SAR image with the same size as the input original SAR image. Based on the feature similarity heat map, an adaptive sample weight loss is introduced to assign a dynamic weight to each training sample, and the loss function for the feature similarity between optical images and SAR images is determined. The mean absolute error between the original and reconstructed SAR images is generated. Based on the loss function for feature similarity and the mean absolute error, a joint loss function is constructed to train the constructed feature matching network model. The grayscale images of the optical image and the SAR image are input into the trained feature matching network model, and the feature similarity heat map is output to match the optical image and the SAR image.

[0011] Preferably, the method for extracting multi-scale pixel-by-pixel depth dense feature maps corresponding to the optical image and the SAR image respectively includes: The pseudo-twin network architecture includes two identical U-Net feature extraction network branches, and the two identical U-Net feature extraction network branches have different network weight parameters; The calculation formula of the pseudo-twin U-Net feature extraction is expressed as: ; Where, and Represents optical images and SAR images Input to different network weight parameters and In the U-Net network; and Represent the multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively.

[0012] Preferably, the method for obtaining a feature similarity heat map corresponding to the optical image and the SAR image comprises the following steps: Fast Fourier transform is introduced based on the traditional pixel-by-pixel sliding strategy to calculate the sum of squared differences. To calculate the similarity between 3D feature maps, the 3D feature map of the image is converted to the frequency domain, and the template sliding operation in the spatial domain is converted to a dot product in the frequency domain. The feature similarity heat map of the sum of squared differences calculated using the pixel-by-pixel sliding strategy is: ; Where, Represents the feature map of the SAR image, Represents the feature map of the optical image, represents a sliding window; The feature similarity heat map of the fast Fourier transform is introduced based on the calculation of the sum of squared differences by pixel-by-pixel sliding strategy: ; The optical image is used as the reference image and the SAR image is used as the template image; The feature similarity heat map of the sum of squared differences is calculated, and the minimum position of the feature similarity heat map of the sum of squared differences is the reference image position corresponding to the template image; and represent the forward and inverse Fourier transform, respectively. express The complex conjugate of .

[0013] Preferably, the determining of the loss function for the similarity of features between the optical image and the SAR image comprises the following steps: Before calculating the similarity score of the feature similarity heat map, the deep dense feature map is L2 normalized; By calculating the similarity score of the feature similarity heat map, the obtained feature similarity heat map is labeled, and the pixels in the label and their adjacent pixels are set as positive samples, and the remaining pixels are set as negative samples; The loss function for calculating the feature similarity between the optical image and the SAR image is: ; Where, and Represent the similarity corresponding to the positive and negative sample features respectively, and Represent the marginal values ​​corresponding to positive and negative samples respectively, and represent adaptive weight factor and scale factor respectively; K n represents the negative sample set, K p represents the positive sample set.

[0014] Preferably, the mean absolute error between the generated original SAR image and the reconstructed SAR image is: ; Where, w and h Represent the width and height of the original SAR image, and represent the original SAR image and the reconstructed SAR image respectively.

[0015] Preferably, the joint loss function is: According to the loss function and mean absolute error of the obtained optical image and SAR image feature similarity, the constructed joint loss function is: ; Where, and Represents the multi-scale pixel-by-pixel depth dense feature map of optical and SAR images, where the superscripts 0, 1, and 2 represent the feature maps of the original size, the width and height of the original size reduced by 1 times, and the width and height of the original size reduced by 2 times, respectively. represents different hyperparameters.

[0016] Preferably, the method for extracting multi-scale pixel-by-pixel depth dense feature maps corresponding to the optical image and the SAR image respectively further includes: Three pairs of optical images and SAR feature maps of different sizes were obtained. The dimensions corresponding to the three pairs of feature maps are W*H*C and w*h*C, 1 / 2W*1 / 2H*C and 1 / 2w*1 / 2h*C, 1 / 4W*1 / 4H*C and 1 / 4w*1 / 4h*C, where W and H represent the width and height of the optical image O, w and h represent the width and height of the SAR image S, and C represents the channel dimension.

[0017] Preferably, a robust matching system for optical and SAR images is also included, comprising: An image acquisition module, used to acquire optical images and SAR images to be matched; The feature matching network model construction module is used to construct a feature matching network model, which includes a feature extraction module, a feature metric learning module and an image reconstruction module; the backbone network of the feature extraction module includes two pseudo-twin network architectures that do not share network weight parameters, which are used to extract multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively; the feature metric learning module introduces fast Fourier transform on the basis of the traditional pixel-by-pixel sliding strategy, and converts the three-dimensional feature map of the multi-scale pixel-by-pixel depth dense feature map into the frequency domain, which is used to obtain the feature similarity heat map corresponding to the optical image and SAR image; the image reconstruction module introduces feature reconstruction analysis The convolutional layer of the reconstruction network branch is connected to the last layer of the decoder of the pseudo-twin network architecture of the optical image to generate a reconstructed SAR image with the same size as the input original SAR image; according to the feature similarity heat map, a dynamic weight is assigned to each training sample by introducing an adaptive sample weight loss to determine the loss function of the feature similarity between the optical image and the SAR image; based on the original SAR image and the reconstructed SAR image, the mean absolute error between the original SAR image and the reconstructed SAR image is generated; based on the loss function of feature similarity and the mean absolute error, a joint loss function is constructed to train the constructed feature matching network model; The image robust matching module is used to input the grayscale images of the optical image and the SAR image into the trained feature matching network model, output the feature similarity heat map, and match the optical image and the SAR image.

[0018] Preferably, a computer device is further included, the computer device comprising a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, causing the processor to perform the following steps: Acquire the optical image and SAR image to be matched; A feature matching network model is constructed, which includes a feature extraction module, a feature metric learning module, and an image reconstruction module. The backbone network of the feature extraction module includes two pseudo-twin network architectures that do not share network weight parameters, which are used to extract multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively. The feature metric learning module introduces fast Fourier transform on the basis of the traditional pixel-by-pixel sliding strategy to convert the three-dimensional feature map of the multi-scale pixel-by-pixel depth dense feature map into the frequency domain to obtain the feature similarity heat map corresponding to the optical image and SAR image. The image reconstruction module introduces a feature reconstruction branch and connects the convolutional layer of the reconstruction network branch to the last layer of the decoder of the pseudo-twin network architecture of the optical image to generate a reconstructed SAR image with the same size as the input original SAR image. Based on the feature similarity heat map, an adaptive sample weight loss is introduced to assign a dynamic weight to each training sample, and the loss function for the feature similarity between optical images and SAR images is determined. The mean absolute error between the original and reconstructed SAR images is generated. Based on the loss function for feature similarity and the mean absolute error, a joint loss function is constructed to train the constructed feature matching network model. The grayscale images of the optical image and the SAR image are input into the trained feature matching network model, and the feature similarity heat map is output to match the optical image and the SAR image.

[0019] Preferably, the present invention further comprises a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the following steps: Acquire the optical image and SAR image to be matched; A feature matching network model is constructed, which includes a feature extraction module, a feature metric learning module, and an image reconstruction module. The backbone network of the feature extraction module includes two pseudo-twin network architectures that do not share network weight parameters, which are used to extract multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively. The feature metric learning module introduces fast Fourier transform on the basis of the traditional pixel-by-pixel sliding strategy to convert the three-dimensional feature map of the multi-scale pixel-by-pixel depth dense feature map into the frequency domain to obtain the feature similarity heat map corresponding to the optical image and SAR image. The image reconstruction module introduces a feature reconstruction branch and connects the convolutional layer of the reconstruction network branch to the last layer of the decoder of the pseudo-twin network architecture of the optical image to generate a reconstructed SAR image with the same size as the input original SAR image. Based on the feature similarity heat map, an adaptive sample weight loss is introduced to assign a dynamic weight to each training sample, and the loss function for the feature similarity between optical images and SAR images is determined. The mean absolute error between the original and reconstructed SAR images is generated. Based on the loss function for feature similarity and the mean absolute error, a joint loss function is constructed to train the constructed feature matching network model. The grayscale images of the optical image and the SAR image are input into the trained feature matching network model, and the feature similarity heat map is output to match the optical image and the SAR image.

[0020] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a robust matching method for optical and SAR images. The proposed feature matching network model uses a backbone network in the feature extraction module, comprising two pseudo-twin network architectures that do not share network weight parameters. This multi-scale feature extraction architecture efficiently extracts multi-scale deep features from optical and SAR image pairs at the pixel level. In the feature metric learning module, a fast Fourier transform is applied to transform multi-scale, pixel-by-pixel, deep dense feature maps and generate feature similarity heatmaps, which help capture more subtle and global feature differences in image processing and computer vision tasks, thereby improving model performance and interpretability. In the image reconstruction module, a feature reconstruction branch is introduced during training to learn shared features between optical and SAR images. The convolutional layer of the reconstruction network branch is connected to the last layer of the decoder of the pseudo-twin network architecture for optical images. This allows the network to reconstruct the SAR image using deep features, enhancing the network's ability to learn more robust shared features. A joint loss function is constructed, which dynamically adjusts weights to accurately match the optical and SAR images, thereby achieving robust matching between the two images. Based on the constructed feature matching network model, multi-scale feature fusion and common feature expression of optical images and SAR images are realized, which improves the robustness of the model and can cope with complex and changeable ground scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of the robust matching method for optical and SAR images of the present invention; Figure 2 The robust matching network architecture for optical and SAR images of the present invention; Figure 3 Optical image and SAR image of the same area; Figure 4 Schematic diagram of the network architecture of the feature extraction module of the present invention; Figure 5 A comparative experiment of different feature matching methods set up for the experimental analysis of the present invention; Figure 6 A heat map comparing the matching results of different matching methods on various ground feature scenes set up for the experimental analysis of the present invention; Figure 7 Ablation experiment set up for the present invention. DETAILED DESCRIPTION

[0022] The following is a combination of the embodiments of the present invention Figure 1-Figure 7 , the technical solutions in the embodiments of the present invention are clearly and completely described. It should be understood that the terms used in the present invention are only used to describe specific implementation methods and are not intended to limit the present invention.

[0023] like Figure 1 As shown, the present invention proposes a robust matching method for optical images and SAR images, comprising the following steps: S1: Obtain the optical image and SAR image to be matched, such as Figure 3 As shown in Figure 1, Figure (a) and Figure (b) correspond to the optical image and SAR image of the same area. Due to the differences in image acquisition characteristics, there is significant nonlinear radiation distortion between the optical image and the SAR image of the same scene, which greatly increases the difficulty of matching.

[0024] S2: If Figure 2 As shown, a feature matching network model is constructed, which includes a feature extraction module, a feature metric learning module and an image reconstruction module; the backbone network of the feature extraction module includes two pseudo-twin network architectures that do not share network weight parameters, which are used to extract multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively; the feature metric learning module introduces fast Fourier transform on the basis of the traditional pixel-by-pixel sliding strategy, and converts the three-dimensional feature map of the multi-scale pixel-by-pixel depth dense feature map into the frequency domain, which is used to obtain the feature similarity heat map corresponding to the optical image and the SAR image; the image reconstruction module introduces a feature reconstruction branch, connects the convolution layer of the reconstruction network branch with the last layer of the decoder of the pseudo-twin network architecture of the optical image, and is used to generate a reconstructed SAR image with the same size as the input original SAR image; S3: Based on the feature similarity heat map, an adaptive sample weight loss is introduced to assign a dynamic weight to each training sample, and the loss function for the feature similarity between the optical image and the SAR image is determined. Based on the original SAR image and the reconstructed SAR image, the mean absolute error between the original SAR image and the reconstructed SAR image is generated. Based on the loss function for feature similarity and the mean absolute error, a joint loss function is constructed to train the constructed feature matching network model. S4: Input the grayscale images of the optical image and the SAR image into the trained feature matching network model, output the feature similarity heat map, and match the optical image and the SAR image.

[0025] Specifically, the backbone network of the feature extraction module consists of two pseudo-twin network architectures that do not share network weight parameters. This helps convert the highly heterogeneous data types of optical and SAR images into a unified high-dimensional space while avoiding mutual interference between the image streams. Specifically, like Figure 4 As shown, the present invention selects U-Net as the backbone network for feature extraction, and the pseudo-twin network architecture includes two identical U-Net feature extraction network branches, and the two identical U-Net feature extraction network branches have different network weight parameters; The calculation formula of the pseudo-twin U-Net feature extraction is expressed as: ; Where, and Represents optical images and SAR images Input to different network weight parameters and In the U-Net network; and Represent the feature maps corresponding to the optical image and SAR image respectively.

[0026] In addition, the commonly used remote sensing image size is 256*256. In order to adapt to the multi-scale feature extraction of the image, the last convolution module in the encoding and decoding stages is removed. That is, the network includes three downsampling and three upsampling. This improvement helps to better balance deep semantic information and shallow detail information. In order to fuse multi-level scale features, in the decoder part, unlike the common single feature layer output, the network design of the present invention allows the simultaneous output of image feature maps of multiple scales, thereby realizing a coarse-to-fine feature matching strategy. Specifically, small-scale image features reflect more abstract and essential common features of the image, while large-scale image features reflect finer texture structure features. By performing similarity measurements on image features of different scales respectively, the matching prediction position is from coarse to fine. Such a multi-scale feature fusion strategy enables the network to learn more robust multi-level image features. This improvement not only enhances the model's ability to capture features of different scales, but also improves the accuracy and robustness of the matching algorithm on complex remote sensing image data.

[0027] Inspired by generative adversarial networks, this paper introduces a feature reconstruction branch during the training process to guide the network to learn more robust common features between optical images and SAR images. Figure 2As shown in Figure 1, the feature reconstruction branch shares weight parameters with the feature extraction network and is connected to the last layer of the optical image network decoder through the convolutional layer of the reconstruction network branch to generate a reconstructed SAR image with the same size as the input original SAR image.

[0028] This feature reconstruction branch implements a translation architecture from optical images to SAR images, enabling the network to extract features that can reconstruct the original SAR image. This feature reconstruction method not only enhances the model's ability to learn features from different types of remote sensing images, but also provides an effective supervisory signal by actually reconstructing the SAR image, helping the network better capture and express the shared features between optical and SAR images. In this way, the network can learn more general feature descriptions with limited training samples, thereby improving the performance of the matching algorithm for multi-scene remote sensing data.

[0029] During training, the optical image is typically used as the reference image, and the SAR image as the template image. Since the template image's position on the reference image is known, the reconstructed SAR image can be cropped to match the template's dimensions. This allows the average error between the reconstructed and SAR images to be calculated. Using this error loss function, the deep feature representation learned by the network is optimized, thereby enhancing the model's robustness and generalization capabilities.

[0030] Extract multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images, including: Multi-scale pixel-by-pixel depth dense feature maps were obtained, including three pairs of optical images and SAR images of different sizes, with dimensions of W*H*C and w*h*C, 1 / 2W*1 / 2H*C and 1 / 2w*1 / 2h*C, 1 / 4W*1 / 4H*C and 1 / 4w*1 / 4h*C. These feature maps represent image information of different scales, where W and H represent the width and height of the optical image O, w and h represent the width and height of the SAR image S, and C represents the channel dimension.

[0031] Obtaining the feature similarity heat map corresponding to the optical image and the SAR image includes the following steps: To calculate the similarity between three-dimensional feature maps, the fast Fourier transform is introduced on the basis of the traditional pixel-by-pixel sliding strategy to calculate the sum of squared differences. By converting the three-dimensional feature map of the image into the frequency domain, the template sliding operation in the spatial domain is converted into a point product in the frequency domain, thereby achieving fast calculation.

[0032] The feature similarity heat map of the sum of squared differences calculated using the pixel-by-pixel sliding strategy is: ; Where, represents the SAR characteristic graph, Represents the optical characteristic diagram, Represents a sliding window.

[0033] The feature similarity heat map of the fast Fourier transform is introduced based on the calculation of the sum of squared differences by pixel-by-pixel sliding strategy: ; Where, The feature similarity heat map of the sum of squared differences is calculated, and the minimum position of the feature similarity heat map of the sum of squared differences is the reference image position corresponding to the template image; and represent the forward and inverse Fourier transform, respectively. express The complex conjugate of .

[0034] After calculating the similarity heatmap between the reference and template image features, the training sample mining strategy has a significant impact on network performance. Simply considering all positive and negative training samples is not optimal for distinguishing features. The triplet loss improves the discriminative power of learned features by mining hard negative samples. However, it relies on only a small amount of data, insufficient to fully describe the global geometric characteristics of the embedding space, and is prone to overfitting.

[0035] Therefore, this paper introduces an adaptive sample weight loss, assigning a dynamic weight to each training sample, so that each sample pair's similarity score has an adaptive penalty strength. For sample pairs whose feature distances are far from the optimal value, a larger weighting coefficient is used, so that these training samples are penalized with larger gradient values. Conversely, for samples that are close to the optimal value, the weighting coefficient is smaller.

[0036] After obtaining multi-scale pixel-by-pixel depth dense feature maps of optical and SAR images, weights are adaptively assigned to positive and negative samples, and the sum of squared differences accelerated by fast Fourier transform (FFT-SSD) is used to calculate the similarity score to measure the similarity between feature pairs.

[0037] Considering the pixel-level matching accuracy and the strategy of mining hard negative samples, the similarity score map obtained by FFT-SSD is ,like Figure 2 As shown in FIG, the present invention sets the pixels in the label and its adjacent pixels as positive samples, and the remaining pixels as negative samples. For similarity score maps with a smaller scale, only the label position is set as a positive sample.

[0038] Calculating feature similarity heatmap Previously, the learned deep dense features After L2 normalization, the loss function of feature similarity is calculated as: ; Where, and Represents the similarity between the positive and negative sample features. and Represent the marginal values ​​corresponding to positive and negative samples respectively, and represent adaptive weight factor and scale factor respectively; K n represents the negative sample set, K p represents the positive sample set.

[0039] The last feature layer of the optical image is connected to the reconstruction network branch, and the average error loss of the original SAR image and the reconstructed SAR image is constructed.

[0040] By calculating the mean absolute error between the generated reconstructed SAR image and the original SAR image, the average error loss function is obtained as: ; Where, and Represent the width and height of the original SAR image, and represent the original SAR image and the reconstructed SAR image respectively.

[0041] According to the obtained similarity loss function and average error loss function of multiple scales, the constructed joint loss function is: ; Where, and Represents the feature maps of optical images and SAR images, where the superscripts 0, 1, and 2 represent the feature maps of the original size, the width and height of the original size reduced by 1 times, and the width and height of the original size reduced by 2 times, respectively; represents different hyperparameters.

[0042] The feature matching network model constructs a joint loss function and learns the similarity measurement of multi-scale features under the constraint of the joint loss function, thereby enhancing the ability to extract robust feature descriptions.

[0043] The matching accuracy of optical images and SAR images is evaluated by calculating the correct matching rate. The correct matching rate refers to the percentage of matching image pairs whose distance to the true value of the label is less than a specific threshold.

[0044] The present invention also proposes a robust matching system for optical and SAR images, comprising: An image acquisition module, used to acquire optical images and SAR images to be matched; The feature matching network model construction module is used to construct a feature matching network model, which includes a feature extraction module, a feature metric learning module and an image reconstruction module; the backbone network of the feature extraction module includes two pseudo-twin network architectures that do not share network weight parameters, which are used to extract multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively; the feature metric learning module introduces fast Fourier transform on the basis of the traditional pixel-by-pixel sliding strategy, and converts the three-dimensional feature map of the multi-scale pixel-by-pixel depth dense feature map into the frequency domain, which is used to obtain the feature similarity heat map corresponding to the optical image and SAR image; the image reconstruction module introduces feature reconstruction analysis The convolutional layer of the reconstruction network branch is connected to the last layer of the decoder of the pseudo-twin network architecture of the optical image to generate a reconstructed SAR image with the same size as the input original SAR image; according to the feature similarity heat map, a dynamic weight is assigned to each training sample by introducing an adaptive sample weight loss to determine the loss function of the feature similarity between the optical image and the SAR image; based on the original SAR image and the reconstructed SAR image, the mean absolute error between the original SAR image and the reconstructed SAR image is generated; based on the loss function of feature similarity and the mean absolute error, a joint loss function is constructed to train the constructed feature matching network model; The image robust matching module is used to input the grayscale images of the optical image and the SAR image into the trained feature matching network model, output the feature similarity heat map, and match the optical image and the SAR image.

[0045] Aiming at the problem of automatic and accurate matching of optical images and SAR images, the present invention proposes a matching method based on a pseudo-twin feature extraction network framework and a U-Net backbone module. The matching method can effectively resist the significant nonlinear radiation differences between optical and SAR images and is suitable for image data containing complex scenes.

[0046] Experimental analysis In order to verify the effectiveness of the method proposed in this invention, the proposed matching method is compared with other representative optical and SAR image matching methods.

[0047] Other representative methods selected include a hand-crafted mutual information (MI) algorithm, channel features of orientated gradients (CFOG), a learning-based deep dense feature network (DDFN), an optical and SAR image matching network (OSMNet), and a siamese U-net with a fast Fourier transform (FFT) correlation layer (FFT U-Net). Accuracy is assessed using the correct match rate, which is the percentage of matching image pairs whose L2 distance to the ground truth label is less than a specified threshold T.

[0048] The experimental data platform processor set up in the present invention is an NVIDIA GeForce RTX 3070 GPU, the training batch size is set to 8, the gradient descent optimizer adopts AdamW, the initial learning rate is set to 0.001, and the weight decay is set to 5e-2.

[0049] Experimental data acquisition This experiment uses two datasets: the publicly available SEN1-2 dataset and a custom dataset of Google Earth optical images and Gaofen-3 SAR images (GE-GF3). The following is a detailed introduction to these two datasets.

[0050] The SEN1-2 dataset was originally a large-scale SAR and optical dataset acquired by the Sentinel-1 and Sentinel-2 satellites with a resolution of 10 meters. The locations of this dataset are distributed all over the world, and land cover types such as forests, deserts, and lakes account for a large proportion. In addition, SEN1-2 contains data for different seasons of spring, summer, autumn, and winter, and the overlap rate of two adjacent images (256×256 pixels) is 50%. Due to the low registration accuracy, in order to further accurately register the dataset, subsequent scholars have published a high-precision dataset with a size of 512×512. Based on this, the present invention cuts the dataset into a size of 256×256, and cuts out 5 images (4 non-overlapping images and 1 center area image) from each 512×512 pixel image, resulting in a total of 20,430 pairs of images.

[0051] The self-made GE-GF3 dataset of the present invention is made of Google Earth optical images and SAR images generated by Gaofen-3 (GF-3), with a spatial resolution of 5 meters and a shooting time of 2021-2022. The optical images are derived from multi-phase and multi-satellite data, and the imaging chromatic aberration of adjacent areas is large, which will further increase the difficulty of matching. In order to evaluate the robustness and generalization ability of the matching algorithm, the present invention collected images of 10 different areas. The longitude and latitude of the study area are widely distributed, and the land cover types are diverse, including cities, suburbs, deserts, mountains, water bodies and other types of land features. Since the geographic information of the collected original data is not fully aligned, the registration error is between a few pixels and dozens of pixels. Therefore, the present invention selects control points by artificial puncture points, and then fits the projection transformation model to accurately align the images. Finally, the images are cropped into non-overlapping pairs of 256×256 size, and a total of 30,000 pairs of images are obtained.

[0052] Both datasets are divided into 70% training set, 20% test set and 10% validation set.

[0053] During training, SAR images were used as template images, with a size of 128×128 pixels; optical images were used as reference images, with a size of 192×192 pixels. The template images were ensured to be completely contained within the reference image, and the search region was set to [-32, 32] pixels. Because the positions of the image patches were randomly selected, the network received different ground truth image inputs in each round of training. This strategy not only expanded the training dataset but also helped improve the network's generalization ability and avoid overfitting.

[0054] Setting up a comparative experiment (1) Quantitative analysis In order to comprehensively evaluate the performance of the proposed matching algorithm, this paper selected five representative optical and SAR image matching methods for comparison. Among them, the parameter settings of the learning-based methods are consistent with the original literature. The performance of all methods was evaluated on the test sets of the two datasets, and their correct matching rates within the 0-5 pixel threshold were counted, as shown in Figure 2. Figure 4 shown.

[0055] like Figure 5Figures (a) and (b) show the experimental results of six methods on the SEN1-2 and GE-GF3 datasets, respectively. First, the figures clearly show that the proposed method significantly outperforms traditional manual methods in terms of matching accuracy, demonstrating that network-learned features are far more capable of expressing common image features than hand-crafted features. Hand-crafted feature descriptions rely on rich image structural information but are difficult to adapt to image data containing complex ground objects. In particular, on the high-resolution GE-GF3 dataset, the correct matching rates of the MI and CFOG algorithms drop significantly due to the reduced amount of information carried by images of the same size. Furthermore, the performance trends of the two datasets show that the correct matching rate within three pixels of the SEN1-2 dataset is significantly better than that of the GE-GF3 dataset. This may indicate that this dataset has higher localization accuracy and a smaller error between the ground truth and the ground truth. Comparing the highest accuracy rates achieved by the matching methods, the proposed method achieves 95% on the SEN1-2 dataset and 65% on the GE-GF3 dataset. This demonstrates that higher image resolution increases the disparity between training datasets, significantly impacting the robustness of deep learning models.

[0056] While learning-based matching methods have achieved impressive matching performance in recent years, the proposed robust matching network for optical and SAR images constrained by depth feature reconstruction (OS_CDFRNet) achieves the highest correct matching rate at thresholds of 0-5 pixels. Compared to classic traditional methods such as CFOG, the proposed method achieves an average correct matching rate that is approximately 30% higher. Notably, within a threshold of 3 pixels, OS_CDFRNet significantly outperforms the second-ranked OSMNet by an average of approximately 3% in correct matching rate, demonstrating its significant advantage in pixel-level matching accuracy. Furthermore, because the training and test data cover a wide range of object types, the feature matching network model constructed in this paper demonstrates greater robustness compared to other learning methods even with limited training data, demonstrating its generalization capabilities across diverse remote sensing imagery datasets.

[0057] (2) Qualitative analysis In order to qualitatively evaluate the robustness of features learned by different methods, this paper presents the feature similarity heat maps generated by six methods, such as Figure 6As shown in the figure. From left to right, images of five different landforms, including cities, suburbs, deserts, mountains, and water bodies, are selected from the GE-GF3 data; from top to bottom, the similarity heat maps calculated by six methods are shown. Among them, the dotted box in the optical image row represents the image range selected from the reference image, and the dotted box in the SAR image row represents the image range of the template image. Since the predicted position using the SSD strategy is represented as the minimum position of the score map, the inversion operation is performed during the display for easy intuitive comparison and observation. The sizes of the reference image and template image are 192 and 128 respectively, so the size of the heat map is the search domain range of 65×65, and the true value of the label is theoretically the center point of the image. The brighter the color on the heat map, the higher the feature similarity score, and vice versa, the greater the feature difference.

[0058] The six methods demonstrated in the present invention all obtained correct matching positions on the image pairs of cities and water bodies. However, in more difficult scenes such as suburbs, deserts, and mountains, the heat maps generated by MI and CFOG are relatively smooth and scattered, and correct matching results cannot be obtained. In contrast, the method proposed in the present invention obtained correct matching results on all five groups of images. Compared with other methods, the heat map generated by the method proposed in the present invention has a smaller and sharper peak range. This shows that the method proposed in the present invention can accurately describe image features, more clearly distinguish between positive and negative samples, and can still maintain robust matching performance on image datasets with complex scenes.

[0059] Ablation experiments In order to verify the effectiveness of the feature matching network model constructed in the method proposed in this invention. Figure 7 As shown in Figure 2, four groups of experiments were conducted on the SEN1-2 data to verify the effects of the feature extraction module and feature reconstruction module introduced by the feature matching network model on the correct matching rate.

[0060] The experimental settings include: Baseline, which uses a pseudo-twin U-Net network; Option 1, adding a feature extraction module based on the Baseline; Option 2, adding a feature reconstruction module based on the Baseline.

[0061] like Figure 7 As shown in Figure 2, the ablation experiment verifies the effectiveness of the module introduced by the present invention when constructing the feature matching network model. Under the same conditions, the solution proposed by the present invention achieves the highest correct matching rate. In addition, through ablation experiments and parameter adjustment, the present invention sets the feature dimension C to 8, The optimal parameter settings are 0.6, 0.3, 0.1, and 0.5 respectively.

[0062] Experimental results show that the proposed optimization strategy effectively improves the model performance, and the matching accuracy of the method proposed in this invention is better than that of several other representative matching methods.

[0063] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

[0064] In addition, unless otherwise specified, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the art to which the present invention belongs. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of any conflict with any incorporated document, the content of this specification shall prevail.

Claims

1. A robust matching method for optical and SAR images, characterized in that: The following steps are involved: Acquire the optical image and SAR image to be matched; A feature matching network model was constructed, which includes a feature extraction module, a feature metric learning module, and an image reconstruction module. The backbone network of the feature extraction module consists of two pseudo-twin network architectures that do not share network weight parameters, which are used to extract multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively. The feature metric learning module introduces fast Fourier transform (FFT) based on the traditional pixel-by-pixel sliding strategy to convert the three-dimensional feature map of the multi-scale pixel-by-pixel depth dense feature map into the frequency domain to obtain the feature similarity heat map corresponding to the optical image and the SAR image. The image reconstruction module introduces a feature reconstruction branch and connects the convolutional layer of the reconstruction network branch to the last layer of the decoder of the pseudo-twin network architecture of the optical image to generate a reconstructed SAR image with the same size as the input original SAR image. Based on the feature similarity heat map, an adaptive sample weight loss is introduced to assign a dynamic weight to each training sample, and the loss function of the feature similarity between the optical image and the SAR image is determined. Based on the original SAR image and the reconstructed SAR image, the mean absolute error between the original SAR image and the reconstructed SAR image is generated. According to the loss function of feature similarity and mean absolute error, the constructed feature matching network model is trained by constructing a joint loss function; The grayscale images of the optical image and the SAR image are input into the trained feature matching network model, and the feature similarity heat map is output to match the optical image and the SAR image.

2. The robust matching method for optical and SAR images according to claim 1, characterized in that: The method for extracting multi-scale pixel-by-pixel depth dense feature maps corresponding to the optical image and the SAR image respectively specifically includes: The pseudo-twin network architecture includes two identical U-Net feature extraction network branches, and the two identical U-Net feature extraction network branches have different network weight parameters; The calculation formula of the pseudo-twin U-Net feature extraction is expressed as: ； Where, and Represents optical images and SAR images Input to different network weight parameters and In the U-Net network; and Represent the multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively.

3. The robust matching method for optical and SAR images according to claim 2, characterized in that: The method for obtaining a feature similarity heat map corresponding to an optical image and a SAR image includes the following steps: Fast Fourier transform is introduced based on the traditional pixel-by-pixel sliding strategy to calculate the sum of squared differences. To calculate the similarity between 3D feature maps, the 3D feature map of the image is converted to the frequency domain, and the template sliding operation in the spatial domain is converted to a dot product in the frequency domain. The feature similarity heat map of the sum of squared differences calculated using the pixel-by-pixel sliding strategy is: ； Where, Represents the feature map of the SAR image, Represents the feature map of the optical image, represents a sliding window; The feature similarity heat map of the fast Fourier transform is introduced based on the calculation of the sum of squared differences by pixel-by-pixel sliding strategy: ； The optical image is used as the reference image and the SAR image is used as the template image; The feature similarity heat map of the sum of squared differences is calculated, and the minimum position of the feature similarity heat map of the sum of squared differences is the reference image position corresponding to the template image; and represent the forward and inverse Fourier transform, respectively. express The complex conjugate of .

4. The robust matching method for optical and SAR images according to claim 3, characterized in that: The method for determining the loss function for similarity between optical image and SAR image features comprises the following steps: Before calculating the similarity score of the feature similarity heat map, the deep dense feature map is L2 normalized; By calculating the similarity score of the feature similarity heat map, the obtained feature similarity heat map is labeled, and the pixels in the label and their adjacent pixels are set as positive samples, and the remaining pixels are set as negative samples; The loss function for calculating the feature similarity between the optical image and the SAR image is: ； Where, and Represent the similarity corresponding to the positive and negative sample features respectively, and Represent the marginal values ​​corresponding to positive and negative samples respectively, and represent adaptive weight factor and scale factor respectively; K n represents the negative sample set, K p represents the positive sample set.

5. The robust matching method for optical and SAR images according to claim 4, characterized in that: The mean absolute error between the original SAR image and the reconstructed SAR image is: ； Where, w and h Represent the width and height of the original SAR image, and represent the original SAR image and the reconstructed SAR image respectively.

6. The robust matching method for optical and SAR images according to claim 5, characterized in that: The joint loss function is: According to the loss function and mean absolute error of the obtained optical image and SAR image feature similarity, the constructed joint loss function is: ； Where, and Represents the multi-scale pixel-by-pixel depth dense feature map of optical and SAR images, where the superscripts 0, 1, and 2 represent the feature maps of the original size, the width and height of the original size reduced by 1 times, and the width and height of the original size reduced by 2 times, respectively. represents different hyperparameters.

7. The robust matching method for optical and SAR images according to claim 2, characterized in that: The method for extracting multi-scale pixel-by-pixel depth dense feature maps corresponding to the optical image and the SAR image respectively specifically includes: Three pairs of optical images and SAR feature maps of different sizes were obtained. The dimensions corresponding to the three pairs of feature maps are W*H*C and w*h*C, 1 / 2W*1 / 2H*C and 1 / 2w*1 / 2h*C, 1 / 4W*1 / 4H*C and 1 / 4w*1 / 4h*C, where W and H represent the width and height of the optical image O, w and h represent the width and height of the SAR image S, and C represents the channel dimension.

8. A robust matching system for optical and SAR images, characterized in that: include: An image acquisition module, used to acquire optical images and SAR images to be matched; A feature matching network model construction module is used to construct a feature matching network model, which includes a feature extraction module, a feature metric learning module, and an image reconstruction module. The backbone network of the feature extraction module includes two pseudo-twin network architectures that do not share network weight parameters, and is used to extract multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively. The feature metric learning module introduces fast Fourier transform (FFT) based on the traditional pixel-by-pixel sliding strategy to convert the three-dimensional feature map of the multi-scale pixel-by-pixel depth dense feature map into the frequency domain to obtain the feature similarity heat map corresponding to the optical image and the SAR image. The image reconstruction module introduces a feature reconstruction branch and connects the convolutional layer of the reconstruction network branch to the last layer of the decoder of the pseudo-twin network architecture of the optical image to generate a reconstructed SAR image with the same size as the input original SAR image. Based on the feature similarity heat map, an adaptive sample weight loss is introduced to assign a dynamic weight to each training sample, and the loss function of the feature similarity between the optical image and the SAR image is determined. Based on the original SAR image and the reconstructed SAR image, the mean absolute error between the original SAR image and the reconstructed SAR image is generated. According to the loss function of feature similarity and mean absolute error, the constructed feature matching network model is trained by constructing a joint loss function; The image robust matching module is used to input the grayscale images of the optical image and the SAR image into the trained feature matching network model, output the feature similarity heat map, and match the optical image and the SAR image.

9. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps: Acquire the optical image and SAR image to be matched; A feature matching network model was constructed, which includes a feature extraction module, a feature metric learning module, and an image reconstruction module. The backbone network of the feature extraction module consists of two pseudo-twin network architectures that do not share network weight parameters, which are used to extract multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively. The feature metric learning module introduces fast Fourier transform (FFT) based on the traditional pixel-by-pixel sliding strategy to convert the three-dimensional feature map of the multi-scale pixel-by-pixel depth dense feature map into the frequency domain to obtain the feature similarity heat map corresponding to the optical image and the SAR image. The image reconstruction module introduces a feature reconstruction branch and connects the convolutional layer of the reconstruction network branch to the last layer of the decoder of the pseudo-twin network architecture of the optical image to generate a reconstructed SAR image with the same size as the input original SAR image. Based on the feature similarity heat map, an adaptive sample weight loss is introduced to assign a dynamic weight to each training sample, and the loss function of the feature similarity between the optical image and the SAR image is determined. Based on the original SAR image and the reconstructed SAR image, the mean absolute error between the original SAR image and the reconstructed SAR image is generated. According to the loss function of feature similarity and mean absolute error, the constructed feature matching network model is trained by constructing a joint loss function; The grayscale images of the optical image and the SAR image are input into the trained feature matching network model, and the feature similarity heat map is output to match the optical image and the SAR image.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor performs the following steps: Acquire the optical image and SAR image to be matched; A feature matching network model was constructed, which includes a feature extraction module, a feature metric learning module, and an image reconstruction module. The backbone network of the feature extraction module consists of two pseudo-twin network architectures that do not share network weight parameters, which are used to extract multi-scale pixel-by-pixel depth dense feature maps corresponding to optical images and SAR images respectively. The feature metric learning module introduces fast Fourier transform (FFT) based on the traditional pixel-by-pixel sliding strategy to convert the three-dimensional feature map of the multi-scale pixel-by-pixel depth dense feature map into the frequency domain to obtain the feature similarity heat map corresponding to the optical image and the SAR image. The image reconstruction module introduces a feature reconstruction branch and connects the convolutional layer of the reconstruction network branch to the last layer of the decoder of the pseudo-twin network architecture of the optical image to generate a reconstructed SAR image with the same size as the input original SAR image. Based on the feature similarity heat map, an adaptive sample weight loss is introduced to assign a dynamic weight to each training sample, and the loss function of the feature similarity between the optical image and the SAR image is determined. Based on the original SAR image and the reconstructed SAR image, the mean absolute error between the original SAR image and the reconstructed SAR image is generated. According to the loss function of feature similarity and mean absolute error, the constructed feature matching network model is trained by constructing a joint loss function; The grayscale images of the optical image and the SAR image are input into the trained feature matching network model, and the feature similarity heat map is output to match the optical image and the SAR image.

Images