Cross-modal different-source image matching method based on unsupervised domain adaptation

Abstract

The invention provides a cross-modal different-source image matching method based on unsupervised domain adaptation, and aims to solve the problem that an existing deep learning method depends on target domain annotation data in cross-modal remote sensing image matching. The conversion of a source domain image and a target domain image is realized through a style migration technology, and domain offset between the source domain image and the target domain image is reduced by using adversarial learning. According to the method, a source domain (with labels) and a target domain (without labels) are aligned from two angles of images and features, knowledge migration is realized, and a matching task in the target domain is solved. The technology fuses unsupervised domain adaptation, focuses on cross-modal different-source image matching, effectively reduces the demand of target domain annotation data, and is of great significance to practical application. Transition from labeled data to unlabeled data is promoted, high-precision target scene image matching is achieved, a new thought is provided for solving the data labeling problem, and the method is expected to be widely applied to various remote sensing image processing scenes.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a cross-modal heterogeneous image matching method based on unsupervised domain adaptation. Background Technology

[0002] With the rapid development of aerospace and remote sensing technologies, the means of acquiring remote sensing images are constantly increasing, and their types are becoming increasingly diverse. Different types of sensors differ significantly in equipment technology and imaging mechanisms, making it difficult for remote sensing images from a single data source to fully reflect the characteristics of ground features. In this context, fully utilizing multi-source remote sensing data acquired by different types of sensors to achieve information integration and complementarity is particularly important. Therefore, effective matching and registration of multi-source remote sensing images has become a key research area.

[0003] Deep learning technology has made significant progress in the field of cross-modal heterogeneous image matching, especially in matching optical images with synthetic aperture radar (SAR) images. Deep learning models can automatically extract image features and be trained on large amounts of data to achieve efficient image matching. However, most existing deep learning models are based on supervised learning frameworks and are highly dependent on labeled data, which brings many challenges in practical applications. For new target scenes (target domains), obtaining a large number of labeled samples is not only time-consuming and laborious but also requires professional knowledge and technical support. In addition, when the model is transferred from a labeled public dataset (source domain) to the target domain, the model performance often drops significantly due to the significant domain differences between the source and target domains. This domain difference is mainly reflected in the distribution of images, that is, the distribution of labeled source domain data is inconsistent with that of unlabeled target domain data. This inconsistency severely affects the model's matching performance in the target domain, making it unable to effectively identify and match image features in the target domain. Summary of the Invention

[0004] The purpose of this invention is to address the problem that existing heterogeneous remote sensing image matching methods based on supervised learning frameworks struggle to obtain a large number of training samples in the target domain. This invention proposes a cross-modal heterogeneous image matching method based on unsupervised domain adaptation, which can fully utilize source domain annotation information even when there are no labeled training samples in the target domain. This solves the problems of existing methods' strong dependence on target domain annotation data and the decline in matching performance caused by domain differences between the source and target domains.

[0005] The technical solution of this invention is: A cross-modal heterogeneous image matching method based on unsupervised domain adaptation includes the following steps: S1. Construct a source domain dataset and a target domain dataset. The source domain dataset includes paired labeled optical and SAR image pairs, and the target domain dataset includes unpaired labeled optical and SAR image pairs. S2. Using a style transfer network, the source domain dataset is transformed into an intermediate domain dataset with the style of the target domain. S3. Construct an image matching network, including a pseudo-twin network, a domain discriminator, and a template matching module. The pseudo-twin network includes a left branch and a right branch with the same structure but different network weights. The input of the left branch is the optical image of the intermediate domain dataset and the optical image of the target dataset. The input of the right branch is the SAR image of the intermediate domain dataset and the SAR image of the target domain dataset. The pseudo-twin network extracts features from the input data and outputs the corresponding deep features of the optical image and the deep features of the SAR image. The domain discriminator is used to discriminate the deep features of the image output by the left and right branches of the pseudo-Twin network. Through adversarial learning, it ensures that the generated intermediate domain image and the target domain image are consistent at the feature level, and further reduces the domain offset between the generated intermediate domain and the target domain from the perspective of feature alignment. The template matching module is used to calculate the similarity score map between the template image and the reference image and determine the matching position, wherein the template image is the optical image deep feature output by the left branch of the pseudo-twin network, and the reference image is the SAR image deep feature output by the right branch of the pseudo-twin network. S4. Train the image matching network using the acquired dataset, defining the loss function used in the training as follows: ,in The loss function for the template matching module is... Using the loss function for the domain adversarial part, we obtain the trained image matching network; S5. Input the reference image and the template image to be matched into the trained image matching network to obtain the matching position of the template image on the reference image.

[0006] Furthermore, S2 includes two style transfer networks corresponding to source and target domain style transfer for optical and SAR respectively. Both style transfer networks are cyclic consistency generative adversarial networks, each including two generators G0 and G1, and two domain discriminators d0 and d1. Generator G0 is used to convert source domain images into target domain styles, generator G1 is used to convert target domain images into source domain styles, domain discriminator d0 is used to determine whether the image generated by generator G0 has the visual style of the target domain, and domain discriminator d1 is used to determine whether the image generated by generator G1 has the visual style of the source domain. The style transfer networks are trained using the source and target domain datasets from S1, so that the generated images are visually consistent with the target domain images while retaining the original structural content of the source domain images.

[0007] Furthermore, in S3, the left branch of the pseudo-Twin network consists of a convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1, three ConvNext V2 block modules, and a hollow spatial pyramid pooling module cascaded sequentially. The right branch of the pseudo-Twin network has the same structure as the left branch. The features obtained after the optical image of the intermediate domain dataset passes through the left branch are defined as follows: The features obtained by passing the right branch through the SAR image in the intermediate domain dataset are: The features obtained by passing the left branch through the optical images of the target domain dataset are defined as follows: The features obtained by passing the right branch of the SAR image in the target domain dataset are: ; Will The input is fed into the domain discriminator, which performs domain discrimination on the input feature representation. Through adversarial learning, domain-invariant features are learned to ensure that the generated intermediate domain image and the original target domain image are consistent at the feature level.

[0008] Furthermore, in S3, the similarity score map calculated between the template image and the reference image uses the Normalized Squared Difference (SSD) as the similarity metric. , Where S is the reference image, T is the template image, M and N are the width and height of the template image, and (i,j) is the coordinate of the top left corner of the corresponding position of the template image sliding on the reference image; In the Fourier domain, the fast computation representation is as follows: , in, Let F represent the forward FFT, the inverse FFT, and the complex conjugate of F, respectively, and let v represent the search position. This indicates a sliding template window. This is shown as a reference diagram; According to the similarity score map, the matching position corresponds to the position with the smallest SSD value.

[0009] Furthermore, in S4, the stated The expression is: ,

[0010] Where corr represents the calculated SSD similarity score map, and label represents the label map that correctly matches the positions within two positive and negative pixels when randomly deselected. Circular loss; The expression is: , Where n is the number of samples in the input domain discriminator. This is the domain label for the input text, with the source domain being 1 and the target domain being 0. It is a domain discriminator for features Domain prediction probability, It is the feature representation of sample i.

[0011] The beneficial effects of this invention are as follows: (1) This invention uses unsupervised domain adaptation technology to make full use of source domain annotation information when there are no labeled training samples in the target domain, which solves the problem of strong dependence of existing methods on target domain annotation data and reduces data acquisition costs and time.

[0012] (2) The present invention extracts pixel-level deep dense features through a pixel deep dense feature extraction network, which reduces the impact of different modal features on matching accuracy and improves the robustness of matching.

[0013] (3) This invention uses CycleGAN to convert the style of the source domain image to the style of the target domain, generating an image with the same appearance style as the target domain, ensuring that the generated image is visually consistent with the target domain image, while preserving the original structural content of the source domain image. Adversarial learning is performed through a domain discriminator to ensure that the target domain style image generated from the source domain image and the target domain image maintain consistency at the pixel-level deep dense feature level, thereby improving the accuracy of image matching. Attached Figure Description

[0014] Figure 1 This is a flowchart of the method in an embodiment.

[0015] Figure 2 This is a schematic diagram of the overall framework of the heterogeneous landscape in the embodiment.

[0016] Figure 3 This is a schematic diagram of the pixel-deep dense feature representation network structure in an embodiment. Detailed Implementation

[0017] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] The method of this invention mainly includes: constructing a cycle GAN model for converting the style of optical and SAR images from a source domain dataset to the style of optical and SAR images from a target domain dataset. Two cycle GAN models are pre-trained to convert the target domain optical images into source domain style optical images and the target domain SAR images into source domain style SAR images, achieving domain adaptation at the image level. A pixel-level deep dense feature representation network is constructed for a pseudo-Twin network matching framework, and adversarial learning is performed at the pixel-level deep dense feature layer to achieve feature-level domain adaptation. Simultaneously, matching training is performed based on the target domain data and labels in the source domain style to further obtain a feature-level domain-adapted pseudo-Twin network matching model. The target domain image is acquired, and the trained pseudo-Twin network matching model is used to perform matching to obtain the target domain matching result.

[0019] Example like Figure 1 As shown, this example includes the following specific steps: S1. Collect and organize two sets of multimodal image datasets, including optical images and SAR images. Preprocess the images, such as cropping and normalization, to ensure consistency in image size and format. Divide the two datasets into a source domain dataset and a target domain dataset. The source domain dataset contains a large number of labeled image pairs, while the target domain dataset contains unlabeled image pairs.

[0020] S2. Construct a CycleGAN (Generative Adversarial Network) to convert the style of a source domain image to the style of a target domain image, generating a source-target domain style image that is consistent with the appearance style of the target domain. Two generators, G0 and G1, are used to convert the source domain image to the target domain style and vice versa. Adversarial training is performed using domain discriminators d0 and d1 to ensure that the generated images are visually consistent with the target domain image while preserving the original structural content of the source domain image.

[0021] S3. Construct a pseudo-Twin network for representing common features of heterogeneous images, used to extract cross-modal invariant features and pixel-level deep dense features. Two branch networks with identical structures but non-shared parameters are used to process optical and SAR images respectively, extracting image feature representations. Feature extraction is performed through an encoder (composed of ConvNeXtV2 blocks) to obtain modality-invariant features, reducing the impact of different modal features on matching accuracy. The original target domain image and the source domain (intermediate domain) image with the generated target domain style are input into the pseudo-Twin network output to obtain the corresponding deep image features.

[0022] S4. Adversarial learning is performed using a domain discriminator d, specifically designed to learn a domain-invariant representation of the deep image features between the generated target domain image and the original target domain image, as output from step S3. Adversarial learning ensures consistency between the source and target domain images at the level of deep image features, avoiding domain difference issues.

[0023] S5. Using Normalized Squared Difference (SSD) as a similarity metric, calculate the similarity score map between the template image and the reference image. Accelerate the calculation process using Fast Fourier Transform (FFT) to improve matching efficiency. Determine the matching position, i.e., the position with the minimum SSD value.

[0024] S6. Use the AdamW optimizer for model training, with a training period of 60 epochs and a batch size of 4. Optimize the learning rate using the Warmup strategy to ensure the model gradually converges during training.

[0025] The above steps constitute the overall execution steps for this example. The trained model is obtained through these steps. The specific execution process also includes the following technical details: The source domain data is defined as XS (optical image XS_opt, SAR image XS_sar), the target domain data is defined as XT (optical image XT_opt, SAR image XT_sar), the synthesized intermediate domain data is defined as XP (optical image XP_opt, SAR image XP_sar), the source domain label data is defined as YS, and the target domain data has no label information.

[0026] The constructed network, such as Figure 2 As shown, it includes: Two CycleGANs are constructed as the basic architecture for image alignment, enabling the conversion of different image styles without requiring a one-to-one correspondence. The purpose of CycleGANs for converting source domain image styles to target domain styles is clearly defined. A CycleGAN model for converting source domain optics to target domain optics style and a CycleGAN model for converting source domain SAR to target domain SAR style are pre-trained.

[0027] The source domain data is input into two trained CycleGAN style transfer network models, and the output is the synthetic intermediate domain data XP (optical image XP_opt, SAR image XP_sar).

[0028] A pseudo-twin network matching framework is constructed, consisting of two pixel deep dense feature representation networks, one for optical images and the other for SAR images. The two pixel deep dense feature representation networks have the same network structure, but the network weights are not shared and they are trained separately.

[0029] Among them, pixel deep dense feature representation network such as Figure 3As shown, it consists of a 3×3 kernel with a stride of 1 and padding of 1, three convolutional blocks (ConvNext V2 block), and a dilated spatial pyramid pooling module, used to extract deep, dense pixel features of the image that fuse richer and more robust multi-scale information.

[0030] The synthesized intermediate domain data XP (optical image XP_opt, SAR image XP_sar) and target data XT (optical image XT_opt, SAR image XT_sar) are input into the pseudo-Twin network matching framework, and the corresponding pixel deep dense features are output. .

[0031] Construct a domain discriminator for feature-level domain adaptation, The input is fed into the domain discriminator, d, which uses the domain discriminator to distinguish between the feature representations of the generated target domain image and the original target domain image. Through adversarial learning, domain-invariant features are learned to ensure that the generated target domain image and the original target domain image maintain consistency at the feature level. The loss function for feature alignment is then calculated. .

[0032] Similarity measurement and matching location determination. Normalized squared difference (SSD) was chosen as the similarity metric to calculate the similarity score between the template image and the reference image. Using the optical image as the reference image and the SAR image as the template image, a 192×192 feature map was randomly extracted from the pixel deep density features of the SAR image and used as the template window for subsequent matching training.

[0033] Calculate the SSD similarity score between the template image and the reference image: , Where S is the reference image and T is the template image. The Fast Fourier Transform (FFT) is used to accelerate the SSD calculation process and improve matching efficiency. In this invention, Fourier Transform is used to accelerate SSD calculation, and the original sliding matching similarity score image becomes: , Where i is the position of each element in the deep dense features of a pixel. This indicates that each search position v is in a larger range. The sliding template window on top Size and Same. Because Since it is constant relative to v, it can be ignored. Additionally, there is... , Therefore, it can be calculated quickly in the Fourier domain: , in, These represent the forward FFT, the inverse FFT, and the complex conjugate of F, respectively.

[0034] Based on the calculated similarity score map, the matching position is determined, which is the position with the minimum SSD value.

[0035] Will The input is fed into the similarity measurement module, which outputs a similarity score map and calculates the loss function for the matching portion based on the matching: , Here, corr represents the calculated SSD similarity score map, label represents the label map that correctly matches the positions within two positive and negative pixels when randomly deducted, and it is a circular loss.

[0036] Finally, by jointly optimizing all the above losses, the total loss of the model is: .

[0037] The AdamW optimizer was chosen as the optimization tool for model training, with a training period of 60 epochs and a batch size of 4 to ensure the model could complete training within a reasonable time. During training, a warmup strategy was used to optimize the learning rate, allowing the model to quickly adapt to the training data in the early stages and then gradually converge, ultimately obtaining model parameters with good performance. Subsequently, the reference image and the template image to be matched were input into a pseudo-Siamese network matching framework. By calculating the SSD similarity score map of the pixel deep features output by both models, the precise matching position on the reference image corresponding to the template was obtained by finding the minimum value position in the SSD similarity score map.

Claims

1. A cross-modal heterogeneous image matching method based on unsupervised domain adaptation, characterized in that: The following steps are involved: S1. Constructing a source domain dataset and a target domain dataset, wherein the source domain dataset includes paired annotated optical and SAR image pairs, and the target domain dataset includes unpaired annotated optical and SAR image pairs; S2, through the style transfer network, the source domain dataset is converted into an intermediate domain dataset in the style of the target domain; S3. Construct an image matching network, including a pseudo-twin network, a domain discriminator, and a template matching module, wherein the pseudo-twin network includes a left branch and a right branch with the same structure but unshared network weights, wherein the input of the left branch is the optical image of the intermediate domain dataset and the optical image of the target dataset, and the input of the right branch is the SAR image of the intermediate domain dataset and the SAR image of the target domain dataset, and the pseudo-twin network extracts features from the input data, and outputs the corresponding deep features of the optical image and the deep features of the SAR image; The domain discriminator is used to discriminate the deep features of the images output by the left and right branches of the pseudo twin network, and ensures the consistency of the generated intermediate domain image and the target domain image at the feature level through adversarial learning, so as to further reduce the domain offset between the generated intermediate domain and the target domain from the perspective of feature alignment; The template matching module is used to calculate the similarity score map between the template image and the reference image, and determine the matching position, wherein the template image is the deep feature of the optical image output by the left branch of the pseudo-twin network, and the reference image is the deep feature of the SAR image output by the right branch of the pseudo-twin network; S4. Use the acquired data set to train the image matching network, and set the loss function used in the training as ,in is the loss function of the template matching module, is the loss function of the domain adversarial part, and the trained image matching network is obtained; S5. Input the reference image and the template image to be matched into the trained image matching network, so as to obtain the matching position of the template image corresponding to the reference image.

2. According to the method of claim 1, the cross-modal heterogeneous image matching method based on unsupervised domain adaptation is characterized in that: S2 includes two style transfer networks corresponding to the source domain style and target domain style conversion of optical and SAR respectively. The two style transfer networks are both cycle-consistent generative adversarial networks, and both include two generators G0 and G1, and two domain discriminators d0 and d1. Generator G0 is used to convert the source domain image into the target domain style, generator G1 is used to convert the target domain image into the source domain style, domain discriminator d0 is used to judge whether the image generated by generator G0 has the visual style of the target domain, and domain discriminator d1 is used to judge whether the image generated by generator G1 has the visual style of the source domain; The style transfer network is trained using the source and target domain datasets of S1, so that the generated images are visually consistent with the target domain images while retaining the original structural content of the source domain images.

3. According to the cross-modal heterogeneous image matching method based on unsupervised domain adaptation of claim 1, it is characterized in that: In S3, the left branch of the pseudo-twin network is composed of a convolutional layer with a kernel size of 3×3, a step size of 1, and a padding of 1, three ConvNext V2 block modules, and a dilated spatial pyramid pooling module, which are cascaded in sequence. The right branch of the pseudo-twin network has the same structure as the left branch. The features of the optical image of the intermediate domain dataset after passing through the left branch are defined as , the features of the SAR image of the middle domain dataset after passing through the right branch are , define the characteristics of the target domain dataset optical image after passing through the left branch as , the features of the target domain dataset SAR image after the right branch are: ; Will The input is sent to the domain discriminator, which uses the domain discriminator to perform domain discrimination on the input feature representation. Through adversarial learning, domain invariant features are learned to ensure that the generated intermediate domain image and the original target domain image remain consistent at the feature level.

4. According to the cross-modal heterogeneous image matching method based on unsupervised domain adaptation according to claim 1, it is characterized in that: In S3, the similarity score map between the template image and the reference image is calculated by selecting the normalized square difference SSD as the similarity measurement indicator: ， Where S is the reference image, T is the template image, M and N are the width and height of the template image, and (i, j) is the upper left corner coordinate of the corresponding position of the template image sliding on the reference image; The fast calculation in Fourier domain is expressed as: ， in, They represent forward FFT, inverse FFT and the complex conjugate of F, respectively, v represents the search position, Represents a sliding template window, Represented as a reference figure; According to the similarity score graph, the matching position corresponds to the position with the smallest SSD value.

5. According to claim 4, a cross-modal heterogeneous image matching method based on unsupervised domain adaptation is characterized in that: In S4, The expression is: ， Among them, corr represents the calculated SSD similarity score map, label represents the label map of the correct matching position within the positive and negative pixels when randomly deducted, is a circular loss; The expression is: ， Where n is the number of samples input to the domain discriminator, is the domain label of the input, the source domain is 1, and the target domain is 0. is the domain discriminator for the feature The predicted probability of the domain, is the feature representation of sample i.

Images