A cross-modal heterogeneous image matching method based on unsupervised domain adaptation

Abstract

The present invention proposes a cross-modal heterogeneous image matching method based on unsupervised domain adaptation, aiming to solve the problem of existing deep learning methods' dependence on target domain labeled data in cross-modal remote sensing image matching. The conversion of source domain and target domain images is achieved through style transfer technology, and adversarial learning is used to reduce the domain offset between the two. This method aligns the source domain (with labels) and the target domain (without labels) from the perspectives of images and features, realizes knowledge transfer, and solves the matching task in the target domain. This technology integrates unsupervised domain adaptation, focuses on cross-modal heterogeneous image matching, effectively reduces the target domain labeled data requirements, and has great significance for practical applications. The present invention promotes the transition from labeled to unlabeled data, realizes high-precision target scene image matching, provides new ideas for solving data labeling problems, and is expected to be widely used in a variety of remote sensing image processing scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a cross-modal heterogeneous image matching method based on unsupervised domain adaptation. Background Art

[0002] With the rapid development of aerospace and remote sensing technology, the means of acquiring remote sensing images are increasing and the types are becoming increasingly rich. Different types of sensors have significant differences in equipment technology and imaging mechanisms, which makes it difficult for remote sensing images from a single data source to fully reflect the characteristics of the ground objects. In this case, it is particularly important to make full use of multi-source remote sensing data obtained by different types of sensors to achieve information integration and complementarity. Therefore, effective matching and registration of multi-source remote sensing images has become the key to research.

[0003] In the field of cross-modal heterogeneous image matching, deep learning technology has made significant progress, especially in the matching task of optical images and synthetic aperture radar (SAR) images. Deep learning models can automatically extract image features and train with a large amount of data to achieve efficient image matching. However, most existing deep learning models are based on supervised learning frameworks and have a strong dependence on labeled data, which brings many challenges in practical applications. For new target scenes (target domains), obtaining a large number of labeled samples is not only time-consuming and labor-intensive, but also requires professional knowledge and technical support. In addition, when the model is migrated from an annotated public dataset (source domain) to the target domain, the model performance often drops significantly due to the significant domain difference between the source domain and the target domain. This domain difference is mainly reflected in the distribution of images, that is, the distribution between the annotated source domain data and the unlabeled target domain data is inconsistent. This inconsistency causes the matching performance of the model in the target domain to be seriously affected, and it is unable to effectively identify and match image features in the target domain. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that it is difficult to obtain a large number of training samples in the target domain in the existing heterogeneous remote sensing image matching methods based on the supervised learning framework. A cross-modal heterogeneous image matching method based on unsupervised domain adaptation is proposed. When there are no labeled training samples in the target domain, the source domain annotation information can be fully utilized to solve the problems of the existing methods' strong dependence on the target domain annotation data and the degradation of matching performance caused by the domain difference between the source domain and the target domain.

[0005] The technical solution of the present invention is:

[0006] A cross-modal heterogeneous image matching method based on unsupervised domain adaptation comprises the following steps:

[0007] S1. Constructing a source domain dataset and a target domain dataset, wherein the source domain dataset includes paired annotated optical and SAR image pairs, and the target domain dataset includes unpaired annotated optical and SAR image pairs;

[0008] S2, through the style transfer network, the source domain dataset is converted into an intermediate domain dataset in the style of the target domain;

[0009] S3. Construct an image matching network, including a pseudo-twin network, a domain discriminator, and a template matching module, wherein the pseudo-twin network includes a left branch and a right branch with the same structure but unshared network weights, wherein the input of the left branch is the optical image of the intermediate domain dataset and the optical image of the target dataset, and the input of the right branch is the SAR image of the intermediate domain dataset and the SAR image of the target domain dataset, and the pseudo-twin network extracts features from the input data, and outputs the corresponding deep features of the optical image and the deep features of the SAR image;

[0010] The domain discriminator is used to discriminate the deep features of the images output by the left and right branches of the pseudo twin network, and ensures the consistency of the generated intermediate domain image and the target domain image at the feature level through adversarial learning, so as to further reduce the domain offset between the generated intermediate domain and the target domain from the perspective of feature alignment;

[0011] The template matching module is used to calculate the similarity score map between the template image and the reference image, and determine the matching position, wherein the template image is the deep feature of the optical image output by the left branch of the pseudo-twin network, and the reference image is the deep feature of the SAR image output by the right branch of the pseudo-twin network;

[0012] S4. Use the acquired data set to train the image matching network, and set the loss function used in the training as ,in is the loss function of the template matching module, is the loss function of the domain adversarial part, and the trained image matching network is obtained;

[0013] S5. Input the reference image and the template image to be matched into the trained image matching network, so as to obtain the matching position of the template image corresponding to the reference image.

[0014] Furthermore, S2 includes two style transfer networks corresponding to the source domain style and target domain style conversion of optical and SAR respectively. The two style transfer networks are both cycle-consistent generative adversarial networks, and both include two generators G0 and G1, and two domain discriminators d0 and d1, wherein the generator G0 is used to convert the source domain image into the target domain style, the generator G1 is used to convert the target domain image into the source domain style, the domain discriminator d0 is used to discriminate whether the image generated by the generator G0 has the visual style of the target domain, and the domain discriminator d1 is used to discriminate whether the image generated by the generator G1 has the visual style of the source domain; the style transfer network is trained using the source domain dataset and the target domain dataset of S1, so that the generated image is visually consistent with the target domain image, while retaining the original structural content of the source domain image.

[0015] Furthermore, in S3, the left branch of the pseudo twin network is composed of a convolutional layer with a kernel size of 3×3, a step size of 1, and a padding of 1, three ConvNext V2 block modules, and a dilated spatial pyramid pooling module, which are cascaded in sequence. The right branch of the pseudo twin network has the same structure as the left branch. The feature obtained after the optical image of the intermediate domain dataset passes through the left branch is defined as , the features of the SAR image of the middle domain dataset after passing through the right branch are , define the characteristics of the target domain dataset optical image after passing through the left branch as , the features of the target domain dataset SAR image after the right branch are: ;

[0016] Will The input is sent to the domain discriminator, which uses the domain discriminator to perform domain discrimination on the input feature representation. Through adversarial learning, domain invariant features are learned to ensure that the generated intermediate domain image and the original target domain image remain consistent at the feature level.

[0017] Furthermore, in S3, the similarity score map between the template image and the reference image is calculated by selecting the normalized square difference SSD as a similarity measurement indicator:

[0018] ,

[0019] Where S is the reference image, T is the template image, M and N are the width and height of the template image, and (i, j) is the upper left corner coordinate of the corresponding position of the template image sliding on the reference image;

[0020] The fast calculation in Fourier domain is expressed as:

[0021] ,

[0022] in, They represent forward FFT, inverse FFT and the complex conjugate of F, respectively, v represents the search position, Represents a sliding template window, Represented as a reference figure;

[0023] According to the similarity score graph, the matching position corresponds to the position with the smallest SSD value.

[0024] Further, in S4, the The expression is:

[0025] ,

[0026] Among them, corr represents the calculated SSD similarity score map, label represents the label map of the correct matching position within the positive and negative pixels when randomly deducted, is a circular loss;

[0027] The expression is:

[0028] ,

[0029] Where n is the number of samples input to the domain discriminator, is the domain label of the input, the source domain is 1, and the target domain is 0. is the domain discriminator for the feature The predicted probability of the domain, is the feature representation of sample i.

[0030] The beneficial effects of the present invention are:

[0031] (1) The present invention uses unsupervised domain adaptation technology to fully utilize the source domain annotation information when there are no labeled training samples in the target domain, thereby solving the problem that the existing methods are highly dependent on the target domain annotation data and reducing the cost and time of data acquisition.

[0032] (2) The present invention extracts pixel-level deep dense features through a pixel-level deep dense feature extraction network, which reduces the impact of different modal features on matching accuracy and improves the robustness of matching.

[0033] (3) The present invention uses CycleGAN to achieve the conversion of source domain image style to target domain style, generate images consistent with the target domain appearance style, ensure that the generated images are visually consistent with the target domain images, and retain the original structural content of the source domain images. Through adversarial learning with a domain discriminator, it is ensured that the target domain style image generated from the source domain image and the target domain image maintain consistency at the pixel-level deep dense feature level, thereby improving the accuracy of image matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The following is a flow chart of a method according to an embodiment.

[0035] Figure 2 Schematic diagram of the overall framework of the heterogeneous scene of the embodiment.

[0036] Figure 3 Schematic diagram of the pixel-deep pixel-deep dense feature expression network structure of the embodiment. DETAILED DESCRIPTION

[0037] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0038] The method of the present invention mainly includes constructing a cycle GAN model for converting the optical and SAR image style of a source domain dataset into the optical and SAR image style of a target domain dataset. Pre-train two cycle GAN models to convert the target domain optical image into a source domain style optical image, and convert the target domain SAR image into a source domain style SAR image, so as to achieve domain adaptation at the image level. Construct a pixel-deep dense feature expression network for a pseudo-twin network matching framework, and perform adversarial learning at the pixel-deep dense feature level to achieve feature-level domain adaptation. At the same time, matching training is performed according to the target domain data and labels of the source domain style to further obtain a pseudo-twin network matching model with feature-level domain adaptation. Obtain the target domain image, and use the trained pseudo-twin network matching model to match and obtain the matching result of the target domain.

[0039] Example

[0040] like Figure 1 As shown, this example includes the following specific steps:

[0041] S1. Collect and organize two sets of multimodal image datasets, including optical images and SAR images. Perform preprocessing on the images, such as cropping and normalization, to ensure the consistency of image size and format. The two sets of datasets are divided into source domain datasets and target domain datasets. The source domain dataset contains a large number of labeled image pairs, and the target domain dataset contains unlabeled image pairs.

[0042] S2. Construct a cycle-consistent generative adversarial network (CycleGAN) to convert the style of the source domain image to the style of the target domain and generate a source-target domain style image that is consistent with the target domain appearance style. Use two generators G0 and G1 to convert the source domain image to the target domain style and the target domain image to the source domain style respectively. Perform adversarial training through domain discriminators d0 and d1 to ensure that the generated image is visually consistent with the target domain image while retaining the original structural content of the source domain image.

[0043] S3. Construct a pseudo twin network for expressing common features of heterogeneous images to extract cross-modal invariant features and pixel-level deep dense features. Use two branch networks with the same structure but no shared parameters to process optical images and SAR images respectively to extract feature representations of the images. Perform feature extraction through an encoder (composed of ConvNeXtV2 block) to obtain modality-invariant features and reduce the impact of different modality features on matching accuracy. Input the original target domain image and the source domain (intermediate domain) image that generates the target domain style into the pseudo twin network output to obtain the corresponding deep features of the image.

[0044] S4, using the domain discriminator d for adversarial learning, specifically used to learn the domain-invariant representation of the deep features of the generated target domain image and the original target domain image output in step S3. Through adversarial learning, it is ensured that the source domain image and the target domain image remain consistent at the level of deep image features to avoid domain differences.

[0045] S5. Use the normalized squared difference (SSD) as the similarity metric to calculate the similarity score map between the template image and the reference image. Use the fast Fourier transform (FFT) to speed up the calculation process and improve the matching efficiency. Determine the matching position, that is, the position with the smallest SSD value.

[0046] S6. Use AdamW optimizer to train the model, with a training cycle of 60 epochs and a batch size of 4. Use the Warmup strategy to optimize the learning rate to ensure that the model gradually converges during the training process.

[0047] The above steps are the overall execution steps of this example. Through the above steps, a trained model is obtained. The specific execution process also includes the following technical details:

[0048] Define the source domain data as XS (optical image XS_opt, SAR image XS_sar), the target domain data as XT (optical image XT_opt, SAR image XT_sar), the synthetic intermediate domain data as XP (optical image XP_opt, SAR image XP_sar), the source domain label data as YS, and the target domain data has no label information.

[0049] The network constructed is as follows Figure 2 As shown, including:

[0050] Two cycle-consistency generative adversarial networks (CycleGANs) are constructed as the basic architecture for image alignment to achieve the conversion of different image styles without the need for one-to-one correspondence. The goal of converting the source domain image style to the target domain style is clarified, and the source domain optical conversion to the target domain optical style cycleGAN model and the source domain SAR conversion to the target domain SAR style cycleGAN model are pre-trained.

[0051] The source domain data is input into two trained cycleGAN style transfer network models, and the output is the synthetic intermediate domain data XP (optical image XP_opt, SAR image XP_sar).

[0052] A pseudo-twin network matching framework is constructed, which consists of two pixel-deep dense feature expression networks, one for optical images and the other for SAR images. The two pixel-deep dense feature expression networks have the same network structure, but the network weights are not shared and are trained separately.

[0053] Among them, the pixel deep dense feature expression network is as follows Figure 3 As shown in the figure, it consists of a convolution with a kernel size of 3×3, a stride of 1 and a padding of 1, three convolution blocks (ConvNext V2 block), and a dilated spatial pyramid pooling module, which are used to extract pixel-deep dense features of the image that integrate richer and more robust multi-scale information.

[0054] The synthetic intermediate domain data XP (optical image XP_opt, SAR image XP_sar) and the target data XT (optical image XT_opt, SAR image XT_sar) are input into the pseudo twin network matching framework, and the corresponding pixel deep dense features are output. .

[0055] Construct a domain discriminator for feature-level domain adaptation. Input to the domain discriminator, use the domain discriminator d to discriminate the feature representations of the generated target domain image and the original target domain image, and learn domain invariant features through adversarial learning to ensure that the generated target domain image and the original target domain image maintain consistency at the feature level. Calculate the loss function of the feature alignment part .

[0056] Similarity measurement and matching position determination. The normalized square difference (SSD) is selected as the similarity measurement indicator to calculate the similarity score map between the template image and the reference image. The optical image is used as the reference image and the SAR image is used as the template image. A 192×192 size feature map is randomly deducted from the pixel deep density feature of the SAR image as a template window for subsequent matching training.

[0057] Calculate the SSD similarity score map between the template image and the reference image:

[0058] ,

[0059] Where S is the reference image and T is the template image. Fast Fourier transform (FFT) is used to accelerate the SSD calculation process to improve the matching efficiency. In the present invention, Fourier transform is used to accelerate the calculation of SSD, and the original sliding matching similarity score map becomes:

[0060] ,

[0061] Where i is the position of each element in the pixel-deep dense feature, Indicates that each search position v is in a larger Sliding template window on the Size and Same. Because is constant with respect to v, so it can be ignored. In addition,

[0062] ,

[0063] So it can be quickly computed in the Fourier domain:

[0064] ,

[0065] in, They represent forward FFT, inverse FFT and the complex conjugate of F respectively.

[0066] According to the calculated similarity score graph, the matching position is determined, that is, the position with the smallest SSD value.

[0067] Will Input to the similarity measurement module, output the similarity score map and calculate the loss function of the matching part based on the matching: ,

[0068] Among them, corr represents the calculated SSD similarity score map, label represents the label map that correctly matches the position within the positive and negative pixels when randomly deducted, which is the circular loss Cricle Loss.

[0069] Finally, the above losses are jointly optimized, and the total loss of the model is: .

[0070] The AdamW optimizer is selected as the optimization tool for model training. The training cycle is set to 60 epochs and the batch size is 4 to ensure that the model can complete the training in a reasonable time. During the training process, the Warmup strategy is used to optimize the learning rate so that the model can quickly adapt to the training data in the early stage of training, and then gradually converge to finally obtain model parameters with good performance. Subsequently, the reference image and the template image to be matched are input into the pseudo-twin network matching framework. By calculating the SSD similarity score map of the pixel deep features output by the two models, the minimum value position in the SSD similarity score map can be obtained to obtain the exact matching position of the template corresponding to the reference image.

Claims

1. A cross-modal heterogeneous image matching method based on unsupervised domain adaptation, characterized in that: The following steps are involved: S1. Constructing a source domain dataset and a target domain dataset, wherein the source domain dataset includes paired annotated optical and SAR image pairs, and the target domain dataset includes unpaired annotated optical and SAR image pairs; S2, through the style transfer network, the source domain dataset is converted into an intermediate domain dataset in the style of the target domain; S3. Construct an image matching network, including a pseudo-twin network, a domain discriminator, and a template matching module, wherein the pseudo-twin network includes a left branch and a right branch with the same structure but unshared network weights, wherein the input of the left branch is the optical image of the intermediate domain dataset and the optical image of the target dataset, and the input of the right branch is the SAR image of the intermediate domain dataset and the SAR image of the target domain dataset, and the pseudo-twin network extracts features from the input data, and outputs the corresponding deep features of the optical image and the deep features of the SAR image; The domain discriminator is used to discriminate the deep features of the images output by the left and right branches of the pseudo twin network, and ensures the consistency of the generated intermediate domain image and the target domain image at the feature level through adversarial learning, so as to further reduce the domain offset between the generated intermediate domain and the target domain from the perspective of feature alignment; The template matching module is used to calculate the similarity score map between the template image and the reference image, and determine the matching position, wherein the template image is the deep feature of the optical image output by the left branch of the pseudo-twin network, and the reference image is the deep feature of the SAR image output by the right branch of the pseudo-twin network; S4. Use the acquired data set to train the image matching network, and set the loss function used in the training as ,in is the loss function of the template matching module, is the loss function of the domain adversarial part, and the trained image matching network is obtained; S5. Input the reference image and the template image to be matched into the trained image matching network, so as to obtain the matching position of the template image corresponding to the reference image.

2. According to the method of claim 1, the cross-modal heterogeneous image matching method based on unsupervised domain adaptation is characterized in that: S2 includes two style transfer networks corresponding to the source domain style and target domain style conversion of optical and SAR respectively. The two style transfer networks are both cycle-consistent generative adversarial networks, and both include two generators G0 and G1, and two domain discriminators d0 and d1. Generator G0 is used to convert the source domain image into the target domain style, generator G1 is used to convert the target domain image into the source domain style, domain discriminator d0 is used to judge whether the image generated by generator G0 has the visual style of the target domain, and domain discriminator d1 is used to judge whether the image generated by generator G1 has the visual style of the source domain; The style transfer network is trained using the source and target domain datasets of S1, so that the generated images are visually consistent with the target domain images while retaining the original structural content of the source domain images.

3. According to the cross-modal heterogeneous image matching method based on unsupervised domain adaptation of claim 1, it is characterized in that: In S3, the left branch of the pseudo-twin network is composed of a convolutional layer with a kernel size of 3×3, a step size of 1, and a padding of 1, three ConvNext V2 block modules, and a dilated spatial pyramid pooling module, which are cascaded in sequence. The right branch of the pseudo-twin network has the same structure as the left branch. The features of the optical image of the intermediate domain dataset after passing through the left branch are defined as , the features of the SAR image of the middle domain dataset after passing through the right branch are , define the characteristics of the target domain dataset optical image after passing through the left branch as , the features of the target domain dataset SAR image after the right branch are: ; Will The input is sent to the domain discriminator, which uses the domain discriminator to perform domain discrimination on the input feature representation. Through adversarial learning, domain invariant features are learned to ensure that the generated intermediate domain image and the original target domain image remain consistent at the feature level.

4. According to the cross-modal heterogeneous image matching method based on unsupervised domain adaptation according to claim 1, it is characterized in that: In S3, the similarity score map between the template image and the reference image is calculated by selecting the normalized square difference SSD as the similarity measurement indicator: ， Where S is the reference image, T is the template image, M and N are the width and height of the template image, and (i, j) is the upper left corner coordinate of the corresponding position of the template image sliding on the reference image; The fast calculation in Fourier domain is expressed as: ， in, They represent forward FFT, inverse FFT and the complex conjugate of F, respectively, v represents the search position, Represents a sliding template window, Represented as a reference figure; According to the similarity score graph, the matching position corresponds to the position with the smallest SSD value.

5. According to claim 4, a cross-modal heterogeneous image matching method based on unsupervised domain adaptation is characterized in that: In S4, The expression is: ， Among them, corr represents the calculated SSD similarity score map, label represents the label map of the correct matching position within the positive and negative pixels when randomly deducted, is a circular loss; The expression is: ， Where n is the number of samples input to the domain discriminator, is the domain label of the input, the source domain is 1, and the target domain is 0. is the domain discriminator for the feature The predicted probability of the domain, is the feature representation of sample i.

Images