Cross-domain remote sensing image retrieval method based on semantic information enhancement model

Abstract

The invention discloses a cross-domain remote sensing image retrieval method based on a semantic information enhancement model, and relates to the technical field of image retrieval. Comprising the steps of obtaining a data set of cross-domain remote sensing images; adding a parallel low-rank decomposition matrix into a Transform module of the original retrieval model to form an improved retrieval model; inputting the data of the cross-domain remote sensing image into the improved retrieval model to obtain an image prototype and a text prototype, and generating a first pseudo tag and a second pseudo tag; and calculating softmax probability values of the first pseudo tag and the second pseudo tag through a Transformer module, aggregating the softmax probability values into an average probability vector, and obtaining a retrieval result of the cross-domain remote sensing image according to an entropy value and a confidence score of the average probability vector. According to the method, the CLIP is used for aligning the false labels of all the label-free images, and the reliability of the false labels in the extracted distinguishable feature representation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image retrieval, and in particular to a cross-domain remote sensing image retrieval method based on a semantic information enhancement model. Background Art

[0002] Cross-domain remote sensing image retrieval plays a key role in environmental monitoring, urban planning, disaster assessment and other fields. In these applications, due to the different data sources and acquisition conditions of remote sensing images, there are often significant differences in the data distribution between the source domain and the target domain. Learning and understanding these distribution differences is crucial for cross-domain retrieval, as it can help us build more robust and accurate retrieval models. By effectively adapting to the data distribution of the target domain, we can improve the accuracy and efficiency of retrieval, thereby better serving various practical application scenarios.

[0003] In the prior art, the UDA method is used to achieve cross-domain retrieval. The UDA method establishes a shared model between the source domain and the target domain, and uses the labeled data of the source domain to train the model. Then, the model is migrated to the unlabeled target domain through feature alignment and domain adaptation methods. The specific process includes: first, preparing data sets of the source domain and the target domain. The source domain data contains labeling information, while the target domain data does not contain labels; then, training a shared feature extractor on the two domains, such as using a convolutional neural network; after the training is completed, the feature distribution is made as similar as possible through feature alignment technology, and domain adaptation is performed on the basis of feature alignment; finally, testing is performed on the target domain to evaluate the model performance.

[0004] The defects of the above-mentioned existing technologies are: it is unrealistic to provide labels for massive remote sensing image samples in real tasks, which seriously limits the practical operability of the UDA method, and the insufficient source domain label data greatly increases the difficulty of learning semantically distinguishable features. Summary of the invention

[0005] Based on this, it is necessary to provide a cross-domain remote sensing image retrieval method based on a semantic information enhancement model to address the above technical problems.

[0006] The embodiment of the present invention provides a cross-domain remote sensing image retrieval method based on a semantic information enhancement model, comprising:

[0007] Obtain a dataset of cross-domain remote sensing images consisting of labeled source domain images, unlabeled source domain images, and unlabeled target domain images;

[0008] In the original retrieval model built based on the semantic information enhancement model CLIP, a parallel low-rank decomposition matrix is ​​added to the Transformer module used to connect the image encoder and the text encoder to form an improved retrieval model;

[0009] The cross-domain remote sensing image dataset is input into the improved retrieval model, and the image prototype is obtained from the category center of the labeled source domain image through the image encoder, and the text prototype is obtained from the category center of the labeled source domain image through the text encoder;

[0010] Use image prototypes to generate class centers for unlabeled source domain images and unlabeled target domain images According to the class center Perform the nearest center point classification to generate the first pseudo label And use the text prototype to generate a second pseudo label for the unlabeled source domain image and the unlabeled target domain image

[0011] Calculate the first pseudo label through the Transformer module and the second pseudo-label The softmax probability value is aggregated into an average probability vector, and the retrieval result of the cross-domain remote sensing image is obtained according to the entropy value and confidence score of the average probability vector.

[0012] Optionally, the cross-domain remote sensing image dataset specifically includes: a small number of labeled source domain images A large number of unlabeled source domain images and unlabeled target domain images Where n sl 、n su and n t are the number of labeled source domain images, unlabeled source domain images, and target domain images, respectively.

[0013] Optionally, the source domain and the target domain have the same label space Y = {1, 2, ..., K}, and the data distribution of the source domain and the target domain is different, that is, P(X sl )≠P(X t ) and P(X su )≠P(X t ).

[0014] Optionally, a parallel low-rank decomposition matrix is ​​added to the Transformer module, specifically including:

[0015] The injection matrix contains three parts of the Transformer module. The first part adds a low-rank matrix to W q , W k and W v To obtain new queries, keys, and values ​​in the attention calculation, the formula is:

[0016] Q=W q zin +B q A q z in

[0017] K=W k z in +B k A k z in

[0018] V=W v z in +B v A v z in

[0019] z′ out =Attention(Q, K, V)

[0020] Among them, z in represents the input sequence of images or text descriptions, z′ out represents the output of the scaled dot product attention mechanism;

[0021] The second part adds trainable matrices A1 and B1 in the linear layer after the attention module, with the formula:

[0022] z out =Linear(z′ out )+B1A1z′ out

[0023] In Section 3, the modified CLIP image encoder f and text encoder g are identified as f(x) and g(t), where x and t are the input image and the corresponding text description, respectively.

[0024] Optionally, an image prototype is obtained from the category center of a labeled source domain image through an image encoder, specifically including:

[0025] Image Prototype It is directly obtained from the category center of the labeled source domain image, and the formula is:

[0026]

[0027] in, Indicates that the labeled source domain sample belongs to the label k class, express The number of samples.

[0028] Optionally, a text prototype is obtained through a text encoder, specifically including:

[0029] By inputting text embedding into the text encoder of CLIP, text prototypes or text semantic features The formula is:

[0030]

[0031] Optionally, use image prototypes to generate class centers for unlabeled source domain images and unlabeled target domain images The formula is:

[0032]

[0033] in, Represents the kth element calculated by the unlabeled image features and image prototypes during softmax.

[0034] Compared with the prior art, the cross-domain remote sensing image retrieval method based on the semantic information enhancement model provided by the embodiment of the present invention has the following beneficial effects:

[0035] In the existing technology, it is unrealistic to provide labels for massive remote sensing image samples in real tasks, which seriously limits the practical operability of the UDA method. The insufficient source domain label data greatly increases the difficulty of learning semantically distinguishable features.

[0036] The present invention uses image prototypes to generate class centers for unlabeled source domain images and unlabeled target domain images. According to the class center Perform the nearest center point classification to generate the first pseudo label And use the text prototype to generate a second pseudo label for the unlabeled source domain image and the unlabeled target domain image The problem of insufficient source domain label data in the existing technology is solved, and the difficulty of learning semantically distinguishable features is overcome.

[0037] In addition, by calculating the first pseudo label in the Transformer module and the second pseudo-label The softmax probability value is aggregated into an average probability vector, and the retrieval result of the cross-domain remote sensing image is obtained according to the entropy value and confidence score of the average probability vector, which improves the credibility and reliability of the cross-domain remote sensing image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a CLIP-CDRSIR framework of a cross-domain remote sensing image retrieval method based on a semantic information enhancement model provided in one embodiment;

[0039] Figure 2 A schematic diagram of a pseudo-label generation mechanism of a cross-domain remote sensing image retrieval method based on a semantic information enhancement model provided in an embodiment;

[0040] Figure 3 A schematic diagram of the Top k accuracy results on DB1 of a cross-domain remote sensing image retrieval method based on a semantic information enhancement model provided in an embodiment;

[0041] Figure 4 A schematic diagram of the Top k accuracy results on DB2 of a cross-domain remote sensing image retrieval method based on a semantic information enhancement model provided in one embodiment;

[0042] Figure 5 A t-SNE schematic diagram of AID→UCMD of a cross-domain remote sensing image retrieval method based on a semantic information enhancement model provided in one embodiment;

[0043] Figure 6 A t-SNE schematic diagram of UCMD→AID of a cross-domain remote sensing image retrieval method based on a semantic information enhancement model provided in one embodiment;

[0044] Figure 7 A schematic diagram of visualization of retrieval results of a cross-domain remote sensing image retrieval method based on a semantic information enhancement model provided in an embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] In one embodiment, a cross-domain remote sensing image retrieval method based on a semantic information enhancement model is provided, the method comprising:

[0047] Obtain a dataset of cross-domain remote sensing images consisting of labeled source domain images, unlabeled source domain images, and unlabeled target domain images.

[0048] In the original retrieval model built based on the semantic information enhancement model CLIP, a parallel low-rank decomposition matrix is ​​added to the Transformer module used to connect the image encoder and the text encoder to form an improved retrieval model.

[0049] The cross-domain remote sensing image dataset is input into the improved retrieval model, and the image prototype is obtained from the category center of the labeled source domain image through the image encoder, and the text prototype is obtained from the category center of the labeled source domain image through the text encoder.

[0050] Use image prototypes to generate class centers for unlabeled source domain images and unlabeled target domain images According to the class center Perform the nearest center point classification to generate the first pseudo label And use the text prototype to generate a second pseudo label for the unlabeled source domain image and the unlabeled target domain image

[0051] Calculate the first pseudo label through the Transformer module and the second pseudo-label The softmax probability value is aggregated into an average probability vector, and the retrieval result of the cross-domain remote sensing image is obtained according to the entropy value and confidence score of the average probability vector.

[0052] 1. Method design

[0053] 1.1 Method Construction

[0054] A novel domain adaptation method CLIP-CDRSIR is constructed, which uses the pre-trained visual language model CLIP to handle the CDRISR task. For the first problem, CLIP is used to align the pseudo-labels of each unlabeled image to improve the reliability of the pseudo-labels in the extracted distinguishable feature representation. In particular, a dual-modality prototype-guided pseudo-labeling mechanism (Dual-Modality Prototypes Guided Pseudo-Labeling, DMPGPL) is set, which generates reliable pseudo-labels for the unlabeled source and target domains from the set image prototypes and text prototypes. At the same time, the confidence scores of these pseudo-labels are calculated to improve their credibility and reliability. For the second problem, the adaptive process of the CLIP model on cross-domain remote sensing images is completed by performing contrastive learning on image features and dual-modal prototypes. In addition, the LRS method is proposed to incorporate the confidence scores as weights into the contrastive loss, thereby mitigating the impact of noisy pseudo-labels on the model. Considering that updating all parameters of the CLIP model requires high computational cost, it is proposed to use the LoRA fine-tuning method to freeze the weights of the CLIP model and add a small number of trainable parameters to each layer in the Transformer framework.

[0055] 1.2 Method representation and definition

[0056] The SSCDRSIR setting consists of three parts: a small number of labeled source domain images A large number of unlabeled source domain images and unlabeled target domain images Where n sl 、n su and n tare the number of labeled source domain images, unlabeled source domain images, and target domain images, respectively. The two domains share the same label space Y = {1, 2, ..., K}, but the data distribution is different, that is, P(X sl )≠P(X t ) and P(X su )≠P(X t ).

[0057] 1.3 Design of domain adaptation method based on CLIP enhancement

[0058] The proposed CLIP-CDRSIR method is introduced in detail from three aspects. The overall framework is shown in the figure below. Figure 1 As shown, Figure 1 (a) is the overall framework diagram. Text and image features are extracted from the image encoder and text encoder of CLIP, and the domain adaptation task is completed by aligning the image features to the corresponding prototype. Figure 1 (b) in the figure shows the details of the Transformer module using the LoRA fine-tuning method. The trainable parallel low-rank decomposition matrix is ​​injected into the frozen Transformer module, in which the details of the multi-head attention mechanism calculation are ignored. First, the LoRA fine-tuning method is introduced according to the framework. Secondly, the construction process of the bimodal prototype and pseudo-label generation mechanism is demonstrated. Subsequently, the confidence score of the pseudo-label is evaluated. Finally, the proposed LRS method is used to perform contrastive learning to complete the category distinguishability feature learning and domain alignment on the cross-domain remote sensing dataset.

[0059] The following is a detailed introduction to the CLIP-CDRSIR method from the perspectives of the CLIP method design based on LoRA fine-tuning, the pseudo-label generation mechanism design guided by bimodal prototypes, the contrastive learning method design of the loss reweighting strategy, and entropy minimization.

[0060] (1) Design of CLIP method based on LoRA fine-tuning. LoRA assumes that there is a low intrinsic rank in the adaptive task, that is, it contains a lot of redundant information. It aims to inject a low-rank matrix into the trainable weights. The generated new vector is shown in formula (1):

[0061] h=W0x+BAx (1)

[0062] Among them, the pre-trained weights Frozen, added matrix and is trainable, with label rank r <<min(d, k). For LoRA in CLIP, a slight change is made from Figure 1 As can be seen from part (b) of the figure, the injection matrix contains three parts of the Transformer module. For the first part, add the low-rank matrix to the pre-trained weight matrix Wq , W k and W v In order to obtain new query, key and value in the attention calculation. Its calculation is shown in formula (2):

[0063] Q=W q z in +B q A q z in

[0064] K=W k z in +B k A k z in

[0065] V=W v z in +B v A v z in

[0066] z′ out =Attention(Q, K, V) (2)

[0067] Among them, z in represents the input sequence of images or text descriptions, Q, K, V are the matrices calculated respectively, and z′ out Represents the output of the scaled dot product attention mechanism. For the second part, add trainable matrices A1 and B1 to the linear layer after the attention module. Linear is a linear layer, and its calculation is shown in formula (3):

[0068] z ou t=Linear(z′ out )+B1A1z′ out (3)

[0069] Among them, z out It refers to the result obtained after processing by the linear layer.

[0070] Finally, for the reparameterization of the feedforward network, the matrix is ​​added to the MLP layer. For simplicity, the modified CLIP image encoder f and text encoder g are denoted as f(x) and g(t), where x and t are the input image and the corresponding text description, respectively.

[0071] (2) Design of pseudo-label generation mechanism guided by bimodal prototype

[0072] Reliable pseudo-labels are constructed to alleviate the serious lack of label supervision, and dual-modality prototypes are constructed to enhance the learning of image features and text semantic information. In order to better use the cross-modal capabilities of CLIP, dual-modality prototypes (DMPs) are constructed, including image prototypes and text prototypes. Image prototype It is directly obtained from the category center of the labeled source domain image, and its calculation is shown in formula (4):

[0073]

[0074] in, Indicates that the labeled source domain sample belongs to the label k class, express For the text prototype, from the text description “This is a satellite image 0fa[CLS]”, [CLS] can be represented as an exact category name in the remote sensing dataset, such as farmland, baseball field, and river, etc. The entire text embedding W = {W1, W2, …, w k By directly embedding text into the text encoder of CLIP, text prototypes or text semantic features It can be generated according to formula (5):

[0075]

[0076] Different from the traditional CLIP zero-shot classification, the pseudo-label generation mechanism is guided by DMP. Given an unlabeled dataset X u ={x|x∈X su} or X u ={x|x∈X t}, first use the image prototype to generate class centers for unlabeled data The calculation method is shown in formula (6):

[0077]

[0078] in, represents the kth element calculated by the unlabeled image features and the image prototype during softmax, and C is the image prototype;

[0079] Then classify it as x according to the nearest center point i Generate pseudo labels As shown in formula (7):

[0080]

[0081] Similarly, the unlabeled sample x iGenerate another pseudo label with the help of text prototype The calculation method is shown in formulas (8) and (9):

[0082]

[0083]

[0084] Figure 2 The pseudo-label generation mechanism above is shown. (a) is the pseudo-label generation mechanism, which generates pseudo-labels for unlabeled samples in two ways. (b) is the pseudo-label confidence score. When the two predicted pseudo-labels are the same, the confidence score is high. In simple terms, first generate two pseudo-labels for the unlabeled sample x. and Secondly, the confidence of the pseudo-label sample is calculated. When the pseudo-label predicted by the unlabeled sample and Based on the above settings, we first use the center point of the unlabeled category to calculate the softmax probability values ​​of the two pseudo labels, marked as and Then it is aggregated into an average probability vector and its entropy is calculated. The calculation formulas are shown in (10) and (11):

[0085]

[0086]

[0087] Among them, K is the number of categories. Finally, the confidence of sample x can be calculated by formula (12):

[0088]

[0089] Depend on Figure 2 It can be seen that when the predicted pseudo label and When the same, for example, sample x a , the unlabeled sample will be closer to the corresponding image prototype and text prototype, making the entropy value smaller and the confidence score higher. On the contrary, when the unlabeled sample is far away from one of the corresponding prototypes, such as sample x b , the predicted and Inconsistency will result in higher entropy values ​​and smaller confidence scores. In brief, two different pseudo labels are generated for unlabeled data (including unlabeled source domain image samples and unlabeled target domain image samples). Considering the modality difference between images and texts, only the pseudo labels calculated from the image prototypes are used as supervisory signals in the training process.

[0090] (3) Design of contrastive learning method for loss reweighting strategy

[0091] We introduce the use of contrastive learning to complete class distinguishable feature learning and domain alignment. First, all image features are aligned with their corresponding DMPs, and the alignment is shown in formulas (13) and (14):

[0092]

[0093]

[0094] Where X = X sl ∪X su ∪X tu , and Represents the sample x i Related to DMP, the supervisory signal of contrast loss comes from labeled source domain images, source domain images with pseudo labels, and target domain image samples with pseudo labels. In order to further distinguish the reliability of pseudo labels, the loss reweighting strategy is integrated into the contrast loss, and the integration method is shown in formula (15):

[0095]

[0096] Among them, X u =X su ∪X tu , w i For x i In LRS, the model captures more semantic information from samples with higher confidence and less semantic information from samples with lower confidence. Therefore, contrastive learning can be expressed by formula (16):

[0097]

[0098] In addition, to reduce the difference between the two modalities, contrastive learning is performed in DMP so that prototypes of the same category are close to each other, while prototype features of different categories are far away from each other, thereby reducing the difference between different modalities, as shown in formula (17):

[0099]

[0100] (4) Entropy minimization

[0101] In order to make the image features close to the corresponding prototype in an unsupervised manner, entropy minimization is used for all samples, and the calculation is shown in formula (18):

[0102]

[0103] The overall loss function is shown in formula (19):

[0104]

[0105] 1.4 Algorithm Design

[0106] In summary, a series of modules are designed to make the CLIP model better adapt to the cross-domain remote sensing image retrieval task, so as to improve the results of cross-domain remote sensing image retrieval. It mainly includes using LoRA to fine-tune CLIP, better utilize relevant information and reduce the large number of parameters generated during training. By proposing a bimodal prototype guidance mechanism to construct pseudo labels, generating bimodal confidence scores to strengthen the alignment supervision of data feature distribution, and proposing the use of loss reweighting strategy to complete the category distinguishability feature learning and domain alignment of the source domain and the target domain, finally using entropy minimization to enhance the learnability of feature prototypes.

[0107] In order to better understand the entire training process, the overall algorithm process is shown.

[0108]

[0109]

[0110] The goal of the algorithm is to fine-tune the CLIP model through LoRA so that it can learn the feature representation of cross-domain remote sensing images and thus achieve cross-domain retrieval. It provides an effective semantic information enhancement and domain alignment mechanism for cross-domain remote sensing image retrieval. To obtain the final cross-domain remote sensing image retrieval results, the algorithm needs to be combined with other modules.

[0111] 2. Experimental results and analysis

[0112] 2.1 Experimental Setup

[0113] In order to evaluate the effectiveness of the proposed CLIP-CDRSIR method on cross-domain remote sensing image retrieval tasks, a large number of comparison and ablation experiments were conducted on 12 cross-domain retrieval tasks. The following is a detailed description of the dataset, comparison method, and experimental settings.

[0114] (1) Datasets. The effectiveness of the CLIP-CDRSIR method is verified by constructing two cross-domain datasets. AID, UCMD, and NWPU are selected to construct DB1. PatternNet, VBing, and VarcGIS are selected to construct DB2. For the convenience of explanation, these six datasets are represented as A, U, N, P, VA, and VB, respectively. For DB1, each dataset contains 1,000 images of 10 categories, that is, each category contains 100 images. Two different datasets are arbitrarily selected as the source domain and the target domain, respectively, so six cross-domain retrieval tasks are generated, namely A→U, U→A, A→N, N→A, U→N, and N→U. Among them, the symbol P→Q indicates that dataset P is used as the source domain and dataset Q is used as the target domain. For DB2, the categories provided by the three datasets are similar, so there is no need to extract subcategories. In the experiment, 800 images are randomly selected from each category and each dataset containing 30,400 images, and then six cross-domain retrieval tasks are constructed, namely P→VB, VB→P, P→VA, VA→P, VB→VA, and VA→VB.

[0115] (2) Comparison method. Six latest DA research methods are selected as comparative experimental methods, namely ADDA, JAN, SE, DATSNET, PCLUDA and BSCA. Among them, ADDA, JAN and SE are traditional UDA methods, and BSCA is a novel SSDA method in the field of remote sensing images.

[0116] (3) Detailed experimental settings. The experiment uses the Pytorch framework. For the two cross-domain datasets, 80% and 20% of each category in the same domain are randomly sampled as training sets and test sets. The training set is re-divided into training data and validation data in a ratio of 8:2. In the training stage, only a small number of labeled source domain images and unlabeled target domain images are used. In the experiment, three label ratios Ratio are set, and the values ​​are set to 0.1, 0.3 and 1.0. When Ratio is 1.0, it means that all labeled source domain images are available. In the cross-domain retrieval stage, the test set extracted from the target domain is used as the query sample, and then 80% of the images are randomly sampled from the source domain as the retrieval dataset. The similarity between the query and all retrieval samples is calculated using the Euclidean distance.

[0117] For fair comparison, all comparison methods use the CLIP image encoder ViT-B / 32 as the model framework, which means that the input image is resized to 224×224 pixels and the extracted feature dimension is 512. For the proposed method, the CLIP model is fine-tuned by setting the LoRA rank coefficient r to 8, and Adam is used for network optimization with a fixed learning rate of le-4. Other hyperparameters such as the training epoch, batchsize, and temperature coefficient τ are set to 30, 32, and 0.1, respectively. At the same time, a FT method is defined to directly use the optimized model to fine-tune the parameters of the entire CLIP to verify the effectiveness of the LoRA fine-tuning mechanism. To comprehensively verify the retrieval performance, the mAP, Topk (P@K) accuracy, and t-SNE high-dimensional feature visualization are used to evaluate the performance of the cross-domain retrieval task.

[0118] 2.2 Comparative experimental results and analysis

[0119] The DB1 comparative experimental results are shown in Table 1, which shows the mAP results of the CLIP-CDRSIR method on DB1 when different label ratios are set.

[0120] It is obvious that as the label coefficient Ratio increases, the retrieval results are gradually improved. When Ratio = 1.0 is compared with Ratio = 0.1, the mAP of cross-domain retrieval is improved by 0.95%, 3.29%, 0.86%, 3.23%, 3.52% and 4.61%, because more real labels in the source domain bring more effective information during the training process, so it can better learn the feature distribution between domains. At the same time, it can also be noted that the proposed method not only performs better than other domain adaptation methods, but also shows strong stability under various label coefficient settings.

[0121] This phenomenon can be attributed to the following reasons: first, the proposed DMPGPL provides more information during training. The cross-modal capabilities of CLIP can be better utilized by performing contrastive learning on image features and text semantic information.

[0122] Secondly, the LRS method also shows a certain effect. By evaluating the reliability of each pseudo-label, the model can pay more attention to the correct label image during training. Finally, compared with the FT and LoRA fine-tuning methods, the LoRA fine-tuning method can not only better demonstrate the generalization ability of the CLIP model, but also reduce the number of parameters in the training process to a greater extent.

[0123] Table 1 Comparative experimental results of DB1 when Ratio=0.1, 0.3, 1.0

[0124]

[0125] In order to demonstrate the operability of the proposed method CLIP-CDRSIR in practical applications, six cross-domain retrieval tasks are performed on the constructed 38-category DB2 retrieval dataset. The mAP results on DB2 are shown in Table 2.

[0126] Table 2 DB2 comparative experimental results when Ratio=0.1, 0.3, 1.0

[0127]

[0128] Experimental results show that the CLIP-CDRSIR method not only shows high retrieval performance on simple 10-category cross-domain tasks, but also shows excellent effectiveness on complex 38-category cross-domain retrieval tasks. Compared with other domain adaptation methods, CLIP-CDRSIR shows better results in mAP results. When Ratio=0.1, 0.3, and 1.0, CLIP-CDRSIR shows strong retrieval performance, which fully demonstrates the feasibility of the constructed semi-supervised cross-domain retrieval scenario task. In particular, when the label coefficient Ratio is set to 0.1, the mAP results of CLIP-CDRSIR are improved by 9.2%, 6.8%, 13.15%, 5.3%, 3.9%, and 5.0% respectively compared with the best results of other methods. With the help of the proposed DMPGPL method, the retrieval results show more robust characteristics as the number of categories increases, indicating that the use of bimodal prototype learning enhances the consistency learning ability of similar image samples in different domains, and the use of LRS also enables the CLIP method to better capture the distinguishable characteristics of categories.

[0129] To further demonstrate the effectiveness of the proposed CLIP-CDRSIR method, Figure 3 and Figure 4The accuracy of the Top k (P@K) results is shown on the constructed DB1 and DB2 cross-domain datasets when the number of retrieval samples ranges from 5 to 100. From the result graphs of DB1 and DB2, it can be seen that the proposed CLIP-CDRSIR method shows strong effectiveness in both simple and complex cross-domain remote sensing image retrieval tasks. In particular, when P = 5, the proposed method can retrieve the image that best matches the input, that is, the probability of the correct retrieval result is much greater than that of other comparative methods, showing strong retrieval accuracy. The use of FT and LoRA fine-tuning can greatly improve the retrieval accuracy, which may be caused by the following reasons: (1) The pseudo-label mechanism guided by the bimodal feature prototype and the loss reweighting strategy can better learn the feature space distribution of the two domains for cross-domain remote sensing image retrieval, and the use of bimodal learning can strengthen the use of pseudo-labels. And generate confidence scores and inject them as weights into the contrastive learning loss of the bimodal prototype, further strengthening the consistency learning of different domains. (2) LoRA fine-tuning improves CLIP’s ability to process and learn massive amounts of data by injecting effective sample information into the three modules of Transformer, that is, injecting low-rank matrices into trainable weights.

[0130] In order to further explore the applicability of the proposed method CLIP-CDRSIR in feature alignment and domain alignment, the t-SNE diagram is used to show the high-dimensional feature visualization diagram of the above comparative experiments in the AID and UCMD cross-domain retrieval tasks, as shown in the figure. Figure 5 and 6The following are visualizations of the A→U and U→A cross-domain retrieval tasks, where the label ratio Ratio is set to 0.1. As can be seen from the figure, the proposed method plays a very important role in domain alignment, and the FT fine-tuning method also shows a certain degree of effectiveness, making the target domain and the source domain similar high-dimensional features highly close, and the high-dimensional features of different classes far away, while other comparison methods still have obvious defects in domain category space alignment. Taking AID→UCMD as an example, in the comparison methods ADDA, JAN, SE, and DATSNET, there are obviously large defects in the source domain and target domain samples, and there are still many classification errors, and the category space alignment ability is poor. The PCLUDA and BSCA methods have better consistency in spatial distribution, can learn the classification boundaries between different classes in the source domain and the target domain, and the distribution between different classes can also be clearly distinguished, but there are still some samples that are difficult to be classified correctly. Compared with other comparison methods, the CLIP-CDRSIR method shows excellent retrieval performance. In particular, compared with the FT fine-tuning method, LoRA is used to fine-tune a small number of useful parameters so that the two domains can learn useful feature semantic information. By freezing the pre-trained weights, LoRA is used to inject a low-rank trainable matrix into the Transformer module to enhance the distinguishable feature learning of the category, which is significantly effective in aligning cross-domain datasets. At the same time, the use of bimodal prototypes can construct more reliable pseudo-labels. Therefore, the above comparative experimental results and visualizations show that the proposed CLIP-CDRSIR method exhibits higher retrieval performance.

[0131] To further verify the effect of this method on cross-domain retrieval tasks, Figure 7 The diagram shows the visualization results of the Top 10 Returned Results retrieval of this method and the comparative experiment on the A→U cross-domain retrieval task. Query represents the input retrieval image sample, Top 10 Returned Results represent the results returned by the retrieval database, and Methods represent the comparative method and the method proposed by CLIP-CDRSIR. In this retrieval visualization, Ratio is set to 0.1, and the green and red borders represent the correct and incorrect returned samples, respectively. Three images are randomly selected as query samples, and then the Top 10 Returned Results results on the A→U cross-domain retrieval task are calculated when Ratio is set to 0.1. Obviously, when a query image is given, the CLIP-CDRSIR method is more capable of retrieving the most similar samples. Compared with other comparative methods, the proposed method can effectively avoid returning image samples that are visually similar but have different semantic representations. This is because the CLIP-CDRSIR method enriches the semantic information of the extracted image features, thereby showing better retrieval performance.

[0132] 2.3 Ablation Experiment Results and Analysis

[0133] The effectiveness of the proposed method CLIP-CDRSIR is verified by ablation experiments. As shown in Table 3, ablation experiments are performed on five variables in the CLIP-CDRSIR method, namely: (1) V1 removes contrastive learning on image features and image prototypes, and only learns discriminative features and domain alignment through text prototypes. (2) V2 removes contrastive learning on image features and text prototypes to reduce the information of text semantics during training. (3) V3 removes (4) V4 removed (5) V5 removes LRS, weakening the ability to distinguish the reliability of pseudo labels. The mAP experimental results on cross-domain remote sensing image retrieval tasks N→U, U→N, VB→VA and VA→VB are shown.

[0134] As can be seen from the table, all parts of CLIP-CDRSIR have a certain impact on the retrieval performance. V1 shows that only text semantic prototypes are not enough in domain adaptation tasks. V2 shows that the lack of text semantic information will weaken the performance results. Therefore, contrastive learning is modified with the help of DMP. Compared with the V3 results of CLIP-CDRSIR, It plays an important role in aligning image prototypes and text prototypes, and can further enhance the ability of contrastive learning. V4 reflects that the entropy minimization used can further enhance the ability of domain alignment and category distinguishability learning in the training phase. The retrieval performance of V5 shows that the proposed LRS method can learn correct semantic information from noisy pseudo-label samples. In general, the contrastive learning loss performed on image and text prototypes, the contrastive learning of bimodal feature prototypes, and the LRS module all have a certain impact on the retrieval performance, and the ablation experiment results can also verify that in the actual cross-domain remote sensing image retrieval, each module can have a significant improvement effect on the CLIP-CDRSIR method.

[0135] Table 3 Ablation experiment results

[0136] Method U→N N→U VB→VA VA→VB V1 0.8893 0.8865 0.8339 0.8896 V2 0.8899 0.8915 0.8287 0.8871 V3 0.9011 0.8939 0.8588 0.9058 V4 0.9001 0.8897 0.8503 0.8916 V5 0.9013 0.9036 0.8409 0.8761 CLIP-CDRSIR 0.9142 0.9106 0.8689 0.9116

[0137] 3. Summary

[0138] A novel domain adaptation scenario task, namely SSCDRSIR, is proposed. To enhance the retrieval performance of SSCDRSIR, a simple and effective domain adaptation method, namely CLIP-CDRSIR, is proposed. This method uses LoRA to fine-tune the CLIP model to complete the task of cross-domain remote sensing image retrieval. To alleviate the problem of severe lack of labeled data, the DMPGPL mechanism is first proposed to generate pseudo labels, propose to construct cross-modal feature prototypes, and use the CLIP model to enhance the network model's semantic information learning of cross-domain remote sensing image samples. Subsequently, confidence scores are estimated for all unlabeled samples. At the same time, in order to further reduce the domain bias between the two domains and better learn category distinguishability features, the LRS method is proposed to drive image features close to their corresponding DMPs. Experimental results on 12 cross-domain remote sensing image retrieval tasks show that the proposed method outperforms the latest related research methods and shows strong retrieval superiority in practical scenarios.

[0139] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A cross-domain remote sensing image retrieval method based on a semantic information enhancement model, characterized in that: include: Obtain a dataset of cross-domain remote sensing images consisting of labeled source domain images, unlabeled source domain images, and unlabeled target domain images; In the original retrieval model built based on the semantic information enhancement model CLIP, a parallel low-rank decomposition matrix is ​​added to the Transformer module used to connect the image encoder and the text encoder to form an improved retrieval model; The cross-domain remote sensing image dataset is input into the improved retrieval model, and the image prototype is obtained from the category center of the labeled source domain image through the image encoder, and the text prototype is obtained from the category center of the labeled source domain image through the text encoder; Use image prototypes to generate class centers for unlabeled source domain images and unlabeled target domain images According to the class center Perform the nearest center point classification to generate the first pseudo label And use the text prototype to generate a second pseudo label for the unlabeled source domain image and the unlabeled target domain image Calculate the first pseudo label through the Transformer module and the second pseudo-label The softmax probability value is aggregated into an average probability vector, and the retrieval result of the cross-domain remote sensing image is obtained according to the entropy value and confidence score of the average probability vector.

2. A cross-domain remote sensing image retrieval method based on a semantic information enhancement model as claimed in claim 1, characterized in that: The cross-domain remote sensing image dataset specifically includes: a small number of source domain images with labels A large number of unlabeled source domain images and unlabeled target domain images where n sl 、n su and n t are the number of labeled source domain images, unlabeled source domain images, and target domain images, respectively.

3. A cross-domain remote sensing image retrieval method based on a semantic information enhancement model as claimed in claim 2, characterized in that: The source domain and the target domain have the same label space Y = {1, 2, ..., K}, and the data distribution of the source domain and the target domain is different, that is, P(X sl )≠P(X t ) and P(X su )≠P(X t ).

4. The cross-domain remote sensing image retrieval method based on a semantic information enhancement model according to claim 1, characterized in that: Adding a parallel low-rank decomposition matrix to the Transformer module specifically includes: The injection matrix contains three parts of the Transformer module. The first part adds a low-rank matrix to W q , W k and W v To obtain new queries, keys, and values ​​in the attention calculation, the formula is: Q=W q z in +B q A q z in K=W k z in +B k A k z in V=W v z in +B v A v z in z′ out =Attention(Q,K,V) Among them, z in is the input sequence of images or text descriptions, z′ out is the output of the scaled dot product attention mechanism, Q, K, V are the calculated matrices; The second part adds trainable matrices A1 and B1 in the linear layer after the attention module, with the formula: from out =Linear(z′ out )+B1A1z′ out Among them, Linear is the linear layer; In Section 3, the modified CLIP image encoder f and text encoder g are identified as f(x) and g(t), where x and t are the input image and the corresponding text description, respectively.

5. The cross-domain remote sensing image retrieval method based on the semantic information enhancement model according to claim 1, characterized in that: The method of obtaining an image prototype from the category center of a labeled source domain image through an image encoder specifically includes: Image Prototype It is directly obtained from the category center of the labeled source domain image, and the formula is: in, Indicates that the labeled source domain sample belongs to the label k class, express The number of samples.

6. The cross-domain remote sensing image retrieval method based on a semantic information enhancement model as claimed in claim 1, characterized in that: The obtaining of the text prototype by the text encoder specifically includes: By inputting text embedding into the text encoder of CLIP, text prototypes or text semantic features The formula is:

7. The cross-domain remote sensing image retrieval method based on a semantic information enhancement model as claimed in claim 1, characterized in that: The method uses image prototypes to generate class centers for unlabeled source domain images and unlabeled target domain images. The formula is: in, Represents the kth element calculated by the unlabeled image features and image prototypes during softmax.

Images