A cross-domain remote sensing image retrieval method based on semantic information enhancement model

Abstract

The application discloses a cross-domain remote sensing image retrieval method based on a semantic information enhancement model, and relates to the technical field of image retrieval. The method comprises the following steps: acquiring a data set of cross-domain remote sensing images; adding a parallel low-rank decomposition matrix to a Transform module of an original retrieval model to form an improved retrieval model; inputting the data of the cross-domain remote sensing images into the improved retrieval model to acquire an image prototype and a text prototype, and to generate a first pseudo label and a second pseudo label; calculating the softmax probability values of the first pseudo label and the second pseudo label through the Transform module, and aggregating the softmax probability values into an average probability vector; and obtaining a retrieval result of the cross-domain remote sensing images according to the entropy value and the confidence score of the average probability vector. The CLIP is used to align the pseudo labels of each unlabeled image, and the reliability of the pseudo labels in the distinguishable feature representation extracted is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image retrieval, in particular to a cross-domain remote sensing image retrieval method based on a semantic information enhancement model. BACKGROUND

[0002] Cross-domain remote sensing image retrieval plays a key role in environmental monitoring, urban planning, disaster assessment and other fields. In these applications, due to the different data sources and collection conditions of remote sensing images, there are often significant differences in the data distribution between the source domain and the target domain. Learning and understanding these distribution differences is crucial for cross-domain retrieval, as it can help us build more robust and accurate retrieval models. By effectively adapting to the data distribution of the target domain, we can improve the accuracy and efficiency of retrieval, thereby better serving various practical application scenarios.

[0003] In the prior art, UDA method is used to realize cross-domain retrieval. UDA method establishes a shared model between the source domain and the target domain, and uses the labeled data of the source domain to train the model, and then migrates the model to the unlabeled target domain through feature alignment and domain adaptation method. The specific process includes: first, prepare the source domain and target domain data sets, the source domain data contains label information, and the target domain data does not contain label; then, train a shared feature extractor on the two domains, for example, use a convolutional neural network; after training, make the feature distribution as similar as possible through feature alignment technology, and perform domain adaptation based on feature alignment; finally, test on the target domain and evaluate the model performance.

[0004] The defect of the above-mentioned prior art is that providing labels for massive remote sensing image samples is not realistic in real tasks, which severely limits the practicality of UDA method, making the insufficient source domain label data greatly increase the difficulty of semantic distinguishable feature learning ability. SUMMARY

[0005] Therefore, it is necessary to provide a cross-domain remote sensing image retrieval method based on a semantic information enhancement model in view of the above technical problems.

[0006] The embodiment of the present application provides a cross-domain remote sensing image retrieval method based on a semantic information enhancement model, which comprises:

[0007] Obtain a data set of cross-domain remote sensing images composed of labeled source domain images, unlabeled source domain images and unlabeled target domain images;

[0008] In the original retrieval model constructed based on the semantic information enhancement model CLIP, a parallel low-rank decomposition matrix is added to the Transformer module used to connect the image encoder and the text encoder to form an improved retrieval model.

[0009] The improved retrieval model is inputted with the cross-domain remote sensing image dataset, and the image prototype is obtained from the class center of the labeled source domain image by the image encoder, and the text prototype is obtained from the class center of the labeled source domain image by the text encoder;

[0010] The class center is generated for the unlabeled source domain image and the unlabeled target domain image using the image prototype According to the class center The nearest center point classification is performed to generate the first pseudo label And the second pseudo label is generated for the unlabeled source domain image and the unlabeled target domain image using the text prototype

[0011] The first pseudo label is calculated by the Transformer module And the softmax probability value of the second pseudo label The softmax probability value is aggregated into an average probability vector, and the cross-domain remote sensing image retrieval result is obtained according to the entropy value and the confidence score of the average probability vector.

[0012] Optionally, the dataset of the cross-domain remote sensing image specifically comprises: a small amount of labeled source domain images A large number of unlabeled source domain images And unlabeled target domain images Wherein n sl , n su And n t are the number of labeled source domain images, unlabeled source domain images and target domain images respectively.

[0013] Optionally, the source domain and the target domain have the same label space Y={1,2, …, K}, and the data distribution of the source domain and the target domain is different, that is, P(X sl )≠P(X t ) and P(X su )≠P(X t ).

[0014] Optionally, a parallel low-rank decomposition matrix is added to the Transformer module, specifically comprising:

[0015] The injection matrix contains three parts of the Transformer module, the first part adds a low-rank matrix to W q , W k And W v , so as to obtain new queries, keys and values in attention calculation, the formula is:

[0016] Q=W q zin +B q A q z in

[0017] K=W k z in +B k A k z in

[0018] V=W v z in +B v A v z in

[0019] z′ out =Attention(Q,K,V)

[0020] where z in denotes the input sequence of images or text descriptions, z′ out denotes the output of the scaled dot-product attention mechanism;

[0021] The second part adds trainable matrices A1 and B1 after the linear layer of the attention module, and the formula is:

[0022] z out =Linear(z′ out )+B1A1z′ out

[0023] The third part identifies the modified CLIP image encoder f and text encoder g as f(x) and g(t), and x and t are the input image and the corresponding text description, respectively.

[0024] Optionally, the image prototype is obtained from the class center of the labeled source domain image by the image encoder, and specifically includes:

[0025] The image prototype is directly obtained from the class center of the labeled source domain image, and the formula is:

[0026]

[0027] wherein, denotes that the labeled source domain sample belongs to the label k class, denotes the number of samples.

[0028] Optionally, the text prototype is obtained by the text encoder, and specifically includes:

[0029] By inputting the text embedding into the text encoder of CLIP, the text prototype or text semantic feature The formula is:

[0030]

[0031] Optionally, an image prototype is used to generate class centers for the unlabeled source domain images and the unlabeled target domain images The formula is:

[0032]

[0033] wherein, represents the kth element calculated by the unlabeled image features and the image prototype when softmax.

[0034] The above-mentioned cross-domain remote sensing image retrieval method based on a semantic information enhanced model provided by the embodiment of the application has the following beneficial effects compared with the prior art:

[0035] In the prior art, providing labels for massive remote sensing image samples is unrealistic in real tasks, which seriously limits the practical operability of the UDA method, and makes the insufficient source domain label data greatly increase the difficulty of learning semantic distinguishable features.

[0036] And the image prototype is used to generate class centers for the unlabeled source domain images and the unlabeled target domain images According to the class centers The nearest center point classification is performed to generate the first pseudo label And the text prototype is used to generate the second pseudo label for the unlabeled source domain images and the unlabeled target domain images The problem of insufficient source domain label data in the prior art is solved, and the difficulty of learning semantic distinguishable features is overcome.

[0037] In addition, the softmax probability values of the first pseudo label And the second pseudo label are calculated in the Transformer module, the softmax probability values are aggregated into an average probability vector, and the cross-domain remote sensing image retrieval result is obtained according to the entropy value and the confidence score of the average probability vector, thereby improving the credibility and reliability of the cross-domain remote sensing image retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 FIG. 1 is a CLIP-CDRSIR framework schematic diagram of a cross-domain remote sensing image retrieval method based on a semantic information enhanced model provided in an embodiment;

[0039] Figure 2 FIG. 2 is a pseudo label generation mechanism schematic diagram of a cross-domain remote sensing image retrieval method based on a semantic information enhanced model provided in an embodiment;

[0040] Figure 3 A top k accuracy result diagram on DB1 of a cross-domain remote sensing image retrieval method based on a semantic information enhanced model provided in an embodiment;

[0041] Figure 4 A top k accuracy result diagram on DB2 of a cross-domain remote sensing image retrieval method based on a semantic information enhanced model provided in an embodiment;

[0042] Figure 5 A t-SNE diagram of AID→UCMD of a cross-domain remote sensing image retrieval method based on a semantic information enhanced model provided in an embodiment;

[0043] Figure 6 A t-SNE diagram of UCMD→AID of a cross-domain remote sensing image retrieval method based on a semantic information enhanced model provided in an embodiment;

[0044] Figure 7 A retrieval result visualization diagram of a cross-domain remote sensing image retrieval method based on a semantic information enhanced model provided in an embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0046] In an embodiment, a cross-domain remote sensing image retrieval method based on a semantic information enhanced model is provided, which comprises:

[0047] A data set of cross-domain remote sensing images composed of labeled source domain images, unlabeled source domain images and unlabeled target domain images is obtained.

[0048] In the original retrieval model constructed based on the semantic information enhanced model CLIP, a parallel low-rank decomposition matrix is added to the Transformer module used to connect the image encoder and the text encoder to form an improved retrieval model.

[0049] The cross-domain remote sensing image data set is input into the improved retrieval model, the image prototype is obtained from the class center of the labeled source domain image through the image encoder, and the text prototype is obtained from the class center of the labeled source domain image through the text encoder.

[0050] The class center is generated for the unlabeled source domain image and the unlabeled target domain image using the image prototype According to the class center The nearest center point classification is performed to generate the first pseudo label and generate the second pseudo-labels for the unlabeled source domain images and the unlabeled target domain images using text prototypes

[0051] The first pseudo-labels are calculated by the Transformer module and the second pseudo-labels The softmax probability values of the first pseudo-labels and the second pseudo-labels are aggregated into an average probability vector, and the retrieval results of the cross-domain remote sensing images are obtained according to the entropy value and the confidence score of the average probability vector.

[0052] 1. Method design

[0053] 1.1 Method construction

[0054] A novel domain adaptation method CLIP-CDRSIR is constructed, which uses a pre-trained visual language model CLIP to process the CDRISR task. For the first problem, the pseudo-labels of each unlabeled image are aligned using CLIP to improve the reliability of the distinguishable feature representations extracted. In particular, a dual-modality prototype guided pseudo-labeling mechanism (DMPGPL) is set, which generates reliable pseudo-labels for the unlabeled source domain and target domain from the set image prototypes and text prototypes. At the same time, the confidence score of these pseudo-labels is calculated to improve their credibility and reliability. For the second problem, the adaptation process of the CLIP model on cross-domain remote sensing images is completed by performing contrastive learning on image features and dual-modality prototypes. In addition, the LRS method is proposed, which integrates the confidence score as a weight into the contrastive loss, thereby reducing the impact of noisy pseudo-labels on the model. Considering that updating all parameters of the CLIP model requires high computational cost, the LoRA fine-tuning method is proposed to freeze the weights of the CLIP model, and a small number of trainable parameters are added to each layer in the Transformer framework.

[0055] 1.2 Method representation and definition

[0056] The SSCDRSIR setting contains three parts: a small number of labeled source domain images a large number of unlabeled source domain images and unlabeled target domain images where n sl , n su and n tare the number of labeled source domain images, unlabeled source domain images, and target domain images. The two domains share the same label space Y = {1, 2, ..., K}, but the data distribution is different, that is, P(X sl )≠P(X t ) and P(X su )≠P(X t ).

[0057] 1.3 Design of domain adaptation method based on CLIP enhancement

[0058] The proposed CLIP-CDRSIR method is introduced in detail from three aspects. The overall framework is shown in the figure below. Figure 1 As shown, Figure 1 (a) is the overall framework diagram. Text and image features are extracted from the image encoder and text encoder of CLIP, and the domain adaptation task is completed by aligning the image features to the corresponding prototype. Figure 1 (b) shows the details of the Transformer module using the LoRA fine-tuning method. The trainable parallel low-rank factorization matrix is ​​injected into the frozen Transformer module, and the details of the multi-head attention mechanism calculation are ignored in this module. First, the LoRA fine-tuning method is introduced according to the framework. Second, the construction process of the bimodal prototype and pseudo-label generation mechanism is demonstrated. Subsequently, the confidence scores of the pseudo-labels are evaluated. Finally, the proposed LRS method is used to perform contrastive learning, completing the learning of class-distinguishable features and domain alignment on a cross-domain remote sensing dataset.

[0059] The following is a detailed introduction to the CLIP-CDRSIR method from the perspectives of the CLIP method design based on LoRA fine-tuning, the pseudo-label generation mechanism design guided by bimodal prototypes, the contrastive learning method design with loss reweighting strategy, and entropy minimization.

[0060] (1) Design of CLIP method based on LoRA fine-tuning. LoRA assumes that there is a low intrinsic rank in the adaptive task, that is, it contains a lot of redundant information. It aims to inject a low-rank matrix into the trainable weights. The generated new vector is shown in formula (1):

[0061] h=W0x+BAx (1)

[0062] Among them, the pre-training weights Frozen, added matrix and is trainable, with a label rank r <<min(d, k). For LoRA in CLIP, a slight change is made from Figure 1 As can be seen from part (b), the injection matrix contains three parts of the Transformer module. For the first part, add the low-rank matrix to the pre-trained weight matrix Wq 、W k and W v In order to obtain new query, key and value in the attention calculation. Its calculation is shown in formula (2):

[0063] Q=W q z in +B q A q z in

[0064] K=W k z in +B k A k z in

[0065] V=W v z in +B v A v z in

[0066] z′ out =Attention(Q, K, V) (2)

[0067] Among them, z in represents the input sequence of images or text descriptions, Q, K, V are the matrices calculated respectively, and z′ out Represents the output of the scaled dot product attention mechanism. For the second part, the trainable matrices A1 and B1 are added to the linear layer after the attention module. Linear is a linear layer, and its calculation is shown in formula (3):

[0068] z ou t=Linear(z′ out )+B1A1z′ out (3)

[0069] Among them, z out It refers to the result obtained after processing by the linear layer.

[0070] Finally, for the reparameterization of the feedforward network, the matrix is ​​added to the MLP layer. For simplicity, the modified CLIP image encoder f and text encoder g are denoted as f(x) and g(t), where x and t are the input image and the corresponding text description, respectively.

[0071] (2) Design of pseudo-label generation mechanism guided by bimodal prototype

[0072] A reliable pseudo-label is constructed to alleviate the problem of severe lack of label supervision, and dual-modality prototypes are constructed to enhance the learning of image features and text semantic information. In order to better use the cross-modal ability of CLIP, dual-modality prototypes (DMP) are constructed, including image prototypes and text prototypes. Image prototypes are directly obtained from the class centers of labeled source domain images, and the calculation is shown in formula (4):

[0073]

[0074] wherein, represents that the labeled source domain sample belongs to the label k class, represents the number of samples. For the text prototype, from the text description“This is a satellite image 0fa[CLS]”, [CLS] can be represented as an exact class name in the remote sensing data set, such as farmland, baseball field and river, etc. According to the constructed text description, the entire text embedding W={w1, w2, …, w k} is obtained. By directly inputting the text embedding into the text encoder of CLIP, the text prototype or text semantic feature can be generated according to formula (5):

[0075]

[0076] Unlike traditional CLIP zero-shot classification, the pseudo-label generation mechanism is guided by DMP. Given a unlabeled dataset X u ={x|x∈X su} or X u ={x|x∈X t}, first use the image prototype to generate the class center for the unlabeled data, and the calculation is shown in formula (6):

[0077]

[0078] wherein, represents the kth element calculated by the unlabeled image feature and the image prototype in the softmax, and C is the image prototype;

[0079] Then, according to the nearest center point classification, the pseudo-label i of x is generated, as shown in formula (7):

[0080]

[0081] Similarly, the pseudo-label of the unlabeled sample x iAnother pseudo-label is generated with the help of the text prototype The calculation method is shown in formulas (8) and (9):

[0082]

[0083]

[0084] Figure 2 The above pseudo-label generation mechanism is shown in (a) Pseudo-label generation mechanism, two pseudo-labels are generated for each unlabeled sample in two ways. (b) Pseudo-label confidence score, the confidence score is high when the two predicted pseudo-labels are the same. In short, first, two pseudo-labels are generated for each unlabeled sample x and Second, the confidence of the pseudo-label sample is calculated, and the predicted pseudo-label and of the unlabeled sample is more reliable when they are consistent. Based on the above setting, first, the softmax probability values of the two pseudo-labels are calculated using the unlabeled class center point, marked as and Then, the average probability vector is aggregated and its entropy is calculated, and the calculation formulas are shown in (10) and (11):

[0085]

[0086]

[0087] where K is the number of classes. Finally, the confidence of sample x can be calculated by formula (12):

[0088]

[0089] As can be seen from Figure 2 , when the predicted pseudo-labels and are the same, for example, sample x a , the unlabeled sample is closer to the corresponding image prototype and text prototype, so that the entropy value is smaller and the confidence score is higher. Conversely, when the unlabeled sample is far away from one of the prototypes, for example, sample x b , the predicted and are inconsistent, which will result in a higher entropy value and a smaller confidence score. In short, two different pseudo-labels are generated for unlabeled data (including unlabeled source domain image samples and unlabeled target domain image samples). Considering the modal difference between images and texts, only the pseudo-label calculated by the image prototype is used as the supervision signal in the training process.

[0090] (3) Loss reweighting strategy-based contrastive learning method design

[0091] The contrastive learning is introduced to complete the class distinguishable feature learning and domain alignment. First, all image features are aligned with the corresponding DMP, and the alignment method is shown in formulas (13) and (14):

[0092]

[0093]

[0094] wherein, X = X sl ∪X su ∪X tu , and DMP related to sample x i The supervision signal of the contrastive loss comes from the labeled source domain image, the source domain image with pseudo label and the target domain image sample with pseudo label. In order to further distinguish the reliability of the pseudo label, the loss reweighting strategy is integrated into the contrastive loss, and the integration method is shown in formula (15):

[0095]

[0096] wherein, X u =X su ∪X tu , w i is the confidence score of x i In LRS, the model captures more semantic information from the samples with higher confidence, and less semantic information from the samples with lower confidence, so the contrastive learning can be represented by formula (16):

[0097]

[0098] In addition, in order to reduce the difference between the two modalities, the contrastive learning is performed in DMP, so that the prototypes of the same class are close to each other, and the prototype features of different classes are far away from each other, so as to reduce the difference between different modalities, which is shown in formula (17):

[0099]

[0100] (4) Entropy minimization

[0101] In order to make the image features close to the corresponding prototypes in an unsupervised manner, entropy minimization is used for all samples, and the calculation is shown in formula (18):

[0102]

[0103] The overall loss function is shown in formula (19):

[0104]

[0105] 1.4 Algorithm design

[0106] To sum up, a series of modules are designed to make the CLIP model better adapt to the cross-domain remote sensing image retrieval task, so as to improve the results of cross-domain remote sensing image retrieval. Mainly including the use of LoRA to fine-tune CLIP, better use of relevant information and reduce the large number of parameters generated in the training process. By proposing a dual-modal prototype guiding mechanism to construct pseudo-labels, generate dual-modal confidence scores to strengthen the alignment supervision of data feature distribution, propose to use loss re-weighting strategy to complete the class distinguishable feature learning and domain alignment of source domain and target domain, finally use entropy minimization to strengthen the learnability of feature prototype.

[0107] In order to better understand the whole training process, the overall algorithm process is shown.

[0108]

[0109]

[0110] The goal of the algorithm is to fine-tune the CLIP model through LoRA, so that it can learn the cross-domain remote sensing image feature representation, and realize cross-domain retrieval. It provides an effective semantic information enhancement and domain alignment mechanism for cross-domain remote sensing image retrieval. To get the final cross-domain remote sensing image retrieval result, the algorithm needs to be combined with other modules.

[0111] 2、Experimental results and analysis

[0112] 2.1 Experimental setup

[0113] To evaluate the effectiveness of the proposed CLIP-CDRSIR method in the cross-domain remote sensing image retrieval task, a large number of comparative and ablation experiments were conducted on 12 cross-domain retrieval tasks. The following details are described from the dataset, comparative method and experimental setup.

[0114] (1) Datasets. Two cross-domain datasets are constructed to verify the effectiveness of the CLIP-CDRSIR method. AID, UCMD and NWPU are selected to construct DB1. PatternNet, VBing and VarcGIS are selected to construct DB2. For convenience of exposition, these six datasets are denoted as A, U, N, P, VA and VB, respectively. For DB1, each dataset contains 1000 images of 10 classes, i.e., 100 images per class. Two different datasets are randomly selected as the source domain and the target domain, respectively, thus generating six cross-domain retrieval tasks, i.e., A→U, U→A, A→N, N→A, U→N and N→U. Among them, the symbol P→Q means that dataset P is the source domain and dataset Q is the target domain. For DB2, the categories provided by the three datasets are similar, so there is no need to extract subcategories. In the experiment, 800 images are randomly selected from each category and each dataset containing 30400 images, and then six cross-domain retrieval tasks are constructed, i.e., P→VB, VB→P, P→VA, VA→P, VB→VA and VA→VB.

[0115] (2) Compared methods. Six latest DA research methods are selected as the compared experimental methods, which are ADDA, JAN, SE, DATSNET, PCLOUDA and BSCA. Among them, ADDA, JAN and SE are traditional UDA methods, and BSCA is a novel SSDA method in the field of remote sensing images.

[0116] (3) Experimental detailed settings. The experiment uses the Pytorch framework. For the two cross-domain datasets, each class in the same domain is randomly sampled with a ratio of 80% and 20% as the training set and the test set. The training set is re-divided into training data and validation data with a ratio of 8:2. In the training phase, only a small part of labeled source domain images and unlabeled target domain images are used. In the experiment, three label ratios Ratio are set, with values set to 0.1, 0.3 and 1.0. When Ratio is 1.0, it means that all labeled source domain images are available. In the cross-domain retrieval phase, the test set extracted from the target domain is used as the query sample, and then 80% of the images are randomly extracted from the source domain as the retrieval dataset. The Euclidean distance is used to calculate the similarity between the query and all retrieval samples.

[0117] For fair comparison, all the compared methods use the CLIP image encoder ViT-B / 32 as the model framework, which means the input image is resized to 224x224 pixels and the extracted feature dimension is 512. For the proposed method, the CLIP model is fine-tuned by setting the LoRA rank coefficient r to 8, and the network is optimized using Adam with a fixed learning rate of le-4. Other hyperparameters such as the number of training epochs, batch size, and temperature coefficient τ are set to 30, 32, and 0.1, respectively. Meanwhile, a FT method is defined to directly fine-tune the entire CLIP parameters using the optimized model to verify the effectiveness of the LoRA fine-tuning mechanism. To comprehensively verify the performance of retrieval, the mAP, Topk (P@K) accuracy, and t-SNE high-dimensional feature visualization are used to evaluate the performance of the cross-domain retrieval task.

[0118] 2.2 Comparison experiment results and analysis

[0119] The results of the DB1 comparison experiment are shown in Table 1, which shows the mAP results of the CLIP-CDRSIR method on DB1 under different label ratio settings.

[0120] It is clear that as the label ratio increases, the retrieval results also gradually improve, with the mAP of the cross-domain retrieval improving by 0.95%, 3.29%, 0.86%, 3.23%, 3.52%, and 4.61% when Ratio = 1.0 compared to Ratio = 0.1. Because more real labels in the source domain provide more effective information during training, the model can better learn the feature distribution between domains. At the same time, it can also be noted that the proposed method not only performs better than other domain adaptation methods, but also exhibits strong stability under various label ratio settings.

[0121] This phenomenon can be attributed to the following reasons. First, the proposed DMPGPL provides more information during training. By performing contrastive learning on image features and text semantic information, the CLIP cross-modal capability can be better utilized.

[0122] Second, the LRS method also shows certain effects. By evaluating the reliability of each pseudo label, the model can pay more attention to correct label images during training. Finally, compared to the FT and LoRA fine-tuning methods, the LoRA fine-tuning method not only better demonstrates the generalization ability of the CLIP model, but also greatly reduces the number of parameters during training.

[0123] Table 1 DB1 comparison experiment results when Ratio = 0.1, 0.3, 1.0

[0124]

[0125] To demonstrate the operability of the proposed method CLIP-CDRSIR in practical applications, six cross-domain retrieval tasks are performed on the constructed 38-class DB2 retrieval dataset. The mAP results on DB2 are shown in Table 2.

[0126] Table 2 DB2 comparison experiment results when Ratio = 0.1, 0.3, 1.0

[0127]

[0128] The experimental results show that the CLIP-CDRSIR method not only shows high retrieval performance on the simple 10-class cross-domain task, but also shows excellent effectiveness on the complex 38-class cross-domain retrieval task. Compared with other domain adaptive methods, CLIP-CDRSIR shows better results in mAP results. When Ratio = 0.1, 0.3, 1.0, CLIP-CDRSIR shows strong retrieval performance, fully demonstrating the practicability of the constructed semi-supervised cross-domain retrieval scene task. In particular, when the label coefficient Ratio is set to 0.1, compared with the best results of other methods, the mAP results of CLIP-CDRSIR are improved by 9.2%, 6.8%, 13.15%, 5.3%, 3.9% and 5.0% respectively. With the help of the proposed DMPGPL method, the retrieval results show more robust characteristics with the increase of classification, which shows that the use of dual-modal prototype learning enhances the consistency learning ability of different domain image samples of the same class, and the use of LRS also makes the CLIP method better capture the class distinguishability features.

[0129] To further demonstrate the effectiveness of the proposed CLIP-CDRSIR method, Figure 3 and Figure 4The accuracy of the Top k(P@K) results is shown on the constructed DB1 and DB2 cross-domain datasets when the number of retrieval samples ranges from 5 to 100. As can be seen from the result figures of DB1 and DB2, the proposed CLIP-CDRSIR method shows strong effectiveness on simple and complex cross-domain remote sensing image retrieval tasks. In particular, when P = 5, the proposed method can retrieve the most matching images with the input, that is, the correct probability of the retrieval result is much higher than that of other comparative methods, showing strong retrieval accuracy. Using FT and LoRA fine-tuning can greatly improve the retrieval accuracy, which can be caused by the following reasons: (1) The pseudo-label mechanism guided by the dual-modal feature prototype and the loss reweighting strategy can better learn the feature space distribution of two domains for cross-domain remote sensing image retrieval, and the dual-modal learning can strengthen the use of pseudo-labels. And generate a confidence score as a weight to inject into the contrastive learning loss of the dual-modal prototype, further strengthening the consistency learning of different domains. (2) LoRA fine-tuning improves the processing and learning ability of CLIP for massive data, and effectively injects sample information into the three modules of Transformer, that is, low-rank matrices are injected into trainable weights.

[0130] To further explore the usability of the proposed method CLIP-CDRSIR in feature alignment and domain alignment, t-SNE figures are used to show the high-dimensional feature visualization figures of the above comparative experiments in the AID and UCMD cross-domain retrieval tasks, as shown in Figure 5 and 6The visualizations are shown respectively on A→U and U→A cross-domain retrieval tasks, where the label ratio Ratio is set to 0.1. As can be seen from the figure, the proposed method plays a great importance in domain alignment, and the FT fine-tuning method also shows a certain effectiveness, so that the high-dimensional features of the same class in the target domain and the source domain are close, and the high-dimensional features of different classes are far away, while other comparison methods still have obvious defects in domain class space alignment. Taking AID→UCMD as an example, in the comparison methods ADDA, JAN, SE, DATSNET, there are obvious defects between the source domain and the target domain samples, and there are still many classification errors, and the class space alignment ability is poor. The PLCUDA and BSCA methods have good consistency in space distribution, can learn the classification boundary between the source domain and the target domain of different classes, and the distribution between different classes can be clearly distinguished, but there are still some samples that are difficult to be correctly classified. The CLIP-CDRSIR method exhibits excellent retrieval performance compared with other comparison methods. In particular, compared with the FT fine-tuning method, the LoRA fine-tuning of a small part of useful parameters makes the two domains learn useful feature semantic information, and by freezing the pre-training weights, the LoRA injects a low-rank trainable matrix into the Transformer module to enhance the class distinguishable feature learning, which has obvious effectiveness in aligning cross-domain datasets. At the same time, the dual-modal prototype can be used to construct more reliable pseudo-labels. Therefore, the above comparison experiment results and visualizations show that the proposed CLIP-CDRSIR method exhibits high retrieval performance.

[0131] To further verify the effectiveness of the proposed method in the cross-domain retrieval task, Figure 7 The Top 10 Returned Results retrieval visualization results of the proposed method and the comparison experiments on the A→U cross-domain retrieval task are shown. Among them, Query represents the input retrieval image sample, Top 10 Returned Results represents the returned results of the retrieval database, and Methods represents the comparison methods and the proposed CLIP-CDRSIR method. In this retrieval visualization, Ratio is set to 0.1, and the green and red frames represent correct and incorrect returned samples, respectively. Randomly select three images as query samples, and then calculate the Top 10 Returned Results results on the A→U cross-domain retrieval task when Ratio is set to 0.1. Obviously, when a query image is given, the CLIP-CDRSIR method can retrieve the most similar samples, and compared with other comparison methods, the proposed method can effectively avoid returning visually similar but semantically different image samples, because the CLIP-CDRSIR method enriches the semantic information of the extracted image features, thereby exhibiting better retrieval performance.

[0132] 2.3 Ablation experiment results and analysis

[0133] The effectiveness of the proposed CLIP-CDRSIR method is verified by ablation experiments. As shown in Table 3, ablation experiments are performed on five variables in the CLIP-CDRSIR method, which are as follows: (1) V1 removes contrastive learning on image features and image prototypes, and only learns discriminative features and domain alignment through text prototypes. (2) V2 removes contrastive learning on image features and text prototypes, reducing the information of text semantic information in training. (3) V3 removes (4) V4 removes (5) V5 removes LRS, weakening the discriminability of the reliability of pseudo-labels. The mAP experimental results on the cross-domain remote sensing image retrieval tasks N→U, U→N, VB→VA and VA→VB are shown.

[0134] As can be seen from the table, all parts of CLIP-CDRSIR have certain influence on retrieval performance. V1 shows that only text semantic prototypes are not enough in domain adaptation tasks. V2 shows that the lack of text semantic information will weaken the performance results. Therefore, with the help of DMP, the contrastive learning is modified, and compared with the V3 results of CLIP-CDRSIR, plays an important role in aligning image prototypes and text prototypes, and can further improve the ability of contrastive learning. V4 reflects that the entropy minimization used can further enhance the ability of domain alignment and class discriminability learning in the training stage. V5 shows that the proposed LRS method can learn correct semantic information from noisy pseudo-label samples. In general, the execution of contrastive learning loss on image and text prototypes, the contrastive learning of dual-modal feature prototypes, and the LRS module all have certain influence on retrieval performance, and the ablation experiment results can also verify that in actual cross-domain remote sensing image retrieval, each module can have a greater improvement effect on the CLIP-CDRSIR method.

[0135] Table 3 Ablation experiment results

[0136] Method U→N N→U VB→VA VA→VB V1 0.8893 0.8865 0.8339 0.8896 V2 0.8899 0.8915 0.8287 0.8871 V3 0.9011 0.8939 0.8588 0.9058 V4 0.9001 0.8897 0.8503 0.8916 V5 0.9013 0.9036 0.8409 0.8761 CLIP-CDRSIR 0.9142 0.9106 0.8689 0.9116

[0137] 3. Summary

[0138] A novel domain adaptation scene task, SSCDRSIR, is proposed. To enhance the retrieval performance of SSCDRSIR, a simple and effective domain adaptation method, CLIP-CDRSIR, is proposed, which fine-tunes the CLIP model by using LoRA to complete the task of cross-domain remote sensing image retrieval. To alleviate the problem of severe lack of labeled data, firstly, the DMPGPL mechanism is proposed to generate pseudo labels, and the cross-modal feature prototype is constructed, and the CLIP model is used to enhance the learning of semantic information of the network model for cross-domain remote sensing image samples. Then, the confidence score of all unlabeled samples is estimated. At the same time, to further reduce the domain bias between the two domains and better learn the class discriminative features, the LRS method is proposed to drive the image features close to the corresponding DMP. The experimental results on 12 cross-domain remote sensing image retrieval tasks show that the proposed method is superior to the latest related research methods, and has strong retrieval superiority in practical scenarios.

[0139] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A cross-domain remote sensing image retrieval method based on a semantic information enhancement model, characterized in that: include: Obtain a dataset of cross-domain remote sensing images consisting of labeled source domain images, unlabeled source domain images, and unlabeled target domain images; In the original retrieval model built based on the semantic information enhancement model CLIP, a parallel low-rank decomposition matrix is ​​added to the Transformer module used to connect the image encoder and text encoder to form an improved retrieval model; The cross-domain remote sensing image dataset is input into the improved retrieval model, and the image prototype is obtained from the category center of the labeled source domain image through the image encoder, and the text prototype is obtained from the category center of the labeled source domain image through the text encoder; The image prototype is obtained from the category center of the labeled source domain image through the image encoder, specifically including: Image Prototype It is directly obtained from the category center of the labeled source domain image, and the formula is: in, Indicates that the labeled source domain sample belongs to the label kind, express The number of samples; Use image prototypes to generate class centers for unlabeled source domain images and unlabeled target domain images , according to the class center Perform the nearest center point classification to generate the first pseudo label , and use text prototypes to generate a second pseudo label for the unlabeled source domain image and the unlabeled target domain image ; The image prototype is used to generate class centers for unlabeled source domain images and unlabeled target domain images. , the formula is: in, Indicates the first value calculated by the unlabeled image features and image prototypes during softmax elements; Calculate the first pseudo label through the Transformer module and the second pseudo-label The softmax probability value is aggregated into an average probability vector, and the retrieval result of the cross-domain remote sensing image is obtained according to the entropy value and confidence score of the average probability vector.

2. The cross-domain remote sensing image retrieval method based on a semantic information enhancement model according to claim 1, characterized in that: The cross-domain remote sensing image dataset specifically includes: a small number of labeled source domain images , a large number of unlabeled source domain images and unlabeled target domain images ,in 、 and are the number of labeled source domain images, unlabeled source domain images, and target domain images, respectively.

3. The cross-domain remote sensing image retrieval method based on a semantic information enhancement model according to claim 2, characterized in that: The source and target domains have the same label space , the data distribution of the source domain and the target domain is different, that is, and .

4. The cross-domain remote sensing image retrieval method based on a semantic information enhancement model according to claim 1, characterized in that: Add a parallel low-rank decomposition matrix to the Transformer module, specifically including: The injection matrix consists of three parts of the Transformer module. The first part adds a low-rank matrix to 、 and To obtain new query, key and value in attention calculation, the formula is: in, is an input sequence of images or text descriptions, is the output of the scaled dot-product attention mechanism, Q 、 K 、 V is the calculated matrix; The second part adds a trainable matrix to the linear layer after the attention module and , the formula is: in, Linear is a linear layer; The third part will modify the CLIP image encoder and text encoders Marked as and , and are the input image and the corresponding text description respectively.

5. The cross-domain remote sensing image retrieval method based on a semantic information enhancement model according to claim 1, characterized in that: Obtain text prototypes through the text encoder, including: By inputting text embedding into CLIP’s text encoder, text prototypes or text semantic features , the formula is: 。

Images