A cross-domain remote sensing image retrieval method based on multi-center point feature prototype

Abstract

The application discloses a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype and relates to the technical field of image retrieval. θ (·) extract all image samples; generate the feature vectors of all image samples in each category of the source domain label data through a memory Memory and perform average operation to obtain the center point feature prototype of each category of the source domain label data; construct pseudo labels for the target domain unlabeled data based on the center point feature prototype, and assign a multi-center feature prototype to each category in the target domain unlabeled data by using the pseudo labels; compare the feature prototype with the multi-center feature prototype to obtain the retrieval result of the cross-domain remote sensing image. The application solves the domain deviation problem caused by the single-center point feature prototype due to category deviation and better explores the internal data distribution of the target domain.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image retrieval, in particular to a cross-domain remote sensing image retrieval method based on multi-center point feature prototype. BACKGROUND

[0002] Cross-domain remote sensing image retrieval technology plays a key role in environmental monitoring, urban planning, disaster assessment and other fields. The core challenge lies in how to effectively learn the data distribution difference between the source domain (known data distribution) and the target domain (to be retrieved data distribution). This is because different remote sensing data sources may be affected by various factors such as sensor type, shooting time, environmental conditions, etc., resulting in significant differences in data distribution. By learning the data distribution between the source domain and the target domain, a more robust and accurate cross-domain retrieval model can be constructed, enabling fast and accurate retrieval of unknown images in the target domain. This not only improves retrieval efficiency, but also broadens the application range of remote sensing images.

[0003] In the prior art, UDA method is used to realize cross-domain retrieval, and the specific process includes: collecting remote sensing image data of source domain (with label) and target domain (without label or with a small amount of label), extracting features from source domain and target domain data, constructing an UDA model containing feature extractor and domain classifier; training the model on the source domain data, and optimizing the model parameters using the domain classifier; using the trained UDA model to perform image retrieval on the target domain to obtain the retrieval result.

[0004] The defects of the above prior art are: when encountering a retrieval model with large cross-domain difference and many categories, the single-center feature prototype cannot effectively represent each category, resulting in that the class alignment method supervised by pseudo-label cannot guarantee reliability, causing the accumulation of false samples due to pseudo-label classification, and unable to strengthen the consistency between the source sample label and the pseudo-label, resulting in a large difference between the source domain and the target domain in data space distribution. SUMMARY

[0005] Therefore, it is necessary to provide a cross-domain remote sensing image retrieval method based on multi-center point feature prototype in view of the above technical problems.

[0006] The embodiment of the present application provides a cross-domain remote sensing image retrieval method based on multi-center point feature prototype, which comprises:

[0007] Obtain a cross-domain remote sensing image dataset comprising source domain label data and target domain unlabeled data;

[0008] In the original retrieval model constructed based on the cycle label consistency method CLC, a memory Memory is added after the feature extractor f θ (·) to form an improved retrieval model;

[0009] The improved retrieval model is inputted with the cross-domain remote sensing image dataset, and the feature extractor f θ (·) extracts all the classified image samples in the source domain labeled data and the target domain unlabeled data;

[0010] The feature vectors of all the image samples in each classification of the source domain labeled data are generated by the memory Memory, and the feature vectors of all the image samples in each classification are averaged to obtain the center point feature prototype C S of each classification of the source domain labeled data; S Pseudo labels are constructed for the target domain unlabeled data, and the pseudo labels are used to assign multi-center feature prototypes C T to each classification in the target domain unlabeled data;

[0011] The center point feature prototype C S of each classification in the source domain labeled data is compared with the multi-center feature prototype C T of each classification in the target domain unlabeled data, and the retrieval result of the cross-domain remote sensing image is obtained.

[0012] Optionally, the memory Memory includes: feature prototypes of each classification of all image samples in the source domain labeled data and multi-center point feature prototypes of each classification of all image samples in the target domain Wherein, there is only one center point feature prototype for each classification of all image samples in the source domain labeled data, and each classification of all image samples in the target domain unlabeled data has P multi-center feature prototypes, and K is the number of categories.

[0013] Optionally, the averaging operation on the feature vectors of all the image samples in each classification specifically includes:

[0014] It is assumed that there are N image samples in each classification of the source domain labeled data, and the feature vector of each image sample is represented as x i , wherein i=1, 2,..., N;

[0015] The average feature vector of each classification in the source domain labeled data is obtained, and the calculation formula is:

[0016]

[0017] Wherein, center is the average feature vector, Sum is the summation operation, N is the total number of image samples, and x i is the feature vector of each image sample;

[0018] The average feature vector of each classification is taken as the center point feature prototype C S of the classification.

[0019] Optionally, the center point feature prototype based on the source domain is used to construct pseudo labels for the unlabeled data in the target domain, and specifically includes:

[0020] Initialize the memory Memory to generate feature prototypes C of each category of all image samples in the source domain label data S , and the average feature vector of all samples in each category is summarized to obtain the center point feature prototype, and the calculation method is:

[0021]

[0022] wherein, is a set of source domain image sample labels belonging to the kth category, is the number of samples in the set, is the ith source domain sample;

[0023] Extract multiple categories of all image samples in the unlabeled data in the target domain, calculate the cosine similarity of each category to the center point feature prototype, and select the most similar center point feature prototype as the pseudo label for the unlabeled data in the target domain;

[0024] Perform k-means clustering processing on the pseudo label to assign multiple center feature prototypes to each category of all image samples in the unlabeled data in the target domain, and the calculation formula is:

[0025]

[0026] wherein, is the target domain sample, is a set of the kth category of the target domain sample data, is the center point feature prototype, argmax is the index operation of finding the maximum value, and Kmeans is the clustering operation.

[0027] Optionally, when the memory Memory is initialized, the center point feature prototype and the multiple center feature prototypes are updated, and specifically includes:

[0028] For the center point feature prototype, after extracting the features of all image samples in one category in each training, momentum update is adopted, and the calculation formula is:

[0029]

[0030] wherein, m s is the update speed coefficient, is a set of source domain image samples belonging to the kth category, is the number of all samples in the set;

[0031] For each sample in the source domain, a pseudo label is assigned, and in each training process k, the weighted momentum is used to update it, and the formula is:

[0032]

[0033]

[0034] wherein m t is the update speed coefficient, is the set of pseudo labels of the target domain image samples belonging to the kth class, is the number of all samples in the set, is the result of the clustering operation.

[0035] Optionally, the center point feature prototype C S of each classification of the obtained source domain label data is obtained.

[0036]

[0037] wherein, is the center point feature prototype corresponding to the class of the sample, τ is the temperature coefficient, and n s is the number of source domain image samples.

[0038] Optionally, it further includes: completing the cross entropy loss between the pseudo label and the real label of the generated source domain sample image by establishing the cyclic label consistency, and the process of establishing the cyclic label consistency specifically includes:

[0039] The similarity probability value prob of the source domain image sample to all multi-center point feature prototypes of the target domain is obtained, and the calculation formula is:

[0040]

[0041] wherein τ is the temperature coefficient, T is the matrix transposition operation, max is the maximum value operation, is the target domain sample, is the result of the clustering operation, p is the source domain sample image feature vector, P is the set of source domain sample image feature vectors, k is the set of the kth class of the source domain sample, K is the total number of classes of the source domain, and j is the index of the feature vector prototype;

[0042] The calculated probability value distribution is used to make the distribution of the same class between the source domain and the target domain similar, and the formula is:

[0043]

[0044] wherein, is an indication function, is a similarity probability value, K is the total number of categories of the source domain, n s is the number of source domain image samples, k is the set of the kth category of the source domain sample.

[0045] Optionally, further comprising: making each sample in the target domain and the respective corresponding feature prototype close through entropy minimization, and the formula of the entropy minimization is:

[0046]

[0047] Based on the optimization loss function is:

[0048] L=L cl +λ1L cycle +λ2L ent

[0049] Wherein, λ1 and λ2 are balance coefficients, E is a hyperparameter, L cl is a contrast loss, L cycle is a cycle loss, L ent is entropy minimization.

[0050] The above-mentioned one kind of cross-domain remote sensing image retrieval method based on multi-center point feature prototype provided by the embodiment of the application has the following beneficial effects compared with the prior art:

[0051] In the prior art, the single-center feature prototype cannot effectively represent each category, resulting in that the class alignment method supervised by the pseudo label cannot guarantee reliability, causing the accumulation of error samples due to pseudo label classification, and the consistency between the labels and the pseudo labels of the source samples cannot be strengthened, so that there is a large difference between the source domain and the target domain in the data space distribution.

[0052] And the center point feature prototype C S is a pseudo label constructed for the target domain unlabeled data, and the pseudo label is used to assign the multi-center feature prototype C T to each classification in the target domain unlabeled data, so as to strengthen the consistency between the labels and the pseudo labels of the source samples, solve the domain deviation problem caused by the category deviation of the single-center point feature prototype, and better explore the internal data distribution of the target domain, reasonably utilize all image samples of the target domain, and reduce the difference between the source domain and the target domain in the data space distribution.

[0053] In addition, when the pseudo label information is used to help construct the multi-center feature prototype of the target domain, the pseudo label does not participate in the entire training process, avoiding the accumulation of error samples caused by pseudo label classification. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1A contrast diagram of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment;

[0055] Figure 2 A cycle label consistency framework diagram of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment;

[0056] Figure 3 A supervised signal diagram of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment;

[0057] Figure 4 An algorithm overall design framework diagram of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment;

[0058] Figure 5 An accuracy result diagram of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment;

[0059] Figure 6 A t-SNE diagram of AID->UCMD of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment;

[0060] Figure 7 A t-SNE diagram of UCMD->AID of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment. DETAILED DESCRIPTION

[0061] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0062] In an embodiment, a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype is provided, and the method comprises:

[0063] 1. Method design

[0064] 1.1 Design based on multi-center point feature prototype

[0065] By constructing multi-center feature prototypes for each class of the target domain, the negative transfer caused by single-center feature prototypes is alleviated, and more robust and accurate labels can be assigned to each class of the target domain using multi-center point features. When constructing multi-center point features using all image samples of the target domain and constructing multi-center point feature prototypes using pseudo-label information, the pseudo-labels are not directly involved in model training, avoiding the direct use of pseudo-labels, better utilizing pseudo-label information, and reducing the noise misleading caused by pseudo-labels. Figure 1 It can be seen that by constructing multi-center feature prototypes, the classification boundaries of each class in the target domain can be better learned, the classification error is greatly reduced, and after learning by the network model, the data distribution between each class is more compact, and the data distribution between different classes is more separated, so that the distribution of the data domain is better learned, thereby improving the performance of retrieval.

[0066] 1.2 Design based on cycle label consistency

[0067] A cycle label consistency method (CLC) is proposed to realize domain alignment, and the algorithm framework diagram is as shown in Figure 2 The CLC method is different from existing UDA research, and directly uses the given real label as network supervision information. As shown in Figure 2 First, the CLC algorithm generates feature prototypes for each classification of the source domain, and the center point feature prototype of each classification is represented by the average feature vector of all samples of the class. According to the center point feature prototype of the source domain, pseudo-labels are constructed for the target domain using nearest neighbor classification.

[0068] The average feature vector is calculated according to the sample feature vectors of each classification in the source domain. The specific operation is to average the feature vectors of all samples under the classification. The confirmation method of the average feature vector includes: for a certain classification, assuming that there are N samples in the source domain, and the feature vector of each sample is represented as x_i, where i=1, 2,..., N. Then the center point feature prototype of the classification can be obtained by calculating the average feature vector, that is:

[0069]

[0070] Where center is the average feature vector, Sum is the summation operation, N is the total number of image samples, and x i is the feature vector of each image sample.

[0071] Then, the target domain center point feature prototypes are updated in the same way, and each source domain sample is assigned to the nearest target domain center point feature prototype to obtain pseudo labels. Finally, cross-entropy is used to strengthen the consistency learning of the pseudo labels and the real labels. In short, the cycle label consistency is only to calculate the target domain center point using the pseudo label and transmit it to the source domain, which alleviates the false information of the mislabeled samples and fully performs the backward information propagation to obtain a more compact data domain distribution.

[0072] Figure 3 For the schematic diagram of the supervised signal, the information of the real label of the source domain is used as the supervised signal to explore the probability distribution of each source domain image sample to the target domain center point feature prototype. The consistency of the probability distribution and the supervised signal is learned using the proposed cycle label consistency method, so that the center point feature prototypes of the same class in the two domains are close to each other, and a consistent expression is achieved in the data distribution space.

[0073] 1.3 Overall design

[0074] The overall design framework is shown in Figure 4 . Given a labeled source domain dataset and an unlabeled target domain n s and n t are the number of source domain and target domain image samples, respectively. The main purpose is to complete the cross-domain remote sensing image retrieval task using multi-center point feature prototypes and the CLC algorithm. In the cross-domain remote sensing image retrieval task, the features of the image samples are first extracted by the feature extractor f θ (·). The network model weights are shared by the source domain and the target domain. Secondly, the multi-center point method is used to generate multiple feature prototypes for each class of the target domain, and the CLC algorithm is used to strengthen the consistency of the source domain and the target domain sample data distribution space. At the same time, a memory Memory is defined to save the center point feature prototypes of the source domain and the target domain during network training, and is used for dynamic updating of the multi-center point. The source domain with label information is used to assign pseudo labels to each image sample of the target domain, and then the pseudo labels are used to assign multiple center feature prototypes to each class of the target domain, thereby reducing the negative transfer caused by the noise of the pseudo labels. At the same time, the CLC algorithm is used to use the real label of the source domain as the supervised signal in training, to strengthen the consistency between the real label and the pseudo label of the sample, so as to better learn the cross-domain data distribution and improve the retrieval performance.

[0075] The overall algorithm design is described in detail below.

[0076] (1) Constructing the memory Memory

[0077] First, the memory Memory is constructed, wherein the memory Memory contains the feature prototypes of each classification of all image samples of the source domain and target domain all image samples each classification of multi-center point feature prototype Wherein the source domain each class only has one feature prototype, the target domain each class has P center point feature prototype, K is the number of categories. Secondly, the memory Memory is initialized, for the feature prototype of the source domain, after extracting the features of all source domain image samples, the average value of each class feature is calculated and saved to the memory Memory, and the calculation method of the source domain center point feature prototype is shown in formula (1):

[0078]

[0079] Wherein, Indicates the set of the source domain image sample label belongs to the kth class, Indicates the number of samples in the set, Indicates the i th source domain sample. For the feature prototype of the target domain, after extracting the features of all target domain samples, the cosine similarity of each feature to the source domain center point feature prototype is calculated, the most similar feature prototype is selected as the pseudo label of the target domain image sample, and then k-means clustering is performed, so that P feature prototypes are obtained for each class of the target domain, and the pseudo label and feature prototype calculation are shown in formula (2) and (3):

[0080]

[0081] Wherein, Indicates the target domain sample, Indicates the set of the kth class of the target domain sample data, arg is a variable, and c k The label of the kth class or the center point of the class.

[0082] Finally, the memory Memory will update the stored source domain and target domain center point feature prototypes in each round of training. For the feature prototype of the source domain, after extracting the features of all image samples in a batch in each training, momentum update is adopted according to the label information, and the update calculation method is shown in formula (4):

[0083]

[0084] Wherein, m s Is the update speed coefficient, Indicates the set of the kth class of the source domain image sample in a batch, Indicates the number of all samples in the set. In addition, for the feature prototype of the target domain, after giving the pseudo label to each sample of the source domain, the weighted momentum is adopted to update it in each round of training k, and the update method is shown in formula (5) and (6):

[0085]

[0086]

[0087] where m t is the update speed coefficient, denotes the set of pseudo-labels of target domain image samples in a batch belonging to the k-th class, denotes the number of all samples in the set.

[0088] (2) Source domain feature prototype contrastive learning

[0089] After calculating the center point feature prototype of each classification of the source domain , it is necessary to strengthen the learning of each image sample and the feature prototype of the classification. Here, a contrastive learning loss function is used to utilize the semantic information learned from the source domain, so that each sample in the source domain is close to the feature prototype of the respective classification. The implementation is shown in equation (7):

[0090]

[0091] where, is the center point feature prototype corresponding to the sample category, τ is the temperature coefficient, exp is the exponential function, and f θ (·) is the operation of the feature extractor.

[0092] (3) Multi-center feature prototype consistency loss

[0093] A. The single-center point feature prototype of the source domain and the multi-center point feature prototype of each classification of the target domain unlabeled data are calculated and stored in the memory. The consistency between the pseudo-labels and the real labels in the source domain needs to be preserved, so the cross-entropy loss between the pseudo-labels and the real labels of the source domain samples is minimized by establishing a cyclic label consistency, so as to use the real labels as a supervision signal to optimize the model.

[0094] The first step is to calculate the similarity probability value of the source domain image sample to all multi-center point feature prototypes of the target domain where The calculation is shown in equation (8):

[0095]

[0096] B. Using the above calculated probability value distribution, the following formula is used to ensure that the distribution of the same category between the source domain and the target domain is similar, and the calculation formula is shown in equation (9):

[0097]

[0098] wherein, is an indicator function, which is 1 when and 0 otherwise. The "soft" soft label is adopted, and compared with the "hard" hard label, the soft label calculation uses more strict constraints, can learn the consistency of the data distribution better, strengthens the learning ability of the network to the sample pseudo label and the real label, makes the distribution of the same class gradually close, and the data distribution not belonging to the same class gradually far away, so that the distribution of each class is more compact.

[0099] The consistency learning of the source domain image sample and the feature prototype thereof is completed by using the contrast loss, the pseudo label of the source domain image is generated by using the multi-center point feature prototype of the target domain, and then the cross entropy learning of the source domain image sample and the pseudo label thereof is completed by using the cycle label consistency. Thus, the same class data distribution gradually closes, and the distribution learning of the same class sample is completed.

[0100] (4) Entropy minimization

[0101] In the algorithm design, in order to further explore the internal structure of the target domain data, each sample in the target domain and the respective corresponding feature prototype are made to be closer, and the entropy minimization is used to achieve the purpose. The calculation is as shown in formula (10):

[0102]

[0103] The final algorithm optimization loss function is as shown in formula (11):

[0104] L = L cl + λ1L cycle + λ2L ent

[0105] Wherein, λ1 and λ2 are balance coefficients.

[0106] 2. Experimental results and analysis

[0107] 2.1 Experimental setting

[0108] In order to verify the feasibility and effectiveness of the multi-center point feature prototype and the CLC method, a large number of comparison experiments and ablation experiments are carried out to verify the effectiveness of the proposed method. The following is the related setting of the experiment, which mainly introduces the data set, the comparison method, the experimental environment and the parameters in detail.

[0109] (1) Datasets. Three remote sensing image datasets, AID, UCMD and NWPU, are used to complete six cross-domain retrieval tasks. Among them, each dataset provides 1000 images of 10 categories, i.e. 100 images per category, including categories such as baseball field, beach, dense residential, forest, parking lot, river, sparse residential and storage tank. In the experiment, two different datasets are randomly selected as the source domain and the target domain, respectively, thereby generating six cross-domain remote sensing image retrieval tasks, AID→UCMD, UCMD→AID, AID→NWPU, NWPU→AID, UCMD→NWPU and NWPU→UCMD, which are briefly identified as A→U, U→A, A→N, N→A, U→N and N→U, respectively. Among them, the symbol P→Q represents that dataset P is used as the source domain and dataset Q is used as the target domain.

[0110] (2) Comparative methods. Seven methods, ADDA, DAN, JAN, MDD, SE, PLCUDA and source only, are selected as comparative methods. Among them, ADDA trains an asymmetric feature extractor through adversarial learning to confuse the domain discriminator; DAN and JAN use MMD and joint MMD for statistical difference minimization to reduce the difference between the two domains; MDD introduces a new difference, called marginal difference, to measure the distribution difference; SE designs a teacher-student framework and reduces the prediction difference between the two networks to align the two domains; PLCUDA reduces the probability distribution difference between the target domain and its perturbed output through pseudo-label self-training and consistency regularization strategy; in addition, a source only method is designed, which only uses the labeled data of the source domain to train the cross-domain remote sensing image retrieval task model.

[0111] (3) Experimental environment and parameter setting. The operating system of the machine used in the experiment is Ubuntu 18.04, the processor is Intel Core 9, the memory is 32 GB, and the graphics card is NVIDIA Tesla T4. The programming language used is Python 3.8, and the deep learning framework is Pytorch 1.13. Each of the above-mentioned methods is carried out in the same experimental environment. For experimental data division, we first uniformly take 80% of the data from both domains, and then take 80% of these data as the training set and 20% as the validation set. For the other 20% of the data, the sample data from the target domain is taken as the query set, and the database is sampled from 80% of the data in the entire source domain. In the experiment, we use the ResNet34 model pre-trained by ImageNet as the feature extractor, so each picture input is resized to 224x224. In addition, we use the TrvialAugment data augmentation strategy to improve the generalization ability of the model. In the training process, the batch size of each batch is set to 64, the learning rate lr is set to 1e-5, and the total number of training rounds epoch is 50. Other hyperparameters such as m s 、m t 、λ1 and λ2 are set to 0.7, 0.7, 0.1, 0.8 and 0.2 respectively. In the testing process, we use the feature extractor to extract the features of the query set and the database samples, and then measure the Euclidean distance between the features of the query set and each database feature to find the most similar sample from the database. For experimental results, we use mAP and Topk (P@K) to measure the results of all methods. Finally, the t-SNE high-dimensional feature distribution graph is shown to intuitively feel the effect of the proposed method.

[0112] 2.2 Comparison of experimental results and analysis

[0113] The comparison of the proposed method and other model methods in six cross-domain remote sensing image retrieval tasks is shown in Table 1.

[0114] Table 1 Comparison of experimental results

[0115]

[0116]

[0117] According to the data in the table, the proposed method can obtain 88.01%, 90.66%, 91.18%, 95.89%, 75.91% and 87.57% results on the mAP index of the six cross-domain remote sensing image retrieval tasks, all of which are better than the other nine comparison methods, which are 5.82%-24.40%, 1.93%-31.63%, 6.39%-22.18%, 3.95%-21.73%, 6.03%-25.62%, 4.38%-29.22%, 7.18%-14.72% and 1.19%-9.59% higher, respectively, which has obvious improvement in retrieval performance. Compared with the latest method PLCUDA, the proposed method explores the data distribution of the target domain by constructing a multi-center point feature prototype, and does not directly use the pseudo label to participate in the training network, while PLCUDA uses the pseudo label to train the network, which has many noise disturbances, making the pseudo label classification for unlabeled samples produce many negative transfers, so the PLCUDA method still has some defects in cross-domain retrieval accuracy. In the comparison experiment, the proposed method is better than the method in the mAP performance index, and the retrieval performance is greatly improved, especially in the U→A, A→N and N→A three comparison methods, the proposed method is compared with the PLCUDA method, which improves by 8.76%, 13.73% and 10.01%, showing a large improvement effect, which shows that the multi-center point can fully explore the data space distribution of the target domain. At the same time, the cycle label consistency algorithm further strengthens the consistency learning between different domains.

[0118] In order to further show the effectiveness of the proposed method, Figures 3-5 The Top k(P@K) results of the six cross-domain remote sensing image retrieval tasks are shown, where K=5, 10, 20, 50, 100. As can be seen from the figure, the proposed method has obvious improvement in the accuracy of Top k(P@K) on the six cross-domain retrieval tasks, which is higher than the other seven comparison methods. From the t-SNE high-dimensional feature visualization, compared with the latest cross-domain retrieval method PLCUDA, when K=5, 10, 20, 50, 100, the proposed method shows strong consistency in class data distribution learning, and the source domain and target domain of the same class data distribution are close, and the distribution of different classes is far away, showing good cross-domain consistency learning, and showing good learning ability in class alignment. The cycle label consistency loss function used strengthens the consistency between the target domain real label and the target domain multi-center point feature prototype, which makes the same class samples between different domains close to the same data distribution, learns the statistical distribution difference between different domains, and avoids the error caused by directly training the network model with pseudo labels, and eliminates the noise interference to the retrieval model.

[0119] The proposed method achieves the best performance in mAP and Top k (P@K) retrieval performance in the experimental results shown. This fully demonstrates that the feature extractor trained by our method has excellent feature extraction capability in the cross-domain remote sensing image retrieval scene. When an input query image is input, our method can more easily retrieve similar samples in the database to the query image. Not only that, these retrieved similar samples can usually occupy a more forward position in the similarity ranking. This advantage is crucial to improving the overall performance of cross-domain remote sensing image retrieval, as it ensures that users can more quickly and accurately obtain highly relevant results to the query image.

[0120] To further study the role of the proposed method in feature alignment, t-SNE is used to visualize the high-dimensional features of the above comparative experimental methods on two datasets (i.e., AID and UCMD), Figures 3-6 and Figures 3-7 are the high-dimensional feature visualization graphs of AID→UCMD and UCMD→AID cross-domain retrieval tasks, respectively. As can be seen from the graphs, the source only method has a large feature confusion between the source domain and the target domain, making it difficult to learn the retrieval boundary of the class and unable to train a better retrieval model. Some DA methods such as DAN, JAN and MDD can weaken the feature bias to some extent, and in these DA methods, the distance between different classes in the target domain is small, the decision boundary is more blurred, and there are many samples that are classified incorrectly, indicating that these contrast methods have obvious defects in class alignment. Obviously, the proposed method successfully overcomes the feature bias between the two domains, and the construction of multi-center point feature prototypes makes the same class distribution closer to each other and the different class space distribution far away from each other, significantly increasing the class separability of the target domain data, better solving the problem of large inter-class similarity of remote sensing images, thereby encouraging the network to learn more discriminative feature representations of the target domain, making the learned network have stronger generalization ability, greatly reducing the number of classification errors. In addition, the cyclic label consistency algorithm strengthens the consistency of the data distribution between the source domain and the target domain during network model training, and through the implementation of cyclic label allocation, the real label of the source domain is used as a supervision signal, and a contrast learning loss function is used to strengthen the consistency between it and the pseudo label, better utilizing the pseudo label information, avoiding the negative transfer result, and greatly improving the performance of cross-domain retrieval.

[0121] 2.3 Ablation experiment results and analysis

[0122] To further verify the algorithms designed by the proposed method, the following modules are subjected to ablation experiments: (1) V1 explores the entropy minimization loss function (2) V2 explores the cyclic label consistency loss function (3) At the same time, explore the number of multi-center point feature prototypes P, V3, V4, V5 respectively take P = 1, 2, 3, study the effectiveness of multiple center points on cross-domain retrieval. Adopting ablation experiment verification on four cross-domain retrieval tasks UCMD→NWPU, NWPU→UCMD, AID→UCMD and UCMD→AID, and the performance evaluation index is mAP. The ablation experiment results are shown in Table 2.

[0123] Table 2 Ablation experiment

[0124]

[0125] From the table data, and The loss function has good feasibility on the six cross-domain retrieval tasks, and greatly improves the accuracy of retrieval. Especially the cycle label consistency loss function, compared with 5.26%, 11.66%, 12.18%, 8.95%, it is obvious that great advantage is obtained on the cross-domain remote sensing image retrieval task, and the retrieval performance can be greatly improved. In addition, for the number of multi-center point feature prototypes, it can be seen that when a single center point is set for retrieval, the retrieval performance is obviously poorer than two and three center points, and when the number of center points P = 3, the mAP value is better than that of two center points. For two center points, the mAP is improved by 3.96%, 0.97%, 0.14%, and 0.06% respectively in four cross-domain retrieval tasks.

[0126] 3. Summary

[0127] A method of using multi-center point feature prototypes and cycle label consistency to complete the cross-domain remote sensing image retrieval task is proposed. The method fully explores the data distribution of the target domain, constructs multiple center point feature prototypes for each classification, then assigns pseudo-labels to the source domain, and then uses the cycle label consistency algorithm to strengthen the consistency of the real label and the pseudo-label of the source domain, learns the domain invariant feature, and makes the data distribution of the same class closer and closer, and the data distribution of different classes far away, so that the learned model is more compact. In the comparison experiment, the ablation experiment, Top k(P@K) and t-SNE results, the proposed method has obvious effect, and the retrieval performance is better than that of the latest cross-domain retrieval method.

[0128] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as limiting the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A cross-domain remote sensing image retrieval method based on multi-center point feature prototype, characterized in that: include: Obtain a cross-domain remote sensing image dataset that includes source domain labeled data and target domain unlabeled data; In the original retrieval model based on the cyclic label consistency method CLC, in the feature extractor Then add memory to form an improved retrieval model; The cross-domain remote sensing image dataset is input into the improved retrieval model and the feature extractor is used to extract the Extract image samples of all categories from source domain labeled data and target domain unlabeled data; Generate the feature vectors of all image samples in each category of the source domain label data through the memory, average the feature vectors of all image samples in each category, and obtain the center point feature prototype of each category of the source domain label data. C S ; Based on the center point feature prototype C S Construct pseudo labels for the target domain unlabeled data, and use the pseudo labels to assign multi-center feature prototypes to each category in the target domain unlabeled data C T ; Construct pseudo labels for unlabeled data in the target domain based on the center point feature prototype of the source domain, which specifically includes: Initialize the memory and generate the feature prototypes of each category of all image samples in the source domain label data C S , sum up the average feature vectors of all samples in each category and get the center point feature prototype, which is calculated as follows: in, The source domain image sample label belongs to A collection of classes, is the number of samples in the set, For the source domain samples; Extract multiple categories of all image samples in the target domain unlabeled data, calculate the cosine similarity of each category to the center point feature prototype, and select the center point feature prototype with the closest similarity to construct a pseudo label for the target domain unlabeled data; Perform k-means clustering on the pseudo labels to assign multi-center feature prototypes to each category of all image samples in the target domain unlabeled data. The calculation formula is: in, is the target domain sample, The target domain sample data A collection of classes, is the prototype of the center point feature, argmax is the index operation with the maximum value, Kmeans is the clustering operation; The center point feature prototype of each category in the source domain label data C S and the multi-center feature prototypes for each category in the target domain unlabeled data C T Compare and obtain the retrieval results of cross-domain remote sensing images; It also includes: completing the cross entropy loss between the pseudo labels and true labels generated by the source domain sample image by establishing cyclic label consistency. The process of establishing cyclic label consistency specifically includes: Obtain the similarity probability value of the source domain image sample to all multi-center point feature prototypes in the target domain prob , and its calculation formula is: in, is the temperature coefficient, T is the matrix transpose operation, max To find the maximum value, is the target domain sample, is the result of the clustering operation, p is the source domain sample image feature vector, P is the set of source domain sample image feature vectors, k is the set of the kth category of source domain samples, K is the total number of categories in the source domain, j is the index of the feature vector prototype; The calculated probability value distribution is used to make the distribution of the same category in the source domain and the target domain similar. The formula for the cycle loss is: in, is the indicator function, is the similarity probability value, K is the total number of categories in the source domain, is the number of source domain image samples, k is the set of the kth category of source domain samples.

2. The cross-domain remote sensing image retrieval method based on multi-center point feature prototype according to claim 1, characterized in that: The memory includes: feature prototypes of each classification of all image samples in the source domain label data And the multi-center feature prototype of each category of all image samples in the target domain , where each category of all image samples in the source domain labeled data has only one center feature prototype, and each category of all image samples in the target domain unlabeled data has A multi-center feature prototype, is the number of categories.

3. The cross-domain remote sensing image retrieval method based on multi-center point feature prototype according to claim 1, characterized in that: The averaging operation of the feature vectors of all image samples in each category specifically includes: Assume that each category in the source domain label data has N image samples, and the feature vector of each image sample is expressed as x i ,in i =1,2,..., N ; Get the average feature vector of each category in the source domain label data. The calculation formula is: in, center is the mean eigenvector, Sum For the sum operation, N is the total number of image samples, x i is the feature vector for each image sample; The average feature vector of each category is used as the center point feature prototype of the category C S .

4. The cross-domain remote sensing image retrieval method based on multi-center point feature prototype according to claim 1, characterized in that: When the memory is initialized, the center point feature prototype and the multi-center feature prototype are updated, which specifically includes: For the center point feature prototype, after extracting the features of all image samples in a category in each training, momentum update is used. The calculation formula is: in, is the update speed coefficient, The source domain image sample belongs to A collection of classes, is the number of all samples in the set; For multi-center feature prototypes, after assigning pseudo labels to each sample in the source domain, The weighted momentum is used to update it, and the formula is: in, is the update speed coefficient, The pseudo label of the target domain image sample belongs to A collection of classes, is the number of all samples in the set, is the result of the clustering operation.

5. The cross-domain remote sensing image retrieval method based on multi-center point feature prototype according to claim 1, characterized in that: The center point feature prototype of each classification of the source domain label data is obtained C S Finally, the contrastive learning loss function is used to make each sample in the source domain close to the feature prototype of its respective classification. The contrastive loss formula is: in, For samples The center point feature prototype corresponding to the category, is the temperature coefficient, is the target domain sample, is the number of source domain image samples.

6. The cross-domain remote sensing image retrieval method based on multi-center point feature prototype according to claim 1, characterized in that: Also includes: By minimizing entropy, each sample in the target domain is brought close to its corresponding feature prototype. The formula for entropy minimization is: Based on the optimization loss function: in, and is the balance coefficient, E is a hyperparameter, L cl is the contrast loss, L cycle is the circulation loss, L ent To minimize entropy.

Images