Cross-domain remote sensing image retrieval method based on multi-center-point feature prototype

Abstract

The invention discloses a cross-domain remote sensing image retrieval method based on a multi-center-point feature prototype, and relates to the technical field of image retrieval. Comprising the following steps: acquiring a data set of cross-domain remote sensing images, and constructing an improved retrieval model; inputting a data set of cross-domain remote sensing images into the retrieval model, and extracting all image samples through a feature extractor f theta (.); generating feature vectors of all image samples in each classification of the source domain label data through a memory, and performing averaging operation to obtain a center point feature prototype of each classification of the source domain label data; constructing a pseudo label for the target domain unlabeled data based on the center point feature prototype, and distributing a multi-center feature prototype for each category in the target domain unlabeled data by using the pseudo label; and comparing the feature prototype with the multi-center feature prototype to obtain a retrieval result of the cross-domain remote sensing image. According to the method, the problem of domain deviation caused by category deviation of a single-center-point feature prototype is solved, and data distribution in a target domain is better explored.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval technology, and in particular to a cross-domain remote sensing image retrieval method based on multi-center point feature prototypes. Background Art

[0002] Cross-domain remote sensing image retrieval technology plays a key role in environmental monitoring, urban planning, disaster assessment and other fields. The core challenge lies in how to effectively learn the data distribution differences between the source domain (known data distribution) and the target domain (data distribution to be retrieved). This is because different remote sensing data sources may be affected by many factors, such as sensor type, shooting time, environmental conditions, etc., resulting in significant differences in data distribution. By learning the data distribution between the source domain and the target domain, a more robust and accurate cross-domain retrieval model can be constructed, thereby achieving fast and accurate retrieval of unknown images in the target domain. This not only improves the retrieval efficiency, but also broadens the application scope of remote sensing images.

[0003] In the prior art, the UDA method is used to realize cross-domain retrieval, and its specific process includes: collecting remote sensing image data of the source domain (with labels) and the target domain (without labels or with a small number of labels), extracting features from the source domain and target domain data, and building a UDA model including a feature extractor and a domain classifier; training the model on the source domain data, and optimizing the model parameters using the domain classifier; using the trained UDA model to perform image retrieval in the target domain to obtain retrieval results.

[0004] The defects of the above-mentioned existing technologies are: when encountering a retrieval model with large cross-domain differences and a large number of categories, the single-center feature prototype cannot effectively represent each category, resulting in the class alignment method supervised by pseudo-labels cannot guarantee reliability, causing the accumulation of erroneous samples classified by pseudo-labels, and unable to strengthen the consistency between the labels of source samples and pseudo-labels, resulting in large differences in data space distribution between the source domain and the target domain. Summary of the invention

[0005] Based on this, it is necessary to provide a cross-domain remote sensing image retrieval method based on multi-center point feature prototype to address the above technical problems.

[0006] The embodiment of the present invention provides a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype, comprising:

[0007] Obtain a cross-domain remote sensing image dataset that includes source domain labeled data and target domain unlabeled data;

[0008] In the original retrieval model based on the cyclic label consistency method CLC, in the feature extractor f θ (·) and then add memory to form an improved retrieval model;

[0009] The cross-domain remote sensing image dataset is input into the improved retrieval model and the feature extractor f θ (·) Extract image samples of all categories in the source domain labeled data and the target domain unlabeled data;

[0010] Generate the feature vectors of all image samples in each category of the source domain label data through the memory, and average the feature vectors of all image samples in each category to obtain the center point feature prototype C of each category of the source domain label data S ; Based on the center point feature prototype C S Construct pseudo labels for the target domain unlabeled data, and use the pseudo labels to assign a multi-center feature prototype C to each classification in the target domain unlabeled data T ;

[0011] The center point feature prototype C of each category in the source domain label data S and the multi-center feature prototype C for each category in the target domain unlabeled data T By comparison, the retrieval results of cross-domain remote sensing images are obtained.

[0012] Optionally, the memory Memory includes: feature prototypes of each classification of all image samples in the source domain label data And the multi-center feature prototypes of each category of all image samples in the target domain Among them, all image samples in the source domain labeled data have only one center point feature prototype for each category, and all image samples in the target domain unlabeled data have P multi-center feature prototypes for each category, and K is the number of categories.

[0013] Optionally, the averaging operation is performed on the feature vectors of all image samples in each category, which specifically includes:

[0014] Assume that each category in the source domain label data has N image samples, and the feature vector of each image sample is represented by x i , where i = 1, 2, ..., N;

[0015] Get the average feature vector of each category in the source domain label data. The calculation formula is:

[0016]

[0017] Among them, center is the average feature vector, Sum is the summation operation, N is the total number of image samples, x i is the feature vector for each image sample;

[0018] The average feature vector of each category is taken as the center point feature prototype C of the category. S .

[0019] Optionally, constructing a pseudo label for unlabeled data in the target domain based on the center point feature prototype of the source domain specifically includes:

[0020] Initialize the memory Memory and generate the feature prototype C of each category of all image samples in the source domain label data S , sum up the average feature vectors of all samples in each category to get the center point feature prototype, which is calculated as:

[0021]

[0022] in, is the set of source domain image sample labels belonging to the kth category, is the number of samples in the set, is the i-th source domain sample;

[0023] Extract multiple categories of all image samples in the target domain unlabeled data, calculate the cosine similarity of each category to the center point feature prototype, and select the center point feature prototype with the closest similarity to construct a pseudo label for the target domain unlabeled data;

[0024] K-means clustering is performed on the pseudo labels to assign multi-center feature prototypes to each category of all image samples in the target domain unlabeled data. The calculation formula is:

[0025]

[0026] in, is the target domain sample, is the set of the kth category of sample data in the target domain, is the center point feature prototype, argmax is the index operation with the maximum value, and Kmeans is the clustering operation.

[0027] Optionally, when the memory Memory is initialized, the center point feature prototype and the multi-center feature prototype are updated, which specifically includes:

[0028] For the center point feature prototype, after extracting the features of all image samples in a category in each training, momentum update is adopted, and the calculation formula is:

[0029]

[0030] Among them, m s is the update speed coefficient, is the set of source domain image samples belonging to the kth category, is the number of all samples in the set;

[0031] For the multi-center feature prototype, after assigning a pseudo label to each sample in the source domain, it is updated using weighted momentum in each round of training process k. The formula is:

[0032]

[0033]

[0034] Among them, m t is the update speed coefficient, is the set of pseudo labels of the target domain image samples belonging to the kth category, is the number of all samples in the set, is the result of the clustering operation.

[0035] Optionally, the center point feature prototype C of each category of the source domain label data is obtained S Finally, the contrastive learning loss function is used to make each sample in the source domain close to the characteristic prototype of its respective classification. The formula is:

[0036]

[0037] in, For sample The center point feature prototype corresponding to the category, τ is the temperature coefficient, n s is the number of source domain image samples.

[0038] Optionally, the method further includes: completing the cross entropy loss between the pseudo label and the real label generated by the source domain sample image by establishing cyclic label consistency, and the process of establishing cyclic label consistency specifically includes:

[0039] Obtain the similarity probability value prob of the source domain image sample to all multi-center feature prototypes in the target domain. The calculation formula is:

[0040]

[0041] Among them, τ is the temperature coefficient, T is the matrix transpose operation, and max is the maximum value operation. is the target domain sample, is the result of the clustering operation, p is the source domain sample image feature vector, P is the set of source domain sample image feature vectors, k is the set of source domain samples of the kth category, K is the total number of source domain categories, and j is the index of the feature vector prototype;

[0042] The calculated probability value distribution is used to make the distribution of the same category in the source domain and the target domain similar. The formula is:

[0043]

[0044] in, is the indicator function, is the similarity probability value, K is the total number of source domain categories, n s is the number of source domain image samples, and k is the set of the kth category of source domain samples.

[0045] Optionally, the method further includes: minimizing entropy so that each sample in the target domain is close to the corresponding feature prototype. The formula for minimizing entropy is:

[0046]

[0047] Based on the optimization loss function:

[0048] L=L cl +λ1L cycle +λ2L ent

[0049] Among them, λ1 and λ2 are balance coefficients, E is a hyperparameter, and L cl is the contrast loss, L cycle is the circulation loss, L ent To minimize entropy.

[0050] Compared with the prior art, the cross-domain remote sensing image retrieval method based on multi-center point feature prototype provided by the embodiment of the present invention has the following beneficial effects:

[0051] In the prior art, a single-center feature prototype cannot effectively represent each category, resulting in the inability of the class alignment method supervised by pseudo-labels to guarantee reliability, causing the accumulation of erroneous samples classified by pseudo-labels, and failing to strengthen the consistency between the source sample labels and pseudo-labels, resulting in large differences in data space distribution between the source domain and the target domain.

[0052] The present invention is based on the central point feature prototype C S Construct pseudo labels for the target domain unlabeled data, and use the pseudo labels to assign a multi-center feature prototype C to each classification in the target domain unlabeled data T , strengthen the consistency between the source sample labels and pseudo labels; solve the domain deviation problem caused by category deviation of the single center point feature prototype, so as to better explore the data distribution within the target domain, reasonably utilize all image samples in the target domain, and narrow the difference in data space distribution between the source domain and the target domain.

[0053] In addition, when using pseudo-label information to help construct a multi-center feature prototype in the target domain, the pseudo-label is not involved in the entire training process, thus avoiding the accumulation of erroneous samples caused by pseudo-label classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1A comparative schematic diagram of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment;

[0055] Figure 2 A schematic diagram of a cyclic label consistency framework of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment;

[0056] Figure 3 A schematic diagram of a supervisory signal of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment;

[0057] Figure 4 A schematic diagram of an overall algorithm design framework of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment;

[0058] Figure 5 A schematic diagram of the accuracy results of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment;

[0059] Figure 6 A t-SNE schematic diagram of AID→UCMD of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in one embodiment;

[0060] Figure 7 A t-SNE schematic diagram of UCMD→AID of a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype provided in an embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0062] In one embodiment, a cross-domain remote sensing image retrieval method based on a multi-center point feature prototype is provided, the method comprising:

[0063] 1. Method design

[0064] 1.1 Design of multi-center feature prototype

[0065] By constructing a multi-center feature prototype for each category in the target domain, the negative transfer caused by the single-center feature prototype can be alleviated. The multi-center feature prototype can assign a more robust and accurate label to each category in the target domain. The multi-center features are constructed using all image samples in the target domain. When the pseudo-label information is used to construct the multi-center feature prototype, the pseudo-label is not directly involved in the model training, avoiding the direct use of pseudo-labels, making better use of pseudo-label information, and reducing the noise misleading caused by pseudo-labels. Figure 1 It can be seen that by constructing a multi-center feature prototype, the classification boundary of each category in the target domain can be better learned, and the classification errors are greatly reduced. After learning through the network model, the data distribution between each category is more compact, and the data distribution between different categories is more separated, thereby better learning the distribution of the data domain, thereby improving the retrieval performance.

[0066] 1.2 Design based on cycle label consistency

[0067] It is proposed to use the Cycle Label-Consistency (CLC) method to achieve domain alignment. The schematic diagram of the algorithm framework is shown in the figure. Figure 2 As shown in Figure 2. The CLC method is different from the existing UDA research and directly uses the given true label as network supervision information. Figure 2 As shown in the figure, first, the CLC algorithm generates a feature prototype for each category in the source domain. The center point feature prototype of each category is represented by the average feature vector of all samples in that category. According to the center point feature prototype of the source domain, the nearest neighbor classification is used to construct a pseudo label for the target domain.

[0068] The average feature vector representation is calculated based on the sample feature vectors of each category in the source domain. The specific operation is to average the feature vectors of all samples under the category. The confirmation method of the average feature vector includes: for a certain category, assume that there are N samples in the source domain, and the feature vector of each sample is represented as x_i, where i = 1, 2, ..., N. Then the center point feature prototype of the category can be obtained by calculating the average feature vector, that is:

[0069]

[0070] Among them, center is the average feature vector, Sum is the summation operation, N is the total number of image samples, x i is the feature vector for each image sample.

[0071] Then, the target domain center feature prototype is updated in the same way, and each source domain sample is assigned to the nearest target domain center feature prototype to obtain a pseudo label. Finally, cross entropy is used to strengthen the consistency learning between pseudo labels and true labels. In simple terms, cyclic label consistency only uses pseudo labels to calculate the target domain center point and transfer it to the source domain, which alleviates the erroneous information of the wrongly labeled samples, fully performs reverse information propagation, and obtains a more compact data domain distribution.

[0072] Figure 3 The schematic diagram of the supervision signal is shown in Figure 1. The information of the true label of the source domain is used as the supervision signal to explore the probability distribution of each source domain image sample to the target domain center point feature prototype. The proposed cyclic label consistency method is used to learn the consistency of the probability distribution and the supervision signal, so that the center point feature prototypes of the same category in the two domains are close to each other, realizing the consistent expression of learning in the data distribution space.

[0073] 1.3 Overall Design

[0074] The overall design framework is as follows Figure 4 Given a labeled source domain dataset and the unlabeled target domain n s and n t are the number of source domain and target domain image samples respectively. The main purpose is to use the multi-center feature prototype and CLC algorithm to complete the cross-domain remote sensing image retrieval task. In the cross-domain remote sensing image retrieval task, the feature extractor f θ (·) Extract features of image samples, and the source domain and target domain share network model weights. Secondly, the multi-center point method is used to generate multiple feature prototypes for each class in the target domain, and the CLC algorithm is used to strengthen the spatial consistency of the source domain and target domain sample data distribution. At the same time, a memory Memory is defined to store the center point feature prototypes of the source domain and target domain during network training, and for dynamic updating of multiple center points. Use the source domain with label information to assign pseudo labels to each image sample in the target domain, and then use the pseudo labels to assign multi-center feature prototypes to each category in the target domain to reduce the negative transfer caused by the noise carried by the pseudo labels. At the same time, using the CLC algorithm, the true label of the source domain is used as a supervisory signal in training to strengthen the consistency between the true label of the sample and its pseudo label, so as to better learn the cross-domain data distribution and improve the retrieval performance.

[0075] The overall algorithm design is introduced in detail below.

[0076] (1) Constructing Memory

[0077] First, construct a memory Memory, where the memory Memory contains the feature prototypes of each classification of all image samples in the source domain And the multi-center feature prototypes of each category of all image samples in the target domain Each category in the source domain has only one feature prototype, and each category in the target domain has P center point feature prototypes, where K is the number of categories. Secondly, the memory is initialized. For the feature prototype of the source domain, all source domain image sample features are extracted, and the average value of each category of features is calculated and saved in the memory. The calculation method of the source domain center point feature prototype is shown in formula (1):

[0078]

[0079] in, Indicates that the source domain image sample label belongs to the set of the kth category, represents the number of samples in the set, Represents the i-th source domain sample. For the target domain feature prototype, first extract all target domain sample features, calculate the cosine similarity of each feature to the feature prototype of the source domain center point, select the feature prototype with the closest similarity to assign pseudo labels to the target domain image samples, and then perform k-means clustering to obtain P feature prototypes for each class of the target domain. The pseudo labels and feature prototypes are calculated as shown in formulas (2) and (3):

[0080]

[0081] in, represents the target domain sample, represents the set of the kth category of sample data in the target domain, arg is a variable, c k is the label of the kth category or the center point of the category.

[0082] Finally, the memory will update the stored source domain and target domain center point feature prototypes in each round of training. For the source domain feature prototype, after extracting the features of all image samples in a batch in each training, momentum is used to update according to the label information. The update calculation method is shown in formula (4):

[0083]

[0084] Among them, m s is the update speed coefficient, Indicates that the source domain image samples in a batch belong to the set of k-th category, Indicates the number of all samples in the set. In addition, for the target domain feature prototype, after assigning a pseudo label to each sample in the source domain, it is updated using weighted momentum in each round of training process k. The update method is shown in formulas (5) and (6):

[0085]

[0086]

[0087] Among them, m t is the update speed coefficient, Indicates that the pseudo labels of the target domain image samples in a batch belong to the set of the kth category, Represents the number of all samples in the set.

[0088] (2) Source domain feature prototype contrast learning

[0089] After calculating the center point feature prototype of each classification in the source domain After that, it is necessary to strengthen the learning of each image sample and the feature prototype of the classification. Here, the contrastive learning loss function is used to use the semantic information learned from the source domain to make each sample in the source domain close to the feature prototype of the respective classification. Its implementation is shown in formula (7):

[0090]

[0091] in, For sample The center point feature prototype corresponding to the category, τ is the temperature coefficient, exp is the exponential function, f θ (·) is the operation of the feature extractor.

[0092] (3) Multi-center feature prototype consistency loss

[0093] A. Calculate the single center point feature prototype of the source domain and the multi-center point feature prototype of each category in the target domain unlabeled data, and store them in the memory. It is necessary to maintain the consistency between the pseudo label in the source domain and its true label, so the cross entropy loss between the pseudo label and the true label of the source domain sample is minimized by establishing the cyclic label consistency, so as to achieve the purpose of using the true label as the supervisory signal to complete the optimization model.

[0094] The first step is to calculate the similarity probability value of the source domain image sample to all multi-center feature prototypes in the target domain. in The calculation is shown in formula (8):

[0095]

[0096] B. Using the probability value distribution calculated above, use the following formula to ensure that the distribution of the same category in the source domain and the target domain is similar. The calculation formula is shown in (9):

[0097]

[0098] in, represents the indicator function, when 1 if the value is 0, otherwise it is 0. Compared with "hard" labels, "soft" labels are calculated with stricter constraints, which can better learn the distribution consistency of data, enhance the network's learning ability of sample pseudo-labels and true labels, make the distribution of the same category gradually closer, and the distribution of data that does not belong to the same category gradually move away, thus making the distribution of each class more compact.

[0099] The contrast loss is used to complete the consistency learning between the source domain image samples and their feature prototypes, and the multi-center feature prototypes of the target domain are used to generate pseudo labels for the source domain images. Then, the cross entropy learning between the source domain image samples and their pseudo labels is completed using the cyclic label consistency. In this way, the distribution of similar data is gradually approached, and the distribution learning of samples of the same category is completed.

[0100] (4) Entropy minimization

[0101] In the algorithm design, in order to further explore the internal structure of the target domain data and make each sample in the target domain closer to its corresponding feature prototype, entropy minimization is used to achieve this goal. The calculation is shown in formula (10):

[0102]

[0103] The final algorithm optimization loss function is as shown in formula (11):

[0104] L=L cl +λ1L cycle +λ2L ent

[0105] Among them, λ1 and λ2 are balance coefficients.

[0106] 2. Experimental results and analysis

[0107] 2.1 Experimental Setup

[0108] In order to verify the feasibility and effectiveness of the multi-center feature prototype and CLC method, a large number of comparison experiments and ablation experiments were conducted to verify the effectiveness of the proposed method. The following is the relevant settings of the experiment, mainly focusing on the data set, comparison method, experimental environment and parameters.

[0109] (1) Datasets. Three remote sensing image datasets AID, UCMD, and NWPU are used to complete six cross-domain retrieval tasks. Each dataset provides 1000 images of 10 categories, that is, 100 images per category, including baseball fields, beaches, densely populated residential areas, forests, parking lots, rivers, sparsely populated residential areas, and storage tanks. In the experiment, two different datasets are randomly selected as the source domain and the target domain, respectively, to generate six cross-domain remote sensing image retrieval tasks, namely AID→UCMD, UCMD→AID, AID→NWPU, NWPU→AID, UCMD→NWPU, and NWPU→UCMD, which are briefly labeled as A→U, U→A, A→N, N→A, U→N, and N→U. The symbol P→Q indicates that dataset P is used as the source domain and dataset Q is used as the target domain.

[0110] (2) Comparison methods. Seven methods, ADDA, DAN, JAN, MDD, SE, PCLUDA and source only, are selected as comparison methods. Among them, ADDA trains an asymmetric feature extractor to confuse the domain discriminator through adversarial learning; DAN and JAN reduce the difference between the two domains by minimizing the statistical difference using MMD and joint MMD; MDD introduces a new difference, called marginal difference, to measure the distribution difference; SE designs a teacher-student framework and aligns the two domains by reducing the prediction difference between the two networks; PCLUDA reduces the probability distribution difference between the target domain and its perturbation output through pseudo-label self-training and consistency regularization strategy; in addition, a source only method is designed, which only uses data labeled by the source domain to train the cross-domain remote sensing image retrieval task model.

[0111] (3) Experimental environment and parameter settings. The operating system of the machine used in the experiment is Ubuntu 18.04, the processor is Intel Core 9, the memory is 32GB, and the graphics card is NVIDIA Tesla T4. The programming language used is Python 3.8, and the deep learning framework is Pytorch 1.13. Each of the methods mentioned above is carried out in the same experimental environment. For the experimental data division, we first evenly extract 80% of the data from the two domains, and then extract 80% of the data as the training set and 20% as the validation set. For the other 20% of the data, the sample data from the target domain is used as the query set, and the database samples 80% of the data from the entire source domain. In the experiment, we use the ResNet34 model pre-trained by ImageNet as the feature extractor, so each image input is resized to 224×224. In addition, we use the TrvialAugment data augmentation strategy to improve the generalization ability of the model. During the training process, each batch size is set to 64, the learning rate lr is set to 1e-5, and the total number of training epochs is 50. Other hyperparameters such as m s 、m t , λ1 and λ2 are taken as 0.7, 0.7, 0.1, 0.8 and 0.2 respectively. During the test, we use the feature extractor to extract the query set and database sample features, and then measure the Euclidean distance between the query set features and each database feature, so as to find the most similar samples from the database. For the experimental results, we use mAP and Topk (P@K) to measure the results of all methods. And finally, the high-dimensional feature distribution diagram of t-SNE is shown to intuitively feel the effect of the proposed method.

[0112] 2.2 Comparative experimental results and analysis

[0113] The comparative experiments of the proposed method and other model methods on six cross-domain remote sensing image retrieval tasks are shown in Table 1.

[0114] Table 1 Comparative experiment

[0115]

[0116]

[0117] According to the data in the table, it can be obtained that the proposed method achieved 88.01%, 90.66%, 91.18%, 95.89%, 75.91% and 87.57% results in the mAP indicators of six cross-domain remote sensing image retrieval tasks, respectively, which are better than the other nine comparison methods, which are 5.82%-24.40%, 1.93%-31.63%, 6.39%-22.18%, 3.95%-21.73%, 6.03%-25.62%, 4.38%-29.22%, 7.18%-14.72% and 1.19%-9.59% higher, respectively, and have a significant improvement in retrieval performance. Compared with the latest method PCLUDA, the proposed method explores the data distribution of the target domain by constructing a multi-center feature prototype, and does not directly use pseudo labels to participate in the training network. PCLUDA uses pseudo labels to train the network, which has many noise perturbations, resulting in many negative transfers when using pseudo labels to classify unlabeled samples. Therefore, the PCLUDA method still has certain defects in cross-domain retrieval accuracy. In the comparative experiments, the proposed method is superior to the method in terms of mAP performance indicators, and the retrieval performance is greatly improved, especially in the three comparative methods of U→A, A→N and N→A. The proposed method is compared with the PCLUDA method, and the improvement reaches 8.76%, 13.73% and 10.01%, showing a significant improvement effect. The results show that by constructing multiple center points, the data space distribution of the target domain can be fully explored. At the same time, the learning of consistency between different domains is further strengthened by using the cyclic label consistency algorithm.

[0118] To further demonstrate the effectiveness of the proposed method, Figure 3-5 The results of Top k (P@K) on six cross-domain remote sensing image retrieval tasks are shown, where K = 5, 10, 20, 50, 100. It can be seen from the figure that in the six cross-domain retrieval tasks, the proposed method has a significant improvement in the accuracy results of Top k (P@K), which is higher than the other seven comparative experimental methods. From the t-SNE high-dimensional feature visualization diagram, it can be seen that compared with the latest cross-domain retrieval method PCLUDA, when K = 5, 10, 20, 50, 100, the proposed method shows strong consistency in the learning of category data distribution. The distribution of similar data in the source domain and the target domain is close, and the distribution of different categories is far away, showing good cross-domain consistency learning and good learning ability in class alignment. The cyclic label consistency loss function used strengthens the consistency between the real label of the target domain and the multi-center feature prototype of the target domain, prompts the same category samples between different domains to approach the same data distribution, learns the statistical distribution differences between different domains, and avoids the errors caused by directly training the network model with pseudo labels, eliminating the interference of noise on the retrieval model.

[0119] In the comparative experimental results shown, the proposed method achieved the best results in both mAP and Top k (P@K) retrieval performance. This fully proves that the feature extractor trained by our method has excellent feature extraction capabilities in cross-domain remote sensing image retrieval scenarios. When a query image is input, our method can more easily retrieve samples in the database that are similar to the query image. Not only that, these retrieved similar samples can usually occupy a higher position in the similarity ranking. This advantage is crucial to improving the overall performance of cross-domain remote sensing image retrieval, because it ensures that users can obtain results that are highly relevant to the query image more quickly and accurately.

[0120] In order to further study the role of the proposed method in feature alignment, t-SNE is used to visualize the high-dimensional features of the above comparative experimental methods on two datasets (i.e., AID and UCMD). Figure 3-6 and Figure 3-7 The high-dimensional feature visualization diagrams of the cross-domain retrieval tasks AID→UCMD and UCMD→AID are shown in Figure 1. As can be seen from the figure, the source only method has a large feature confusion between the source domain and the target domain, making it difficult to learn the retrieval boundary of the category and unable to train a good retrieval model. Some of the DA methods, such as DAN, JAN, and MDD, can weaken the feature deviation to a certain extent. In these DA methods, the distance between different categories in the target domain is very small, the decision boundary is more blurred, and there are many samples in the target domain that are misclassified, indicating that these comparison methods have obvious defects in class alignment. Obviously, the proposed method successfully overcomes the feature offset between the two domains, constructs a multi-center feature prototype to make the distributions of the same category close to each other, and the spatial distributions of different categories far apart, and the inter-class separability of the target domain data is significantly increased, which better solves the problem of large inter-class similarity of remote sensing images, thereby motivating the network to learn more discriminative feature representations of the target domain, so that the learned network has stronger generalization ability and greatly reduces the misclassified samples. In addition, the cyclic label consistency algorithm strengthens the consistency of data distribution between the source domain and the target domain during the network model training process. By realizing the cyclic allocation of labels, the real labels of the source domain are used as supervision signals, and the contrastive learning loss function is used to strengthen the consistency between them and the pseudo-labels, the pseudo-label information is better utilized, the negative transfer results are avoided, and the performance of cross-domain retrieval is greatly improved.

[0121] 2.3 Ablation Experiment Results and Analysis

[0122] To further verify the algorithm designed by the proposed method, ablation experiments are performed on the following modules: (1) V1 explores the entropy minimization loss function (2) V2 explores the cycle label consistency loss function (3) At the same time, the number of multi-center point feature prototypes P is explored. V3, V4, and V5 are set to P = 1, 2, and 3 respectively, and the effectiveness of multiple center points for cross-domain retrieval is studied. Ablation experiments are conducted on four cross-domain retrieval tasks UCMD→NWPU, NWPU→UCMD, AID→UCMD, and UCMD→AID, and the performance evaluation indicator is mAP. The ablation experiment results are shown in Table 2.

[0123] Table 2 Ablation experiment

[0124]

[0125] From the data in the table, it can be concluded that and The loss function has good feasibility in the six cross-domain retrieval tasks and greatly improves the retrieval accuracy. In particular, the cycle label consistency loss function has improved by 5.26%, 11.66%, 12.18%, and 8.95%. It is obvious that It has achieved great advantages in cross-domain remote sensing image retrieval tasks and can greatly improve retrieval performance. In addition, for the number of multi-center point feature prototypes, it can be seen that when a single center point is set for retrieval, the retrieval performance is significantly worse than two and three center points, and when the number of center points P = 3, the mAP value is better than the effect of taking two center points. In the four cross-domain retrieval tasks, for two center points, the mAP increased by 3.96%, 0.97%, 0.14%, and 0.06%, respectively.

[0126] 3. Summary

[0127] It is proposed to use multi-center feature prototypes and cyclic label consistency methods to complete cross-domain remote sensing image retrieval tasks. This method fully explores the target domain data distribution, constructs multi-center feature prototypes for each classification, and then assigns pseudo labels to the source domain. The cyclic label consistency algorithm is then used to strengthen the consistency between the source domain real label and the pseudo label, learn domain invariant features, make the distribution of data in the same category closer and the distribution of data in different categories farther, so that the learned model is more compact. In the comparative experiments, ablation experiments, Top k (P@K) and t-SNE results, the proposed method has achieved significant results, and the retrieval performance is better than the latest cross-domain retrieval method.

[0128] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A cross-domain remote sensing image retrieval method based on multi-center point feature prototype, characterized in that: include: Obtain a cross-domain remote sensing image dataset that includes source domain labeled data and target domain unlabeled data; In the original retrieval model based on the cyclic label consistency method CLC, in the feature extractor f θ (·) and then add memory to form an improved retrieval model; The cross-domain remote sensing image dataset is input into the improved retrieval model and the feature extractor f θ (·) Extract image samples of all categories in the source domain labeled data and the target domain unlabeled data; Generate the feature vectors of all image samples in each category of the source domain label data through the memory, and average the feature vectors of all image samples in each category to obtain the center point feature prototype C of each category of the source domain label data S ; Based on the center point feature prototype C S Construct pseudo labels for the target domain unlabeled data, and use the pseudo labels to assign a multi-center feature prototype C to each classification in the target domain unlabeled data T ; The center point feature prototype C of each category in the source domain label data S and the multi-center feature prototype C for each category in the target domain unlabeled data T By comparison, the retrieval results of cross-domain remote sensing images are obtained.

2. A cross-domain remote sensing image retrieval method based on multi-center point feature prototypes as claimed in claim 1, characterized in that: The memory Memory includes: feature prototypes of each classification of all image samples in the source domain label data And the multi-center feature prototypes of each category of all image samples in the target domain Among them, all image samples in the source domain labeled data have only one center point feature prototype for each category, and all image samples in the target domain unlabeled data have P multi-center feature prototypes for each category, and K is the number of categories.

3. A cross-domain remote sensing image retrieval method based on multi-center point feature prototypes as claimed in claim 1, characterized in that: The averaging operation of the feature vectors of all image samples in each category specifically includes: Assume that each category in the source domain label data has N image samples, and the feature vector of each image sample is represented by x i , where i = 1, 2, ..., N; Get the average feature vector of each category in the source domain label data. The calculation formula is: Among them, center is the average feature vector, Sum is the summation operation, N is the total number of image samples, x i is the feature vector for each image sample; The average feature vector of each category is taken as the center point feature prototype C of the category. S .

4. The cross-domain remote sensing image retrieval method based on multi-center point feature prototype according to claim 1, characterized in that: The method of constructing a pseudo label for the target domain unlabeled data based on the center point feature prototype of the source domain specifically includes: Initialize the memory Memory and generate the feature prototype C of each category of all image samples in the source domain label data S , sum up the average feature vectors of all samples in each category to get the center point feature prototype, which is calculated as: in, is the set of source domain image sample labels belonging to the kth category, is the number of samples in the set, is the i-th source domain sample; Extract multiple categories of all image samples in the target domain unlabeled data, calculate the cosine similarity of each category to the center point feature prototype, and select the center point feature prototype with the closest similarity to construct a pseudo label for the target domain unlabeled data; K-means clustering is performed on the pseudo labels to assign multi-center feature prototypes to each category of all image samples in the target domain unlabeled data. The calculation formula is: in, is the target domain sample, is the set of the kth category of sample data in the target domain, is the center point feature prototype, argmax is the index operation with the maximum value, and Kmeans is the clustering operation.

5. A cross-domain remote sensing image retrieval method based on multi-center point feature prototypes as claimed in claim 4, characterized in that: When the memory Memory is initialized, the center point feature prototype and the multi-center feature prototype are updated, which specifically includes: For the center point feature prototype, after extracting the features of all image samples in a category in each training, momentum update is adopted, and the calculation formula is: Among them, m s is the update speed coefficient, is the set of source domain image samples belonging to the kth category, is the number of all samples in the set; For the multi-center feature prototype, after assigning a pseudo label to each sample in the source domain, it is updated using weighted momentum in each round of training process k. The formula is: Among them, m t is the update speed coefficient, is the set of pseudo labels of the target domain image samples belonging to the kth category, is the number of all samples in the set, is the result of the clustering operation.

6. The cross-domain remote sensing image retrieval method based on multi-center point feature prototype according to claim 1, characterized in that: The center point feature prototype C of each classification of the source domain label data is obtained S Finally, the contrastive learning loss function is used to make each sample in the source domain close to the feature prototype of its respective classification. The contrastive loss formula is: in, For sample The center point feature prototype corresponding to the category, τ is the temperature coefficient, is the target domain sample, n s is the number of source domain image samples.

7. The cross-domain remote sensing image retrieval method based on multi-center point feature prototype according to claim 1, characterized in that: Also includes: The cross entropy loss between the pseudo labels and the real labels generated by the source domain sample images is completed by establishing cyclic label consistency. The process of establishing cyclic label consistency specifically includes: Obtain the similarity probability value prob of the source domain image sample to all multi-center feature prototypes in the target domain. The calculation formula is: Among them, τ is the temperature coefficient, T is the matrix transpose operation, and max is the maximum value operation. is the target domain sample, is the result of the clustering operation, p is the source domain sample image feature vector, P is the set of source domain sample image feature vectors, k is the set of source domain samples of the kth category, K is the total number of source domain categories, and j is the index of the feature vector prototype; The calculated probability value distribution is used to make the distribution of the same category in the source domain and the target domain similar. The formula for the cycle loss is: in, is the indicator function, is the similarity probability value, K is the total number of source domain categories, n s is the number of source domain image samples, and k is the set of the kth category of source domain samples.

8. The cross-domain remote sensing image retrieval method based on multi-center point feature prototype according to claim 1, characterized in that: Also includes: By minimizing entropy, each sample in the target domain is brought close to its corresponding feature prototype. The formula for entropy minimization is: Based on the optimization loss function: Among them, λ1 and λ2 are balance coefficients, E is a hyperparameter, and L cl is the contrast loss, L cycle is the circulation loss, L ent To minimize entropy.

Images