Closed set domain adaptive remote sensing image classification retrieval method fusing space domain and frequency domain

Abstract

The embodiment of the invention provides a closed set domain adaptive remote sensing image classification retrieval method fusing a space domain and a frequency domain, which belongs to the technical field of data processing, and specifically comprises the following steps: establishing a closed set domain adaptive model fusing the space domain and the frequency domain of a remote sensing image; establishing a model training strategy; dividing the sample data set into a training set, a test set and a verification set according to a preset proportion, training a closed set domain adaptation model through the training set and a model training strategy, and testing and verifying the closed set domain adaptation model through the test set and the verification set to obtain a trained classification retrieval model; and inputting the target remote sensing image into the trained classification retrieval model to obtain a classification result and a retrieval result. By means of the scheme, the problem that due to the influence of remote sensing image background complexity and multi-source data distribution difference, a complex remote sensing image cross-domain classification retrieval task only focuses on spatial domain information, and a good generalization result cannot be obtained is solved, and classification retrieval accuracy and adaptability are improved.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of data processing technology, and in particular to a closed-set domain adaptive remote sensing image classification and retrieval method that integrates spatial domain and frequency domain. Background Art

[0002] With the continuous advancement of remote sensing technology, satellite and aerial sensors have become more efficient in earth observation and data capture, generating unprecedented amounts of remote sensing data. The large-scale acquisition of high-resolution remote sensing images has laid a big data foundation for applications in various fields, and massive amounts of data bring a huge amount of information. How to quickly and efficiently classify and retrieve images that meet user needs with a small number of labels or even no labels has become a major problem restricting the sharing and effective use of remote sensing image data.

[0003] In response to the problem of data distribution shift, domain adaptation (DA) learning has been proposed to transfer knowledge from a labeled source domain to an unlabeled target domain under the premise that the data distribution of the source domain (training set) and the target domain (test set) are different but similar, so as to improve the generalization ability of the model in the target domain. According to whether there is labeled information in the target domain, domain adaptation can be divided into supervised domain adaptation and unsupervised domain adaptation. Among them, unsupervised domain adaptation (UDA) is the most challenging and also the research direction closest to the actual scenario in domain adaptation learning, that is, the target domain does not contain any labeled information. According to whether the composition of the label (category) space between domains is the same, domain adaptation can be divided into domain adaptation with consistent label space and domain adaptation with inconsistent label space. Among them, domain adaptation with consistent label space is also called closed set domain adaptation, which refers to the domain adaptation problem with the same label space of the source domain and the target domain, and is also the focus of most existing domain adaptation research. At present, some mainstream closed-set domain adaptation methods are mainly divided into three types: ① Based on distribution difference: deep features are obtained through the model, and then the distribution difference between domains is reduced based on the distribution difference method to reduce the generalization error. ② Based on adversarial learning: borrowing the idea of ​​generative adversarial network, the adversarial training of feature extractor and domain discriminator is used to learn domain invariant feature representation. ③ Based on self-training: domain adaptation is achieved by using unlabeled target domain data, which generates pseudo labels for unlabeled data and retrains the network by combining source domain labels and target domain pseudo labels.

[0004] At present, domain adaptation is mainly based on image spatial domain features, that is, using deep neural networks to bring the source domain and target domain spatial domain information closer. However, due to the fact that remote sensing images contain a variety of ground object types and complex background information, the classification and retrieval accuracy of existing methods in complex multi-source scenes still has room for improvement.

[0005] At present, studies have shown that the frequency domain features of images will not have significant feature shifts between different data sets, and the frequency domain features contain detailed information such as texture and structure. Studies have applied them to target recognition, change detection, etc. Combining frequency features may produce more accurate predictions in the target domain. Therefore, how to solve the confusion of cross-domain sample categories, how to make full use of spatial and frequency domain information, and improve the closed set domain adaptation effect are issues that need to be solved urgently. Summary of the invention

[0006] In view of this, the embodiments of the present disclosure provide a closed-set domain adaptive remote sensing image classification retrieval method that integrates the spatial domain and the frequency domain, which at least partially solves the problems of poor classification retrieval accuracy and adaptability in the prior art.

[0007] The present disclosure provides a closed-set domain adaptive remote sensing image classification retrieval method integrating spatial domain and frequency domain, including:

[0008] Step 1, establishing a closed set domain adaptation model for fusing the space domain and frequency domain of remote sensing images, wherein the closed set domain adaptation model for fusing the space domain and frequency domain of remote sensing images includes a data enhancement module, a space domain feature extraction module, a frequency domain feature extraction module, a space-frequency cross attention fusion module, a feedforward and a classifier;

[0009] Step 2: Establish a model training strategy;

[0010] Step 3, dividing the sample data set into a training set, a test set and a validation set according to a preset ratio, training a closed set domain adaptation model that integrates the spatial domain and frequency domain of remote sensing images through the training set and the model training strategy, and testing and validating it through the test set and the validation set to obtain a trained classification retrieval model;

[0011] Step 4: Input the target remote sensing image into the trained classification retrieval model to obtain the classification result and retrieval result.

[0012] According to a specific implementation of the embodiment of the present disclosure, the data enhancement module is used to weakly enhance the source domain image to obtain a source domain weakly enhanced view, and to weakly enhance and strongly enhance the target domain to obtain a target domain weakly enhanced image and a target domain strongly enhanced view;

[0013] The spatial domain feature extraction module is used to sequentially input the source domain weak enhancement map, the target domain weak enhancement map, and the strong enhancement map into the spatial domain encoder to obtain the spatial domain feature

[0014] The frequency domain feature extraction module is used to perform Fourier transform on the source domain weak enhancement image, the target domain weak enhancement image, and the strong enhancement image to obtain the image frequency domain, and then set the frequency domain filter to input the frequency domain information corresponding to the source domain weak enhancement image, the target domain weak enhancement image, and the target domain strong enhancement image obtained by the transformation into the frequency domain encoder in sequence to obtain the frequency domain feature extraction module.

[0015] The space-frequency cross attention fusion module is used to generate queries, keys and values ​​from spatial domain features and frequency domain features through linear transformation, and use the query value of the spatial domain feature and the key value of the frequency domain feature for cross calculation, use the multi-head attention mechanism to linearly transform the query, key and value respectively, and then divide them into multiple heads, independently calculate the attention of each head, and finally connect the outputs of all heads, and then generate the final output through a linear transformation, and the final output is added to the spatial domain feature through a residual connection, and layer normalization is performed, and finally a feedforward neural network is passed to obtain the fused high-level features;

[0016] The classifier is used to obtain the category probabilities of different categories of the image by linearly transforming the high-level features.

[0017] According to a specific implementation of the embodiment of the present disclosure, the expression for generating query, key and value is:

[0018]

[0019] Among them, Q sq , K sq and V sq are the query, key and value of spatial domain features respectively, Q fre , K fre and V fre are the query, key and value of frequency domain features respectively, is the parameter of the linear transformation;

[0020] The expression of the crossover calculation is:

[0021]

[0022] The final output expression is MultiHead(Q,K,V)=Concat(head 1 ,…,head h )W O

[0023]

[0024] Among them, MultiHead(·) is a multi-head attention mechanism;

[0025] According to a specific implementation of the embodiment of the present disclosure, step 2 specifically includes:

[0026] Step 2.1, the maximum mean difference method is used to measure the spatial domain distribution difference loss, and the high-order moment matching method is used to measure the frequency domain distribution difference loss;

[0027] Step 2.2, cross entropy loss is used as the source domain classification loss;

[0028] Step 2.3, pseudo-label consistency learning is used as the classification loss of the weak enhancement map and the strong enhancement map of the target domain;

[0029] Step 2.4, use minimum class confusion learning on the class probability distribution obtained by the classifier to obtain the class confusion loss;

[0030] In step 2.5, the total training loss is obtained based on the spatial domain distribution difference loss, the frequency domain distribution difference loss, the source domain classification loss, the target domain weak enhancement map and strong enhancement map classification loss, and the class confusion loss.

[0031] According to a specific implementation of the embodiment of the present disclosure, the spatial domain distribution difference loss The expression is

[0032]

[0033] in, represents the reproducing kernel Hilbert space, Used to label sample features And unlabeled sample features Projected onto the reproducing kernel Hilbert space, Defines a set of Continuous functions in ;

[0034] The expression of the frequency domain distribution difference loss is:

[0035]

[0036] Where p is the order of high-order moment matching, Same as MDD, representing the reproducing kernel Hilbert space, represents the frequency domain Hilbert space feature representation of the i-th source domain and target domain sample, L p is the number of neurons in the hidden layer, n s is the number of samples in each batch, That is and Represents the random sampling value of the high-order tensor of frequency domain features. The formula is:

[0037]

[0038] Where k(x,y)=exp(-γ∥xy∥ 2 ) is the RBF kernel function. When p = 1, the high-order moment matching method metric is equivalent to the maximum mean difference metric;

[0039] The expression of the source domain classification loss is

[0040]

[0041] in, Representation sample The true probability distribution of is, the class label is 1, and the other class values ​​are 0. express The predicted probability distribution of the sample input is the category probability distribution

[0042] The expression of the classification loss of the weakly enhanced graph and the strongly enhanced graph of the target domain is:

[0043]

[0044] Among them, B represents the batch size, μ is the proportion of pseudo samples in a batch that meet the threshold selection requirements, and H represents the cross entropy loss of two probability distributions;

[0045] The expression of the class confusion loss is

[0046]

[0047] Among them, W jj represents the weight of the j-th sample for class confusion modeling, represents the probability distribution entropy of the sample, B is the batch size, R cc' is the class confusion after sample weighting, It is R cc' The result of normalized calculation is is the class confusion loss, is the number of sample categories;

[0048] The expression of the total training loss is

[0049]

[0050] Among them, α and β are hyperparameters that balance the optimization objectives of the model.

[0051] According to a specific implementation of the embodiment of the present disclosure, step 4 specifically includes:

[0052] Step 4.1: Input the target remote sensing image into the trained classification retrieval model to obtain the sample category probability distribution pi , the category corresponding to the maximum value of the sample category probability is taken as the classification result of the target remote sensing image;

[0053] Step 4.2: Input all images in the retrieval database into the trained classification retrieval model to obtain the retrieval data feature library F = [f 1 ,f 2 ,…,f n ], input the target remote sensing image into the trained classification retrieval model to obtain the target high-level features The cosine similarity is used to calculate the similarity between the target high-level features and all image features in the retrieval data feature library, and the similarity is sorted from high to low, and the most similar image list is returned as the retrieval result.

[0054] According to a specific implementation method of the embodiment of the present disclosure, the expression of the classification result is:

[0055]

[0056] The expression of the similarity is:

[0057]

[0058] Among them, f i Any image feature of the retrieval data feature library.

[0059] The closed-set domain adaptation remote sensing image classification and retrieval scheme of the fusion of spatial domain and frequency domain in the disclosed embodiment includes: step 1, establishing a closed-set domain adaptation model of the fusion of spatial domain and frequency domain of remote sensing images, wherein the closed-set domain adaptation model of the fusion of spatial domain and frequency domain of remote sensing images includes a data enhancement module, a spatial domain feature extraction module, a frequency domain feature extraction module, a space-frequency cross-attention fusion module, a feedforward and a classifier; step 2, establishing a model training strategy; step 3, dividing the sample data set into a training set, a test set and a verification set according to a preset ratio, training the closed-set domain adaptation model of the fusion of spatial domain and frequency domain of remote sensing images through the training set and the model training strategy, and testing and verifying it through the test set and the verification set to obtain a trained classification and retrieval model; step 4, inputting the target remote sensing image into the trained classification and retrieval model to obtain classification results and retrieval results.

[0060] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, spatial domain information and frequency domain information are obtained through a spatial domain encoder and a frequency domain encoder, and a spatial domain and frequency domain feature difference measurement method is introduced to reduce the data offset between the source domain and the target domain. At the same time, a space-frequency cross-attention fusion module is designed on the basis of the above, which not only retains the spatial features and rich semantic features of the spatial domain features, but also effectively introduces additional frequency domain information. Finally, a pseudo-label learning method enhanced by consistency regularization is introduced to adaptively adjust the decision boundary according to the posterior class distribution information of the target domain. And the minimization class confusion learning strategy is embedded in the pseudo-label consistency learning branch to reduce the confusion of similar classes around the decision boundary, so that the classifier can give a higher confidence prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 A flowchart of a closed-set domain adaptive remote sensing image classification and retrieval method that integrates the spatial domain and the frequency domain is provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0064] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0065] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0066] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0067] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0068] The disclosed embodiment provides a closed-set domain adaptive remote sensing image classification and retrieval method that integrates the spatial domain and the frequency domain. The method can be applied to the classification and retrieval of remote sensing images in scenarios such as urban management and geological monitoring.

[0069] See also Figure 1 , is a flow chart of a closed-set domain adaptive remote sensing image classification and retrieval method that integrates the spatial domain and the frequency domain provided by the embodiment of the present disclosure. Figure 1 As shown, the method mainly comprises the following steps:

[0070] Step 1, establishing a closed set domain adaptation model for fusing the space domain and frequency domain of remote sensing images, wherein the closed set domain adaptation model for fusing the space domain and frequency domain of remote sensing images includes a data enhancement module, a space domain feature extraction module, a frequency domain feature extraction module, a space-frequency cross attention fusion module, a feedforward and a classifier;

[0071] Furthermore, the data enhancement module is used to weakly enhance the source domain image to obtain a source domain weakly enhanced view, and to weakly enhance and strongly enhance the target domain to obtain a target domain weakly enhanced image and a target domain strongly enhanced view;

[0072] The spatial domain feature extraction module is used to sequentially input the source domain weak enhancement map, the target domain weak enhancement map, and the strong enhancement map into the spatial domain encoder to obtain the spatial domain feature

[0073] The frequency domain feature extraction module is used to perform Fourier transform on the source domain weak enhancement image, the target domain weak enhancement image, and the strong enhancement image to obtain the image frequency domain, and then set the frequency domain filter to input the frequency domain information corresponding to the source domain weak enhancement image, the target domain weak enhancement image, and the target domain strong enhancement image obtained by the transformation into the frequency domain encoder in sequence to obtain the frequency domain feature extraction module.

[0074] The space-frequency cross attention fusion module is used to generate queries, keys and values ​​from spatial domain features and frequency domain features through linear transformation, and use the query value of the spatial domain feature and the key value of the frequency domain feature for cross calculation, use the multi-head attention mechanism to linearly transform the query, key and value respectively, and then divide them into multiple heads, independently calculate the attention of each head, and finally connect the outputs of all heads, and then generate the final output through a linear transformation, and the final output is added to the spatial domain feature through a residual connection, and layer normalization is performed, and finally a feedforward neural network is passed to obtain the fused high-level features;

[0075] The classifier is used to obtain the category probabilities of different categories of the image by linearly transforming the high-level features.

[0076] Furthermore, the expression for generating query, key and value is

[0077]

[0078] Among them, Q sq , K sq and V sq are the query, key and value of spatial domain features respectively, Q fre , K fre and V fre are the query, key and value of frequency domain features respectively, is the parameter of the linear transformation;

[0079] The expression of the crossover calculation is:

[0080]

[0081] The final output expression is:

[0082] MultiHead(Q,K,V)=Concat(head 1 ,…,head h )W O

[0083]

[0084] Among them, MultiHead(·) is a multi-head attention mechanism;

[0085] In specific implementation, the specific process of establishing a closed set domain adaptation model in the spatial frequency domain of remote sensing images can be as follows:

[0086] Step 1.1: Data and data augmentation: weakly augmented view of the source domain, weakly augmented and strongly augmented views of the target domain.

[0087] A formal definition of the closed domain adaptation problem of remote sensing images is given: given a closed domain containing n s The source domain of the samples and a containing n t Target domain of the sample in and They represent labeled samples and unlabeled samples respectively. represent The corresponding label, C s represents the number of categories in the source domain. It should be noted that the target domain shares the same label space as the source domain, but its samples do not have annotation information. The purpose of domain adaptation learning is to use the labeled source domain D s and the unlabeled target domain D t Learn a classification model so that it can achieve good results in the target domain.

[0088] For the source domain image, horizontal flipping and random cropping with a certain probability are performed in sequence to obtain a weakly enhanced view. For the target domain, a weakly enhanced view is obtained in the same way, and a strongly enhanced view is obtained by RandAugment enhancement.

[0089] Step 1.2: Spatial domain feature extraction.

[0090] The source domain weak enhancement, target domain weak enhancement, and strong enhancement samples are sequentially input into the spatial domain encoder to obtain the spatial domain features. The spatial domain encoder disclosed in the present invention is proposed to adopt Vi T-B16.

[0091] Step 1.3: Frequency domain feature extraction.

[0092] Perform Fourier transform on the weakly enhanced source domain, weakly enhanced target domain, and strongly enhanced samples to obtain the image frequency domain. The formula is as follows:

[0093]

[0094] Set the frequency domain filter Mask, whose value is to obtain the high-frequency and low-frequency parts of the sample. The present disclosure intends to use the phase part θ(u) of the low-frequency frequency domain, and the formula is as follows, where h and w are the height and width of the sample, and b is the hyperparameter of the control filter.

[0095]

[0096] F * (u)=Mask·F(u)

[0097]

[0098] The frequency domain information obtained by weak enhancement of the source domain, weak enhancement of the target domain, and strong enhancement is input into the frequency domain encoder in sequence to obtain the frequency domain features. The frequency domain encoder disclosed in the present invention intends to adopt Resnet-50.

[0099] Step 1.4: Spatial-frequency cross-attention fusion module.

[0100] Through linear transformation, the spatial domain and frequency domain features are converted into queries, keys, and values:

[0101]

[0102]

[0103] The query value of the spatial domain feature and the key value of the frequency domain feature are cross-calculated, which can guide the model to effectively capture the complex dependencies between the spatial domain and frequency domain information. The formula is as follows:

[0104]

[0105] Use the multi-head attention mechanism to capture more detailed information. Perform linear transformations on the query, key, and value, and then divide them into multiple heads. Calculate the attention of each head independently, and finally connect the outputs of all heads, and generate the final output through a linear transformation:

[0106] MultiHead(Q,K,V)=Concat(head 1 ,…,head h )W O

[0107]

[0108] Residual connection and normalization, the output is added to the spatial domain features through residual connection, and layer normalization is performed to ensure gradient stability. The formula is as follows:

[0109] x=LayerNorm(V sq+MultiHead(Q,K,V)

[0110] Finally, a feedforward neural network is used to obtain the fused high-level features. The feedforward neural network includes two fully connected layers and an activation function. The formula is as follows:

[0111] FFN(x)=ReLU(xW 1 +b 1 )W 2 +b 2

[0112] f fusion =LayerNorm(x+FFN(x))

[0113] Step 1.5: Classifier.

[0114] The high-level features obtained by cross attention are transformed linearly to obtain the category probabilities of different categories of the image. The formula is as follows:

[0115] p i =head(f fusion ).

[0116] Step 2: Establish a model training strategy;

[0117] Based on the above embodiment, step 2 specifically includes:

[0118] Step 2.1, the maximum mean difference method is used to measure the spatial domain distribution difference loss, and the high-order moment matching method is used to measure the frequency domain distribution difference loss;

[0119] Step 2.2, cross entropy loss is used as the source domain classification loss;

[0120] Step 2.3, pseudo-label consistency learning is used as the classification loss of the weak enhancement map and the strong enhancement map of the target domain;

[0121] Step 2.4, use minimum class confusion learning on the class probability distribution obtained by the classifier to obtain the class confusion loss;

[0122] In step 2.5, the total training loss is obtained based on the spatial domain distribution difference loss, the frequency domain distribution difference loss, the source domain classification loss, the target domain weak enhancement map and strong enhancement map classification loss, and the class confusion loss.

[0123] Furthermore, the spatial domain distribution difference loss The expression is

[0124]

[0125] in, represents the reproducing kernel Hilbert space, Used to label sample features And unlabeled sample features Projected onto the reproducing kernel Hilbert space, Defines a set of Continuous functions in ;

[0126] The expression of the frequency domain distribution difference loss is:

[0127]

[0128] Where p is the order of high-order moment matching, Same as MDD, representing the reproducing kernel Hilbert space, represents the frequency domain Hilbert space feature representation of the i-th source domain and target domain sample, L p is the number of neurons in the hidden layer, n s is the number of samples in each batch, That is and Represents the random sampling value of the high-order tensor of frequency domain features. The formula is:

[0129]

[0130] Where k(x,y)=exp(-γ∥xy∥ 2 ) is the RBF kernel function. When p = 1, the high-order moment matching method metric is equivalent to the maximum mean difference metric;

[0131] The expression of the source domain classification loss is

[0132]

[0133] in, Representation sample The true probability distribution of is, the class label is 1, and the other class values ​​are 0. express The predicted probability distribution of the sample input is the category probability distribution

[0134] The expression of the classification loss of the weakly enhanced graph and the strongly enhanced graph of the target domain is:

[0135]

[0136] Among them, B represents the batch size, μ is the proportion of pseudo samples in a batch that meet the threshold selection requirements, and H represents the cross entropy loss of two probability distributions;

[0137] The expression of the class confusion loss is

[0138]

[0139] Among them, W jj represents the weight of j samples for class confusion modeling, represents the probability distribution entropy of the sample, B is the batch size, R cc' is the class confusion after sample weighting, It is R cc' The result of normalized calculation is is the class confusion loss, is the number of sample categories;

[0140] The expression of the total training loss is

[0141]

[0142] Among them, α and β are hyperparameters that balance the optimization objectives of the model.

[0143] In specific implementation, the specific process of setting the model training strategy can be as follows:

[0144] (1) Feature metric difference loss:

[0145] When there is data drift between the source domain and the target domain, it will lead to low classification and retrieval accuracy. Therefore, the present disclosure adopts the spatial domain-frequency domain dual-domain feature measurement difference.

[0146] The randomness in the strong enhancement process makes the distribution of strong enhancement samples more stable and generalized. Therefore, a spatial domain distribution difference function is established for the weak enhancement of the source domain and the strong enhancement of the target domain.

[0147] The present disclosure intends to use MMD (maximum mean difference) for the spatial domain, which is a measurement method of distribution difference commonly used in domain adaptation. s and the target domain D t ,The formulas of MMD and spatial domain distribution difference loss are as follows:

[0148]

[0149] in, represents the reproducing kernel Hilbert space (RKHS), Used to and Projected onto the reproducing kernel Hilbert space, Defines a set of In essence, it is to find a mapping relationship to minimize the difference between the maximum mean of the spatial domain features of the projected target domain and the source domain.

[0150] Different from the spatial domain, strong enhancement will significantly change the frequency domain phase of the image. Therefore, a frequency domain distribution difference function is established for the frequency domain features of weak enhancement in the source domain and weak enhancement in the target domain.

[0151] The present disclosure intends to use HoMM (high-order moment matching) for the frequency domain, which can achieve fine-grained domain alignment through a more complex non-Gaussian distribution. s and the target domain D t ,The formula of HoMM is as follows:

[0152]

[0153] Where p is the order of high-order moment matching, Same as MDD, representing the reproducing kernel Hilbert space, represents the frequency domain Hilbert space feature representation of the i-th source domain and target domain sample, L p is the number of neurons in the hidden layer, n s is the number of samples in each batch, That is and Represents the random sampling value of the high-order tensor of frequency domain features. The formula is:

[0154]

[0155] Where k(x,y)=exp(-γ∥xy∥ 2 ) is the RBF kernel function. When p = 1, HoMM is equivalent to K-MMD, and the high-order moment matching method metric is equivalent to the maximum mean difference metric.

[0156] (2) Source domain classification loss:

[0157] After the sample input, the source domain category probability distribution is obtained. Since the source domain data has corresponding annotation information, supervised learning is performed to ensure that the samples in the source domain can be accurately identified. The cross entropy loss is used as the source domain classification loss, and the formula is as follows:

[0158]

[0159] In the formula, Representation sample The true probability distribution of is, the class label is 1, and the other class values ​​are 0. express The predicted probability distribution of the sample input is the category probability distribution

[0160] (3) Pseudo-label consistency learning:

[0161] After the above process, the source domain samples can be accurately classified, while reducing the overall deviation of the source domain and target domain feature distribution. However, the decision boundary of the classifier depends on the data distribution of the source domain, and cannot effectively obtain the inter-class discrimination information of the target domain. Therefore, the present invention adopts a pseudo-label consistency learning strategy, uses the prediction results of the classifier to select high-confidence samples in the target domain, and then obtains pseudo-labels through label hardening. Weakly enhanced samples in the target domain The softmax output of is described as the predicted category probability distribution of the sample, and its formal expression is defined as Generally speaking, samples with higher confidence levels will generate higher probability values ​​for their categories, while noise samples have lower prediction probabilities for each category and tend to be evenly distributed because the classifier cannot give reliable predictions. Therefore, the category labels whose maximum category probability is higher than the predefined threshold τ are retained as hard labels. The process formula is:

[0162]

[0163] The decision boundary of the classifier is perturbed by the strongly enhanced samples, and the pseudo-labels generated by the weakly enhanced samples are used as the supervisory information for the corresponding strongly enhanced samples. The pseudo-label classification loss of the consistent regularized enhanced is:

[0164]

[0165] Where B represents the batch size, μ is the proportion of pseudo samples in a batch that meet the threshold selection requirements, and H represents the cross entropy loss of two probability distributions.

[0166] (4) Minimum Class Confusion Learning:

[0167] Pseudo-label consistency learning improves the discrimination ability of the classifier through high-confidence samples in the target domain. However, due to the lack of constraints on the distance between feature classes, difficult samples near the decision boundary are easily confused, and the significant intra-class variance and inter-class similarity of remote sensing images further aggravate this problem. Therefore, the present disclosure adopts minimum class confusion learning.

[0168] Minimum class confusion learning determines the weights of target domain samples based on the entropy of the predicted class probabilities. Definition represents the probability that all samples in the batch come from class c. According to information theory, the entropy of the distribution p The probability distribution entropy of the target domain samples predicted by the classifier is:

[0169]

[0170] in, It represents the probability that the jth sample in the target domain belongs to the cth class. The entropy value is inverted and then logarithmized to ensure that the samples with high confidence in model prediction have a greater impact on class confusion modeling. The weight formula, i.e., the class confusion after sample weighting, is:

[0171]

[0172] This disclosure uses category normalization to calculate the class confusion matrix and obtains the class confusion loss, the formula is as follows:

[0173]

[0174] The total loss of pre-training is:

[0175]

[0176] Among them, α and β are hyperparameters that balance the optimization objectives of the model.

[0177] Step 3, dividing the sample data set into a training set, a test set and a validation set according to a preset ratio, training a closed set domain adaptation model that integrates the spatial domain and frequency domain of remote sensing images through the training set and the model training strategy, and testing and validating it through the test set and the validation set to obtain a trained classification retrieval model;

[0178] In specific implementation, the samples can be divided into training set, test set and validation set according to the ratio of 8:1:1. The training set cannot contain samples existing in the validation set and the test set, and the validation set cannot contain samples existing in the test set.

[0179] Then the training set source domain and target domain samples are obtained through the closed set domain adaptation model of remote sensing image space-frequency domain. After feature extraction, the feature fusion module is used to obtain the source domain, target domain weak enhancement, and target domain strong enhancement high-level features. And input it into the classifier to get the category probability distribution

[0180] Finally, the test set and validation set remote sensing images are input into the remote sensing image classification-retrieval model that has been trained with the training set. The hyperparameters in the model are fine-tuned according to the classification accuracy and retrieval accuracy to obtain the most stable classification and retrieval model.

[0181] Step 4: Input the target remote sensing image into the trained classification retrieval model to obtain the classification result and retrieval result.

[0182] Based on the above embodiment, step 4 specifically includes:

[0183] Step 4.1: Input the target remote sensing image into the trained classification retrieval model to obtain the sample category probability distribution p i, the category corresponding to the maximum value of the sample category probability is taken as the classification result of the target remote sensing image;

[0184] Step 4.2: Input all images in the retrieval database into the trained classification retrieval model to obtain the retrieval data feature library F = [f 1 ,f 2 ,…,f n ], input the target remote sensing image into the trained classification retrieval model to obtain the target high-level features The cosine similarity is used to calculate the similarity between the target high-level features and all image features in the retrieval data feature library, and the similarity is sorted from high to low, and the most similar image list is returned as the retrieval result.

[0185] Furthermore, the expression of the classification result is

[0186]

[0187] The expression of the similarity is:

[0188]

[0189] Among them, f i Any image feature of the retrieval data feature library.

[0190] In specific implementation, the process of remote sensing image classification and retrieval can be as follows:

[0191] Step 4.1 Remote sensing image classification

[0192] The remote sensing image is input into the remote sensing image classification-retrieval model that has been trained with the training set to obtain the sample category probability distribution p i .

[0193] The category corresponding to the maximum value of the sample category probability is the classification result of the sample, and the formula is:

[0194]

[0195] Step 4.2 Remote sensing image retrieval

[0196] Input all images in the retrieval database into the remote sensing image classification-retrieval model that has been trained with the training set, and obtain the retrieval data feature library F = [f 1 ,f 2 ,…,f n ]

[0197] Input the remote sensing image into the remote sensing image classification-retrieval model that has been trained with the training set to obtain the sample high-level features

[0198] Use cosine similarity to calculate the similarity between the query image features and all stored image features. The formula is as follows:

[0199]

[0200] Sort by similarity from high to low (or distance from low to high), and return a list of the most similar images.

[0201] The closed-set domain adaptation remote sensing image classification and retrieval method provided in this embodiment integrates the spatial domain and the frequency domain. At the feature level, the spatial domain and frequency domain features are obtained through the spatial domain and frequency domain encoders, and a space-frequency cross-attention fusion module is designed to obtain high-level features of the image. Compared with other domain adaptation methods, it not only retains the spatial features and rich semantic features of the spatial domain features, but also effectively introduces additional frequency domain information; in terms of model training strategy, it introduces a spatial domain and frequency domain feature difference measurement method, a consistency regularized enhanced pseudo-label learning method, and minimized class confusion learning. Compared with the prior art, the present disclosure optimizes the model from multiple aspects such as data offset, category decision boundary, similar class confusion, etc., which is more conducive to remote sensing image classification and retrieval tasks in complex cross-domain situations, and solves the problem that complex remote sensing image cross-domain classification-retrieval tasks only focus on spatial domain information and cannot obtain good generalization results due to the influence of remote sensing image background complexity and multi-source data distribution differences.

[0202] The method of the present invention will be further illustrated below in conjunction with a specific embodiment. By using the public remote sensing datasets UCMD and NWPU, an experimental analysis was carried out in the constructed UCMD→NWPU complex cross-domain scenario by comparing Source (without domain adaptation), MCC (minimum class confusion), and PCLUDA (a closed-set domain adaptation remote sensing image retrieval method based on pseudo-label consistency learning). As shown in Tables 1 and 2, Table 1 shows the classification accuracy of different methods in the UCMD→NWPU cross-domain scenario, and Table 2 shows the retrieval accuracy mAP (%) of different methods in the UCMD→NWPU cross-domain scenario. The results show that the closed-set domain adaptation remote sensing image classification and retrieval method that integrates the spatial domain and the frequency domain proposed in the present invention is superior to the existing domain adaptation method.

[0203] Table 1

[0204]

[0205] Table 2

[0206]

[0207] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.

[0208] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A closed-domain adaptive remote sensing image classification and retrieval method integrating spatial domain and frequency domain, It is characterized in that include: Step 1, establishing a closed set domain adaptation model for fusing the space domain and frequency domain of remote sensing images, wherein the closed set domain adaptation model for fusing the space domain and frequency domain of remote sensing images includes a data enhancement module, a space domain feature extraction module, a frequency domain feature extraction module, a space-frequency cross attention fusion module, a feedforward and a classifier; Step 2: Establish a model training strategy; Step 3, dividing the sample data set into a training set, a test set and a validation set according to a preset ratio, training a closed set domain adaptation model that integrates the spatial domain and frequency domain of remote sensing images through the training set and the model training strategy, and testing and validating it through the test set and the validation set to obtain a trained classification retrieval model; Step 4: Input the target remote sensing image into the trained classification retrieval model to obtain the classification result and retrieval result.

2. The method according to claim 1, Features ,The data enhancement module is used to weakly enhance the source domain image to obtain the source domain weakly enhanced view, and ,to weakly enhance and strongly enhance the target domain to obtain the target domain weakly enhanced image and the target domain strongly enhanced view; The spatial domain feature extraction module is used to sequentially input the source domain weak enhancement map, the target domain weak enhancement map, and the strong enhancement map into the spatial domain encoder to obtain the spatial domain feature The frequency domain feature extraction module is used to perform Fourier transform on the source domain weak enhancement image, the target domain weak enhancement image, and the strong enhancement image to obtain the image frequency domain, and then set the frequency domain filter to input the frequency domain information corresponding to the source domain weak enhancement image, the target domain weak enhancement image, and the target domain strong enhancement image obtained by the transformation into the frequency domain encoder in sequence to obtain the frequency domain feature extraction module. The space-frequency cross attention fusion module is used to generate queries, keys and values ​​from spatial domain features and frequency domain features through linear transformation, and use the query value of the spatial domain feature and the key value of the frequency domain feature for cross calculation, use the multi-head attention mechanism to linearly transform the query, key and value respectively, and then divide them into multiple heads, independently calculate the attention of each head, and finally connect the outputs of all heads, and then generate the final output through a linear transformation, and the final output is added to the spatial domain feature through a residual connection, and layer normalization is performed, and finally a feedforward neural network is passed to obtain the fused high-level features; The classifier is used to obtain the category probabilities of different categories of the image by linearly transforming the high-level features.

3. The method according to claim 2, Features , the expression for generating query, key and value is Among them, Q sq , K sq and V sq are the query, key and value of spatial domain features respectively, Q fre , K fre and V fre are the query, key and value of frequency domain features respectively, is the parameter of the linear transformation; The expression of the crossover calculation is: The final output expression is: MultiHead(Q,K,V)=Concat(head 1 ,…,head h )W O Among them, MultiHead(·) is a multi-head attention mechanism.

4. The method according to claim 3, Features , the step 2 specifically includes: Step 2.1, the maximum mean difference method is used to measure the spatial domain distribution difference loss, and the high-order moment matching method is used to measure the frequency domain distribution difference loss; Step 2.2, cross entropy loss is used as the source domain classification loss; Step 2.3, pseudo-label consistency learning is used as the classification loss of the weak enhancement map and the strong enhancement map of the target domain; Step 2.4, use minimum class confusion learning on the class probability distribution obtained by the classifier to obtain the class confusion loss; In step 2.5, the total training loss is obtained based on the spatial domain distribution difference loss, the frequency domain distribution difference loss, the source domain classification loss, the target domain weak enhancement map and strong enhancement map classification loss, and the class confusion loss.

5. The method according to claim 4, Features , the spatial domain distribution difference loss The expression is in, represents the reproducing kernel Hilbert space, Used to label sample features And unlabeled sample features Projected onto the reproducing kernel Hilbert space, Defines a set of Continuous functions in ; The expression of the frequency domain distribution difference loss is: Where p is the order of high-order moment matching, Same as MDD, representing the reproducing kernel Hilbert space, represents the frequency domain Hilbert space feature representation of the i-th source domain and target domain sample, L p is the number of neurons in the hidden layer, n s is the number of samples in each batch, That is and Represents the random sampling value of the high-order tensor of frequency domain features. The formula is: Where k(x,y)=exp(-γ∥xy∥ 2 ) is the RBF kernel function. When p = 1, the high-order moment matching method metric is equivalent to the maximum mean difference metric; The expression of the source domain classification loss is in, Representation sample The true probability distribution of is, the class label is 1, and the other class values ​​are 0. express The predicted probability distribution of the sample input is the category probability distribution The expression of the classification loss of the weakly enhanced graph and the strongly enhanced graph of the target domain is: Among them, B represents the batch size, μ is the proportion of pseudo samples in a batch that meet the threshold selection requirements, and H represents the cross entropy loss of two probability distributions; The expression of the class confusion loss is Among them, W jj represents the weight of the j-th sample for class confusion modeling, represents the probability distribution entropy of the sample, B is the batch size, R cc' is the class confusion after sample weighting, It is R cc' The result of normalized calculation is is the class confusion loss, is the number of sample categories; The expression of the total training loss is Among them, α and β are hyperparameters that balance the optimization objectives of the model.

6. The method according to claim 5, Features , the step 4 specifically includes: Step 4.1: Input the target remote sensing image into the trained classification retrieval model to obtain the sample category probability distribution p i , the category corresponding to the maximum value of the sample category probability is taken as the classification result of the target remote sensing image; Step 4.2: Input all images in the retrieval database into the trained classification retrieval model to obtain the retrieval data feature library F = [f 1 ,f 2 ,…,f n ], input the target remote sensing image into the trained classification retrieval model to obtain the target high-level features The cosine similarity is used to calculate the similarity between the target high-level features and all image features in the retrieval data feature library, and the similarity is sorted from high to low, and the most similar image list is returned as the retrieval result.

7. The method according to claim 6, Features , the expression of the classification result is The expression of the similarity is: Among them, f i Any image feature of the retrieval data feature library.

Images