Closed-set domain adaptation method for remote sensing image classification and retrieval by fusing spatial and frequency domains

Abstract

The embodiment of the present disclosure provides a closed set domain adaptive remote sensing image classification retrieval method fusing a spatial domain and a frequency domain, and belongs to the technical field of data processing. Specifically, the method comprises the following steps: establishing a closed set domain adaptive model fusing a spatial domain and a frequency domain of a remote sensing image; establishing a model training strategy; dividing a sample data set into a training set, a test set and a verification set according to a preset proportion; training the closed set domain adaptive model through the training set and the model training strategy, and testing and verifying the closed set domain adaptive model through the test set and the verification set to obtain a trained classification retrieval model; and inputting a target remote sensing image into the trained classification retrieval model to obtain a classification result and a retrieval result. Through the scheme of the present disclosure, the problem that a complex remote sensing image cross-domain classification retrieval task can only focus on spatial domain information and cannot obtain a good generalization result due to the influence of remote sensing image background complexity and multi-source data distribution difference is solved, and the classification retrieval precision and adaptability are improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to a closed-set domain adaptive remote sensing image classification and retrieval method that integrates spatial domain and frequency domain. Background Art

[0002] With the continuous advancement of remote sensing technology, Earth observation and data capture by satellite and aerial sensors have become increasingly efficient, generating unprecedented amounts of remote sensing data. The large-scale acquisition of high-resolution remote sensing imagery has laid a big data foundation for applications across various fields. This massive amount of data brings with it a tremendous amount of information. The challenge of quickly and efficiently classifying and retrieving images that meet user needs, with minimal or no labels, has become a major obstacle to the sharing and effective use of remote sensing image data.

[0003] To address the problem of data distribution shift, domain adaptation (DA) learning has been proposed to transfer knowledge from a labeled source domain to an unlabeled target domain, under the premise that the data distributions of the source domain (training set) and the target domain (test set) are different but similar, thereby improving the model's generalization ability in the target domain. Depending on whether the target domain has labeled information, domain adaptation can be divided into supervised domain adaptation and unsupervised domain adaptation. Unsupervised domain adaptation (UDA) is the most challenging and most practical research direction in domain adaptation learning, meaning that the target domain does not contain any labeled information. Depending on whether the structure of the label (category) space between domains is the same, domain adaptation can be divided into domain adaptation with consistent label spaces and domain adaptation with inconsistent label spaces. Domain adaptation with consistent label spaces, also known as closed-set domain adaptation, refers to domain adaptation problems where the source and target domains have the same label space and is the focus of most existing domain adaptation research. Currently, some mainstream closed-set domain adaptation methods can be divided into three main types: ① Distribution difference-based: Deep features are obtained through the model, and then the distribution difference-based method is used to reduce the distribution difference between domains to reduce generalization error. ② Adversarial learning-based: Drawing on the concept of generative adversarial networks, domain-invariant feature representations are learned through adversarial training between feature extractors and domain discriminators. ③ Self-training-based: Domain adaptation is achieved by utilizing unlabeled target domain data. This method generates pseudo-labels for the unlabeled data and retrains the network by combining the source domain labels with the target domain pseudo-labels.

[0004] Currently, domain adaptation is primarily based on spatial image features, using deep neural networks to bring the spatial information of the source and target domains closer together. However, due to the diverse range of object types and complex background information contained in remote sensing images, existing methods still have room for improvement in classification and retrieval accuracy in complex multi-source scenarios.

[0005] Current research has demonstrated that frequency domain features of images do not exhibit significant feature shifts between different datasets. Furthermore, frequency domain features contain detailed information such as texture and structure. Research has applied these features to target recognition and change detection, and combining them with frequency features may yield more accurate predictions in the target domain. Therefore, addressing cross-domain sample category confusion and leveraging spatial and frequency domain information to improve closed-domain adaptation are pressing challenges. Summary of the Invention

[0006] In view of this, the embodiments of the present disclosure provide a closed-set domain adaptive remote sensing image classification and retrieval method that integrates the spatial domain and the frequency domain, which at least partially solves the problems of poor classification and retrieval accuracy and adaptability in the prior art.

[0007] The present disclosure provides a closed-set domain adaptive remote sensing image classification and retrieval method that integrates spatial domain and frequency domain, including:

[0008] Step 1, establishing a closed-set domain adaptation model for fusing the spatial domain and frequency domain of remote sensing images, wherein the closed-set domain adaptation model for fusing the spatial domain and frequency domain of remote sensing images includes a data enhancement module, a spatial domain feature extraction module, a frequency domain feature extraction module, a space-frequency cross-attention fusion module, a feedforward and a classifier;

[0009] Step 2: Establish a model training strategy;

[0010] Step 3: Divide the sample data set into a training set, a test set, and a validation set according to a preset ratio. Use the training set and model training strategy to train a closed-set domain adaptation model that integrates the spatial domain and frequency domain of remote sensing images. Test and verify it using the test set and validation set to obtain a trained classification retrieval model.

[0011] Step 4: Input the target remote sensing image into the trained classification retrieval model to obtain the classification results and retrieval results.

[0012] According to a specific implementation of the embodiment of the present disclosure, the data enhancement module is used to weakly enhance the source domain image to obtain a source domain weakly enhanced view, and to weakly enhance and strongly enhance the target domain to obtain a target domain weakly enhanced image and a target domain strongly enhanced view;

[0013] The spatial domain feature extraction module is used to sequentially input the source domain weak enhancement map, the target domain weak enhancement map, and the strong enhancement map into the spatial domain encoder to obtain the spatial domain feature.

[0014] The frequency domain feature extraction module is used to perform Fourier transform on the source domain weak enhancement image, the target domain weak enhancement image, and the strong enhancement image to obtain the image frequency domain, and then set the frequency domain filter to input the frequency domain information corresponding to the source domain weak enhancement image, the target domain weak enhancement image, and the target domain strong enhancement image obtained by the transformation into the frequency domain encoder in sequence to obtain the frequency domain feature.

[0015] The spatial-frequency cross-attention fusion module is used to generate queries, keys, and values ​​from spatial domain features and frequency domain features through linear transformation, and use the query value of the spatial domain feature and the key value of the frequency domain feature for cross calculation. The query, key, and value are linearly transformed using a multi-head attention mechanism, and then divided into multiple heads. The attention of each head is independently calculated, and finally the outputs of all heads are connected. The final output is then generated through a linear transformation. The final output is added to the spatial domain feature through a residual connection and layer normalization is performed. Finally, a feedforward neural network is used to obtain the fused high-level features.

[0016] The classifier is used to obtain the category probabilities of different categories of the image by linearly transforming the high-level features.

[0017] According to a specific implementation of the embodiment of the present disclosure, the expression for generating query, key and value is:

[0018]

[0019] Among them, Q sq , K sq and V sq are the query, key and value of spatial domain features respectively, Q fre , K fre and V fre are the query, key and value of frequency domain features respectively, is the parameter of the linear transformation;

[0020] The expression of the cross calculation is:

[0021]

[0022] The final output expression is MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0023]

[0024] Among them, MultiHead(·) is a multi-head attention mechanism;

[0025] According to a specific implementation of the embodiment of the present disclosure, step 2 specifically includes:

[0026] Step 2.1: The maximum mean difference method is used to measure the spatial domain distribution difference loss, and the high-order moment matching method is used to measure the frequency domain distribution difference loss.

[0027] In step 2.2, cross entropy loss is used as the source domain classification loss;

[0028] Step 2.3: Use pseudo-label consistency learning as the classification loss for the weak and strong enhancement maps of the target domain.

[0029] Step 2.4: Apply minimum class confusion learning to the class probability distribution obtained by the classifier to obtain the class confusion loss.

[0030] In step 2.5, the total training loss is obtained based on the spatial domain distribution difference loss, the frequency domain distribution difference loss, the source domain classification loss, the target domain weak enhancement map and strong enhancement map classification loss, and the class confusion loss.

[0031] According to a specific implementation of the embodiment of the present disclosure, the spatial domain distribution difference loss The expression is

[0032]

[0033] in, represents the reproducing kernel Hilbert space, Used to label sample features and unlabeled sample features Projected onto the reproducing kernel Hilbert space, Defines a set of Continuous functions in ;

[0034] The expression of the frequency domain distribution difference loss is:

[0035]

[0036] Where p is the order of high-order moment matching, Same as MDD, representing the reproducing kernel Hilbert space, Represents the frequency domain Hilbert space feature representation of the i-th source domain and target domain samples, L p is the number of neurons in the hidden layer, n s is the number of samples in each batch, That is and Represents the random sampling value of the high-order tensor of frequency domain features. The formula is:

[0037]

[0038] Where k(x,y)=exp(-γ∥xy∥2) is the RBF kernel function. When p=1, the high-order moment matching method is equivalent to the maximum mean difference metric.

[0039] The expression of the source domain classification loss is:

[0040]

[0041] in, Represents a sample The true probability distribution of , that is, the class label is 1, and the rest of the class values ​​are 0, express The predicted probability distribution, that is, the category probability distribution obtained by sample input

[0042] The expression of the classification loss of the target domain weak enhancement map and strong enhancement map is:

[0043]

[0044] Where B represents the batch size, μ is the proportion of pseudo samples in a batch that meet the threshold selection requirements, and H represents the cross entropy loss of the two probability distributions;

[0045] The expression of the class confusion loss is

[0046]

[0047] Among them, W jj represents the weight of the j-th sample for modeling class confusion, represents the probability distribution entropy of the sample, B is the batch size, R cc' is the class confusion after sample weighting, It is R cc' The result of normalized calculation is is the class confusion loss, is the number of sample categories;

[0048] The expression of the total training loss is

[0049]

[0050] Among them, α and β are hyperparameters that balance the model optimization objectives.

[0051] According to a specific implementation of the embodiment of the present disclosure, step 4 specifically includes:

[0052] Step 4.1: Input the target remote sensing image into the trained classification retrieval model to obtain the sample category probability distribution p i, the category corresponding to the maximum value of the sample category probability is taken as the classification result of the target remote sensing image;

[0053] Step 4.2: Input all images in the retrieval database into the trained classification retrieval model to obtain the retrieval data feature library F = [f 1 ,f 2 ,…,f n ], input the target remote sensing image into the trained classification retrieval model to obtain the target high-level features The cosine similarity is used to calculate the similarity between the target high-level features and all image features in the retrieval data feature library, and the similarity is sorted from high to low according to the similarity, and a list of the most similar images is returned as the retrieval result.

[0054] According to a specific implementation of the embodiment of the present disclosure, the expression of the classification result is

[0055]

[0056] The expression of the similarity is:

[0057]

[0058] Among them, f i Any image feature in the retrieval data feature library.

[0059] The closed-set domain adaptation remote sensing image classification and retrieval scheme for fusing spatial and frequency domains in the disclosed embodiment includes: step 1, establishing a closed-set domain adaptation model for fusing spatial and frequency domains of remote sensing images, wherein the closed-set domain adaptation model for fusing spatial and frequency domains of remote sensing images includes a data enhancement module, a spatial domain feature extraction module, a frequency domain feature extraction module, a space-frequency cross-attention fusion module, a feedforward and a classifier; step 2, establishing a model training strategy; step 3, dividing the sample data set into a training set, a test set and a validation set according to a preset ratio, training the closed-set domain adaptation model for fusing spatial and frequency domains of remote sensing images through the training set and the model training strategy, and testing and verifying it through the test set and the validation set to obtain a trained classification and retrieval model; step 4, inputting the target remote sensing image into the trained classification and retrieval model to obtain classification results and retrieval results.

[0060] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, spatial domain information and frequency domain information are obtained through a spatial domain encoder and a frequency domain encoder, and a spatial domain and frequency domain feature difference measurement method is introduced to reduce the data offset between the source domain and the target domain. At the same time, a space-frequency cross-attention fusion module is designed on the basis of the above, which not only retains the spatial features and rich semantic features of the spatial domain features, but also effectively introduces additional frequency domain information. Finally, a pseudo-label learning method with consistency regularization enhancement is introduced to adaptively adjust the decision boundary according to the posterior class distribution information of the target domain. And the minimization class confusion learning strategy is embedded in the pseudo-label consistency learning branch to reduce the confusion of similar classes around the decision boundary, so that the classifier can give a higher confidence prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] Figure 1 A flowchart of a closed-set domain adaptive remote sensing image classification and retrieval method that integrates spatial and frequency domains is provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0064] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0065] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0066] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0067] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0068] The embodiments of the present disclosure provide a closed-set domain adaptive remote sensing image classification and retrieval method that integrates the spatial domain and the frequency domain. The method can be applied to the classification and retrieval of remote sensing images in scenarios such as urban management and geological monitoring.

[0069] See also Figure 1 , is a flow chart of a closed-set domain adaptive remote sensing image classification and retrieval method that integrates the spatial domain and the frequency domain provided by the embodiment of the present disclosure. Figure 1 As shown, the method mainly includes the following steps:

[0070] Step 1, establishing a closed-set domain adaptation model for fusing the spatial domain and frequency domain of remote sensing images, wherein the closed-set domain adaptation model for fusing the spatial domain and frequency domain of remote sensing images includes a data enhancement module, a spatial domain feature extraction module, a frequency domain feature extraction module, a space-frequency cross-attention fusion module, a feedforward and a classifier;

[0071] Furthermore, the data enhancement module is used to weakly enhance the source domain image to obtain a source domain weakly enhanced view, and to weakly enhance and strongly enhance the target domain to obtain a target domain weakly enhanced image and a target domain strongly enhanced view;

[0072] The spatial domain feature extraction module is used to sequentially input the source domain weak enhancement map, the target domain weak enhancement map, and the strong enhancement map into the spatial domain encoder to obtain the spatial domain feature.

[0073] The frequency domain feature extraction module is used to perform Fourier transform on the source domain weak enhancement image, the target domain weak enhancement image, and the strong enhancement image to obtain the image frequency domain, and then set the frequency domain filter to input the frequency domain information corresponding to the source domain weak enhancement image, the target domain weak enhancement image, and the target domain strong enhancement image obtained by the transformation into the frequency domain encoder in sequence to obtain the frequency domain feature.

[0074] The spatial-frequency cross-attention fusion module is used to generate queries, keys, and values ​​from spatial domain features and frequency domain features through linear transformation, and use the query value of the spatial domain feature and the key value of the frequency domain feature for cross calculation. The query, key, and value are linearly transformed using a multi-head attention mechanism, and then divided into multiple heads. The attention of each head is independently calculated, and finally the outputs of all heads are connected. The final output is then generated through a linear transformation. The final output is added to the spatial domain feature through a residual connection and layer normalization is performed. Finally, a feedforward neural network is used to obtain the fused high-level features.

[0075] The classifier is used to obtain the category probabilities of different categories of the image by linearly transforming the high-level features.

[0076] Furthermore, the expression for generating query, key and value is

[0077]

[0078] Among them, Q sq , K sq and V sq are the query, key and value of spatial domain features respectively, Q fre , K fre and V fre are the query, key and value of frequency domain features respectively, is the parameter of the linear transformation;

[0079] The expression of the cross calculation is:

[0080]

[0081] The final output expression is

[0082] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0083]

[0084] Among them, MultiHead(·) is a multi-head attention mechanism;

[0085] In specific implementation, the specific process of establishing a closed set domain adaptation model in the spatial frequency domain of remote sensing images can be as follows:

[0086] Step 1.1: Data and data augmentation: weakly augmented view of the source domain, weakly augmented and strongly augmented views of the target domain.

[0087] A formal definition of the closed domain adaptation problem for remote sensing images is given: given a set containing n s The source domain of the samples and a containing n t Target domain of the sample in and denote labeled samples and unlabeled samples respectively. represent The corresponding label, C s It should be noted that the target domain shares the same label space as the source domain, but its samples do not have annotation information. The purpose of domain adaptation learning is to use the labeled source domain D s and the unlabeled target domain D t Learn a classification model so that the model can achieve good results in the target domain.

[0088] For the source domain image, horizontal flipping and random cropping with a certain probability are performed in sequence to obtain a weakly enhanced view. For the target domain, the weakly enhanced view is obtained in the same way, and the strongly enhanced view is obtained by RandAugment enhancement.

[0089] Step 1.2: Spatial domain feature extraction.

[0090] The source domain weak enhancement, target domain weak enhancement, and strong enhancement samples are sequentially input into the spatial domain encoder to obtain spatial domain features. The spatial domain encoder disclosed in this disclosure is intended to adopt Vi T-B16.

[0091] Step 1.3: Frequency domain feature extraction.

[0092] Perform Fourier transform on the weakly enhanced source domain, weakly enhanced target domain, and strongly enhanced samples to obtain the image frequency domain. The formula is as follows:

[0093]

[0094] Set the frequency domain filter Mask, whose value is the high-frequency and low-frequency parts of the sample. This disclosure intends to use the phase part θ(u) of the low-frequency frequency domain, as follows, where h and w are the height and width of the sample, and b is the hyperparameter controlling the filter.

[0095]

[0096] F * (u)=Mask·F(u)

[0097]

[0098] The frequency domain information obtained by weak enhancement of the source domain, weak enhancement of the target domain, and strong enhancement is input into the frequency domain encoder in sequence to obtain the frequency domain features. The frequency domain encoder disclosed in this invention intends to adopt Resnet-50.

[0099] Step 1.4: Spatial-frequency cross attention fusion module.

[0100] Through linear transformation, the spatial domain and frequency domain features are converted into query, key and value:

[0101]

[0102]

[0103] Using the query value of the spatial domain feature and the key value of the frequency domain feature for cross-calculation can help the model effectively capture the complex dependencies that may exist between the spatial domain and frequency domain information. The formula is as follows:

[0104]

[0105] Use the multi-head attention mechanism to capture more detailed information. The query, key, and value are linearly transformed, then divided into multiple heads, and the attention of each head is calculated independently. Finally, the outputs of all heads are concatenated and a linear transformation is performed to generate the final output:

[0106] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0107]

[0108] Residual connection and normalization, the output is added to the spatial domain features through residual connection, and layer normalization is performed to ensure gradient stability. The formula is as follows:

[0109] x=LayerNorm(V sq +MultiHead(Q,K,V)

[0110] Finally, a feedforward neural network is used to obtain the fused high-level features. The feedforward neural network consists of two fully connected layers and an activation function. The formula is as follows:

[0111] FFN(x)=ReLU(xW1+b1)W2+b2

[0112] f fusion =LayerNorm(x+FFN(x))

[0113] Step 1.5: Classifier.

[0114] The high-level features obtained by cross attention are linearly transformed to obtain the category probabilities of different categories of the image. The formula is as follows:

[0115] p i =head(f fusion ).

[0116] Step 2: Establish a model training strategy;

[0117] Based on the above embodiment, step 2 specifically includes:

[0118] Step 2.1: The maximum mean difference method is used to measure the spatial domain distribution difference loss, and the high-order moment matching method is used to measure the frequency domain distribution difference loss.

[0119] In step 2.2, cross entropy loss is used as the source domain classification loss;

[0120] Step 2.3: Use pseudo-label consistency learning as the classification loss for the weak and strong enhancement maps of the target domain.

[0121] Step 2.4: Apply minimum class confusion learning to the class probability distribution obtained by the classifier to obtain the class confusion loss.

[0122] In step 2.5, the total training loss is obtained based on the spatial domain distribution difference loss, the frequency domain distribution difference loss, the source domain classification loss, the target domain weak enhancement map and strong enhancement map classification loss, and the class confusion loss.

[0123] Furthermore, the spatial domain distribution difference loss The expression is

[0124]

[0125] in, represents the reproducing kernel Hilbert space, Used to label sample features and unlabeled sample features Projected onto the reproducing kernel Hilbert space, Defines a set of Continuous functions in ;

[0126] The expression of the frequency domain distribution difference loss is:

[0127]

[0128] Where p is the order of high-order moment matching, Same as MDD, representing the reproducing kernel Hilbert space, Represents the frequency domain Hilbert space feature representation of the i-th source domain and target domain samples, L p is the number of neurons in the hidden layer, n s is the number of samples in each batch, That is and Represents the random sampling value of the high-order tensor of frequency domain features. The formula is:

[0129]

[0130] Where k(x,y)=exp(-γ∥xy∥2) is the RBF kernel function. When p=1, the high-order moment matching method is equivalent to the maximum mean difference metric.

[0131] The expression of the source domain classification loss is:

[0132]

[0133] in, Represents a sample The true probability distribution of , that is, the class label is 1, and the rest of the class values ​​are 0, express The predicted probability distribution, that is, the category probability distribution obtained by sample input

[0134] The expression of the classification loss of the target domain weak enhancement map and strong enhancement map is:

[0135]

[0136] Where B represents the batch size, μ is the proportion of pseudo samples in a batch that meet the threshold selection requirements, and H represents the cross entropy loss of the two probability distributions;

[0137] The expression of the class confusion loss is

[0138]

[0139] Among them, W jj represents the weight of j samples for class confusion modeling, represents the probability distribution entropy of the sample, B is the batch size, R cc' is the class confusion after sample weighting, It is R cc' The result of normalized calculation is is the class confusion loss, is the number of categories of samples;

[0140] The expression of the total training loss is

[0141]

[0142] Among them, α and β are hyperparameters that balance the model optimization objectives.

[0143] In specific implementation, the specific process of setting the model training strategy can be as follows:

[0144] (1) Feature metric difference loss:

[0145] When there is data drift between the source domain and the target domain, it will lead to low classification and retrieval accuracy. Therefore, this disclosure adopts the spatial domain and frequency domain dual-domain feature measurement difference.

[0146] The randomness in the strong enhancement process makes the distribution of strong enhancement samples more stable and generalized. Therefore, a spatial domain distribution difference function is established for the weak enhancement of the source domain and the strong enhancement of the target domain.

[0147] This disclosure proposes to use MMD (Maximum Mean Difference) for the spatial domain, which is a distribution difference measurement method commonly used in domain adaptation. s and target domain D t ,The formulas of MMD and spatial domain distribution difference loss are as follows:

[0148]

[0149] in, represents the reproducing kernel Hilbert space (RKHS), Used to and Projected onto the reproducing kernel Hilbert space, Defines a set of In essence, it is to find a mapping relationship that minimizes the difference between the maximum mean of the spatial domain features of the projected target domain and the source domain.

[0150] Different from the spatial domain, strong enhancement will significantly change the frequency domain phase of the image. Therefore, a frequency domain distribution difference function is established for the frequency domain features of weak enhancement in the source domain and weak enhancement in the target domain.

[0151] The present disclosure proposes to use HoMM (high-order moment matching) for the frequency domain, which can achieve fine-grained domain alignment through more complex non-Gaussian distribution. s and target domain D t , the formula of HoMM is as follows:

[0152]

[0153] Where p is the order of high-order moment matching, Same as MDD, representing the reproducing kernel Hilbert space, Represents the frequency domain Hilbert space feature representation of the i-th source domain and target domain samples, L p is the number of neurons in the hidden layer, n s is the number of samples in each batch, That is and Represents the random sampling value of the high-order tensor of frequency domain features. The formula is:

[0154]

[0155] Where k(x,y)=exp(-γ||xy||2) is the RBF kernel function. When p=1, HoMM is equivalent to K-MMD, and the high-order moment matching method metric is equivalent to the maximum mean difference metric.

[0156] (2) Source domain classification loss:

[0157] After the sample input, the source domain category probability distribution is obtained. Since the source domain data has corresponding annotation information, supervised learning is performed on it to ensure that the samples in the source domain can be accurately identified. The cross entropy loss is used as the source domain classification loss, and the formula is as follows:

[0158]

[0159] Where, Represents a sample The true probability distribution of , that is, the class label is 1, and the rest of the class values ​​are 0, express The predicted probability distribution, that is, the category probability distribution obtained by sample input

[0160] (3) Pseudo-label consistency learning:

[0161] After the above process, the source domain samples can be accurately classified, while reducing the overall deviation of the source domain and target domain feature distribution. However, the decision boundary of the classifier depends on the data distribution of the source domain, and cannot effectively obtain the inter-class discrimination information of the target domain. Therefore, the present invention adopts a pseudo-label consistency learning strategy, using the prediction results of the classifier to select high-confidence samples in the target domain, and then obtains pseudo-labels through label hardening. Weakly enhanced samples in the target domain The softmax output is described as the predicted category probability distribution of the sample, and its formal expression is defined as Generally speaking, samples with higher confidence levels will generate higher probability values ​​for their respective categories, while noise samples, because the classifier cannot give reliable predictions, have relatively low predicted probabilities for each category and tend to be evenly distributed. Therefore, the category labels whose maximum category probability is higher than the predefined threshold τ are retained as hard labels. The process formula is:

[0162]

[0163] The decision boundary of the classifier is perturbed by the strong enhancement sample, and the pseudo-label generated by the weak enhancement sample is used as the supervision information for the corresponding strong enhancement sample. The pseudo-label classification loss of the consistency regularization enhancement is:

[0164]

[0165] Where B represents the batch size, μ is the proportion of pseudo samples in a batch that meet the threshold selection requirements, and H represents the cross entropy loss of the two probability distributions.

[0166] (4) Minimum class confusion learning:

[0167] Pseudo-label consistency learning improves the classifier's discriminative ability by using high-confidence samples in the target domain. However, the lack of inter-class distance constraints makes difficult samples near the decision boundary easily confused. This problem is further exacerbated by the significant intra-class variance and inter-class similarity of remote sensing images. Therefore, this paper adopts minimum class confusion learning.

[0168] Minimum class confusion learning determines the weights of target domain samples based on the entropy of the predicted class probabilities. Definition represents the probability that all samples in the batch come from class c. According to information theory, the entropy of the distribution p The probability distribution entropy of the target domain samples predicted by the classifier is:

[0169]

[0170] in, It represents the probability that the jth sample in the target domain belongs to the cth class. The entropy value is inverted and then logarithmized to ensure that samples with high model prediction confidence have a greater impact on class confusion modeling. Its weight formula, that is, the class confusion after sample weighting, is:

[0171]

[0172] This paper uses category normalization to calculate the class confusion matrix and obtains the class confusion loss. The formula is as follows:

[0173]

[0174] The total loss of pre-training is:

[0175]

[0176] Among them, α and β are hyperparameters that balance the model optimization objectives.

[0177] Step 3: Divide the sample data set into a training set, a test set, and a validation set according to a preset ratio. Use the training set and model training strategy to train a closed-set domain adaptation model that integrates the spatial domain and frequency domain of remote sensing images. Test and verify it using the test set and validation set to obtain a trained classification retrieval model.

[0178] In specific implementation, the samples can be divided into training set, test set, and validation set according to the ratio of 8:1:1. The training set cannot contain samples that exist in the validation set and test set, and the validation set cannot contain samples that exist in the test set.

[0179] Then the training set source domain and target domain samples are obtained through the closed set domain adaptation model of remote sensing image space-frequency domain. After feature extraction, the feature fusion module is used to obtain the source domain, target domain weak enhancement, and target domain strong enhancement high-level features. And input it into the classifier to get the category probability distribution

[0180] Finally, the test set and validation set remote sensing images are input into the remote sensing image classification-retrieval model that has been trained with the training set. The hyperparameters in the model are fine-tuned based on the classification accuracy and retrieval accuracy to obtain the most stable classification and retrieval model.

[0181] Step 4: Input the target remote sensing image into the trained classification retrieval model to obtain the classification results and retrieval results.

[0182] Based on the above embodiment, step 4 specifically includes:

[0183] Step 4.1: Input the target remote sensing image into the trained classification retrieval model to obtain the sample category probability distribution p i, the category corresponding to the maximum value of the sample category probability is taken as the classification result of the target remote sensing image;

[0184] Step 4.2: Input all images in the retrieval database into the trained classification retrieval model to obtain the retrieval data feature library F = [f 1 ,f 2 ,…,f n ], input the target remote sensing image into the trained classification retrieval model to obtain the target high-level features The cosine similarity is used to calculate the similarity between the target high-level features and all image features in the retrieval data feature library, and the similarity is sorted from high to low according to the similarity, and a list of the most similar images is returned as the retrieval result.

[0185] Furthermore, the expression of the classification result is

[0186]

[0187] The expression of the similarity is:

[0188]

[0189] Among them, f i Any image feature in the retrieval data feature library.

[0190] In specific implementation, the process of remote sensing image classification and retrieval can be as follows:

[0191] Step 4.1 Remote sensing image classification

[0192] Input the remote sensing image into the remote sensing image classification-retrieval model that has been trained with the training set to obtain the sample category probability distribution p i .

[0193] The category corresponding to the maximum value of the sample category probability is the classification result of the sample. The formula is:

[0194]

[0195] Step 4.2 Remote Sensing Image Retrieval

[0196] Input all images in the retrieval database into the remote sensing image classification-retrieval model that has been trained with the training set, and obtain the retrieval data feature library F = [f 1 ,f 2 ,…,f n ]

[0197] Input the remote sensing image into the remote sensing image classification-retrieval model that has been trained with the training set to obtain the sample high-level features

[0198] Use cosine similarity to calculate the similarity between the query image features and all stored image features. The formula is as follows:

[0199]

[0200] Sort by similarity from high to low (or distance from low to high) and return a list of the most similar images.

[0201] The closed-set domain adaptation remote sensing image classification and retrieval method provided in this embodiment integrates the spatial domain and the frequency domain. At the feature level, spatial domain and frequency domain features are obtained through spatial domain and frequency domain encoders, and a space-frequency cross-attention fusion module is designed to obtain high-level features of the image. Compared with other domain adaptation methods, it not only retains the spatial features and rich semantic features of the spatial domain features, but also effectively introduces additional frequency domain information. In terms of model training strategy, it introduces a spatial domain and frequency domain feature difference measurement method, a consistency regularization enhanced pseudo-label learning method, and minimized class confusion learning. Compared with the existing technology, the present disclosure optimizes the model from multiple aspects such as data offset, category decision boundary, and similar class confusion, which is more conducive to remote sensing image classification and retrieval tasks in complex cross-domain situations. It solves the problem that due to the influence of the complexity of the remote sensing image background and the differences in the distribution of multi-source data, complex remote sensing image cross-domain classification-retrieval tasks cannot obtain good generalization results by focusing only on spatial domain information.

[0202] The method disclosed herein will be further illustrated below with reference to a specific embodiment. By using the public remote sensing datasets UCMD and NWPU, an experimental analysis was conducted by comparing Source (without domain adaptation), MCC (minimum class confusion), and PCLUDA (a closed-set domain adaptation remote sensing image retrieval method based on pseudo-label consistency learning) in a complex cross-domain scenario of UCMD→NWPU. As shown in Tables 1 and 2, Table 1 shows the classification accuracy of different methods in the UCMD→NWPU cross-domain scenario, and Table 2 shows the retrieval accuracy mAP (%) of different methods in the UCMD→NWPU cross-domain scenario. The results show that the closed-set domain adaptation remote sensing image classification and retrieval method that integrates spatial and frequency domains proposed in the present invention is superior to the existing domain adaptation method.

[0203] Table 1

[0204]

[0205] Table 2

[0206]

[0207] It should be understood that various parts of the present disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0208] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A closed-domain adaptive remote sensing image classification and retrieval method that integrates spatial domain and frequency domain, characterized in that: include: Step 1, establishing a closed-set domain adaptation model for fusing the spatial domain and frequency domain of remote sensing images, wherein the closed-set domain adaptation model for fusing the spatial domain and frequency domain of remote sensing images includes a data enhancement module, a spatial domain feature extraction module, a frequency domain feature extraction module, a space-frequency cross-attention fusion module, a feedforward and a classifier; The data enhancement module is used to weakly enhance the source domain image to obtain a source domain weakly enhanced view, and to weakly enhance and strongly enhance the target domain to obtain a target domain weakly enhanced image and a target domain strongly enhanced view; The spatial domain feature extraction module is used to sequentially input the source domain weak enhancement map, the target domain weak enhancement map, and the strong enhancement map into the spatial domain encoder to obtain the spatial domain feature. , ; The frequency domain feature extraction module is used to perform Fourier transform on the source domain weak enhancement image, the target domain weak enhancement image, and the strong enhancement image to obtain the image frequency domain, and then set the frequency domain filter to input the frequency domain information corresponding to the source domain weak enhancement image, the target domain weak enhancement image, and the target domain strong enhancement image obtained by the transformation into the frequency domain encoder in sequence to obtain the frequency domain feature. , , ; The spatial-frequency cross-attention fusion module is used to generate queries, keys, and values ​​from spatial domain features and frequency domain features through linear transformation, and use the query value of the spatial domain feature and the key value of the frequency domain feature for cross calculation. The query, key, and value are linearly transformed using a multi-head attention mechanism, and then divided into multiple heads. The attention of each head is independently calculated, and finally the outputs of all heads are connected. The final output is then generated through a linear transformation. The final output is added to the spatial domain feature through a residual connection and layer normalization is performed. Finally, a feedforward neural network is used to obtain the fused high-level features. The classifier is used to obtain the category probabilities of different categories of the image by linearly transforming the high-level features; Step 2: Establish a model training strategy; Step 3: Divide the sample data set into a training set, a test set, and a validation set according to a preset ratio. Use the training set and model training strategy to train a closed-set domain adaptation model that integrates the spatial domain and frequency domain of remote sensing images. Test and verify it using the test set and validation set to obtain a trained classification retrieval model. Step 4: Input the target remote sensing image into the trained classification retrieval model to obtain the classification results and retrieval results.

2. The method according to claim 1, characterized in that , the step 2 specifically includes: Step 2.1: The maximum mean difference method is used to measure the spatial domain distribution difference loss, and the high-order moment matching method is used to measure the frequency domain distribution difference loss. In step 2.2, cross entropy loss is used as the source domain classification loss; Step 2.3: Use pseudo-label consistency learning as the classification loss for the weak and strong enhancement maps of the target domain. Step 2.4: Apply minimum class confusion learning to the class probability distribution obtained by the classifier to obtain the class confusion loss. In step 2.5, the total training loss is obtained based on the spatial domain distribution difference loss, the frequency domain distribution difference loss, the source domain classification loss, the target domain weak enhancement map and strong enhancement map classification loss, and the class confusion loss.

3. The method according to claim 2, characterized in that , the step 4 specifically includes: Step 4.1: Input the target remote sensing image into the trained classification retrieval model to obtain the sample category probability distribution , the category corresponding to the maximum value of the sample category probability is taken as the classification result of the target remote sensing image; Step 4.2: Input all images in the retrieval database into the trained classification retrieval model to obtain the retrieval data feature library , input the target remote sensing image into the trained classification retrieval model to obtain the target high-level features , use cosine similarity to calculate the similarity between the target high-level features and all image features of the retrieval data feature library, and sort them from high to low according to the similarity, and return the most similar image list as the retrieval result, where, Indicates retrieving features from the data feature library.

Images