Passive domain adaptation remote sensing image scene classification method and system based on self-training

Abstract

The present invention belongs to the field of image processing technology and relates to a self-training-based passive domain adaptation remote sensing image scene classification method and system. The present invention uses the total loss of source domain data as an objective function to guide a passive domain adaptation network to learn semantic information in the source domain data, thereby obtaining a pre-trained network on the source domain data set; initializes a network on a target domain data set using the pre-trained network on the source domain data set, and divides the target domain data into source domain similar samples and uncertain samples based on the initialized network on the target domain data set; obtains the target domain total loss based on the source domain similar samples and uncertain samples; uses a teacher-student network to optimize the network on the target domain data set using the target domain total loss and the target domain data, thereby obtaining a classification network adapted to the target domain data set; and uses the classification network adapted to the target domain data set to perform unsupervised classification on the target domain data to be classified.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and relates to a passive domain adaptation remote sensing image scene classification method and system based on self-training. Background Art

[0002] Domain adaptation is one of the current approaches used to address distribution shift and missing target domain labels. It is primarily achieved through two components: distribution alignment and target domain semantic learning. Common solutions for distribution alignment include adversarial learning and mean difference loss. Adversarial learning extracts domain-invariant features through adversarial training of the discriminator and generator. Mean difference loss aligns features from two domains by minimizing distribution differences. For semantic learning, pseudo-labeling is a common solution. High-confidence model predictions are selected as pseudo-labels to supervise network training.

[0003] For passive domain adaptation methods, existing pseudo-label-based approaches suffer from high noise levels, which can cause the network to learn the noise in the pseudo-labels and produce biased predictions. This problem can lead to network degradation during transfer learning, meaning that the final network performance is worse than the initial network after learning.

[0004] Existing remote sensing fields are used to solve the problem of missing labeled information, such as small sample learning. It still requires the existence of labeled data in the target field, which is not applicable to some situations with strong restrictions.

[0005] In summary, the existing remote sensing field has the following problems:

[0006] 1. Traditional pseudo-label-based methods have the problem of high noise, which may cause network degradation during transfer learning;

[0007] 2. Other methods for dealing with insufficient label information, such as small-sample learning and semi-supervised learning, still require the existence of labeled information in the target domain and are not applicable when labeled data is extremely scarce or completely missing. Summary of the Invention

[0008] The purpose of the present invention is to provide a passive domain adaptation remote sensing image scene classification method and system based on self-training to solve the technical problems that the existing remote sensing image scene classification methods have high noise and are not suitable for extremely scarce or completely missing labeled data.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect, the present invention provides a method for remote sensing image scene classification based on passive domain adaptation and self-training, comprising the following steps:

[0011] The total loss of the source domain data is used as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain a network pre-trained on the source domain dataset;

[0012] Initialize the network on the target domain dataset with the network pre-trained on the source domain dataset, and divide the target domain data into source domain similar samples and uncertain samples based on the network on the initialized target domain dataset;

[0013] Based on similar samples and uncertain samples in the source domain, we obtain the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss;

[0014] The total loss of the target domain is obtained based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss;

[0015] The teacher-student network is used to optimize the network on the target domain dataset using the total loss and data of the target domain, and a classification network adapted to the target domain dataset is obtained.

[0016] The classification network adapted to the target domain dataset is used to perform unsupervised classification on the target domain data to be classified.

[0017] In a second aspect, the present invention provides a self-training-based passive domain adaptation remote sensing image scene classification system, comprising:

[0018] Pre-trained network acquisition module: used to use the total loss of source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain a network pre-trained on the source domain dataset;

[0019] Target domain data partitioning module: used to initialize the network on the target domain dataset with the network pre-trained on the source domain dataset, and divide the target domain data into source domain similar samples and uncertain samples based on the network on the initialized target domain dataset;

[0020] Target domain loss acquisition module: used to obtain the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss based on similar samples and uncertain samples in the source domain;

[0021] Target domain total loss acquisition module: used to obtain the target domain total loss based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss;

[0022] Classification network acquisition module: used to use the teacher-student network to optimize the network on the target domain dataset using the target domain total loss and target domain data, and obtain a classification network adapted to the target domain dataset;

[0023] Unsupervised classification module: used to perform unsupervised classification on the target domain data to be classified using the classification network adapted to the target domain dataset.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The method of the present invention uses the total loss of the source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, obtain a network pre-trained on the source domain dataset, and improve the generalization performance of the model in the new domain. The network pre-trained on the source domain dataset is used to initialize the network on the target domain dataset, and the knowledge on the source domain dataset is loaded to pave the way for the subsequent transfer learning process. This helps to adopt different training strategies for different samples in the future, improving the pertinence and efficiency of training. Based on similar samples and uncertain samples in the source domain, the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data comparative learning training loss, and the target domain data mixed loss are obtained. These losses together form the basis of the total loss of the target domain and reflect the performance of the model in the target domain. The total loss of the target domain is obtained based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data comparative learning training loss, and the target domain data mixed loss. Integrating loss information from different sources provides a clear goal for subsequent model optimization. By minimizing the total loss of the target domain, the performance of the model in the target domain can be promoted, further improving the performance of the model. The target domain data to be classified is unsupervisedly classified using a classification network adapted to the target domain dataset, with high classification accuracy. The source domain data is adapted to the target domain dataset and the classification of the target domain data to be classified is completed. In the case where the source domain dataset is inaccessible and the supervision information of the target domain dataset is accurate, the framework can accurately complete the scene classification task. The present invention is designed based on a self-training method. Thanks to the division of the target domain dataset, the present invention can reduce the problem of network degradation caused by excessive sample label noise. The present invention transfers the knowledge in the source domain to the target domain dataset in a self-training manner. Therefore, for the target domain data to be classified, this method can realize classification without relying on labeled data, solving the technical problem that the existing remote sensing image scene classification method is not suitable for extremely scarce or completely missing labeled data.

[0026] 2. The system of the present invention includes: a pre-trained network acquisition module, a target domain data division module, a target domain loss acquisition module, a target domain total loss acquisition module, a classification network acquisition module and an unsupervised classification module. The pre-trained network acquisition module is used to use the total loss of the source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain the pre-trained network on the source domain data set; the target domain data partitioning module is used to initialize the network on the target domain data set with the network pre-trained on the source domain data set, and divide the target domain data into source domain similar samples and uncertain samples according to the network on the initialized target domain data set; the target domain loss acquisition module is used to obtain the supervised training loss of the target domain credible samples, the unsupervised training loss of the uncertain samples, the target domain data comparative learning training loss and the target domain data mixed loss according to the source domain similar samples and the uncertain samples; the target domain total loss acquisition module is used to obtain the target domain total loss according to the supervised training loss of the target domain credible samples, the unsupervised training loss of the uncertain samples, the target domain data comparative learning training loss and the target domain data mixed loss; the classification network acquisition module is used to use the teacher-student network to optimize the network on the target domain data set using the target domain total loss and the target domain data, and obtain the classification network adapted to the target domain data set; the unsupervised classification module is used to use the classification network adapted to the target domain data set to perform unsupervised classification on the target domain data to be classified. The various modules work together to achieve the classification of data to be classified in the target field without relying on labeled data, and can solve the technical problem that the existing remote sensing image scene classification method is not suitable for the extremely scarce or completely missing labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flowchart of passive field optical remote sensing image scene classification based on self-training according to an embodiment of the present invention;

[0028] Figure 2 This is a source domain training network diagram according to an embodiment of the present invention;

[0029] Figure 3 This is a network framework diagram of the domain knowledge migration process according to an embodiment of the present invention;

[0030] Figure 4 is a flow chart of the method of the present invention;

[0031] Figure 5 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," and the like in the description of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0034] The present invention is described in further detail below with reference to the accompanying drawings:

[0035] See also Figure 4 The present invention discloses a passive domain adaptation remote sensing image scene classification method based on self-training, comprising the following steps:

[0036] S1 uses the total loss of source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, obtain a network pre-trained on the source domain dataset, and improve the generalization performance of the model in the new domain.

[0037] S2 uses the pre-trained network on the source domain dataset to initialize the target domain network, loading the knowledge from the source domain dataset and paving the way for subsequent transfer learning. Using the initialized network on the target domain dataset, the target domain data is divided into samples similar to the source domain and samples with uncertainty. This facilitates the subsequent adoption of different training strategies for different samples, improving the relevance and efficiency of training.

[0038] S3, based on similar samples and uncertain samples in the source domain, obtains the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data comparative learning training loss, and the target domain data mixed loss. These losses together form the basis of the total loss in the target domain and reflect the performance of the model in the target domain.

[0039] S4 calculates the total target domain loss based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss. Integrating loss information from different sources provides a clear goal for subsequent model optimization. By minimizing the total target domain loss, we can improve the model's performance in the target domain.

[0040] S5, using the teacher-student network to optimize the network on the target domain dataset using the total loss of the target domain and the target domain data, to obtain a classification network adapted to the target domain dataset, which can further improve the performance of the model.

[0041] S6 uses a classification network adapted to the target domain dataset to perform unsupervised classification of the target domain data to be classified, achieving high classification accuracy. This adapts the source domain data to the target domain dataset and completes the classification of the target domain data to be classified. Even when the source domain dataset is unavailable and the target domain dataset has sufficient supervision information, the framework can accurately complete the scene classification task.

[0042] Example 1:

[0043] See also Figure 4 The present invention discloses a passive domain adaptation remote sensing image scene classification method based on self-training, comprising the following steps:

[0044] S1 uses the total loss of the source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtains the network pre-trained on the source domain dataset, as follows:

[0045] After rotating the source domain data, the source domain feature extractor and the rotation predictor are used to obtain the prediction result of the rotation predictor. The rotation prediction loss is obtained based on the prediction result of the rotation predictor and the randomly rotated one-hot label of the source domain data.

[0046] Cross entropy is used to combine source domain data to obtain data mixing loss;

[0047] After data enhancement of the source domain data, the source domain data classification prediction results are obtained through the source domain feature extractor and the source domain classifier. The label smoothed cross entropy loss is obtained based on the classifier prediction results and the one-hot encoding of the source domain data, as follows:

[0048] After data enhancement of the source domain data, the source domain data classification prediction results are obtained through the source domain feature extractor and source domain classifier;

[0049] Obtain soft labels based on the one-hot encoding of the source domain data;

[0050] Obtain label-smoothed cross entropy loss based on the classifier prediction results and soft labels.

[0051] The total loss of source domain data is obtained based on the rotation prediction loss, data mixing loss, and label smoothing cross entropy loss. The formula for obtaining the total loss of source domain data is as follows:

[0052]

[0053] in, and is a hyperparameter, is the total loss of source domain data, is the label smoothed cross entropy loss, is the rotation prediction loss, is the data mixing loss.

[0054] The total loss of the source domain data is used to optimize the source domain network to obtain an optimized source domain network. The optimized source domain network is used to learn the semantic information in the source domain data to obtain a network pre-trained on the source domain dataset.

[0055] S2: Initialize the network on the target domain dataset using the pre-trained network on the source domain dataset. Then, divide the target domain data into source domain similar samples and uncertain samples based on the initialized network on the target domain dataset. The details are as follows:

[0056] Obtain the prediction confidence change of the target domain data based on the network on the initialized target domain dataset;

[0057] The target domain data is divided into source domain similar samples and uncertain samples according to the change in prediction confidence of the target domain data.

[0058] S3, based on similar samples and uncertain samples in the source domain, obtains the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss, as follows:

[0059] The pseudo-label supervision method is used to train similar samples in the source domain to obtain the supervised training loss of credible samples in the target domain, as follows:

[0060] Obtain the logical value of the source domain similar sample based on the network's predicted probability of the source domain similar sample, the neighbor set of the source domain similar sample, and the neighbor weight;

[0061] Obtain pseudo labels of similar samples in the source domain based on the logical values ​​of similar samples in the source domain;

[0062] The supervised training loss of the target domain credible samples is obtained based on the pseudo labels of similar samples in the source domain and the predicted probability of similar samples in the source domain. The formula for obtaining the supervised training loss of the target domain credible samples is as follows:

[0063]

[0064] in, is the supervised training loss of credible samples in the target domain, Pseudo labels for samples in the target domain data The first one-hot encoding of The value of the dimension, is the pseudo label of the sample in the target domain data, is the number of categories, For the Vidi The predicted probability of similar samples in the source domain.

[0065] The unsupervised training method is used to train uncertain samples to obtain the unsupervised training loss of uncertain samples, which is as follows:

[0066] The unsupervised training loss of uncertain samples is obtained according to the predicted probability of weak enhancement of uncertain samples and the predicted probability of strong enhancement of uncertain samples. The formula for obtaining the unsupervised training loss of uncertain samples is as follows:

[0067]

[0068] in, is the unsupervised training loss for uncertain samples, is the number of categories, Indicates the The predicted probability of weak enhancement of uncertain samples, Indicates the The predicted probability of a strongly enhanced uncertain sample.

[0069] The noise-modified contrastive learning strategy learns the target domain data to obtain the target domain data contrastive learning training loss, which is as follows:

[0070] The target domain data contrastive learning training loss is obtained based on the prediction head output of the target domain data, the mapping head output of the target domain data, the probability prediction result of the target domain data, and the similarity coefficient. The formula for obtaining the target domain data contrastive learning training loss is as follows:

[0071]

[0072] in, training loss for comparative learning of target domain data, Indicates the The prediction head output of target domain data, Indicates the The mapping header output of the target domain data, Indicates the The prediction head output of target domain data, Indicates the The mapping header output of the target domain data, express The transpose of Indicates the first The probability prediction results of target domain data, Indicates the first The probability prediction results of target domain data, The similarity coefficient is used to measure the similarity between two samples and is set based on experience. , Indicates the dot product operation between two vectors. Represents the similarity measure of two vectors, where vector dot product is used.

[0073] The target domain data is learned using pseudo-label supervision to obtain the target domain data mixed loss, as follows:

[0074] Similar samples in the source domain and Mixup technology are combined with pseudo labels to generate mixed loss of target domain data. The cross entropy loss function is used for calculation in the process of generating mixed loss of target domain data.

[0075] S4, based on the supervised training loss of the target domain’s credible samples, the unsupervised training loss of the uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss, the total loss of the target domain is obtained. The formula for obtaining the total loss of the target domain is as follows:

[0076]

[0077] in, 、 and are all hyperparameters, is the total loss in the target area, is the supervised training loss of credible samples in the target domain, is the unsupervised training loss for uncertain samples, training loss for comparative learning of target domain data, is the target domain data mixing loss.

[0078] S5: The teacher-student network is used to optimize the network on the target domain dataset using the total loss of the target domain and the target domain data, and a classification network adapted to the target domain dataset is obtained, as follows:

[0079] Initialize the teacher network and student network using the pre-trained parameters of the source domain;

[0080] The student network parameters are updated in the student network, and the exponential average move is used to smoothly update the student network parameters to the teacher network to obtain a classification network adapted to the target domain dataset.

[0081] S6, uses the classification network adapted to the target domain dataset to perform unsupervised classification on the target domain data to be classified.

[0082] Example 2:

[0083] See also Figure 1 The present invention discloses a remote sensing image scene classification method based on self-training and passive domain adaptation, which is divided into two processes: source domain training and target domain adaptation.

[0084] First, the present invention aims to train the model in a supervised manner in the source domain, and then transfer the source domain knowledge to the target domain dataset through transfer learning to achieve the target domain data scene classification task without labeled information in the target domain. In order for the model to achieve the expected effect in the target domain, training a model with strong classification performance in the source domain is a problem worth exploring. Taking into account the characteristics of remote sensing images and the requirements of domain generalization performance, the present invention designs a source domain network training method to enhance the model's learning of domain invariant knowledge and improve generalization performance. After completing the pre-training in the source domain, the network needs to adjust the parameters in the absence of the target domain dataset to complete the scene classification task of the target domain dataset. The second part is the target domain adaptation process. Figure 3 Give an explanation of the process. In order to better illustrate this process, we divide it into four parts: sample partitioning, source domain similar sample learning, uncertain sample learning, and training process. Sample partitioning aims to divide similar samples and uncertain samples from the source domain and design specific learning methods for different data features to fully learn the internal relationship of the data and avoid network degradation caused by noise. The training process is to better coordinate the learning of the two parts of samples and add some additional modules to achieve efficient adaptation of the target domain scene classification task. The details of the self-training-based passive domain adaptation optical remote sensing image scene classification method proposed in this study are as follows:

[0085] S1, source domain training:

[0086] The training process of the source domain network is as follows Figure 2, considering that the performance of domain adaptation is affected by the learning effect of the model in the source domain, the present invention designs a training method with strong generalization performance. In order to make the model adapt to the domain migration task, the present invention adopts a general passive domain adaptation network design model, and adds batch normalization and weight normalization to the backbone network and classifier to reduce the data distribution bias brought about by the training process and the similarity calculation bias caused by the classification of new data. In order to enhance the generalization performance of the source domain network, label smoothing cross entropy loss is used as the objective function to guide the network to learn the semantic information in the source domain data set. The process is as follows:

[0087]

[0088]

[0089]

[0090] in, Indicates a soft tag. represents soft parameters, Represents source domain data No. The unique encoding value of the category, set according to experience In this embodiment , Represents source domain data No. The predicted probability of the class, represents a classifier, represents the feature extractor, is the number of categories, is the label smoothed cross entropy loss, is the label smoothing loss identifier, is the source domain identifier, is the source domain classifier, is the source domain feature extractor, For the source field data.

[0091] Inspired by unsupervised learning, in order to enable the network to learn universal representation capabilities, a rotation prediction classifier is introduced here to guide the network to extract features with domain invariance. The process is expressed as:

[0092]

[0093]

[0094] in, is the rotation prediction loss, A pointer to traverse each dimension of the angle prediction result. Indicates that for source domain data Randomly rotated one-hot labels, represents the prediction result of the rotation predictor, Represents source domain data The predicted probability of the jth dimension after random rotation, Represents a rotation operation, Represents the rotation predictor, the rotation angle range is , the image is randomly rotated before each input to the network, is the source domain feature extractor.

[0095] In order to further enhance the generalization performance of the network and improve its robustness, data enhancement and mixed loss Mixup are used here to improve the robustness of the network. The mixed loss Mixup is expressed as The cross entropy is used to implement it. The overall objective function of the training process in the final source domain is:

[0096]

[0097] in, and is a hyperparameter, is the total loss of source domain data, is the label smoothed cross entropy loss, is the rotation prediction loss, is the data mixing loss.

[0098] S2, domain adaptation on the target dataset;

[0099] Sample division: Considering the large difference in distribution between the target domain and the source domain, directly using pseudo-label supervised training will lead to the accumulation of noise and cause network degradation. To this end, the present invention divides the target domain data into source domain similar samples and uncertain samples. For source domain similar samples, we use pseudo-label supervised training methods to obtain supervised training losses of credible samples in the target domain, which can achieve semantic learning while reducing noise. For uncertain samples, we use unsupervised training methods to obtain unsupervised training losses of uncertain samples, guiding the network to learn the potential relationship between samples to avoid the impact of noise.

[0100] According to the deep learning hypothesis, after the target domain data is input into the feature extractor and mapped to the latent space, it should show a clustered distribution according to the category, and different categories should have clear boundaries. Based on this, we designed a simple and novel solution: for sample data in the target domain data, samples whose prediction confidence changes less than a threshold before and after the network parameter update should be regarded as similar samples in the source domain, while samples whose changes are greater than the threshold are uncertain samples. Let's use an intuitive example to explain: when the confidence of the sample prediction in the target domain data changes significantly, we can assume that the network is not confident enough in the prediction result of the sample, resulting in a large displacement of the sample in the latent space or a large change in the confidence of the sample mapping on the classification boundary due to the change of category. According to this principle, our sample partitioning expression is as follows, and we will use different training methods based on the partitioning results here.

[0101]

[0102]

[0103] in, represents the oscillation factor, Represents the number of steps of forward propagation of samples in the target domain data, Represents the predicted probability, that is, the predicted result of the sample, Represents the forward propagation of samples in the target domain data The predicted probability of the step, Represents the forward propagation of samples in the target domain data The predicted probability of the step, represents the prediction confidence of samples in the target domain data, Represents the division result of samples in the target domain data. When , it means that the target domain data is similar to the source domain sample, otherwise , indicating that the target domain data is an uncertain sample, Gets the maximum value in the vector.

[0104] Learning of similar samples in the source domain: Similar samples in the source domain can be obtained by sample division. For this type of samples, they can be regarded as being close to the domain-invariant space, so the network prediction results of these samples are relatively reliable, and using them for supervised training can achieve better results. Supervised training requires labels, considering that directly using the prediction results of samples as pseudo labels may be biased due to noise. The present invention uses a neighbor sample label fusion method to generate pseudo labels for similar samples in the source domain. The idea of ​​neighbor sample label fusion is to select samples from the latent space of features and average the predicted probabilities of the nearest neighbor samples to obtain the predicted probability of the sample. In order to better filter the noise, we do not use the mean but entropy weighting to apply different weights to trusted neighbors and untrusted neighbors and then fuse them. The generation process of pseudo labels for similar samples in the source domain is as follows:

[0105]

[0106]

[0107]

[0108] in, is the logical value of similar samples in the source domain, is a pseudo-label identifier, Represents the neighbor set of similar samples in the source domain. Here, cosine similarity is used to measure the distance between potential space samples. is the sample number in the dataset, is the neighbor weight, is the network’s predicted probability for similar samples in the source domain, represents the first node extracted from the neighbor set samples, are pseudo labels of similar samples in the source domain, is the maximum independent variable point set function, is a normalization function used to convert logical values ​​into probability vectors. represents the predicted probability of the neighbor sample, represents the information entropy calculation function, represents the entropy of the prediction results of similar samples in the source domain, is the number of categories.

[0109] After the pseudo labels are generated, their hard labels are used as supervised training samples from the source domain to guide the network to learn the semantic information of the target domain.

[0110]

[0111] in, is the supervised training loss of credible samples in the target domain, Pseudo labels for samples in the target domain data The first one-hot encoding of The value of the dimension, is the pseudo label of the sample in the target domain data, is the number of categories, For the Vidi The predicted probability of similar samples in the source domain is the predicted result of the sample.

[0112] Uncertain sample learning: Uncertain samples are screened out because the network makes unconfident predictions about them. These samples are often located in a domain-specific distribution. Using pseudo-label training will inevitably introduce a lot of noise interference, causing network degradation. Therefore, an unsupervised training method is used for uncertain samples to avoid noise interference. We perform two types of processing on this type of samples: strong enhancement and weak enhancement. Obviously, the prediction results of these two samples should be the same category. We consider using mean square error loss to bring the prediction results of the two data enhancements closer. This objective function can be expressed in the latent space as the sample and the perturbation samples around it produce the same prediction. Therefore, it can play a tightening role in both the decision boundary and the cluster structure to achieve representation learning of uncertain samples. Due to the introduction of unsupervised learning, the network will learn a representation with confident judgment for such difficult samples, so these samples will be propagated to the learning of similar samples in the source domain for semantic learning, thereby increasing the number of similar samples in the source domain. The expression of uncertain sample learning is as follows:

[0113]

[0114] in, is the unsupervised training loss for uncertain samples, is the number of categories, Indicates the The predicted probability of weak enhancement of uncertain samples, Indicates the The predicted probability of a strongly enhanced uncertain sample.

[0115] During the training process, source domain sample learning achieves semantic learning of high-confidence samples, while uncertain sample learning achieves consistent prediction of sample perturbations in the latent space. These two modules work together to propagate uncertain samples to similar samples in the source domain. However, using these two modules alone cannot yield a clear decision boundary in the target domain. Therefore, to ensure the desired scene classification performance in the target domain, contrastive learning and a mixed loss (Mixup) are introduced to assist these two processes.

[0116] For learning from uncertain samples, simply narrowing the distance between samples and perturbed samples in the latent space is clearly insufficient. This does not address the problem of class cohesion. This method introduces contrastive learning and uses pseudo-labels as supervision to guide the network to produce a more cohesive class mapping for samples of the same class, thereby achieving discriminative feature extraction. To mitigate the influence of noise, a contrastive learning strategy based on noise improvement is introduced, expressed as follows:

[0117]

[0118] in, training loss for comparative learning of target domain data, Indicates the The prediction head output of target domain data, Indicates the The mapping header output of the target domain data, Indicates the The prediction head output of target domain data, Indicates the The mapping header output of the target domain data, express The transpose of Indicates the first The probability prediction results of target domain data, Indicates the first The probability prediction results of target domain data, The similarity coefficient is used to measure the similarity between two samples and is set based on experience. , Indicates the dot product operation between two vectors. Represents the similarity measure of two vectors, where vector dot product is used.

[0119] For samples similar to the source domain, pseudo-label supervised learning is used. Although high-confidence pseudo-labels can guide the network to learn accurate semantic information, the network still cannot accurately extract features due to insufficient samples and small amounts of noise. Here, data mixing operations are used to expand the samples to form smooth decision boundaries between samples and improve the robustness of model predictions. The target domain data and the Mixup technique are combined with pseudo-labels to generate the target domain data mixing loss. The cross-entropy loss function is used to calculate the target domain data mixing loss.

[0120] In the target domain adaptation process, the overall objective function of our method is as follows:

[0121]

[0122] in, 、 and are all hyperparameters, is the total loss in the target area, is the supervised training loss of credible samples in the target domain, is the unsupervised training loss for uncertain samples, training loss for comparative learning of target domain data, is the target domain data mixing loss.

[0123] In order to ensure the stability of the sample division process, the teacher-student network design is adopted here. Figure 3 During initialization, the teacher network and student network are initialized with the source domain pre-trained parameters. Network parameter updates are performed in the student network, and the student network parameters are smoothly updated to the teacher network using exponential moving averages. The sample partitioning of the target domain data to be classified is completed by the teacher network. Due to the introduction of exponential moving averages, the teacher network update amplitude is small, thus ensuring the stability of the sample partitioning and improving the robustness of the overall network prediction. The exponential moving average process update formula is as follows:

[0124]

[0125] in, is the exponential moving average weight, which is set to 0.9 based on experience. For Network parameters at the moment.

[0126] The present invention introduces an unsupervised method to extract domain-invariant features, enhance the generalization ability of the source domain model, and improve the performance of subsequent domain migration tasks.

[0127] The present invention uses a sample partitioning method to perceive the confidence fluctuation of target domain samples and screen out similar samples and uncertain samples in the source domain, accurately providing a credible sample set and an uncredible sample set for subsequent use.

[0128] The present invention utilizes a pseudo-label generation method to avoid the error caused by network prediction of a single sample by fusing neighbor samples and adopts entropy weighting to filter the noise in the neighbor samples to achieve accurate generation of pseudo-labels.

[0129] The present invention achieves the adaptation of source domain knowledge to target domain datasets and completes scene classification tasks. Even when the source domain dataset is unavailable and the target domain dataset has reliable supervision information, the framework can accurately complete the scene classification task.

[0130] For domain adaptation methods, this invention can solve the problem of inaccessibility in the source domain, using only a pre-trained model in the source domain to fine-tune parameters in the target domain. This not only reduces model training costs but also reduces manual annotation costs, enabling scene classification in the absence of annotations in the target domain.

[0131] For passive domain adaptation, the present invention improves the generalization ability of the source domain model and provides an effective source domain pre-training model for the domain adaptation process.

[0132] Compared with the small sample method, the present invention can achieve classification when the target domain is completely unlabeled, solving the strong limitation problem of the target domain having no dataset annotations.

[0133] Based on the above method, the present invention also discloses a passive domain adaptation remote sensing image scene classification system based on self-training, see Figure 5 ,include:

[0134] Pre-trained network acquisition module: used to use the total loss of source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain a network pre-trained on the source domain dataset;

[0135] Target domain data partitioning module: used to initialize the network on the target domain dataset with the network pre-trained on the source domain dataset, and divide the target domain data into source domain similar samples and uncertain samples based on the network on the initialized target domain dataset;

[0136] Target domain loss acquisition module: used to obtain the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss based on similar samples and uncertain samples in the source domain;

[0137] Target domain total loss acquisition module: used to obtain the target domain total loss based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss;

[0138] Classification network acquisition module: used to use the teacher-student network to optimize the network on the target domain dataset using the target domain total loss and target domain data, and obtain a classification network adapted to the target domain dataset;

[0139] Unsupervised classification module: used to perform unsupervised classification on the target domain data to be classified using the classification network adapted to the target domain dataset.

[0140] The system of the present invention can classify the data to be classified in the target field without relying on labeled data, and can solve the technical problem that the existing remote sensing image scene classification method is not suitable for the extremely scarce or completely missing labeled data.

[0141] The present invention is mainly used in remote sensing scene classification tasks, which is a basic task in remote sensing image intelligent interpretation technology, and solves the problem of high cost and long cycle of manual labeling. The introduction of transfer learning technology realizes the transfer of knowledge from the source field to the target field, improves the application value of existing labeled data sets, and overcomes the problem that the source field data set is inaccessible and edge computing cannot be retrained. The present invention can be widely used in environmental protection, earth observation, urban planning and other fields, bringing certain social benefits. In addition, the ideas and processes of the present invention can be applied to tasks such as hyperspectral classification and semantic segmentation to provide new ideas for the application of transfer learning in the intelligent interpretation of remote sensing images.

[0142] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A self-training-based passive domain adaptation remote sensing image scene classification method, characterized by: The following steps are involved: The total loss of the source domain data is used as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain the network pre-trained on the source domain dataset, as follows: After rotating the source domain data, the source domain feature extractor and the rotation predictor are used to obtain the prediction result of the rotation predictor. The rotation prediction loss is obtained based on the prediction result of the rotation predictor and the randomly rotated one-hot label of the source domain data. Cross entropy is used to combine source domain data to obtain data mixing loss; After data enhancement of the source domain data, the source domain data classification prediction results are obtained through the source domain feature extractor and the source domain classifier. The label smoothed cross entropy loss is obtained based on the classifier prediction results and the one-hot encoding of the source domain data. Obtain the total loss of source domain data based on rotation prediction loss, data mixing loss, and label smoothing cross entropy loss; The total loss of the source domain data is used to optimize the source domain network to obtain an optimized source domain network. The optimized source domain network is used to learn the semantic information in the source domain data to obtain a network pre-trained on the source domain dataset. Initialize the network on the target domain dataset with the network pre-trained on the source domain dataset, and divide the target domain data into source domain similar samples and uncertain samples based on the network on the initialized target domain dataset; Based on similar samples and uncertain samples in the source domain, we obtain the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss; The total loss of the target domain is obtained based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss; The teacher-student network is used to optimize the network on the target domain dataset using the total loss and data of the target domain, and a classification network adapted to the target domain dataset is obtained. The classification network adapted to the target domain dataset is used to perform unsupervised classification on the target domain data to be classified.

2. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 1, characterized in that: After data enhancement is performed on the source domain data, the source domain data classification prediction result is obtained through the source domain feature extractor and the source domain classifier. The label smoothed cross entropy loss is obtained according to the classifier prediction result and the one-hot encoding of the source domain data, as follows: After data enhancement of the source domain data, the source domain data classification prediction results are obtained through the source domain feature extractor and source domain classifier; Obtain soft labels based on the one-hot encoding of the source domain data; Obtain label-smoothed cross entropy loss based on the classifier prediction results and soft labels.

3. The self-training-based passive domain adaptation remote sensing image scene classification method according to claim 1, characterized in that: The target domain data is divided into source domain similar samples and uncertain samples according to the network on the initialized target domain data set, as follows: Obtain the prediction confidence change of the target domain data based on the network on the initialized target domain dataset; The target domain data is divided into source domain similar samples and uncertain samples according to the change in prediction confidence of the target domain data.

4. The self-training-based passive domain adaptation remote sensing image scene classification method according to claim 1, characterized in that: The supervised training loss of the target domain credible samples, the unsupervised training loss of the uncertain samples, the target domain data comparative learning training loss and the target domain data mixed loss are obtained based on the similar samples and uncertain samples in the source domain, as follows: The pseudo-label supervision method is used to train similar samples in the source domain to obtain the supervised training loss of credible samples in the target domain; Unsupervised training methods are used to train uncertain samples to obtain unsupervised training losses of uncertain samples; The target domain data is learned based on the noise-improved contrastive learning strategy to obtain the target domain data contrastive learning training loss; The target domain data is learned using pseudo-label supervision to obtain the target domain data mixed loss.

5. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 4, characterized in that: The pseudo-label supervision method is used to train similar samples in the source domain to obtain the supervised training loss of credible samples in the target domain. Specifically, Obtain the logical value of the source domain similar sample based on the network's predicted probability of the source domain similar sample, the neighbor set of the source domain similar sample, and the neighbor weight; Obtain pseudo labels of similar samples in the source domain based on the logical values ​​of similar samples in the source domain; The supervised training loss of credible samples in the target domain is obtained according to the pseudo labels of similar samples in the source domain and the predicted probabilities of similar samples in the source domain.

6. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 4, characterized in that: The unsupervised training method is used to train uncertain samples to obtain the unsupervised training loss of uncertain samples, which is as follows: The unsupervised training loss of uncertain samples is obtained according to the predicted probability of weak enhancement of uncertain samples and the predicted probability of strong enhancement of uncertain samples.

7. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 4, characterized in that: The noise-improved contrastive learning strategy learns the target domain data to obtain the target domain data contrastive learning training loss, and uses pseudo-label supervised learning to obtain the target domain data mixed loss, which is as follows: Obtain the target domain data comparative learning training loss based on the prediction head output of the target domain data, the mapping head output of the target domain data, the probability prediction result of the target domain data, and the similarity coefficient; Similar samples in the source domain and Mixup technology are combined with pseudo labels to generate mixed loss of target domain data. The cross entropy loss function is used for calculation in the process of generating mixed loss of target domain data.

8. The self-training-based passive domain adaptation remote sensing image scene classification method according to claim 1, characterized in that: The teacher-student network is used to optimize the network on the target domain dataset using the target domain total loss and target domain data to obtain a classification network adapted to the target domain dataset, specifically as follows: Initialize the teacher network and student network using the pre-trained parameters of the source domain; The student network parameters are updated in the student network, and the exponential average move is used to smoothly update the student network parameters to the teacher network to obtain a classification network adapted to the target domain dataset.

9. A self-training-based passive domain adaptation remote sensing image scene classification system, characterized by: include: Pre-trained network acquisition module: It is used to use the total loss of source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data and obtain the network pre-trained on the source domain dataset. The details are as follows: After rotating the source domain data, the source domain feature extractor and the rotation predictor are used to obtain the prediction result of the rotation predictor. The rotation prediction loss is obtained based on the prediction result of the rotation predictor and the randomly rotated one-hot label of the source domain data. Cross entropy is used to combine source domain data to obtain data mixing loss; After data enhancement of the source domain data, the source domain data classification prediction results are obtained through the source domain feature extractor and the source domain classifier. The label smoothed cross entropy loss is obtained based on the classifier prediction results and the one-hot encoding of the source domain data. Obtain the total loss of source domain data based on rotation prediction loss, data mixing loss, and label smoothing cross entropy loss; The total loss of the source domain data is used to optimize the source domain network to obtain an optimized source domain network. The optimized source domain network is used to learn the semantic information in the source domain data to obtain a network pre-trained on the source domain dataset. Target domain data partitioning module: used to initialize the network on the target domain dataset with the network pre-trained on the source domain dataset, and divide the target domain data into source domain similar samples and uncertain samples based on the network on the initialized target domain dataset; Target domain loss acquisition module: used to obtain the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss based on similar samples and uncertain samples in the source domain; Target domain total loss acquisition module: used to obtain the target domain total loss based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss; Classification network acquisition module: used to use the teacher-student network to optimize the network on the target domain dataset using the target domain total loss and target domain data, and obtain a classification network adapted to the target domain dataset; Unsupervised classification module: used to perform unsupervised classification on the target domain data to be classified using the classification network adapted to the target domain dataset.

Images