Self-training-based passive domain adaptive remote sensing image scene classification method and system

Abstract

The invention belongs to the technical field of image processing, and relates to a self-training-based passive domain adaptive remote sensing image scene classification method and system. According to the method, the total loss of source field data is adopted as a target function to guide a passive field adaptive network to learn semantic information in the source field data, and a pre-trained network on a source field data set is obtained; initializing a network on the target domain data set by using a pre-trained network on the source domain data set, and dividing target domain data into source domain similar samples and uncertain samples according to the initialized network on the target domain data set; obtaining the total loss of the target domain according to the similar samples and the uncertain samples of the source domain; optimizing a network on the target domain data set by using the total loss of the target domain and the data of the target domain by adopting a teacher-student network to obtain a classification network adaptive to the target domain data set; and performing unsupervised classification on the to-be-classified target domain data by using the classification network adaptive to the target domain data set.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a passive domain adaptation remote sensing image scene classification method and system based on self-training. Background Art

[0002] Domain adaptation is one of the current methods to solve the distribution shift and target domain label missing. It is mainly achieved through two parts: distribution alignment and target domain semantic learning. For distribution alignment, common solutions include adversarial learning and mean difference loss. Adversarial learning extracts domain-invariant features through adversarial training of the discriminator and the generator. Mean difference loss aligns features of two domains by minimizing the distribution difference. For semantic learning, a common solution is pseudo-labeling. Select the model prediction results with high confidence as pseudo-labels to supervise the training network.

[0003] For passive domain adaptation methods, existing pseudo-label-based methods have the problem of high noise, which will cause the network to learn the noise information in the pseudo-label and produce biased prediction results. This problem may cause network degradation in the transfer learning process, that is, the network performance after the final learning is worse than the initial network.

[0004] The existing remote sensing field is used to solve the problem of missing labeled information, such as small sample learning. It still requires the existence of labeled data in the target field, which is not applicable to some situations with strong restrictions.

[0005] In summary, the existing remote sensing field has the following problems: 1. Traditional pseudo-label-based methods have the problem of high noise, which may cause network degradation in the transfer learning process; 2. Other methods for dealing with insufficient label information, such as small sample learning, semi-supervised learning, etc., still require the existence of labeled information in the target domain, and are not applicable when labeled data is extremely scarce or completely missing. Summary of the invention

[0006] The purpose of the present invention is to provide a passive domain adaptation remote sensing image scene classification method and system based on self-training, so as to solve the technical problems that the existing remote sensing image scene classification methods have large noise and are not suitable for extremely scarce or completely missing labeled data.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a passive domain adaptation remote sensing image scene classification method based on self-training, comprising the following steps: The total loss of source domain data is used as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain a network pre-trained on the source domain data set; Initialize the network on the target domain dataset with the network pre-trained on the source domain dataset, and divide the target domain data into source domain similar samples and uncertain samples according to the network on the initialized target domain dataset; Based on similar samples and uncertain samples in the source domain, the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss are obtained; The total loss of the target domain is obtained based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss; The teacher-student network is used to optimize the network on the target domain dataset using the total loss of the target domain and the target domain data, and a classification network adapted to the target domain dataset is obtained; The classification network adapted to the target domain dataset is used to perform unsupervised classification on the target domain data to be classified.

[0008] In a second aspect, the present invention provides a passive domain adaptation remote sensing image scene classification system based on self-training, comprising: Pre-trained network acquisition module: used to use the total loss of source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain the network pre-trained on the source domain data set; Target domain data division module: used to initialize the network on the target domain data set with the network pre-trained on the source domain data set, and divide the target domain data into source domain similar samples and uncertain samples according to the network on the initialized target domain data set; Target domain loss acquisition module: used to obtain the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss based on similar samples and uncertain samples in the source domain; Target domain total loss acquisition module: used to obtain the target domain total loss based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss and the target domain data mixed loss; Classification network acquisition module: used to use the teacher-student network to optimize the network on the target domain dataset using the total loss of the target domain and the target domain data, and obtain a classification network adapted to the target domain dataset; Unsupervised classification module: used to perform unsupervised classification on the target domain data to be classified using a classification network adapted to the target domain dataset.

[0009] Compared with the prior art, the present invention has the following beneficial effects: 1. The method of the present invention uses the total loss of source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, obtain the pre-trained network on the source domain data set, and improve the generalization performance of the model in the new domain. The network on the target domain data set is initialized with the network pre-trained on the source domain data set, and the knowledge on the source domain data set is loaded to pave the way for the subsequent transfer learning process. It is helpful to adopt different training strategies for different samples in the future to improve the pertinence and efficiency of training. According to the similar samples and uncertain samples in the source domain, the supervised training loss of the credible samples in the target domain, the unsupervised training loss of the uncertain samples, the target domain data comparison learning training loss and the target domain data mixed loss are obtained. These losses together constitute the basis of the total loss of the target domain and reflect the performance of the model in the target domain. The total loss of the target domain is obtained according to the supervised training loss of the credible samples in the target domain, the unsupervised training loss of the uncertain samples, the target domain data comparison learning training loss and the target domain data mixed loss. Integrating the loss information from different sources provides a clear goal for subsequent model optimization. By minimizing the total loss in the target domain, the performance of the model in the target domain can be promoted, and the performance of the model can be further improved. The target domain data to be classified is unsupervisedly classified using a classification network adapted to the target domain data set, with high classification accuracy. The source domain data is adapted to the target domain data set and the classification of the target domain data to be classified is completed. When the source domain data set is inaccessible and the supervision information of the target domain data set is accurate, the framework can accurately complete the scene classification task. The present invention is designed based on a self-training method. Thanks to the division of the target domain data set, the present invention can reduce the problem of network degradation caused by excessive sample label noise. The present invention transfers the knowledge in the source domain to the target domain data set by self-training. Therefore, for the target domain data to be classified, this method can realize classification without relying on labeled data, solving the technical problem that the existing remote sensing image scene classification method is not suitable for extremely scarce or completely missing labeled data.

[0010] 2. The system of the present invention includes: a pre-trained network acquisition module, a target domain data division module, a target domain loss acquisition module, a target domain total loss acquisition module, a classification network acquisition module and an unsupervised classification module. The pre-trained network acquisition module is used to use the total loss of source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain the pre-trained network on the source domain data set; the target domain data division module is used to initialize the network on the target domain data set with the network pre-trained on the source domain data set, and divide the target domain data into source domain similar samples and uncertain samples according to the network on the initialized target domain data set; the target domain loss acquisition module is used to obtain the supervised training loss of target domain credible samples, the unsupervised training loss of uncertain samples, the target domain data contrast learning training loss and the target domain data mixed loss according to the source domain similar samples and uncertain samples; the target domain total loss acquisition module is used to obtain the target domain total loss according to the supervised training loss of target domain credible samples, the unsupervised training loss of uncertain samples, the target domain data contrast learning training loss and the target domain data mixed loss; the classification network acquisition module is used to use the teacher-student network to optimize the network on the target domain data set using the target domain total loss and the target domain data, and obtain the classification network adapted to the target domain data set; the unsupervised classification module is used to use the classification network adapted to the target domain data set to perform unsupervised classification on the target domain data to be classified. The various modules work together to achieve the classification of data to be classified in the target field without relying on labeled data, and can solve the technical problem that the existing remote sensing image scene classification method is not suitable for the extremely scarce or completely missing labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flowchart of passive field optical remote sensing image scene classification based on self-training according to an embodiment of the present invention; Figure 2 A source domain training network diagram of an embodiment of the present invention; Figure 3 A network framework diagram of the domain knowledge migration process of an embodiment of the present invention; Figure 4 is a flow chart of the method of the present invention; Figure 5 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0012] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0013] It should be noted that the terms "first", "second", etc. in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0014] The present invention is further described in detail below in conjunction with the accompanying drawings: See also Figure 4 The present invention discloses a passive domain adaptation remote sensing image scene classification method based on self-training, comprising the following steps: S1, uses the total loss of source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, obtains the network pre-trained on the source domain data set, and improves the generalization performance of the model in the new domain.

[0015] S2, initialize the network on the target domain dataset with the network pre-trained on the source domain dataset, and load the knowledge on the source domain dataset to pave the way for the subsequent transfer learning process. According to the initialized network on the target domain dataset, the target domain data is divided into similar samples and uncertain samples in the source domain, which helps to adopt different training strategies for different samples in the future, and improve the pertinence and efficiency of training.

[0016] S3, based on similar samples and uncertain samples in the source domain, obtains the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data comparative learning training loss and the target domain data mixed loss. These losses together form the basis of the total loss in the target domain and reflect the performance of the model in the target domain.

[0017] S4, based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrast learning training loss and the target domain data mixed loss, obtains the total loss of the target domain. Integrating the loss information from different sources provides a clear goal for subsequent model optimization. By minimizing the total loss of the target domain, the performance of the model in the target domain can be improved.

[0018] S5, using the teacher-student network to optimize the network on the target domain dataset using the target domain total loss and target domain data, and obtain a classification network adapted to the target domain dataset, which can further improve the performance of the model.

[0019] S6, uses the classification network adapted to the target domain dataset to perform unsupervised classification on the target domain data to be classified, with high classification accuracy. The source domain data is adapted to the target domain dataset and the classification of the target domain data to be classified is completed. When the source domain dataset is not accessible and the supervision information of the target domain dataset is accurate, the framework can accurately complete the scene classification task.

[0020] Embodiment 1: See also Figure 4 The present invention discloses a passive domain adaptation remote sensing image scene classification method based on self-training, comprising the following steps: S1, using the total loss of source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain the network pre-trained on the source domain data set, as follows: After rotating the source domain data, the prediction result of the rotation predictor is obtained through the source domain feature extractor and the rotation predictor, and the rotation prediction loss is obtained according to the prediction result of the rotation predictor and the randomly rotated unique hot label of the source domain data; Cross entropy is used to combine source domain data to obtain data mixing loss; After data enhancement of the source domain data, the source domain data classification prediction results are obtained through the source domain feature extractor and the source domain classifier. The label smoothed cross entropy loss is obtained according to the classifier prediction results and the unique hot encoding of the source domain data, as follows: After data enhancement is performed on the source domain data, the source domain data classification prediction results are obtained through the source domain feature extractor and the source domain classifier; Get soft labels based on the one-hot encoding of the source domain data; Obtain label smoothed cross entropy loss based on classifier predictions and soft labels.

[0021] The total loss of source domain data is obtained based on the rotation prediction loss, data mixing loss, and label smoothing cross entropy loss. The formula for obtaining the total loss of source domain data is as follows:

[0022] in, and is a hyperparameter, is the total data loss in the source domain, is the label smoothed cross entropy loss, is the rotation prediction loss, is the data mixing loss.

[0023] The total loss of source domain data is used to optimize the source domain network to obtain an optimized source domain network. The optimized source domain network is used to learn the semantic information in the source domain data to obtain a network pre-trained on the source domain data set.

[0024] S2, initialize the network on the target domain dataset with the network pre-trained on the source domain dataset, and divide the target domain data into source domain similar samples and uncertain samples according to the network on the initialized target domain dataset, as follows: Obtain the prediction confidence change of the target domain data based on the network on the initialized target domain data set; The target domain data is divided into source domain similar samples and uncertain samples according to the change in prediction confidence of the target domain data.

[0025] S3, based on similar samples and uncertain samples in the source domain, obtains the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss, as follows: The pseudo-label supervision method is used to train similar samples in the source domain to obtain the supervised training loss of credible samples in the target domain, as follows: Obtain the logical value of the source domain similar samples according to the network's predicted probability of the source domain similar samples, the neighbor set of the source domain similar samples, and the neighbor weight; Obtain pseudo labels of similar samples in the source domain according to the logical values ​​of similar samples in the source domain; The supervised training loss of credible samples in the target domain is obtained based on the pseudo labels of similar samples in the source domain and the predicted probabilities of similar samples in the source domain. The formula for obtaining the supervised training loss of credible samples in the target domain is as follows:

[0026] in, is the supervised training loss of credible samples in the target domain, Pseudo labels for samples in the target domain data The first one-hot encoding of The value of the dimension, is the pseudo label of the sample in the target domain data, is the number of categories, For the Vidi The predicted probability of similar samples in the source domain.

[0027] The unsupervised training method is used to train uncertain samples to obtain the unsupervised training loss of uncertain samples, as follows: The unsupervised training loss of uncertain samples is obtained according to the predicted probability of weak enhancement of uncertain samples and the predicted probability of strong enhancement of uncertain samples. The formula for obtaining the unsupervised training loss of uncertain samples is as follows:

[0028] in, is the unsupervised training loss for uncertain samples, is the number of categories, Indicates The predicted probability of weak enhancement of uncertain samples, Indicates The predicted probability of a strongly enhanced uncertain sample.

[0029] The target domain data is learned based on the noise-improved contrastive learning strategy to obtain the target domain data contrastive learning training loss, which is as follows: The target domain data contrast learning training loss is obtained according to the prediction head output of the target domain data, the mapping head output of the target domain data, the probability prediction result of the target domain data and the similarity coefficient. The formula for obtaining the target domain data contrast learning training loss is as follows:

[0030] in, is the training loss for comparative learning of target domain data, Indicates The prediction head output of target domain data is Indicates The mapping header output of the target domain data, Indicates The prediction head output of target domain data is Indicates The mapping header output of the target domain data, express The transpose of Indicates the first The probability prediction results of target domain data, Indicates the first The probability prediction results of target domain data, The similarity coefficient is used to measure the similarity between two samples and is set based on experience. , It represents the dot product operation of two vectors. Represents the similarity measure of two vectors, where vector dot product is used.

[0031] The target domain data is supervised by pseudo-label learning to obtain the mixed loss of the target domain data, as follows: Similar samples in the source domain and Mixup technology are used in combination with pseudo labels to generate mixed loss of target domain data. The cross entropy loss function is used for calculation in the process of generating mixed loss of target domain data.

[0032] S4, the total loss of the target domain is obtained according to the supervised training loss of the credible samples in the target domain, the unsupervised training loss of the uncertain samples, the target domain data contrastive learning training loss and the target domain data mixed loss. The formula for obtaining the total loss of the target domain is as follows:

[0033] in, , and are all hyperparameters, is the total loss in the target area, is the supervised training loss of credible samples in the target domain, is the unsupervised training loss for uncertain samples, is the training loss for comparative learning of target domain data, is the target domain data mixing loss.

[0034] S5, the teacher-student network is used to optimize the network on the target domain dataset using the total loss of the target domain and the target domain data, and a classification network adapted to the target domain dataset is obtained, as follows: Initialize the teacher network and student network using the pre-trained parameters of the source domain; The student network parameters are updated in the student network, and the exponential average move is used to smoothly update the student network parameters to the teacher network to obtain a classification network adapted to the target domain dataset.

[0035] S6, uses the classification network adapted to the target domain dataset to perform unsupervised classification on the target domain data to be classified.

[0036] Embodiment 2: See also Figure 1 The present invention discloses a passive domain adaptation remote sensing image scene classification method based on self-training, which is divided into two processes: source domain training and target domain adaptation.

[0037] First, the present invention aims to train the model in a supervised manner in the source domain, and then transfer the source domain knowledge to the target domain data set through transfer learning to achieve the target domain data scene classification task without labeled information in the target domain. In order for the model to achieve the expected effect in the target domain, training a model with strong classification performance in the source domain is a problem worth exploring. Taking into account the characteristics of remote sensing images and the requirements of domain generalization performance, the present invention designs a source domain network training method to enhance the model's learning of domain invariant knowledge and improve generalization performance. After completing the pre-training in the source domain, the network needs to adjust parameters in the absence of the target domain data set to complete the scene classification task of the target domain data set. The second part is the target domain adaptation process. Figure 3 Give an explanation of the process. In order to better illustrate this process, we divide it into four parts: sample partitioning, source domain similar sample learning, uncertain sample learning, and training process. Sample partitioning aims to divide similar samples and uncertain samples in the source domain and design specific learning methods for different data features to fully learn the internal relationship of the data and avoid network degradation caused by noise. The training process is to better coordinate the learning of the two parts of the samples and add some additional modules to achieve efficient adaptation of the target domain scene classification task. The details of the passive domain adaptation optical remote sensing image scene classification method based on self-training proposed in this study are as follows: S1, source domain training: The training process of the source domain network is as follows: Figure 2 Considering that the performance of domain adaptation is affected by the learning effect of the model in the source domain, the present invention designs a training method with strong generalization performance. In order to adapt the model to the domain migration task, the present invention adopts a general passive domain adaptation network design model, and adds batch normalization and weight normalization to the backbone network and classifier to reduce the data distribution bias caused by the training process and the similarity calculation bias caused by the classification of new data. In order to enhance the generalization performance of the source domain network, the label smoothed cross entropy loss is used as the objective function to guide the network to learn the semantic information in the source domain data set. The process is as follows:

[0038]

[0039]

[0040] in, Indicates a soft tag. represents soft parameters, Represents source domain data No. The unique hot encoding value of the category is set based on experience In this embodiment , Represents source domain data No. The predicted probability of the class, represents the classifier, represents the feature extractor, is the number of categories, is the label smoothed cross entropy loss, is the label smoothing loss identifier, is the source domain identifier, is the source domain classifier, is the source domain feature extractor, Source field data.

[0041] Inspired by unsupervised learning, in order to enable the network to learn universal representation capabilities, a rotation prediction classifier is introduced here to guide the network to extract features with domain invariance. The process is expressed as:

[0042]

[0043] in, is the rotation prediction loss, A pointer to traverse each dimension of the angle prediction result. Indicates that for source domain data Randomly rotated one-hot labels, represents the prediction result of the rotation predictor, Represents source domain data The predicted probability of the jth dimension after random rotation, Represents a rotation operation. Represents the rotation predictor, the rotation angle range is , the image is randomly rotated before each input to the network, is the source domain feature extractor.

[0044] In order to further enhance the generalization performance of the network and improve its robustness, data enhancement and mixed loss Mixup are used here to improve the robustness of the network. The mixed loss Mixup is expressed as The cross entropy is used to implement it. The overall objective function of the training process in the final source domain is:

[0045] in, and is a hyperparameter, is the total data loss in the source domain, is the label smoothed cross entropy loss, is the rotation prediction loss, is the data mixing loss.

[0046] S2, domain adaptation on the target dataset; Sample division: Considering the large difference in distribution between the target domain and the source domain, directly using pseudo-label supervised training will lead to the accumulation of noise and cause network degradation. To this end, the present invention divides the target domain data into source domain similar samples and uncertain samples. For source domain similar samples, we use pseudo-label supervised training methods to obtain supervised training losses of credible samples in the target domain, which can achieve semantic learning while reducing noise. For uncertain samples, we use unsupervised training methods to obtain unsupervised training losses of uncertain samples, guiding the network to learn the potential relationship between samples to avoid the impact of noise.

[0047] According to the deep learning hypothesis, the target domain data should be mapped to the latent space after being input into the feature extractor and should show a clustered distribution according to the category and different categories should have clear boundaries. Based on this, we designed a simple and novel solution: for a sample data in the target domain data, the sample whose prediction confidence changes less than the threshold before and after the network parameter update should be regarded as a similar sample in the source domain, and the sample whose change is greater than the threshold is an uncertain sample. Let's use an intuitive example to explain: when the confidence of the sample prediction in the target domain data changes significantly, we can think that the network is not confident enough in the prediction result of the sample, resulting in a large displacement of the sample in the latent space or a large change in the confidence of the sample mapping on the classification boundary due to the change of category. According to this principle, our sample partitioning expression is as follows, and we will use different methods for training according to the partitioning results here.

[0048]

[0049]

[0050] in, represents the oscillation factor, Represents the number of steps of forward propagation of samples in the target domain data, represents the predicted probability, that is, the predicted result of the sample, Represents the forward propagation of samples in the target domain data The predicted probability of the step, Represents the forward propagation of samples in the target domain data The predicted probability of the step, Represents the prediction confidence of samples in the target domain data, Represents the division result of samples in the target domain data. , it means that the target domain data is similar to the source domain data, otherwise , indicating that the target domain data is an uncertain sample, Gets the largest value in the vector.

[0051] Learning of similar samples in the source domain: Similar samples in the source domain can be obtained by sample division. This type of sample can be regarded as being close to the domain-invariant space, so the network prediction results of this part of the samples are relatively reliable, and using them for supervised training can achieve better results. Supervised training requires labels, considering that directly using the prediction results of the samples as pseudo-labels may be biased due to noise. The present invention uses a neighbor sample label fusion method to generate pseudo-labels for similar samples in the source domain. The idea of ​​neighbor sample label fusion is to select samples from the latent space of the features and average the predicted probabilities of the nearest neighbor samples to obtain the predicted probability of the sample. In order to better filter the noise, we do not use the mean but an entropy weighted method to apply different weights to trusted neighbors and untrusted neighbors and then fuse them. The generation process of pseudo-labels for similar samples in the source domain is as follows:

[0052]

[0053]

[0054] in, is the logical value of similar samples in the source domain, is a pseudo-label identifier, Represents the neighbor set of similar samples in the source domain. Here, cosine similarity is used to measure the distance of latent space samples. is the sample number in the data set, is the neighbor weight, is the network’s prediction probability for similar samples in the source domain, represents the first samples, are pseudo labels of similar samples in the source domain, is the maximum independent variable point set function, is a normalization function used to convert logical values ​​into probability vectors. represents the predicted probability of neighbor samples, represents the information entropy calculation function, represents the entropy of the prediction results of similar samples in the source domain, is the number of categories.

[0055] After the pseudo labels are generated, their hard labels are used as supervised training samples in the source domain to guide the network to learn the semantic information of the target domain.

[0056]

[0057] in, is the supervised training loss of credible samples in the target domain, Pseudo labels for samples in the target domain data The first one-hot encoding of The value of the dimension, is the pseudo label of the sample in the target domain data, is the number of categories, For the Vidi The predicted probability of similar samples in the source domain is the predicted result of the sample.

[0058] Uncertain sample learning: Uncertain samples are selected because the network makes unconfident predictions about them. These samples are often located in a domain-specific distribution. Using pseudo-label training will inevitably introduce a lot of noise interference, causing network degradation. Therefore, an unsupervised training method is used for uncertain samples to avoid noise interference. We perform two types of processing on this type of samples: strong enhancement and weak enhancement. Obviously, the prediction results of these two samples should be of the same category. We consider using mean square error loss to bring the prediction results of the two data enhancements closer. This objective function can be expressed in the latent space as the same prediction for the sample and the perturbed samples around it. Therefore, it can play a tightening role in both the decision boundary and the cluster structure to achieve the representation learning of uncertain samples. Due to the introduction of unsupervised learning, the network will learn a representation with confident judgment for such difficult samples, so these samples will be propagated to the learning of similar samples in the source domain for semantic learning, thereby increasing the number of similar samples in the source domain. The expression of uncertain sample learning is as follows:

[0059] in, is the unsupervised training loss for uncertain samples, is the number of categories, Indicates The predicted probability of weak enhancement of uncertain samples, Indicates The predicted probability of a strongly enhanced uncertain sample.

[0060] Training process: Source domain sample learning realizes semantic learning of high-confidence samples, and uncertain sample learning completes the consistent prediction of sample perturbations in the latent space. The two coordinate to complete the transfer of uncertain samples to similar samples in the source domain. However, using only these two modules cannot obtain a clear decision boundary in the target domain. Therefore, in order to ensure that the target domain has the expected scene classification performance, contrastive learning and mixed loss Mixup are introduced here to assist these two processes.

[0061] For the learning of uncertain samples, it is obviously not enough to simply shorten the distance between the sample and the perturbed sample in the latent space. This cannot solve the problem of category cohesion. This method introduces contrastive learning and uses pseudo labels as supervision to guide the network to produce a more cohesive category mapping for samples of the same category, thereby achieving the extraction of discriminative features. In order to avoid the influence of noise, a contrastive learning strategy based on noise improvement is introduced here, and the expression is as follows:

[0062] in, is the training loss for comparative learning of target domain data, Indicates The prediction head output of target domain data is Indicates The mapping header output of the target domain data, Indicates The prediction head output of target domain data is Indicates The mapping header output of the target domain data, express The transpose of Indicates the first The probability prediction results of target domain data, Indicates the first The probability prediction results of target domain data, The similarity coefficient is used to measure the similarity between two samples and is set based on experience. , It represents the dot product operation of two vectors. Represents the similarity measure of two vectors, where vector dot product is used.

[0063] For samples similar to the source domain, pseudo-label supervised learning is used. Although high-confidence pseudo-labels can guide the network to learn accurate semantic information. However, due to problems such as insufficient samples and a small amount of noise, the network still cannot accurately extract features. Here, data mixing operations are used to expand the samples to form a smooth decision boundary between samples to improve the robustness of model prediction. The target domain data and Mixup technology are combined with pseudo-labels to generate the target domain data mixed loss. The cross entropy loss function is used for calculation in the process of generating the target domain data mixed loss.

[0064] In the target domain adaptation process, the overall objective function of our method is as follows:

[0065] in, , and are all hyperparameters, is the total loss in the target area, is the supervised training loss of credible samples in the target domain, is the unsupervised training loss for uncertain samples, is the training loss for comparative learning of target domain data, is the target domain data mixing loss.

[0066] In order to ensure the stability of the sample division process, the teacher-student network design is adopted here. Figure 3 , the teacher network and the student network are initialized with the source domain pre-trained parameters during initialization. The network parameter update is performed in the student network and the exponential average move is used to smoothly update the student network parameters to the teacher network. The sample division in the target domain data to be classified is completed by the teacher network. Due to the introduction of the exponential average move, the update amplitude of the teacher network is small, thus ensuring the stability of the sample division and improving the robustness of the overall network prediction. The exponential average move process update formula is as follows:

[0067] in, is the exponential moving average weight, which is set to 0.9 based on experience. For Network parameters at the time.

[0068] The present invention introduces an unsupervised method to extract domain-invariant features, enhance the generalization ability of the source domain model, and improve the performance of subsequent domain migration tasks.

[0069] The present invention utilizes a sample partitioning method to perceive the confidence fluctuation of samples in the target domain and screen out similar samples and uncertain samples in the source domain, accurately providing a credible sample set and an uncredible sample set for subsequent use.

[0070] The present invention utilizes a pseudo-label generation method to avoid the error caused by network prediction of a single sample by fusing neighbor samples and adopts entropy weighting to filter the noise in the neighbor samples to achieve accurate generation of pseudo-labels.

[0071] The present invention realizes the adaptation of source domain knowledge to target domain data set and completes the scene classification task. When the source domain data set is not accessible and the supervision information of the target domain data set is accurate, the framework can accurately complete the scene classification task.

[0072] For the domain adaptation method, the present invention can solve the problem of the inaccessibility of the source domain, and only use the model pre-trained in the source domain to complete the parameter fine-tuning in the target domain. This not only reduces the cost of model training, but also reduces the cost of manual annotation to achieve scene classification without annotation in the target domain.

[0073] For passive domain adaptation, the present invention improves the generalization ability of the source domain model and provides an effective source domain pre-training model for the domain adaptation process.

[0074] Compared with the small sample method, the present invention can achieve classification when the target domain is completely unlabeled, solving the strong restriction problem that the target domain has no data set annotations at all.

[0075] Based on the above method, the present invention also discloses a passive domain adaptation remote sensing image scene classification system based on self-training, see Figure 5 ,include: Pre-trained network acquisition module: used to use the total loss of source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain the network pre-trained on the source domain data set; Target domain data division module: used to initialize the network on the target domain data set with the network pre-trained on the source domain data set, and divide the target domain data into source domain similar samples and uncertain samples according to the network on the initialized target domain data set; Target domain loss acquisition module: used to obtain the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss based on similar samples and uncertain samples in the source domain; Target domain total loss acquisition module: used to obtain the target domain total loss based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss and the target domain data mixed loss; Classification network acquisition module: used to use the teacher-student network to optimize the network on the target domain dataset using the total loss of the target domain and the target domain data, and obtain a classification network adapted to the target domain dataset; Unsupervised classification module: used to perform unsupervised classification on the target domain data to be classified using a classification network adapted to the target domain dataset.

[0076] The system of the present invention can classify the data to be classified in the target field without relying on labeled data, and can solve the technical problem that the existing remote sensing image scene classification method is not suitable for the extremely scarce or completely missing labeled data.

[0077] The present invention is mainly used in remote sensing scene classification tasks, which is a basic task in remote sensing image intelligent interpretation technology, and solves the problem of high cost and long cycle of manual labeling. The introduction of transfer learning technology realizes the migration of knowledge from the source field to the target field, improves the application value of the existing labeled data set, and overcomes the problem that the source field data set is inaccessible and edge computing cannot be retrained. The present invention can be widely used in environmental protection, earth observation, urban planning and other fields, bringing certain social benefits. In addition, the ideas and processes of the present invention can be applied to tasks such as hyperspectral classification and semantic segmentation to provide new ideas for the application of transfer learning in the intelligent interpretation of remote sensing images.

[0078] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A passive domain adaptation remote sensing image scene classification method based on self-training, characterized in that: The following steps are involved: The total loss of source domain data is used as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain a network pre-trained on the source domain data set; Initialize the network on the target domain dataset with the network pre-trained on the source domain dataset, and divide the target domain data into source domain similar samples and uncertain samples according to the network on the initialized target domain dataset; Based on similar samples and uncertain samples in the source domain, the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss are obtained; The total loss of the target domain is obtained based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss; The teacher-student network is used to optimize the network on the target domain dataset using the total loss of the target domain and the target domain data, and a classification network adapted to the target domain dataset is obtained; The classification network adapted to the target domain dataset is used to perform unsupervised classification on the target domain data to be classified.

2. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 1, characterized in that: The total loss of the source domain data is used as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain a network pre-trained on the source domain data set, as follows: After rotating the source domain data, the prediction result of the rotation predictor is obtained through the source domain feature extractor and the rotation predictor, and the rotation prediction loss is obtained according to the prediction result of the rotation predictor and the randomly rotated unique hot label of the source domain data; Cross entropy is used to combine source domain data to obtain data mixing loss; After data enhancement of the source domain data, the source domain data classification prediction results are obtained through the source domain feature extractor and the source domain classifier, and the label smoothed cross entropy loss is obtained according to the classifier prediction results and the unique hot encoding of the source domain data; Obtain the total loss of source domain data based on rotation prediction loss, data mixing loss, and label smoothing cross entropy loss; The total loss of source domain data is used to optimize the source domain network to obtain an optimized source domain network. The optimized source domain network is used to learn the semantic information in the source domain data to obtain a network pre-trained on the source domain data set.

3. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 2, characterized in that: After data enhancement is performed on the source domain data, the source domain data classification prediction result is obtained through the source domain feature extractor and the source domain classifier, and the label smoothed cross entropy loss is obtained according to the classifier prediction result and the unique hot encoding of the source domain data, as follows: After data enhancement is performed on the source domain data, the source domain data classification prediction results are obtained through the source domain feature extractor and the source domain classifier; Get soft labels based on the one-hot encoding of the source domain data; Obtain label smoothed cross entropy loss based on classifier predictions and soft labels.

4. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 1, characterized in that: The target domain data is divided into source domain similar samples and uncertain samples according to the network on the initialized target domain data set, as follows: Obtain the prediction confidence change of the target domain data based on the network on the initialized target domain data set; The target domain data is divided into source domain similar samples and uncertain samples according to the change in prediction confidence of the target domain data.

5. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 1, characterized in that: The supervised training loss of the target domain credible samples, the unsupervised training loss of the uncertain samples, the target domain data contrast learning training loss and the target domain data mixed loss are obtained according to the similar samples and uncertain samples in the source domain, as follows: The pseudo-label supervision method is used to train similar samples in the source domain to obtain the supervised training loss of credible samples in the target domain; An unsupervised training method is used to train uncertain samples to obtain an unsupervised training loss for uncertain samples; The target domain data is learned based on the noise-improved contrastive learning strategy to obtain the target domain data contrastive learning training loss; Pseudo-label supervised learning of target domain data is used to obtain the target domain data mixed loss.

6. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 5, characterized in that: The pseudo-label supervision method is used to train similar samples in the source domain to obtain the supervised training loss of credible samples in the target domain, which is as follows: Obtain the logical value of the source domain similar samples according to the network's predicted probability of the source domain similar samples, the neighbor set of the source domain similar samples, and the neighbor weight; Obtain pseudo labels of similar samples in the source domain according to the logical values ​​of similar samples in the source domain; The supervised training loss of credible samples in the target domain is obtained according to the pseudo labels of similar samples in the source domain and the predicted probabilities of similar samples in the source domain.

7. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 5, characterized in that: The unsupervised training method is used to train uncertain samples to obtain unsupervised training losses of uncertain samples, which are specifically as follows: The unsupervised training loss of uncertain samples is obtained according to the predicted probability of weak enhancement of uncertain samples and the predicted probability of strong enhancement of uncertain samples.

8. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 5, characterized in that: The noise-improved contrastive learning strategy learns the target domain data to obtain the target domain data contrastive learning training loss, and the pseudo-label supervised learning target domain data is used to obtain the target domain data mixed loss, which is as follows: Obtain the target domain data comparative learning training loss according to the prediction head output of the target domain data, the mapping head output of the target domain data, the probability prediction result of the target domain data and the similarity coefficient; Similar samples in the source domain and Mixup technology are used in combination with pseudo labels to generate mixed loss of target domain data. The cross entropy loss function is used for calculation in the process of generating mixed loss of target domain data.

9. The method for remote sensing image scene classification based on self-training and passive domain adaptation according to claim 1, characterized in that: The teacher-student network is used to optimize the network on the target domain dataset using the total loss of the target domain and the target domain data to obtain a classification network adapted to the target domain dataset, as follows: Initialize the teacher network and student network using the pre-trained parameters of the source domain; The student network parameters are updated in the student network, and the exponential average move is used to smoothly update the student network parameters to the teacher network to obtain a classification network adapted to the target domain dataset.

10. A passive domain adaptation remote sensing image scene classification system based on self-training, characterized in that: include: Pre-trained network acquisition module: used to use the total loss of source domain data as the objective function to guide the passive domain adaptation network to learn the semantic information in the source domain data, and obtain the network pre-trained on the source domain data set; Target domain data division module: used to initialize the network on the target domain data set with the network pre-trained on the source domain data set, and divide the target domain data into source domain similar samples and uncertain samples according to the network on the initialized target domain data set; Target domain loss acquisition module: used to obtain the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss, and the target domain data mixed loss based on similar samples and uncertain samples in the source domain; Target domain total loss acquisition module: used to obtain the target domain total loss based on the supervised training loss of credible samples in the target domain, the unsupervised training loss of uncertain samples, the target domain data contrastive learning training loss and the target domain data mixed loss; Classification network acquisition module: used to use the teacher-student network to optimize the network on the target domain dataset using the total loss of the target domain and the target domain data, and obtain a classification network adapted to the target domain dataset; Unsupervised classification module: used to perform unsupervised classification on the target domain data to be classified using a classification network adapted to the target domain dataset.

Images