A remote sensing image cross-domain small sample classification method based on pseudo-label uncertainty perception

Abstract

The application discloses a remote sensing image cross-domain small sample classification method based on pseudo-label uncertainty perception, and comprises the following steps: acquiring source domain remote sensing image data and target domain remote sensing image data; pre-training a deep learning backbone network model by using the source domain remote sensing image data to obtain pre-training initialization parameters of the deep learning backbone network model; constructing a teacher-student network model for semi-supervised cross-domain training; inputting the source domain remote sensing image data and the target domain remote sensing image data into the teacher-student network model for training to obtain a trained deep learning backbone network model; and inputting to-be-tested remote sensing image data into the trained deep learning backbone network model to obtain a classification result of the to-be-tested remote sensing image data. The application can overcome the defect that high-uncertainty pseudo labels damage model cross-domain learning in the prior art, and improve the performance of a deep learning method in remote sensing image cross-domain small sample classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image cross-domain small sample classification, and particularly relates to a remote sensing image cross-domain small sample classification method based on pseudo-label uncertainty perception. BACKGROUND

[0002] Scene classification is a basic problem in the intelligent interpretation task of remote sensing images, that is, through computer vision algorithm to extract the semantic information rich in remote sensing images, and to assign correct semantic labels to remote sensing images, so as to realize the classification of the scene displayed by the remote sensing images. This is the basis for carrying out intelligent interpretation tasks of remote sensing data such as urban intelligent planning, land statistical analysis, and regional automatic extraction. With the wide application of deep learning, convolutional neural networks have shown strong automatic feature extraction capabilities, which has significantly improved the ability of remote sensing scene classification. However, most of the current scene classification methods based on deep learning are based on the assumption that the training set and the test set are sampled from the same data distribution, which is difficult to meet in the actual application of remote sensing images. Because remote sensing images come from different satellite sensors, affected by factors such as sensor type, illumination angle, and shooting position, the remote sensing image data obtained has differences in imaging, resolution, and illumination, etc. in terms of data distribution; and because different types of remote sensing data sets have different purposes, the categories they contain are also different, that is, the categories are not aligned, which forms a domain shift. And the existing deep learning model based on supervised learning has poor generalization ability, and for the real open scene, the training data and the measured data have domain shift, the measured labeled samples are limited, the model has poor generalization ability, and it is difficult to adapt to new data distribution and new emerging categories, which is difficult to meet the needs of the actual application of remote sensing scene classification. Therefore, how to train the model from the existing labeled data set and transfer the learned knowledge to improve the generalization ability of the model so that it can adapt to the classification task of other distribution and category different data sets in the open environment with less labeled and cross-scene has become a challenging problem in remote sensing scene classification.

[0003] Based on the above reasons, it is necessary to consider the cross-domain small sample classification of remote sensing images, that is, the model learns knowledge from the existing labeled remote sensing image dataset (source domain), and then on a new remote sensing image dataset (target domain), only through a small number of labeled training samples of the target domain, accurate classification results can be obtained, wherein there is a domain shift between the source domain data and the target domain data, including data distribution difference and class misalignment. At present, advanced cross-domain small sample classification is mainly based on the transfer learning paradigm, which uses part of the unlabeled data of the target domain as additional data, introduces the target domain information by labeling it with pseudo labels, and at the same time uses the labeled data of the source domain, combines source domain supervised learning with target domain self-supervised learning, and achieves good cross-domain small sample classification performance. However, the output of the deep learning model for data or classes that have not been seen before is uncertain. Due to the existence of domain shift, the output pseudo label of the model for the target domain will have great uncertainty, and directly using the pseudo label with high uncertainty for training will seriously affect the cross-domain learning of the model, resulting in poor cross-domain small sample classification performance. SUMMARY

[0004] In order to solve the above problems, the present application provides a remote sensing image cross-domain small sample classification method based on pseudo label uncertainty perception, which can overcome the defect that high uncertainty pseudo label damages the cross-domain learning of the model in the prior art, and improve the performance of the deep learning method for remote sensing image cross-domain small sample classification.

[0005] In order to achieve the purpose of the present application, the following technical solutions are adopted:

[0006] A remote sensing image cross-domain small sample classification method based on pseudo-label uncertainty perception, comprising the following steps: S101: acquiring source domain remote sensing image data and target domain remote sensing image data; wherein the source domain remote sensing image data carries a first number of labels, and the target domain remote sensing image data does not carry labels; the labels indicate the category information of the remote sensing image data; S102: pre-training a deep learning backbone network model using the source domain remote sensing image data to obtain pre-training initialization parameters of the deep learning backbone network model; S103: constructing a teacher-student network model for semi-supervised cross-domain training; wherein the teacher-student network model includes a teacher model and a student model, the teacher model includes the deep learning backbone network model and a teacher mapper, the student model includes the deep learning backbone network model and a student mapper, and the deep learning backbone network model loads the pre-training initialization parameters; S104: inputting the source domain remote sensing image data and the target domain remote sensing image data into the teacher-student network model for training to obtain a trained deep learning backbone network model; S105: inputting to-be-tested remote sensing image data into the trained deep learning backbone network model to obtain a classification result of the to-be-tested remote sensing image data, the classification result indicating the category of the to-be-tested remote sensing image; wherein the to-be-tested remote sensing image data carries a second number of labels, and the second number is less than the first number.

[0007] Further, the constructing a teacher-student network model for semi-supervised cross-domain training comprises: S103.1: loading the pre-training initialization parameters into the deep learning backbone network model in the teacher model and the deep learning backbone network model in the student model; S103.2: the student model updates the student model pre-training initialization parameters using gradient descent; S103.3: the teacher model updates the teacher model pre-training initialization parameters using exponential moving average, and the exponential moving average parameter update expression is as follows:

[0008]

[0009] wherein θ t is the teacher model pre-training initialization parameter, θ s is the student model pre-training initialization parameter, and m is a momentum parameter and a is the current iteration number.

[0010] Further, inputting the source domain remote sensing image data and the target domain remote sensing image data into the teacher-student network model for training to obtain a trained deep learning backbone network model comprises: S104.1: inputting the source domain remote sensing image data into the student model for source domain supervised learning, so that the student model synchronously learns source domain knowledge; wherein the source domain knowledge at least includes medium-low layer general feature information of the source domain remote sensing image; S104.2: performing weak enhancement processing and strong enhancement processing on the target domain remote sensing image data to obtain weak enhancement data and strong enhancement data respectively; S104.3: inputting the weak enhancement data into the teacher model, and performing processing through the teacher model to obtain teacher prediction probability and teacher mapping result; S104.4: inputting the strong enhancement data into the student model, and performing processing through the student model to obtain student prediction probability and student mapping result; S104.5: performing sharpening processing on the teacher prediction probability to obtain pseudo labels; S104.6: performing consistency regularization processing on the pseudo labels and the student prediction probability; S104.7: constructing an uncertainty-aware target domain difficult example sample mining strategy to mine difficult example samples; wherein the difficult example samples are remote sensing image data that are difficult to classify; S104.8: performing teacher external feature mapping processing on the teacher mapping result to obtain teacher external mapping result; performing student external feature mapping on the student mapping result to obtain student external mapping result; S104.9: performing contrastive learning processing on the teacher mapping result and the student external mapping result, and performing contrastive learning processing on the teacher external mapping result and the student mapping result; S104.10: updating the pre-training initialization parameters based on the source domain supervised learning, the consistency regularization processing, the contrastive learning processing and the difficult example sample mining strategy to obtain the trained deep learning backbone network model.

[0011] Further, the inputting the source domain remote sensing image data into the student model for the source domain supervised learning comprises: S104.1.1: performing feature extraction on the source domain remote sensing image data to obtain source domain data feature result; S104.1.2: performing classification prediction on the source domain data feature result to obtain source domain data classification prediction probability wherein, is the source domain remote sensing image sample, i is a source domain sample index, f s is a feature extractor, c s is a classifier; S104.1.3: optimizing the student model by using a source domain supervised loss function, wherein the source domain supervised loss function L S The expression is as follows:

[0012]

[0013] wherein, is a cross-entropy loss function, is a label corresponding to the source domain remote sensing image data, N S is the number of source domain remote sensing image data.

[0014] Further, the weak enhancement processing and the strong enhancement processing on the first target domain remote sensing image data include that the weak enhancement processing includes: S104.2.11: performing random cropping processing on the first target domain remote sensing image data to obtain a first cropped image; S104.2.12: performing horizontal flip processing on the first cropped image to obtain a first flipped image; S104.2.13: performing normalization processing on the first flipped image to obtain first normalized image data; and the strong enhancement processing includes: S104.2.21: performing random cropping processing on the first target domain remote sensing image data to obtain a second cropped image; S104.2.22: performing horizontal flip processing on the second cropped image to obtain a second flipped image; S104.2.23: performing color jitter processing on the second flipped image to obtain a second color changed image; S104.2.24: performing Gaussian blur processing on the second color changed image to obtain a second blurred image; S104.2.25: performing random grayscale transformation processing on the second blurred image to obtain a second random grayscale image; and S104.2.26: performing normalization processing on the second random grayscale image to obtain second normalized image data.

[0015] Further, the weak enhancement data inputting the teacher model and being processed by the teacher model includes: S104.3.1: performing feature extraction on the weak enhancement data to obtain a teacher feature map; S104.3.2: performing classification prediction on the teacher feature map to obtain a teacher prediction probability; and S104.3.3: performing feature mapping on the teacher feature map to obtain a teacher mapping result.

[0016] Further, the strong enhancement data inputting the student model and being processed by the student model includes: S104.4.1: performing feature extraction on the strong enhancement data to obtain a student feature map; S104.4.2: performing classification prediction on the student feature map to obtain a student prediction probability; and S104.4.3: performing feature mapping on the student feature map to obtain a student mapping result.

[0017] Further, the consistency regularization processing on the pseudo label and the student prediction probability includes: performing consistency regularization processing by a consistency regularization loss function, the consistency regularization loss function L CR The expression is as follows:

[0018]

[0019] wherein, L CE is a cross-entropy loss function, is a pseudo label, is a student prediction probability, N T is a first target domain remote sensing image sample number, T represents a target domain, and j is a target domain sample index.

[0020] Further, the uncertainty-aware target domain hard example mining strategy comprises: S104.7.1: calculating the uncertainty Unc j of the teacher prediction probability:

[0021]

[0022] wherein, L SE is a self-entropy, is the teacher prediction probability, and j is a sample index of target domain remote sensing image data;

[0023] S104.7.2: sorting the uncertainty from high to low to obtain a sorted sample index value j' of the target domain remote sensing image data, j' = argsort(Unc j ), wherein argsort is a sorting function; S104.7.3: extracting the first N H =B·r samples as hard example samples, the index value corresponding to the hard example samples is j H =j'[0:B·r], wherein N H is a hard example sample number, H represents a hard example, B is a sample number in a training small batch, and r is a selection ratio;

[0024] S104.7.4: calculating a hard example loss L UH of the hard example samples:

[0025]

[0026] wherein, is the pseudo label, is a student prediction probability N H is a hard example sample number, H represents a hard example, j is a target domain sample index, and j H is an index value corresponding to the hard example samples.

[0027] Further, the contrastive learning processing comprises: the contrastive learning processing is performed by a contrastive loss function L Con :

[0028]

[0029] wherein, is a student external mapping result, is a teacher mapping result, is a teacher external mapping result, is a student mapping result, p d is an external mapper, p t is a mapper, D(q,z)=-q / ||q||·z / ||z|| is a negative cosine similarity calculation formula, ||·|| represents a modulus operation, N T is a first target domain remote sensing image sample number, T represents a target domain, and j is a target domain sample index.

[0030] Further, the updating of the pre-training initialization parameter includes: updating of the pre-training initialization parameter based on a total loss function L:

[0031] L=L S +λ(L CR +L UH )+L Con

[0032] wherein,

[0033]

[0034]

[0035]

[0036] wherein, L S is the source domain supervised loss function, L CR is a target domain consistency regularization loss function, L UH is a hard example loss function, L Con is a contrastive loss function, and λ is a hyperparameter.

[0037] Based on the defects of high-uncertainty pseudo-label damage model cross-domain learning in the prior art, the scheme claimed in the present application is proposed. A deep learning backbone network model is obtained by pre-training of source domain remote sensing image data, and then a semi-supervised cross-domain training teacher-student network model is constructed. At the same time, source domain labeled remote sensing image data and target domain remote sensing image data are used, combined with a pseudo-label uncertainty perception target domain hard example sample mining strategy, to perform cross-domain learning of the deep learning backbone network model. Finally, the trained deep learning backbone network model is used for fine-tuning to complete high-performance small sample classification on the target domain.

[0038] It is to be understood that the description of the background of the application included herein is not to be taken as an admission that the information provided herein at any time constitutes prior art with respect to any present application. Unless otherwise indicated, the description of the application contained herein does not constitute an BRIEF DESCRIPTION OF DRAWINGS

[0039] The above and other features, aspects, and advantages of embodiments of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like reference numerals refer to like elements, wherein:

[0040] Figure 1 is a schematic diagram of the method of the present application;

[0041] Figure 2 is a schematic diagram of the source domain remote sensing image data deep learning backbone network model pre-training of the present application;

[0042] Figure 3 is a schematic diagram of the teacher-student network model for semi-supervised cross-domain training of the present application;

[0043] Figure 4 is a schematic diagram of the classification test of the present application.

[0044] Figure 5 is a schematic diagram of the uncertainty-aware target domain hard example mining strategy of the present application. DETAILED DESCRIPTION

[0045] Embodiments of the present application will now be described in detail with reference to the drawings. Although specific embodiments of the application are illustrated in the drawings, it is understood that the application can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art. It should be understood that the drawings and the embodiments are only for illustrative purposes and are not intended to limit the scope of protection of the present application.

[0046] In the description of embodiments of the present application, the term "including" and its derivatives, shall be understood as comprising, i.e., open-ended, and do not exclude the presence of additional elements. The term "based on" shall be understood as "based, at least in part, on." The term "one embodiment" or "an embodiment" shall be understood as "at least one embodiment." The term "some embodiments" shall be understood as "at least some embodiments." Other explicit or implicit definitions can also be included below.

[0047] In the following, only certain exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are considered to be exemplary in nature and not limiting. There is no order requirement for the description of the method steps, as long as they can be implemented, all within the scope of the present application.

[0048] As Figure 1 illustrated, the present application provides a pseudo-label uncertainty-aware remote sensing image cross-domain few-shot classification method, comprising the following steps:

[0049] S1: obtaining source domain remote sensing image data and target domain remote sensing image data;

[0050] Wherein, the source domain remote sensing image data carries a first number of labels, and the target domain remote sensing image data does not carry labels; the labels indicate the category information of the remote sensing image data;

[0051] S2: pre-training a deep learning backbone network model using the source domain remote sensing image data to obtain pre-training initialization parameters of the deep learning backbone network model;

[0052] S3: constructing a teacher-student network model for semi-supervised cross-domain training;

[0053] Wherein, the teacher-student network model includes a teacher model and a student model, the teacher model includes the deep learning backbone network model and a teacher mapper, the student model includes the deep learning backbone network model and a student mapper, and the deep learning backbone network model loads the pre-training initialization parameters;

[0054] S4: inputting the source domain remote sensing image data and the target domain remote sensing image data into the teacher-student network model for training to obtain a trained deep learning backbone network model;

[0055] S5: inputting the to-be-tested remote sensing image data into the trained deep learning backbone network model to obtain a classification result of the to-be-tested remote sensing image data, the classification result indicating the category of the to-be-tested remote sensing image;

[0056] Wherein, the to-be-tested remote sensing image data carries a second number of labels, and the second number is less than the first number.

[0057] According to another embodiment of the present application, the present application provides a pseudo-label uncertainty-aware remote sensing cross-domain few-shot scene classification method, comprising:

[0058] S101: obtaining source domain remote sensing image data and target domain remote sensing image data.

[0059] The source domain remote sensing image data carries a first number of labels, and the target domain remote sensing image data does not carry labels; the labels indicate category information of the remote sensing image data, and the categories include scene semantic categories, target model categories, and the like.

[0060] There is a certain degree of domain difference between the source domain remote sensing image data and the target domain remote sensing image data, including data distribution difference and category misalignment. The data distribution difference mainly reflects the difference in imaging such as color, illumination, resolution, and the like caused by differences in remote sensing platform sensors. The category misalignment is caused by differences in themes focused by various data sets, and there are two cases of partial category misalignment and complete category misalignment. The category misalignment causes the two to have at least partially different image data, such as images of the categories of factory and river in the target domain, but no images of the categories in the source domain. The present application especially focuses on the classification prediction of images with domain difference, and effectively improves the accuracy of the classification results of these images.

[0061] S102: pre-training a deep learning backbone network model using the source domain remote sensing image data to obtain pre-training initialization parameters of the deep learning backbone network model; wherein the deep learning backbone network model includes a feature extractor and a classifier.

[0062] In some embodiments, the source domain labeled remote sensing image data is used to pre-train a deep learning backbone network model under a supervised learning paradigm to obtain pre-training initialization parameters of a deep learning feature extraction network model, wherein the deep learning feature extraction network model uses a small deep neural network ResNet-10. Supervised learning refers to model training through a large number of known image-label data pairs, and classification prediction on test image data without labels.

[0063] The pre-training of the deep learning backbone network model mainly includes the following steps:

[0064] S102.1: inputting the source domain remote sensing image data into the deep learning backbone network model to perform feature extraction on the source domain remote sensing image data to obtain a feature result

[0065] S102.2: performing classification prediction on the feature result to obtain a classification prediction probability

[0066] wherein, is the source domain remote sensing image sample, i is the source domain sample index, f s is the feature extractor, c sFor the classifier, softmax(·) is a normalized exponential function commonly used in multi-classification tasks, which is a common way in the prior art and will not be described here;

[0067] S102.3: In the supervised learning paradigm, the classification prediction probability and the label of the source domain remote sensing image data are optimized by using a cross-entropy loss function, and the deep learning backbone network model is optimized, and the cross-entropy loss function is as follows:

[0068]

[0069] wherein, is the label corresponding to the image sample, N S is the number of source domain samples, and s represents the source domain;

[0070] S102.4: The parameter optimization is performed by using the cross-entropy loss function to obtain the pre-training initialization parameters of the deep learning backbone network model.

[0071] S103: A teacher-student network model for semi-supervised cross-domain training is constructed; wherein the teacher-student network model includes a teacher model and a student model, the teacher model includes the deep learning backbone network model and a teacher mapper, the student model includes the deep learning backbone network model and a student mapper, the deep learning backbone network model loads the pre-training initialization parameters, and the teacher mapper and the student mapper are both composed of two fully connected layers.

[0072] Under the semi-supervised learning paradigm, based on the teacher-student network model framework, the teacher-student network model is constructed by simultaneously using the labeled source domain remote sensing image data and part of the unlabeled target domain remote sensing image data to realize the training of the deep learning backbone network model. The initialization parameters are loaded into the deep learning backbone network model in the teacher model and the deep learning backbone network model in the student model, the student model uses gradient descent to update the model parameters, and the teacher model uses exponential moving average to update the parameters.

[0073] The exponential moving average parameter update expression is as follows:

[0074]

[0075] wherein, θ t is the teacher model parameter, θ s is the student model parameter, a is the current iteration number, m is the momentum hyperparameter, when m=1, the teacher model is fixed; when m=0, the teacher and student model parameters are the same, and the experience value of the hyperparameter can be m=0.99, or other values that can realize the scheme, which is not limited here.

[0076] The gradient descent updates the model parameters by calculating the loss function, determining the gradient of the loss function, and updating the model parameters in the reverse direction. This method is commonly used in the prior art and will not be described here.

[0077] S104: inputting the source domain remote sensing image data into the student model for supervised learning, so that the student model synchronously learns the source domain knowledge; wherein the source domain knowledge at least includes middle-low layer general feature information of the source domain image.

[0078] The middle-low layer general feature information can be middle-low layer texture, edge and color general feature information.

[0079] For the source domain remote sensing image data carrying labels, input it into the student model, and through supervised learning, make the model synchronously learn the source domain knowledge, establish the knowledge transfer between the source domain and the target domain, and ensure the convergence of model training. The specific steps are as follows:

[0080] S104.1: performing feature extraction on the source domain remote sensing image data to obtain a feature result

[0081] S104.2: performing classification prediction on the feature result to obtain a classification prediction probability

[0082] Wherein is the source domain remote sensing image sample, i is the source domain sample index, f s is a feature extractor, c s is a classifier.

[0083] S104.3: under the supervised learning paradigm, the classification prediction probability and the label of the source domain remote sensing image data are used to optimize the student model by using the source domain supervised loss function, and the source domain supervised loss function L S The expression is as follows:

[0084]

[0085] Wherein, is a cross-entropy loss function, is the corresponding label of the source domain remote sensing image data, N S is the number of source domain remote sensing images, and s represents the source domain.

[0086] S105: performing weak enhancement processing and strong enhancement processing on the first target domain remote sensing image data to obtain weak enhancement data and strong enhancement data, respectively.

[0087] The weak enhancement only performs simple transformation on the image, such as cropping, rotating, etc., without changing the image properties; and the strong enhancement performs property transformation such as color, definition, etc. on the basis of the simple transformation.

[0088] The weak enhancement processing includes: performing random cropping processing on the first target domain remote sensing image to obtain a first cropped image; performing horizontal flipping processing on the first cropped image to obtain a first flipped image; and performing normalization processing on the first flipped image to obtain first normalized image data.

[0089] The strong enhancement processing includes: performing random cropping processing on the first target domain remote sensing image to obtain a second cropped image; performing horizontal flipping processing on the second cropped image to obtain a second flipped image; performing color dithering processing on the second flipped image to obtain a second color changed image; performing Gaussian blur processing on the second color changed image to obtain a second blurred image; performing random grayscale transformation processing on the second blurred image to obtain a second random grayscale image; and performing normalization processing on the second random grayscale image to obtain second normalized image data.

[0090] S106: The weak enhancement data is input into the teacher model, and processed by the teacher model to obtain a teacher prediction probability and a teacher mapping result.

[0091] This step can be implemented in the following way:

[0092] S106.1: The weak enhancement data is processed by feature extraction to obtain a teacher feature map wherein, is a first target domain remote sensing image data weak enhancement sample, f t is a teacher feature extractor, and j represents a target domain remote sensing image data sample index value;

[0093] S106.2: The teacher feature map is processed by classification prediction to obtain a teacher prediction probability wherein, c t is a teacher classifier, is a teacher feature map, and softmax(·) is a normalized exponential function;

[0094] S106.3: The teacher feature map is processed by feature mapping to obtain a teacher mapping result wherein, is a teacher feature map, and p t is a teacher mapper.

[0095] S107: The strong enhancement data is input into the student model, and processed by the student model to obtain a student prediction probability and a student mapping result.

[0096] This step can be implemented in the following way:

[0097] S107.1: The strong enhancement data is subjected to feature extraction to obtain a student feature map Wherein, is a target domain remote sensing image data strong enhancement sample, f s is a student feature extractor;

[0098] S107.2: The student feature map is subjected to classification prediction to obtain a student prediction probability Wherein, c s is a student classifier, is a student feature map;

[0099] S107.3: The student feature map is subjected to feature mapping to obtain a student mapping result

[0100] S108: The teacher prediction probability is subjected to sharpening processing to obtain a pseudo label p j ′.

[0101]

[0102] Wherein, is a teacher feature map, and τ is a temperature parameter used for sharpening the teacher prediction probability.

[0103] S109: The pseudo label and the student prediction probability are subjected to consistency regularization processing.

[0104] The consistency regularization processing is performed through a consistency regularization loss function, wherein the consistency regularization loss function L CR is expressed as follows:

[0105]

[0106] Wherein, L CE is a cross-entropy loss function, p′ j is a pseudo label, is a student prediction probability, N T is a first target domain remote sensing image sample quantity, T represents a target domain, and j is a target domain sample index. S1010: An uncertainty-aware target domain hard example sample mining strategy is constructed, and the uncertainty-aware target domain hard example sample mining strategy is used for hard example mining on the hard example samples.

[0107] The uncertainty-aware target domain hard example sample mining strategy includes:

[0108] S1010.1: The uncertainty Unc of the teacher prediction probability corresponding to each sample is calculated j :

[0109]

[0110] wherein, L SE is the self-entropy, is the teacher prediction probability, and j is the target domain sample index;

[0111] S1010.2: Sort the uncertainty from high to low to obtain the sorted sample index value j': j' = argsort(Unc j ), wherein argsort is an ordering function;

[0112] S1010.3: Extract the first N H =B·r samples as the hard example samples, the index value corresponding to the hard example samples is j H =j'[0:B·r], wherein N H is the number of hard example samples, H represents a hard example, B is the number of samples in a training mini-batch, and r is a selection ratio, which can be an empirical value of r=0.5 or other numerical values, again without limitation S1010.4: Calculate the hard example loss L UH of the hard example samples:

[0113]

[0114] wherein, is a pseudo-label, is the student prediction probability, and N H is the number of hard example samples, and H represents a hard example. S1011: Perform teacher external feature mapping processing on the teacher mapping result to obtain a teacher external mapping result wherein d t is a teacher external mapper, is the teacher mapping result; perform student external feature mapping on the student mapping result to obtain a student external mapping result wherein d s is a student external mapper, is the student mapping result;

[0115] S1012: Perform contrastive learning processing on the teacher mapping result and the student external mapping result, and perform contrastive learning processing on the teacher external mapping result and the student mapping result;

[0116] The contrastive learning processing is performed through a contrastive loss function, wherein the contrastive loss function expression is as follows:

[0117]

[0118] wherein, Map the result for the student externally, Map the result for the teacher, Map the result for the teacher externally, Map the result for the student, D(q,z) = -q / ||q||·z / ||z|| is a negative cosine similarity calculation formula, ||·|| represents the modulus operation, N T is the number of remote sensing image samples in the first target domain, T represents the target domain, and j is the target domain sample index. S1013: Based on the consistency regularization processing, the contrastive learning processing and the difficult example sample mining strategy, the pre-training initialization parameter is updated to obtain a trained deep learning backbone network model.

[0119] Updating the pre-training initialization parameter includes:

[0120] Updating the initialization parameter based on the total loss function, the total loss function L expression is as follows:

[0121] L = L S + lambda (L CR + L UH ) + L Con

[0122] Wherein,

[0123]

[0124]

[0125] Wherein, L S is the source domain supervised loss function, L CR is the target domain consistency regularization loss function, L UH is the difficult example loss function, L Con is the contrastive loss function, and lambda is a hyperparameter. The experience value of the hyperparameter in the above process is preferably lambda = 0.5, which can also be other values that can realize the scheme, and is not limited here.

[0126] S1014: Input the to-be-tested remote sensing image data into the first optimization model to obtain the classification result of the to-be-tested remote sensing image data, and the classification result indicates the category of the to-be-tested remote sensing image. The to-be-tested remote sensing image data carries a second number of labels, and the second number is less than the first number.

[0127] In some embodiments, the first number can be 100% of the sample size, and the second number can be 15% of the sample size. In other embodiments, the first number can also be 100% of the sample size, and the second number can be 20% of the sample size. As long as the second number is less than or much less than the first number, the present application does not limit in this aspect.

[0128] In some embodiments, the remote sensing image data to be tested can come from the target domain.

[0129] This step can be implemented in the following way:

[0130] S1014.1: divide the remote sensing image data to be tested into a support set and a query set, wherein the support set data carries a second number of labels, and the query set has no label;

[0131] S1014.2: input the support set data into the trained deep learning backbone network model, wherein the feature extractor weight parameters are frozen and not updated, and the classifier parameters are fine-tuned to obtain a trained deep learning backbone network model adapted to the remote sensing image of the target domain;

[0132] Input the query set data into the trained deep learning backbone network model to obtain the classification result of the query set data, which indicates the category of the query set data.

[0133] In summary, the remote sensing cross-domain small sample scene classification method based on pseudo-label uncertainty perception can classify small samples of remote sensing images under cross-domain conditions, and is suitable for open scenes, small samples and few labeled conditions in real remote sensing scene classification applications. Through the semi-supervised cross-domain training paradigm based on the teacher-student network model, the deep learning backbone network model can learn from the source domain and the target domain at the same time to achieve cross-domain knowledge transfer from the source domain to the target domain. Through consistency regularization processing, the deep learning backbone network model can learn feature representation from target domain remote sensing image data without label information, improving the robustness and stability of model learning. Through the uncertainty perception target domain difficult example mining strategy, the uncertainty of cross-domain learning caused by domain difference and the input disturbance caused by data augmentation can be reduced to achieve more reliable cross-domain knowledge transfer. Through contrastive learning processing, the generalization of the deep learning backbone network model can be enhanced.

[0134] Compared with the prior art, the present application has the following advantages:

[0135] 1) Compared with the prior art, the method of the present application can better classify small samples of remote sensing images under cross-domain conditions, and can achieve better classification performance under the constraints of open scenes, small samples and few labels in real remote sensing scene classification applications.

[0136] 2) Through the target domain difficult example sample mining strategy of uncertainty perception, the cross-domain remote sensing image small sample classification task with larger domain difference can be processed, including the cross-domain scene between natural, visible light and SAR image data, and the manual and time cost of remote sensing data labeling can be reduced by using the unlabeled data and small sample learning method.

[0137] 3) The method uses a small backbone network, has a fast training speed, ensures high classification performance of the cross-domain small sample, and has good practical application value.

[0138] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, used to illustrate the technical solutions of the present application, and are not limited thereto, the protection scope of the present application is not limited thereto, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: any person skilled in the art within the technical range disclosed by the present application, can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features, and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A remote sensing image cross-domain small sample classification method based on pseudo-label uncertainty perception, characterized in that, The method comprises the following steps: S101: acquiring source domain remote sensing image data and target domain remote sensing image data; wherein the source domain remote sensing image data carries a first number of labels, and the target domain remote sensing image data does not carry labels; the labels indicate category information of the remote sensing image data; S102: pre-training a deep learning backbone network model by using the source domain remote sensing image data to obtain pre-training initialization parameters of the deep learning backbone network model; S103: constructing a teacher-student network model for semi-supervised cross-domain training; wherein the teacher-student network model comprises a teacher model and a student model, the teacher model comprises the deep learning backbone network model and a teacher mapper, the student model comprises the deep learning backbone network model and a student mapper, and the deep learning backbone network model loads the pre-training initialization parameters; S104: inputting the source domain remote sensing image data and the target domain remote sensing image data into the teacher-student network model for training to obtain a trained deep learning backbone network model; inputting the source domain remote sensing image data and the target domain remote sensing image data into the teacher-student network model for training to obtain a trained deep learning backbone network model comprises: S104.1: inputting the source domain remote sensing image data into the student model for source domain supervised learning to enable the student model to learn source domain knowledge synchronously; wherein the source domain knowledge at least comprises medium-low layer general feature information of source domain remote sensing images; S104.2: performing weak enhancement processing and strong enhancement processing on the target domain remote sensing image data to obtain weak enhancement data and strong enhancement data, respectively; S104.3: inputting the weak enhancement data into the teacher model for processing to obtain teacher prediction probability and teacher mapping results; S104.4: inputting the strong enhancement data into the student model for processing to obtain student prediction probability and student mapping results; S104.5: performing sharpening processing on the teacher prediction probability to obtain pseudo labels; S104.6: performing consistency regularization processing on the pseudo labels and the student prediction probability; S104.7: constructing an uncertainty-aware target domain difficult example sample mining strategy to mine difficult example samples; wherein the difficult example samples are remote sensing image data that are difficult to classify; S104.8: performing teacher external feature mapping processing on the teacher mapping results to obtain teacher external mapping results; and performing student external feature mapping on the student mapping results to obtain student external mapping results; S104.9: performing comparative learning processing on the teacher mapping results and the student external mapping results, and performing comparative learning processing on the teacher external mapping results and the student mapping results; S104.10: updating the pre-training initialization parameters based on the source domain supervised learning, the consistency regularization processing, the comparative learning processing and the difficult example sample mining strategy to obtain a trained deep learning backbone network model; S105: input the to-be-tested remote sensing image data into the trained deep learning backbone network model to obtain a classification result of the to-be-tested remote sensing image data, the classification result indicating a category of the to-be-tested remote sensing image; Wherein, the to-be-tested remote sensing image data carries a second number of labels, and the second number is less than the first number.

2. The method of claim 1, wherein, The teacher-student network model for semi-supervised cross-domain training comprises: S103.1: load the pre-training initialization parameters into the deep learning backbone network model in the teacher model and the deep learning backbone network model in the student model; S103.2: the student model updates the student model pre-training initialization parameters using gradient descent; S103.3: the teacher model updates the teacher model pre-training initialization parameters using exponential moving average, and the exponential moving average parameter update expression is as follows: where θ t is the pre-training initialization parameter of the teacher model, θ s is the pre-training initialization parameter of the student model, m is a momentum parameter, and a is the current iteration number.

3. The method of claim 1, wherein, The input of the source domain remote sensing image data into the student model for the source domain supervised learning comprises: S104.1.1: feature extraction is performed on the source domain remote sensing image data to obtain a source domain data feature result; S104.1.2: performing a classification prediction on the source domain data feature result to obtain a source domain data classification prediction probability wherein, is the source domain remote sensing image sample, i is the source domain sample index, f s is a feature extractor, c s is a classifier; S104.1.3: optimizing the student model with a source domain supervised loss function, L S The expression is as follows: wherein, is a cross-entropy loss function, is a label corresponding to the source domain remote sensing image data, N S is the number of source domain remote sensing images.

4. The method of claim 1, wherein, The weak enhancement processing and the strong enhancement processing on the target domain remote sensing image data comprise: The weak enhancement processing comprises: S104.2.11: the first target domain remote sensing image data is subjected to random cropping processing to obtain a first cropped image; S104.2.12: the first cropped image is subjected to horizontal flip processing to obtain a first flipped image; S104.2.13: the first flipped image is subjected to normalization processing to obtain first normalized image data; The strong enhancement processing comprises: S104.2.21: the first target domain remote sensing image data is subjected to random cropping processing to obtain a second cropped image; S104.2.22: the second cropped image is subjected to horizontal flip processing to obtain a second flipped image; S104.2.23: the second flipped image is subjected to color dithering processing to obtain a second color-variation image; S104.2.24: the second color-variation image is subjected to Gaussian blur processing to obtain a second blurred image; S104.2.25: the second blurred image is subjected to random gray scale transformation processing to obtain a second random gray scale image; S104.2.26: the second random gray scale image is subjected to normalization processing to obtain second normalized image data.

5. The method of claim 1, wherein, The input of the weak enhancement data into the teacher model and the processing by the teacher model comprise: S104.3.1: the weak enhancement data is subjected to feature extraction to obtain a teacher feature map; S104.3.2: the teacher feature map is subjected to classification prediction to obtain a teacher prediction probability; S104.3.3: the teacher feature map is subjected to feature mapping to obtain a teacher mapping result.

6. The method of claim 3, wherein, The input of the strong enhancement data into the student model and the processing by the student model comprise: S104.4.1: the strong enhancement data is subjected to feature extraction to obtain a student feature map; S104.4.2: the student feature map is subjected to classification prediction to obtain a student prediction probability; S104.4.3: mapping the student feature map to obtain a student mapping result.

7. The method of claim 1, wherein, The consistency regularization processing on the pseudo label and the student prediction probability includes: The consistency regularization is performed by a consistency regularization loss function L CR The expression is as follows: wherein L CE is a cross-entropy loss function, is a pseudo label, is a student prediction probability, N T is a first target domain remote sensing image sample number, T represents a target domain, and j is a target domain sample index, is a target domain remote sensing image data strong enhancement sample, f s is a student feature extractor, c s is a student classifier.

8. The method of claim 1, wherein, The constructing of the uncertainty-aware target domain hard example mining strategy includes: S104.7.1: Calculate the uncertainty Unc of the teacher prediction probability j : wherein L SE is the self-entropy, is the teacher prediction probability, and j is the sample index of the target domain remote sensing image data. S104.7.2: Sort the uncertainty from high to low, and obtain the sorted target domain remote sensing image data sample index value j', j' = argsort(Unc j ), wherein argsort is an ordering function; S104.7.3: N before extraction H =B·r samples as hard example samples, the index value corresponding to the hard example samples is j H =j'[0:B·r], wherein N H is the number of hard example samples, H represents a hard example, B is the number of samples in a training mini-batch, and r is a selection ratio; S104.7.4: Calculate the hard example loss L of the hard example sample UH : wherein, is the pseudo label, is the student prediction probability N H is the number of hard example samples, H represents a hard example, j is a target domain sample index, j H is an index value corresponding to the hard example sample, is a target domain remote sensing image data strong augmented sample, f s is a student feature extractor, c s is a student classifier.

9. The method of claim 1, wherein, The contrastive learning processing includes: The consistency regularization processing on the pseudo label and the student prediction probability includes: The constructing of the uncertainty-aware target domain hard example mining strategy includes: The contrastive learning processing includes: The contrastive learning process proceeds by a contrastive loss function L Con is performed: wherein, is a student external mapping result, is a teacher mapping result, is a teacher external mapping result, is a student mapping result, p d is an external mapper, p t is a mapper, D(q,z) = -q / ||q|| · z / ||z|| is a negative cosine similarity calculation formula, ||·|| represents a modulo operation, N T is a first target domain remote sensing image sample number, T represents a target domain, j is a target domain sample index, is a target domain remote sensing image data strong augmented sample, f s is a student feature extractor, is a target domain remote sensing image data weak augmented sample, f t is a teacher feature extractor, j represents a target domain remote sensing image data sample index value.

Images