Domain-Adaptive Optical Remote Sensing Image Classification Method Based on Prototype Comparison Learning

Abstract

A domain-adaptive optical remote sensing image classification method based on prototype contrastive learning is proposed. This method acquires source domain image datasets and target domain image datasets, and performs two different levels of data augmentation on the target domain images. A deep learning model is built, and a feature extractor extracts features from both the source and target domains. The shallow and deep features from the source domain in the feature extractor are clustered within classes to obtain specific classification prototypes Ps and classification-independent prototypes Pa. The source domain features are clustered to obtain target domain prototypes Pt. A prototype loss is constructed, consisting of the loss between source domain features and Pt, and the loss between high-confidence samples from the target domain and Ps. The contrastive loss is calculated by perturbing low-confidence sample features from the target domain using Pa, and the network is trained accordingly. After model training, the target domain image is input into the feature extractor and then into the classifier to obtain the classification result. This invention improves the model's transfer efficiency and generalization ability, thereby enhancing the accuracy of predictions in the target domain.

Description

Technical Field

[0001] This invention belongs to the fields of transfer learning and image classification technology, and specifically relates to a domain-adaptive optical remote sensing image classification method based on prototype contrastive learning. Background Technology

[0002] Remote sensing scene classification is a crucial method in remote sensing image interpretation, aiming to categorize ground features in remote sensing images into different scene types. On one hand, with continuous advancements in sensor technology, the spatial resolution of remote sensing images is constantly improving, leading to more precise data acquisition and presenting both opportunities and challenges for remote sensing scene classification. On the other hand, the rapid development of deep learning technology has significantly advanced remote sensing scene classification, particularly the application of convolutional neural networks (CNNs), which enables more accurate extraction of high-level semantic features from remote sensing images. However, the success of deep learning models relies on time-consuming and expensive data annotation, and their generalization ability is poor when faced with new or differently distributed data. Due to differences in conditions such as illumination, reflectivity, and geographical location during data acquisition, there are discrepancies between different optical remote sensing datasets. For example, a river image acquired in region A may appear as thin stripes, while a river image acquired in region B may have large, irregular areas. Applying a deep learning model trained on the former image to classify the latter image will yield poor results. Domain adaptation was proposed to better utilize existing knowledge, namely labeled source domain images and models, to alleviate the gap between domains with different feature distributions, thereby improving the model's generalization ability. Domain adaptation methods generally used for optical remote sensing images have two main drawbacks: first, they fail to consider the inherent characteristics of optical remote sensing images; second, they neglect another major problem during data acquisition—the imbalance of class numbers. This makes these methods prone to misclassification, especially when faced with significant differences in the number of classes, where the model tends to favor the class with more samples. Summary of the Invention

[0003] To overcome the shortcomings of the prior art, the present invention aims to provide a domain-adaptive optical remote sensing image classification method based on prototype contrastive learning. This method addresses the problem in existing deep learning classification techniques where the source and target domains are identical in label space but have different feature distributions, resulting in poor generalization ability of the source domain model when transferred to the target domain. It also aims to improve classification accuracy when the number of categories in the domain is unevenly distributed.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] Firstly, a domain-adaptive optical remote sensing image classification method based on prototype contrastive learning is provided, comprising the following steps:

[0006] Step 1. Obtain the source domain image dataset and target domain image dataset Two different levels of data augmentation were performed on the target domain image; Represents the i-th source domain image. express The tag, n s Indicates the number of source domain images. Let n represent the i-th target domain image. t Indicates the number of images in the target domain;

[0007] Step 2. Build a deep learning model, which includes a feature extractor G and a classifier C. Use the feature extractor G to extract features from the source domain. Extracting features from the target domain

[0008] Step 3. Perform intra-class clustering on the shallow features of the source domain in the feature extractor G to obtain the classification-independent prototype Pa, and perform intra-class clustering on the deep features of the source domain in the feature extractor G to obtain the specific classification prototype Ps;

[0009] Step 4. For Clustering is performed using a bidirectional weighted prototype strategy to obtain the target domain prototype Pt;

[0010] Step 5. Construction The loss between Pt and the loss between high-confidence samples in the target domain and Ps constitutes the prototype loss L. proto ;

[0011] Step 6. Introduce a memory in the target domain to store low-confidence samples. Use Pa to perturb the features of low-confidence samples in the target domain, construct anchor points, positive samples, and negative samples, and use them to calculate the contrastive loss L. cont ;

[0012] Step 7. Apply consistency loss L to the image features of the target domain with two different levels of data augmentation. cons The total loss for the target domain is obtained by weighted summation of the loss described in the above steps, and cross-entropy classification loss is applied to the source domain to train the network.

[0013] Step 8. After the model training is completed, input the target domain image into the feature extractor G and then into the classifier C to obtain the classification result.

[0014] In a second aspect, a computer-readable storage medium is provided for storing a computer program that causes a computer to perform the methods as described in the first aspect or in various embodiments thereof.

[0015] Thirdly, a computer program product is provided, including computer program instructions that cause a computer to perform the methods as described in the first aspect or in various embodiments thereof.

[0016] Compared with the prior art, the beneficial effects of the present invention are:

[0017] As described above, this invention discloses a domain-adaptive optical remote sensing image classification method based on prototype contrastive learning. This method leverages the characteristic that the transferability of features in a feature extraction network varies with the convolutional layers, developing classification-independent prototypes and classification-specific prototypes with different purposes in the source domain. The former is used to perturb low-confidence features in the target domain, constructing positive and negative samples to complete contrastive learning, which helps generate larger boundaries between low-confidence sample features, making them less prone to confusion; the latter is used for prototype learning, bringing the mapping between source domain features and target domain features closer in high-dimensional space. In the target domain, this method designs a bidirectional weighted prototype strategy to generate diverse and robust prototypes for target domain image features, assisting in pseudo-label generation and reducing inter-domain differences. Furthermore, the consistency loss incorporated in this invention can further enhance the model's robustness to distribution variations. Attached Figure Description

[0018] Figure 1 This is a flowchart of a domain adaptive optics remote sensing image classification method based on prototype contrastive learning in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the structure of a deep learning model in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram illustrating the construction of anchor points and positive and negative samples in the contrastive learning method of this invention. Detailed Implementation

[0021] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0022] This invention presents a domain-adaptive optical remote sensing image classification method based on prototype contrastive learning. The prototype is an abstraction of representative samples for each category, which can be implemented using techniques such as averaging, clustering, and kernel methods. New, unseen samples are assigned to the category represented by the most similar prototype by comparing their similarity or distance with learned prototypes. Domain adaptation addresses the issue of supervised models performing poorly on downstream tasks under new feature distributions, and is therefore proposed to align different feature distributions or extract domain-invariant features. In domain adaptation, the domain refers to datasets with different feature distributions but the same task. The source domain contains images and labels, while the target domain contains only images. Domain adaptation aims to utilize this information and deep learning methods to better perform downstream tasks such as classification on the target domain. Therefore, as... Figure 1 As shown, the classification method of the present invention mainly includes the following steps:

[0023] Step 1. Obtain the source domain image dataset and target domain image dataset Two different levels of data augmentation were performed on the target domain image. Represents the i-th source domain image. express The tag, n s Indicates the number of source domain images. Let n represent the i-th target domain image. t This indicates the number of images in the target domain.

[0024] In this invention, a domain refers to a dataset with a specific feature distribution. The source domain image refers to a domain with existing knowledge, meaning that a deep learning model can be trained based on its image and label information for classification. The target domain only contains images without labels and needs to be labeled by the model. However, directly applying the model trained on the source domain to classify the target domain is ineffective because even if the categories are the same, the feature distributions of the two are different. For example, a model trained on European face data will perform differently when used for Asian face recognition because, although they are both faces, the features of European and Asian faces are different.

[0025] In an embodiment of the present invention, this step involves inputting the source domain image and the target domain image into the network model in parallel batches. Among the two different levels of data augmentation, the weak augmentation T... w This can generally be achieved through random cropping and random horizontal flipping, strongly enhancing T. s Implemented using RandAugment technology, it refers to randomly applying two of the enhancement methods to an image, including translation, flipping, brightness adjustment, contrast adjustment, and Gaussian blur.

[0026] Step 2. Build a deep learning model, refer to... Figure 2The model includes a feature extractor G and a classifier C, where the feature extractor G is used to extract features from the source domain. Extracting features from the target domain

[0027] In an embodiment of the present invention, the feature extractor G is implemented by the ResNet50 feature extraction part and the dimensionality reduction layer, and the classifier C is composed of fully connected layers, with the source domain and the target domain sharing model parameters.

[0028] This invention pre-trains the feature extractor G and then uses it for feature extraction. Here, pre-training refers to training the source domain image using label smoothing and cross-entropy loss. The cross-entropy loss is defined as:

[0029]

[0030] In the above formula, δ k (·) represents the k-th element output by softmax, q k The value is 1 if the category is correct, and 0 otherwise. K is the number of categories, and x is the value of x. s Represents the source domain image, y s Indicates the source domain label, X s Y represents the total source domain image. s G(x) represents the overall source domain label. s (referring to x) s The features extracted after the feature extractor, C(G(x) s )) refers to G(x s The softmax output after passing through the classifier.

[0031] For samples in the source domain Samples in the target domain Features are extracted using the pre-trained feature extractor G.

[0032] Step 3. Perform intra-class clustering on the shallow features of the source domain in the feature extractor G to obtain the classification-independent prototype Pa, and perform intra-class clustering on the deep features of the source domain in the feature extractor G to obtain the specific classification prototype Ps.

[0033] The shallow features of feature extractor G refer to the features extracted in the first stage of the ResNet50 backbone network. In other words, deep features refer to the features extracted after the dimensionality reduction layer, that is, the features extracted by the entire feature extractor G. All dimensions are 256. Intra-class clustering is implemented using the k-means clustering algorithm, performing unsupervised clustering within each class in the source domain. Each class contains one class-independent prototype Pa, and each class-specific prototype Ps contains three class-specific prototypes Ps. That is, in the source domain, one class-independent prototype Pa and three class-specific prototypes Ps are generated for each class. The formula for obtaining the source domain class-independent prototype Pa and class-specific prototypes Ps is formalized as follows:

[0034]

[0035] In the above formula, Pa i Represents the classification-independent prototype of the i-th class in the source domain;

[0036]

[0037] In the above formula This represents the prototype of the j-th specific category of the i-th class in the source domain;

[0038] Step 4. Extract features from the target domain using the feature extractor. Clustering is performed to obtain the target domain prototype Pt.

[0039] In an embodiment of the present invention, this step is... Clustering is performed using a bidirectional weighted prototype clustering strategy to obtain the target domain prototype Pt. The steps of the bidirectional weighted prototype clustering strategy are as follows:

[0040] Step 4.1: Collect reliable sample features for prototype clustering (i.e., the aforementioned intra-class clustering). Based on the softmax result output by the model from the target domain image, set a threshold σ, and collect sample features whose argmax classification probability is greater than the threshold σ as a set S. h ;

[0041]

[0042] In the formula, h t () represents C(G()), which is the softmax output obtained after the target domain image passes through the feature extractor and classifier.

[0043] Step 4.2: Collect reliable sample features for prototype clustering. For the target domain image, using the softmax result output by the model, set the number N, and collect the top N sample features with the highest confidence in each class from the softmax output for that class as a set S. v ;

[0044]

[0045] Step 4.3, set S h With set Sv Taking the union of sets yields S. bi ;

[0046] S bi =S h ∪S v (6)

[0047] At the same time, the number of features in the union of samples is limited, when S bi The number exceeds S v When the sample size is α times the sample size, S is truncated. bi The softmax output of the set ranks α times the confidence level of the class. v The number of sample features as a set S′ bi ;

[0048]

[0049] Step 4.4, set S′ bi We perform weighted k-means clustering on the sample features to generate m prototypes for each category as the target domain prototype Pt. The formula for obtaining the prototypes is formalized as follows:

[0050]

[0051] In the above formula, This represents the j-th prototype of the i-th class in the target domain;

[0052] Step 4.5: Calculate the Euclidean squared distance between the target domain sample and all target domain prototypes, and assign pseudo labels to the target domain sample.

[0053] Step 5, Construct source domain features The prototype loss L consists of the loss between the target domain prototype Pt and the loss between high-confidence samples in the target domain and specific classification prototypes Ps in the source domain. proto The high-confidence and low-confidence sample features in the target domain are determined by a threshold σ. Only the loss between the high-confidence sample features in the target domain and the specific classification prototype in the source domain, as well as the loss between the source domain features and the target domain prototype, are calculated. The prototype loss L proto The formula for calculating the squared Euclidean distance is as follows:

[0054]

[0055] In the above formula, ||·||² represents the squared Euclidean distance, σ is the threshold for evaluating the classification confidence of the target domain samples, which is generally taken as 0.9, c is the number of classes, and m is the number of prototypes for a specific class, which is set to 3 as mentioned above. δ represents the j-th specific classification prototype of the i-th class in the source domain. i This refers to the softmax output value in the i-th class.

[0056] Step 6. Introduce a memory in the target domain to store low-confidence samples. Use Pa to perturb the features of low-confidence samples in the target domain, construct anchor points, positive samples, and negative samples, and use them to calculate the contrastive loss L. cont .like Figure 3 The specific steps will be as follows:

[0057] Step 6.1: In the target domain, a memory is introduced to store the quantitative low-confidence sample features r. The classification-independent prototype Pa of any class is weighted with the features of a certain low-confidence sample and used as an anchor point.

[0058]

[0059] In the above formula, μ is a uniform distribution ranging from 0.9 to 1, ensuring that the features of low-confidence samples in the target domain dominate; the classification-irrelevant prototypes Pa of other classes are weighted together with the features of the low-confidence samples to form the positive sample r. + ;

[0060]

[0061] Calculate the similarity between the features of the low-confidence sample and other low-confidence samples, and take the M least similar sample features as negative samples r. - ;

[0062]

[0063] In the above formula, d(·) represents the Euclidean distance, Pa i Pa represents the classification-independent prototype of the i-th class in the source domain. j Let i represent the class-independent prototype of class j in the source domain, and i ≠ j, meaning j is any class other than class i.

[0064] Step 6.2: Calculate the contrast loss L using the positive and negative samples obtained in Step 6.1. cons The formula is formalized as follows:

[0065]

[0066] In the above formula, C - This indicates that the number of categories is reduced by 1, which means the selected category is removed because a certain selected category has been excluded. h() represents cosine similarity.

[0067] Step 7: Apply consistency loss L to the image features of the target domain with two different levels of data augmentation. consThe total loss for the target domain is obtained by weighted summation of the losses described in the previous steps, and cross-entropy classification loss is applied to the source domain to train the network. This cross-entropy classification loss is consistent with the loss used during pre-training of the source domain, and the cross-entropy loss has been provided in step 2. The consistency loss is applied to the weakly enhanced and strongly enhanced versions of high-confidence samples in the target domain. The pseudo-labels output by the feature classification of the weakly enhanced image are used as the labels of the strongly enhanced image, and the consistency loss L is calculated using the cross-entropy loss. cons The formula is as follows:

[0068]

[0069] In the above formula, σ represents the threshold. h w with h s These are the softmax outputs of weakly and strongly enhanced features, respectively, X. t x is the total set of images in the target domain. i For the target domain image.

[0070] Step 8. After the model training is completed, input the target domain image into the feature extractor G and then into the classifier C to obtain the classification result.

[0071] To verify the effectiveness of the present invention, six types of image data common to four optical remote sensing classification datasets, namely UCM, WHU-RS, AID, and RSSCN7, were selected as different domains, including residential, farmland, forest, industrial, parking lot, and river. Domain adaptation was performed between each pair of domains.

[0072] Table 1: Accuracy of Experimental Classification (Unit: %)

[0073] Method A→R A→U A→W R→A R→U R→W U→A U→R U→W W→A W→R W→U Avg ResNet 76.88 74.86 98.42 85.18 80.71 86.71 66.44 56.50 71.20 93.11 71.25 75.14 78.03 DDC 75.63 78.29 99.21 84.68 79.71 89.24 66.26 57.71 71.84 90.09 70.13 78.00 78.40 DAN 76.88 80.29 99.37 85.32 82.86 88.92 67.07 57.50 74.68 92.79 72.79 80.14 79.88 JAN 76.83 80.71 99.05 85.50 83.86 90.19 67.48 55.83 72.47 92.30 72.75 80.86 79.82 RevGrad 78.13 82.86 98.10 84.28 81.57 88.61 69.19 61.89 76.58 92.29 73.58 81.86 80.75 MRAN 78.96 84.71 99.73 93.87 83.06 93.35 82.99 64.78 78.52 94.06 79.16 81.71 84.47 AMRAN 81.00 88.16 99.68 94.48 87.56 94.59 84.80 69.02 89.56 94.82 79.50 85.20 87.36 Ours 81.79 92.14 99.37 98.38 91.86 98.42 98.51 80.38 97.78 96.49 82.21 89.29 92.36

[0074] This invention represents a significant improvement over existing algorithms, outperforming ResNet, DDC, DAN, JAN, RevGrad, MRAN, and AMRAN by 14.33%, 13.96%, 12.48%, 12.54%, 11.61%, 7.89%, and 5.00%, respectively. Except for a slight gap compared to AMRAN in the A→W transfer task, this invention achieves significantly better results than AMRAN in all other transfer tasks, particularly in the U→A and U→R transfer tasks, where it outperforms AMRAN by more than ten percentage points, demonstrating the effectiveness of this invention.

[0075] In summary, this invention provides a domain-adaptive optical remote sensing image classification method based on prototype contrastive learning. The core of this method is that it can improve transfer efficiency and model prediction accuracy, and has high promotional value.

Claims

1. A domain adaptation optical remote sensing image classification method based on prototype contrast learning, characterized in that, Includes the following steps: Step 1. Obtain the source domain image dataset and target domain image dataset Two different levels of data augmentation were performed on the target domain image; Indicates the first One source domain image, express The tag, Indicates the number of source domain images. Indicates the first A target domain image, Indicates the number of images in the target domain; Step 2. Build a deep learning model, which includes a feature extractor G and a classifier C. Use the feature extractor G to extract features from the source domain. Extracting features from the target domain ; Step 3. Perform intra-class clustering on the shallow features of the source domain in the feature extractor G to obtain classification-independent prototypes. Perform intra-class clustering on the deep features of the source domain in the feature extractor G to obtain a specific classification prototype. ; Step 4. For Clustering is performed using a two-way weighted prototype strategy to obtain the target domain prototype. The method is as follows: Step 4.1: Collect reliable sample features for prototype clustering, and set a threshold based on the softmax result output by the model from the target domain image. Collect argmax classification probabilities greater than the threshold The sample features as a set ; In the formula, express ; Step 4.2: Collect reliable sample features for prototype clustering. For the target domain image, set the quantity based on the softmax result output by the model. N Collect the softmax output of each category and rank the confidence scores for that category. N The sample features as a set ; Step 4.3, set With sets Taking the union yields At the same time, the number of features in the union of samples is limited. Quantity exceeds Sample size When doubled, truncate The softmax output of the set ranks the confidence scores for that class. times Number of sample features as a set ; Step 4.4, set We perform weighted k-means clustering on the sample features to generate m prototypes for each category as target domain prototypes. ; Step 5. Construction and The loss between and the target domain high-confidence samples The loss between them constitutes the prototype loss. ; Step 6. Introduce a memory in the target domain to store low-confidence samples, and use... By interfering with low-confidence sample features in the target domain, anchor points, positive samples, and negative samples are constructed to calculate the contrastive loss. ; Step 7. Apply consistency loss to image features of the target domain with two different levels of data augmentation. The total loss for the target domain is obtained by weighted summation of the loss described in the above steps, and cross-entropy classification loss is applied to the source domain to train the network. Step 8. After the model training is complete, input the target domain image into the feature extractor G and then into the classifier C to obtain the classification result.

2. The domain adaptive optical remote sensing image classification method based on prototype contrastive learning according to claim 1, characterized in that, In step 1, the source domain image and the target domain image are input into the network model in batches in parallel. Two different levels of data augmentation are involved: weak augmentation and... Achieved through random cropping and random horizontal flipping, significantly enhanced. Implemented using RandAugment technology.

3. The domain adaptive optical remote sensing image classification method based on prototype contrastive learning according to claim 1, characterized in that, In step 2, the feature extractor G consists of a ResNet50 and a dimensionality reduction layer, and the classifier C consists of a fully connected layer. The feature extractor G is pre-trained and then used for feature extraction. The pre-training refers to training the source domain image using label smoothing and cross-entropy loss. The cross-entropy loss is defined as: In the formula, This represents the output of softmax. k One element, It is 1 if the category is correct, and 0 otherwise. K It is the number of categories. Represents the source domain image. Indicates the source domain tag. Represents the overall source domain image. This represents the overall source domain tag. refer to Features extracted after passing through the feature extractor refer to The softmax output after passing through the classifier.

4. The domain adaptive optical remote sensing image classification method based on prototype contrastive learning according to claim 1, characterized in that, In step 3, the shallow features and deep features refer to the features extracted from the first stage of the ResNet50 backbone network and the features extracted after the dimensionality reduction layer, respectively, both with a dimension of 256. Intra-class clustering is implemented using the k-means clustering algorithm, that is, in the source domain, one classification-independent prototype is generated for each category. and 3 specific class prototypes .

5. The domain adaptive optics remote sensing image classification method based on prototype contrastive learning according to claim 1, characterized in that, In step 5, the sample features of high-confidence and low-confidence target domains are determined by a threshold. Determine the prototype loss. The formula for calculating the squared Euclidean distance is as follows: in, For the number of categories, For a specific category of prototypes, Indicates the target domain. i The class of j A prototype, Represents the source domain. i The class of j A specific category prototype, This refers to the softmax output value in the i-th class.

6. The domain adaptive optical remote sensing image classification method based on prototype contrastive learning according to claim 1, characterized in that, Step 6 includes: Step 6.1: Introduce a memory in the target domain to store quantitative features of low-confidence samples in the target domain. Take the classification-independent prototype of any class. Weighted by features of a low-confidence sample, used as an anchor point ;Remove the class-independent prototype of other classes except this one. The sample with low confidence level features is weighted and used as a positive sample. Calculate the similarity between the features of the low-confidence sample and other low-confidence samples, and take the M least similar sample features as negative samples. ; In the above formula, The data is taken from a uniform distribution between 0.9 and 1, ensuring that features from low-confidence samples dominate in the target domain. Represents Euclidean distance; Represents the source domain. i Class classification is unrelated to prototype. Represents the source domain. j The classification of classes is independent of their prototypes, and i ≠ j ,Right now j Except for the first i Other classes of the class; Step 6.2: Calculate the contrast loss using the positive and negative samples obtained in Step 6.

1. ; In the above formula, Decrease the number of categories by one, which means removing the selected category. Represents cosine similarity.

7. The domain adaptive optical remote sensing image classification method based on prototype contrastive learning according to claim 1, characterized in that, In step 7, the consistency loss is applied to the weakly enhanced and strongly enhanced versions of the high-confidence samples in the target domain. The pseudo-labels output by the feature classification of the weakly enhanced image are used as the labels of the strongly enhanced image, and the consistency loss is calculated using cross-entropy loss. The formula is as follows: In the above formula, Indicates the threshold. , and These are the softmax outputs for weak and strong enhancement features, respectively. The total set of images in the target domain. For the target domain image.

8. A computer-readable storage medium, characterized in that, Used to store computer programs, the computer programs causing the computer to perform as claimed in claim 1 The method described in any one of the 7.

9. A computer program product containing instructions, characterized in that, When the instructions are executed on a computer, the computer causes the computer to perform claim 1. The method described in any of 7.

Images