Adaptive clustering remote sensing scene classification method based on cross-domain cross enhancement

Abstract

The invention relates to a self-adaptive clustering remote sensing scene classification method based on cross-domain cross enhancement, and belongs to the field of remote sensing cross-domain scene classification. The method comprises the following steps: firstly, providing a cross-domain cross enhancement algorithm, performing cross enhancement on labeled and unlabeled data, combining data information of a source domain and a target domain, and bridging the distribution difference of the two domains; then, high-confidence pseudo-label propagation is promoted through consistency regularization, and feature alignment between a source domain and a target domain is achieved; finally, a self-adaptive clustering method is introduced for the target domain data with the pseudo labels, the target domain data are clustered and grouped, feature alignment of the clustering level is achieved, and the quality of the pseudo labels is further improved. The method effectively solves the problem that cross-domain alignment is difficult and knowledge migration is blocked due to significant distribution difference between a remote sensing image source domain and a target domain, improves remote sensing cross-domain scene classification performance, and has good practical application value.

Description

Technical Field

[0001] The present invention relates to the technical field of optical remote sensing image scene classification, and in particular to an adaptive clustering remote sensing scene classification method based on cross-domain cross enhancement. Background Art

[0002] Remote sensing image scene classification aims to assign semantic labels to images based on their content, enabling scene information analysis of large-scale remote sensing data, thereby supporting the intelligent application of remote sensing data and playing an important role in land surveys, urban planning, agricultural assessments, and forest monitoring.

[0003] However, in practical applications, the distribution of image data generated by different satellites varies significantly due to factors such as sensor type, geographic location, lighting conditions, and seasonal variations. For example, sensor differences can affect image resolution and color saturation, resulting in different image styles; changes in lighting conditions can also lead to differences in image brightness. Deep learning-based scene classification models trained on one dataset often experience significant performance degradation when applied to another dataset with a significantly different distribution. Therefore, models trained on source domain data are difficult to directly transfer to unlabeled target domain data with domain differences. Although manual labeling of large amounts of data can improve the model's classification capabilities for target domain data, large-scale, high-quality manual labeling of rapidly growing remote sensing observation data not only consumes significant manpower and resources but can also introduce human errors, thereby affecting model performance.

[0004] Therefore, enhancing the generalization performance of the model for unlabeled target domain data with significant distribution differences is a key challenge in improving the intelligent interpretation capabilities of remote sensing data. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings and defects of the existing technology, and to solve the problem that the distribution difference between the source domain and the target domain in the optical remote sensing image scene classification is too large, resulting in poor classification performance of the model in the target domain. An adaptive clustering remote sensing scene classification method based on cross-domain cross enhancement is creatively proposed.

[0006] The method of the present invention is achieved through the following technical solutions:

[0007] A remote sensing scene classification method based on adaptive clustering and cross-domain enhancement, including:

[0008] Step 1: Build a domain adaptation network, which includes a feature extractor and a classifier module;

[0009] Step 2: Perform cross-domain cross-enhancement on the input data to create enhanced data from labeled source domain data and target domain data to unlabeled target domain data;

[0010] Step 3: Using the feature extractor to extract features from the target domain data after cross-enhancement, using the classifier to obtain pseudo labels for the target domain data, and promoting the propagation of high-confidence pseudo labels through consistency regularization;

[0011] Step 4: Design an adaptive clustering loss to group the features of the target domain data into clusters, further improve the pseudo-label quality during the propagation process, and align cluster features between the source and target domains;

[0012] Step 5: Construct an overall objective loss function based on the data, and optimize the overall objective loss function to iteratively train the domain adaptation network.

[0013] In step 2, the method for performing cross-domain cross-enhancement on the input data is:

[0014] Using the CutMix operation, weakly enhanced data is generated by replacing unlabeled target domain data regions with patches of labeled target domain data; strongly enhanced data is generated by replacing unlabeled target domain data regions with patches of labeled source domain data. In this way, enhanced data that combines information from both the source and target domains is obtained.

[0015] In step 3, the method for promoting the propagation of high-confidence pseudo-labels through consistency regularization is:

[0016] Based on the classic semi-supervised FixMatch framework, this approach combines weak and strong augmentation strategies within a consistent regularization process. First, a weakly augmented image of each image is obtained from the unlabeled target domain image, and the model's predictions are used as pseudo-labels for supervised training. Then, predictions are obtained from the strongly augmented version of the same image to increase the model's input diversity. Finally, to ensure consistent regularization during training, a standard cross-entropy loss is enforced on the output of the strongly augmented image model.

[0017] Based on the calculation of the cross-augmented pseudo-label loss L PL .

[0018] Further, wherein the cross-enhancement pseudo-label loss L is calculated based on the PL include:

[0019]

[0020] Where M is the number of target samples per input batch of unlabeled target domain images. j ′=p(x′ j )=σ(F(G(x′ j))) is the training result after the weakly enhanced target domain image passes through the model, where G represents the feature extractor in the domain adaptation network, F represents the classifier in the domain adaptation network, and the prediction result is converted into a single point hard label Then used as pseudo label. j ′)=σ(F(G(x′ j ′))) represents the model's strong enhancement of the target domain image x′ j ′’s predictions, and τ is a scalar confidence threshold that determines the subset of pseudo labels that should be retained during model training.

[0021] In step 4, a method for designing an adaptive clustering loss is:

[0022] Based on the assumption that samples in the same cluster have similar features and belong to the same category label, the features extracted from the target domain image are clustered to achieve alignment of the source domain image features with the target domain image features at the cluster level.

[0023] Furthermore, the method for clustering the features extracted from the target domain image is:

[0024] After the cross-enhanced target domain image passes through the feature extractor, the feature vectors are sorted from large to small, and the sorted results are used as indicators to evaluate the similarity of paired features. If the first k indices in the sorted lists of the feature vectors of two samples are the same, then the two paired samples belong to the same category, and the pairwise similarity label s is set. ij =1; otherwise s ij = 0. Based on the method, pairwise comparisons between target domain images are calculated to obtain the adaptive clustering loss L AC .

[0025] Furthermore, the adaptive clustering loss L is obtained AC include:

[0026] The binary cross entropy loss is used to establish pairwise comparisons between target domain images, which is expressed as follows:

[0027]

[0028] Where M is the number of target samples per input batch of unlabeled target domain images, Represents the unlabeled target domain images in the input batch The prediction results after passing the model. j ′=p(x′ j )=σ(F(G(x′ j ))) represents the weakly enhanced target domain image x′ j The prediction result. i T Pj ′ can be regarded as a similarity score and used to predict and x′ j Whether they have the same category label.

[0029] In step 5, constructing the overall objective loss function includes:

[0030] Construct the cross-domain cross entropy loss, and the expression is shown in the following formula (4):

[0031]

[0032] Among them, S is the source domain, L is the labeled target domain, x represents the image sample, and y represents the label predicted by the model.

[0033] Using the cross-augmented pseudo-label loss L PL , the adaptive clustering loss L AC and the cross-domain cross entropy loss L CE The overall objective loss function is established, and the expressions are shown in the following equations (5) and (6):

[0034]

[0035] Here, G represents the feature extractor of the constructed domain adaptation network, and F represents the classifier of the constructed domain adaptation network. λ is a scalar hyperparameter that controls the balance between the cross-domain cross entropy loss and the proposed cross-domain adaptive clustering loss. Gradient reversal is applied between the feature extractor and the classifier.

[0036] Furthermore, the application of gradient reversal between the feature extractor and the classifier includes:

[0037] Directly minimize the adaptive clustering loss L AC This approach encourages feature clustering within the target domain. However, due to the dominance of a large number of source labels during supervised training, the feature representations learned in the target domain tend to be biased towards the source domain. Directly minimizing the loss on unlabeled target domain data exacerbates this bias and increases the risk of overfitting. To address this issue, a gradient reversal layer is employed to reverse the gradients between the feature extractor and the classifier, ensuring that the classifier performs accurate classification in the target domain.

[0038] Beneficial effects

[0039] (1) Efficient feature alignment in cross-domain scenarios. This paper uses a cross-domain cross-enhancement method to effectively combine data information from the source and target domains, bridging the distribution differences between the two domains. Compared with traditional methods that ignore the distribution differences between different domains, this method significantly improves the accuracy of feature alignment in the target domain and enhances the generalization performance of the classification model in cross-domain scenarios.

[0040] (2) Significant optimization of pseudo-label quality. Through a consistency regularization strategy and an adaptive clustering algorithm, the present invention can improve the confidence and accuracy of pseudo-labels. In particular, when the target domain data is unlabeled, the designed adaptive clustering loss further ensures the quality of cluster feature alignment and pseudo-label propagation, overcoming the limitation of low pseudo-label quality on classification performance in the existing technology.

[0041] (3) Comprehensive improvement of classification performance. This paper comprehensively adopts cross-augmentation pseudo-label loss, cross-domain cross entropy loss, and adaptive clustering loss to construct an efficient target loss function, achieving effective optimization of knowledge transfer between the source domain and the target domain. Compared with existing remote sensing scene classification technologies, the accuracy of target domain classification is significantly improved, and the problem of model performance degradation caused by distribution differences is reduced, showing excellent application value in the classification of diverse remote sensing image scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the overall flow chart of the method of the present invention;

[0043] Figure 2 A schematic diagram of a data processing overview of the present invention;

[0044] Figure 3 Schematic diagram of the model architecture and training process of the present invention;

[0045] Figure 4 Partial comparison of the data set for the embodiment;

[0046] Figure 5 Schematic diagram comparing the target domain feature distribution before and after the adaptive clustering algorithm proposed in this method. DETAILED DESCRIPTION

[0047] The following describes embodiments of the present invention in more detail with reference to the accompanying drawings. It should be noted that the present invention can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0048] like Figure 1-Figure 5 As shown in FIG, an adaptive clustering remote sensing scene classification method based on cross-domain cross enhancement includes the following steps:

[0049] Step 1: Build a domain adaptation network, which includes a feature extractor and a classifier module;

[0050] Step 2: Perform cross-domain cross-enhancement on the input data to create enhanced data from labeled source domain data and target domain data to unlabeled target domain data;

[0051] Among them, the method for cross-domain cross-enhancement of input data is:

[0052] Using the CutMix operation, weakly enhanced data is generated by replacing unlabeled target domain data regions with patches of labeled target domain data; strongly enhanced data is generated by replacing unlabeled target domain data regions with patches of labeled source domain data. In this way, enhanced data that combines information from both the source and target domains is obtained.

[0053] Step 3: Using the feature extractor to extract features from the target domain data after cross-enhancement, using the classifier to obtain pseudo labels for the target domain data, and promoting the propagation of high-confidence pseudo labels through consistency regularization;

[0054] Among them, the method to promote the propagation of high-confidence pseudo-labels through consistency regularization is:

[0055] Based on the classic semi-supervised FixMatch framework, this approach combines weak and strong augmentation strategies within a consistent regularization process. First, a weakly augmented image of each image is obtained from the unlabeled target domain image, and the model's predictions are used as pseudo-labels for supervised training. Then, predictions are obtained from the strongly augmented version of the same image to increase the model's input diversity. Finally, to ensure consistent regularization during training, a standard cross-entropy loss is enforced on the output of the strongly augmented image model.

[0056] Based on the calculation of the cross-augmented pseudo-label loss L PL .

[0057] Further, wherein the cross-enhancement pseudo-label loss L is calculated based on the PL include:

[0058]

[0059] Where M is the number of target samples per input batch of unlabeled target domain images. j ′=p(x′ j )=σ(F(G(x′ j ))) is the training result after the weakly enhanced target domain image passes through the model, where G represents the feature extractor in the domain adaptation network, F represents the classifier in the domain adaptation network, and the prediction result is converted into a single point hard label Then used as pseudo label. j ′)=σ(F(G(x′ j ′))) represents the model's strong enhancement of the target domain image x′ j′’s predictions, and τ is a scalar confidence threshold that determines the subset of pseudo labels that should be retained during model training.

[0060] Step 4: Design an adaptive clustering loss to group the features of the target domain data into clusters, further improve the pseudo-label quality during the propagation process, and align cluster features between the source and target domains;

[0061] Among them, the method of designing an adaptive clustering loss is:

[0062] Based on the assumption that samples in the same cluster have similar features and belong to the same category label, the features extracted from the target domain image are clustered to achieve alignment of the source domain image features with the target domain image features at the cluster level.

[0063] Furthermore, the method for clustering the features extracted from the target domain image is:

[0064] After the cross-enhanced target domain image passes through the feature extractor, the feature vectors are sorted from large to small, and the sorted results are used as indicators to evaluate the similarity of paired features. If the first k indices in the sorted lists of the feature vectors of two samples are the same, then the two paired samples belong to the same category, and the pairwise similarity label s is set. ij =1; otherwise s ij = 0. Based on the method, pairwise comparisons between target domain images are calculated to obtain the adaptive clustering loss L AC .

[0065] Furthermore, the adaptive clustering loss L is obtained AC include:

[0066] The binary cross entropy loss is used to establish pairwise comparisons between target domain images, which is expressed as follows:

[0067]

[0068] Where M is the number of target samples per input batch of unlabeled target domain images, Represents the unlabeled target domain images in the input batch The prediction results after passing the model. j ′=p(x′ j )=σ(F(G(x′ j ))) represents the weakly enhanced target domain image x′ j The prediction result. i T P j ′ can be regarded as a similarity score and used to predict and x′ j Whether they have the same category label.

[0069] Step 5: Construct an overall objective loss function based on the data, and optimize the overall objective loss function to iteratively train the domain adaptation network.

[0070] Among them, constructing the overall target loss function includes:

[0071] Construct the cross-domain cross entropy loss, and the expression is shown in the following formula (4):

[0072]

[0073] Among them, S is the source domain, L is the labeled target domain, x represents the image sample, and y represents the label predicted by the model.

[0074] Using the cross-augmented pseudo-label loss L PL , the adaptive clustering loss L AC and the cross-domain cross entropy loss L CE The overall objective loss function is established, and the expressions are shown in the following equations (5) and (6):

[0075]

[0076]

[0077] Here, G represents the feature extractor of the constructed domain adaptation network, and F represents the classifier of the constructed domain adaptation network. λ is a scalar hyperparameter that controls the balance between the cross-domain cross entropy loss and the proposed cross-domain adaptive clustering loss. Gradient reversal is applied between the feature extractor and the classifier.

[0078] Furthermore, the application of gradient reversal between the feature extractor and the classifier includes:

[0079] Directly minimize the adaptive clustering loss L AC This approach encourages feature clustering within the target domain. However, due to the dominance of a large number of source labels during supervised training, the feature representations learned in the target domain tend to be biased towards the source domain. Directly minimizing the loss on unlabeled target domain data exacerbates this bias and increases the risk of overfitting. To address this issue, a gradient reversal layer is employed to reverse the gradients between the feature extractor and the classifier, ensuring that the classifier performs accurate classification in the target domain.

[0080] The following experimental verification is carried out.

[0081] We first choose to build a remote sensing cross-domain scene classification task based on two different open source remote sensing datasets, namely the NWPU-RESISC45 dataset and the AID dataset. Figure 4As shown, there is a significant distribution difference between the two datasets. We visually extract the shared categories between the two datasets and create two cross-scene tasks, called NWPU→AID and AID→NWPU. Table 1 lists the number of labeled source domain images and unlabeled target domain images and the number of shared categories in the two cross-domain scene classification tasks.

[0082] Table 1 Cross-domain scene classification task data table

[0083] Cross-domain scene classification task Number of source domain images Number of target domain images Total number of categories NWPU→AID 16100 7740 23 AID→NWPU 7740 16100 23

[0084] The method proposed in the present invention is implemented based on the PyTorch library of the Python language. There are other ways to implement this solution, which are not limited here. In addition, we also used other domain adaptation methods to conduct experimental evaluations on the constructed cross-domain scene classification task, including domain adaptation correlation alignment network (DeepCORAL), deep domain adaptation network (DAN), adversarial discriminant domain adaptation (ADDA), adaptive clustering network (CDAC), attention-based multi-scale residual adaptation network (AMRAN) and cross-domain Transformer (CDTrans), for comparison with the adaptive clustering remote sensing scene classification method based on cross-domain cross enhancement of the present invention. The present invention uses the overall classification accuracy to evaluate each cross-domain scene classification task, and calculates the average value of the overall classification accuracy as the overall evaluation index. The experimental results are shown in Table 2 below. The results are expressed as percentages, with the highest value in bold and the second highest value in underline.

[0085] Table 2 Classification accuracy evaluation table of different methods

[0086] method NWPU→AID AID→NWPU average value DeepCORAL 81.62 82.67 82.15 DAN 85.44 76.61 81.03 ADDA 77.2 87.3 82.25 CDAC 90.29 83.42 86.86 AMRAN 90.95 83.94 87.45 CDTrans 92.03 82.14 87.09 The present invention 93.84 84.07 88.96

[0087] From the experimental results in Table 2, the proposed method achieves the highest NWPU→AID and average accuracy in the cross-domain remote sensing scene classification task, and the second highest accuracy in AID→NWPU. The overall performance is better than the existing technologies given. Figure 5 The figure shows the feature distribution comparison of the target domain data before and after the adaptive clustering algorithm of the present invention. It can be seen that the algorithm effectively achieves good clustering of the target domain data features, thereby further achieving cross-domain cluster alignment.

[0088] The examples show that the present invention can bridge the distribution differences between the two domains by cross-enhancing labeled and unlabeled data and combining data information from the source and target domains. Then, consistency regularization is used to promote the propagation of high-confidence pseudo-labels, thereby achieving feature alignment between the source and target domains. Finally, an adaptive clustering method is introduced for target domain data with pseudo-labels to cluster and group the target domain data, achieve feature alignment at the cluster level, and further improve the quality of pseudo-labels. The present invention effectively solves the problems of cross-domain alignment difficulties and knowledge transfer obstructions caused by significant distribution differences between the source and target domains of remote sensing images, thereby improving the classification performance of remote sensing cross-domain scenes.

[0089] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An adaptive clustering remote sensing scene classification method based on cross-domain cross-enhancement, characterized by , including the following steps: Step 1: Build a domain adaptation network, which includes a feature extractor and a classifier module; Step 2: Perform cross-domain cross-enhancement on the input data to create enhanced data from labeled source domain data and target domain data to unlabeled target domain data; Step 3: Using the feature extractor to extract features from the target domain data after cross-enhancement, using the classifier to obtain pseudo labels for the target domain data, and promoting the propagation of high-confidence pseudo labels through consistency regularization; Step 4: Design an adaptive clustering loss to group the features of the target domain data into clusters, further improve the pseudo-label quality during the propagation process, and align cluster features between the source and target domains; Step 5: Construct an overall objective loss function based on the data, and optimize the overall objective loss function to iteratively train the domain adaptation network.

2. The adaptive clustering remote sensing scene classification method based on cross-domain cross-enhancement according to claim 1, characterized in that: In step 2, the method for performing cross-domain cross-enhancement on the input data is: Using the CutMix operation, weakly enhanced data is generated by replacing the unlabeled target domain data region with patches of labeled target domain data; strongly enhanced data is generated by replacing the unlabeled target domain data region with patches of labeled source domain data; thus, enhanced data that combines information from both the source and target domains is obtained.

3. The adaptive clustering remote sensing scene classification method based on cross-domain cross-enhancement according to claim 1 is characterized in that: In step 3, the method for promoting the propagation of high-confidence pseudo-labels through consistency regularization is: Based on the classic semi-supervised FixMatch framework, this method combines weak and strong augmentation strategies in a consistency regularization process. First, a weakly enhanced image of each image is obtained from the unlabeled target domain image, and the prediction results of the model are used as pseudo labels in supervised training. Then, we obtain the prediction results from the strongly enhanced images of the same image to increase the input diversity of the model. Finally, to ensure the consistency regularization in training, we enforce the standard cross entropy loss on the output of the strongly enhanced image model. Based on the above calculation, we can get the cross-enhancement pseudo-label loss L. PL .

4. The adaptive clustering remote sensing scene classification method based on cross-domain cross-enhancement according to claim 3 is characterized by: The cross-enhancement pseudo-label loss L is calculated based on the PL include: Where M is the number of target samples per input batch of unlabeled target domain images; P j ′=p(x′ j )=σ(F(G(x′ j ))) is the training result after the weakly enhanced target domain image passes through the model, where G represents the feature extractor in the domain adaptation network, F represents the classifier in the domain adaptation network, and the prediction result is converted into a single point hard label Then used as pseudo label; p(x′ j ′)=σ(F(G(x′ j ′))) represents the model's strong enhancement of the target domain image x′ j ′’s predictions, and τ is a scalar confidence threshold that determines the subset of pseudo labels that should be retained during model training.

5. The adaptive clustering remote sensing scene classification method based on cross-domain cross-enhancement according to claim 1, characterized in that: In step 4, a method for designing an adaptive clustering loss is: Based on the assumption that samples in the same cluster have similar features and belong to the same category label, the features extracted from the target domain image are clustered to achieve alignment of the source domain image features with the target domain image features at the cluster level.

6. The adaptive clustering remote sensing scene classification method based on cross-domain cross-enhancement according to claim 5, characterized in that: The method for clustering the features extracted from the target domain image is: After the cross-enhanced target domain image passes through the feature extractor, the feature vectors are sorted from large to small, and the sorted results are used as indicators to evaluate the similarity of paired features; if the first k indices in the sorted lists of the feature vectors of two samples are the same, the two paired samples belong to the same category, and the pairwise similarity label s is set ij =1; Otherwise ij = 0; Based on the method, the pairwise comparison between the target domain images is calculated to obtain the adaptive clustering loss L AC .

7. The adaptive clustering remote sensing scene classification method based on cross-domain cross-enhancement according to claim 6, characterized in that: The adaptive clustering loss L is obtained AC include: The binary cross entropy loss is used to establish pairwise comparisons between target domain images, which is expressed as follows: Where M is the number of target samples per input batch of unlabeled target domain images, Represents the unlabeled target domain images in the input batch The prediction results after passing the model; P j ′=p(x′ j )=σ(F(G(x′ j ))) represents the weakly enhanced target domain image x′ j The prediction result of Can be regarded as a similarity score for prediction and x′ j Whether they have the same category label.

8. The adaptive clustering remote sensing scene classification method based on cross-domain cross-enhancement according to claim 1, characterized in that: In step 5, constructing the overall objective loss function includes: Construct the cross-domain cross entropy loss, and the expression is shown in the following formula (4): Among them, S is the source domain, L is the labeled target domain, x represents the image sample, and y represents the label predicted by the model; Using the cross-augmented pseudo-label loss L PL , the adaptive clustering loss L AC and the cross-domain cross entropy loss L CE The overall objective loss function is established, and the expressions are shown in the following equations (5) and (6): Where G represents the feature extractor of the constructed domain adaptation network, F represents the classifier of the constructed domain adaptation network; λ is a scalar hyperparameter used to control the balance between the cross-domain cross entropy loss and the proposed cross-domain adaptive clustering loss; gradient reversal is applied between the feature extractor and the classifier.

9. The adaptive clustering remote sensing scene classification method based on cross-domain cross-enhancement according to claim 8, characterized in that: The gradient reversal is applied between the feature extractor and the classifier including: Directly minimize the adaptive clustering loss L AC Feature clustering is encouraged within the target domain; however, due to the dominance of a large number of source labels during supervised training, the feature representations learned in the target domain are often biased towards the source domain; directly minimizing the loss of unlabeled target domain data will exacerbate this bias and increase the risk of overfitting; to address this issue, a gradient reversal layer is used to reverse the gradient between the feature extractor and the classifier to ensure that the classifier performs accurate classification in the target domain.

Images