Remote sensing image unsupervised domain adaptation method based on comparative learning and multi-prototype alignment

Abstract

The invention belongs to the technical field of remote sensing image imaging, and provides a remote sensing image unsupervised domain adaptation method based on comparative learning and multi-prototype alignment, and the method specifically comprises the steps: constructing a teacher-student network, and setting a cross-domain multi-prototype alignment module and a pixel-level cross-domain comparative learning module which are needed in training in the teacher-student network; during teacher-student network training, a cross-domain multi-prototype alignment module is used for extracting prototypes of all categories in a source domain and a target domain and minimizing the feature distance of the prototypes, and a pixel-level cross-domain contrast learning module is used for explicitly modeling the global semantic relationship of cross-domain pixel samples; and an embedding space with high intra-class compactness and high inter-class separability is established.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image imaging, and in particular relates to a remote sensing image unsupervised domain adaptation method and device based on contrastive learning and multi-prototype alignment. Background Art

[0002] Rapid advances in remote sensing technology have significantly enhanced the ability to automatically identify and classify land surface types. Semantic segmentation, which aims to segment remote sensing images into regions with clear semantic information, has been widely used in multiple domains such as resource exploration, change detection, land use strategy, and disaster warning. Although deep learning technology has achieved significant progress, its effectiveness still relies heavily on large-scale, detailed annotated datasets. However, compiling and annotating these datasets is usually time-consuming and cumbersome, especially in professional domains. Therefore, using existing annotated datasets to train target datasets provides a feasible strategy. However, remote sensing data from different sensors, periods, and regions show high variability and complexity. This inter-domain difference often leads to reduced effectiveness when neural networks are trained only with pre-annotated datasets in the actual target dataset environment. To address these challenges, unsupervised domain adaptation (UDA) methods use existing annotated datasets and unlabeled target data to promote the adaptation of the network to the target domain. At present, the widely used unsupervised domain adaptation (UDA) methods are mainly divided into three categories: style transfer, adversarial learning, and self-training.

[0003] Style transfer methods use neural networks to synthesize source domain images that are visually similar to the target domain, thereby enhancing the generalization ability of models trained on the source domain to the target domain. Gatys et al. pioneered the use of convolutional neural networks to explicitly capture the content and style attributes of an image. By minimizing the composite content and style loss function, they achieved style transfer from one image to another, fusing the style of the target with the content of the source image. Karras et al. introduced CycleGAN based on cycle consistency loss, which allows images to be transformed in style and then restored to their original form, thereby ensuring content consistency. Li et al. explored unsupervised style transfer in remote sensing through the DualGAN method, using the transformed images and the labels of the source domain to enhance the network's understanding of the class characteristics of the target domain remote sensing data. Zhao et al. introduced an in-network regulator module and residual connections in the widely used DualGAN model and designed two new methods: residual DualGAN (RDG) and deep residual DualGAN (DRDG).

[0004] Adversarial learning techniques aim to reduce the difference in domain distribution by aligning feature-level or pixel-level information of each domain. AdaptSegNet, which uses the GAN network framework, innovatively includes a domain discriminator in its output space. This discriminator evaluates whether the output is from the source domain or the target domain, thereby reducing the difference in pixel predictions across domains. The CLAN model proposes a category-level adversarial training method that uses different classifiers to produce local alignment maps. These maps align each category separately, thus avoiding the "negative transfer" that is common in the global alignment process. Zheng et al. recognized the differences between remote sensing and natural images and adapted HR-Net and incorporated it into the adversarial domain adaptation scheme.

[0005] Self-training methods seek to use pseudo-labels as a means of training models in the target domain. The Mean Teacher method organizes training by synchronizing the predictions of the student model with the output of the teacher model, thereby enhancing the generalization ability of the model. CBST uses confidence scores of classification normalization to select and generate pseudo-labels with balanced class distribution, effectively solving the class imbalance problem inherent in pseudo-labels. Innovatively, DAFormer integrates the Transformer architecture into UDA and significantly improves the segmentation accuracy by using a teacher network for self-training in the target domain. HRDA and MIC effectively improve the robustness of the model by cropping images with different resolutions and random mask processing, respectively.

[0006] Despite the multitude of UDA methods, there is still a significant performance gap compared to fully supervised training. A common problem with most UDA methods is that they rely on implicit adaptation strategies without establishing explicit pixel-level associations between different domains. Therefore, the performance of implicit adaptation methods often degrades significantly when remote sensing images exhibit significant domain differences and severe class imbalance. For example, adversarial training methods mainly aim to minimize global distribution differences, which may not ensure accurate alignment of class-specific feature distributions. Especially in remote sensing images, simple alignment may not capture all relevant feature differences due to high intra-class variability and inter-class similarity. Self-training methods generate pseudo-labels of target domain images and continue to train or fine-tune the model to adapt to the target data distribution, but significant domain differences may lead to inaccurate pseudo-labels and increase the risk of overfitting of the model. In addition, this method may ignore rare categories by generating too many pseudo-labels for over-represented categories. In addition, relying solely on cross-entropy loss to measure the difference between predictions and labels also has its limitations. This method penalizes the prediction of individual pixels and ignores the relationship between pixels. Summary of the invention

[0007] In view of this, the present invention provides an unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment, which can provide an effective solution to the significant domain difference and obvious category imbalance problems in unsupervised domain adaptation (UDA) of remote sensing images, thereby significantly improving the UDA performance of remote sensing images.

[0008] To achieve the above object, the present invention is achieved through the following technical solutions:

[0009] An unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment. The specific process is as follows:

[0010] Build a teacher-student network and set up the cross-domain multi-prototype alignment module and pixel-level cross-domain contrast learning module required for training in the teacher-student network;

[0011] During the teacher-student network training, the cross-domain multi-prototype alignment module is used to extract prototypes of each category in the source and target domains and minimize their feature distances. The pixel-level cross-domain contrastive learning module is used to explicitly model the global semantic relationship of cross-domain pixel samples to establish an embedding space with high intra-class compactness and high inter-class separability.

[0012] Furthermore, the specific process of the method of the present invention is:

[0013] Build a teacher-student network and set up the cross-domain multi-prototype alignment module, pixel-level cross-domain contrast learning module and discriminator required for training in the teacher-student network;

[0014] Input source domain image X S To the student network to obtain fusion features And the prediction results, and based on the prediction results, construct a cross entropy loss for source domain supervision Input target domain image X T To the teacher network to obtain fusion features and prediction results, and calculating high confidence pseudo labels based on the prediction results

[0015] Using source domain data (X S ,Y S ) and target domain data Building hybrid data Input mixed data into the student network to obtain fusion features And prediction results, construct prediction results and pseudo labels The cross entropy loss between

[0016] Using the cross-domain multi-prototype alignment module to align source domain features Source domain label Y S and target domain features Target domain label Align and get the source domain prototype Proto S and the target domain prototype Proto T , and further obtain the global prototype Proto G ; Build domain prototype Proto S and Proto T With Proto G The weighted feature distance loss L between FD ;

[0017] The discriminator is used to identify the input target domain prototype and / or source domain prototype to obtain whether the prototype extracted in the current batch is from the source domain or the target domain, and the discriminator loss L is constructed. D , and further based on the output of the discriminator to the source domain, construct the minimum loss L Main ;

[0018] Using pixel-level cross-domain contrast learning module, mixed domain fusion features Compare pixel feature similarity with mixed domain labels and construct pixel-level contrast loss

[0019] The above losses are minimized during the training process, and after the training is completed, the teacher-student network model with the best performance indicators is selected for the unsupervised domain adaptation task of optical remote sensing images.

[0020] Furthermore, the cross entropy loss of the present invention for:

[0021]

[0022] Among them, Y S represents the source domain label, S(X S ) represents the prediction result.

[0023] Furthermore, the teacher network of the present invention is updated based on the student network parameters in the following manner:

[0024] T t+1 =αT t +(1-α)S t

[0025] Among them, S t represents the parameters of the student network at iteration number t, T t It represents the parameters of the teacher network at the iteration number t, and α represents the set weight threshold.

[0026] Furthermore, the mixed number construction process of the present invention is: randomly select Y SSet these category regions to 1 and the others to 0 to form a mask M. S ,Y S ) pixel by pixel, M is negated and recorded as M′ and Multiply them and then sum the results separately.

[0027] Furthermore, the prediction result S(X) constructed in the present invention Mix ) and pseudo labels The cross entropy loss between for:

[0028]

[0029] Furthermore, the present invention also constructs the similarity between the feature and the global prototype to determine the category of the feature Minimize calculations and pseudo labels The cross entropy loss between

[0030]

[0031] Furthermore, the cross-domain multi-prototype alignment module of the present invention obtains the global prototype Proto G for:

[0032]

[0033] Wherein, α and β represent preset thresholds.

[0034] Furthermore, the construction domain prototype Proto S and Proto T With Proto G The weighted feature distance loss L between FD for:

[0035]

[0036] Among them, the weight ω S and ω T According to the source domain label Y S and the target domain label The number of pixel samples in is calculated using the formula:

[0037]

[0038]

[0039] Furthermore, the discriminator loss L of the present invention is D for:

[0040]

[0041] Among them, D(P) represents the output of the discriminator for the prototype, and y represents the corresponding label.

[0042] Furthermore, the output of the discriminator of the present invention to the source domain is used to construct the minimum loss L Main

[0043]

[0044] Furthermore, the pixel-level contrast loss of the present invention for:

[0045]

[0046] Among them, v represents the feature vector map obtained from the encoder, v i represents the feature vector of the i-th pixel, τ is a temperature parameter used to scale the dot product of the feature, and D represents the feature similarity of two pixels, which is characterized by the exponential cosine similarity.

[0047] Beneficial effects:

[0048] First, the present invention sets up a neural network framework based on contrastive learning and multi-prototype alignment during teacher-student network training. The framework includes a pixel-level cross-domain contrastive learning module and a cross-domain multi-prototype alignment module, which respectively provide effective solutions to the significant domain differences and obvious category imbalance problems in unsupervised domain adaptation (UDA) of remote sensing images, thereby significantly improving the UDA performance of remote sensing images.

[0049] Second, the present invention utilizes a cross-domain multi-prototype alignment method to alleviate the adverse effects of category imbalance by extracting prototypes of each category in the source domain and the target domain and minimizing their feature distances to ensure the similarity of prototypes of the same category in the two domains.

[0050] Third, the present invention dynamically adjusts the prototype weights according to the pixel ratios of the categories, especially assigning higher weights to underrepresented categories, guiding the model to pay more attention to rare category samples during training, thereby improving the model's ability to handle imbalanced data.

[0051] Fourth, the present invention uses feature prototypes to guide learning in the target domain and combines them with pseudo labels to calculate the cross entropy loss. This encourages the model to produce outputs consistent with the pseudo labels, thereby reducing the impact of label noise.

[0052] Fourth, the present invention uses pixel-level cross-domain contrastive learning to maintain the consistency of similar features in the latent space while clearly separating different features by explicitly establishing pixel-level associations between the source and target domains. This method constructs a more compact space with more distinct features, making it easier to classify. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A schematic diagram of the overall structure of an unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment provided by the present invention;

[0055] Figure 2 A schematic diagram of the structure of the cross-domain multi-prototype alignment module provided by the present invention; DETAILED DESCRIPTION

[0056] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0057] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0058] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0059] This application is based on two key prior knowledge: (1) pixels of the same category should maintain the consistency of deep semantic features; (2) feature embedding spaces with high intra-class compactness and high inter-class dispersion are more conducive to feature differentiation. A mixed domain dataset is constructed by combining data from the source domain and the target domain through image stitching technology, and contrastive learning is applied to regard pixel features of the same category in different domains as positive samples and pixel features of different categories as negative samples. According to the prior knowledge (1), contrastive learning promotes the similarity between positive data and the difference between negative data. Feature learning is enhanced by explicitly reducing the gap in pixel embeddings in the same category while increasing the distinction between different categories. According to the prior knowledge (2), contrastive loss is applied in the source domain to further explore domain-specific knowledge, aiming to create a richer, more compact and more clearly defined feature space.

[0060] In view of the limitations of implicit adaptive methods, the present application proposes an unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment. This method aims to improve the unsupervised domain adaptation (UDA) performance of remote sensing images by explicitly establishing pixel-level connections between the source domain and the target domain to help identify and understand the commonalities and differences of images in different domains. The specific process is as follows:

[0061] A teacher-student network is constructed, and the cross-domain multi-prototype alignment module and pixel-level cross-domain contrastive learning module required for training in the teacher-student network are set up. During the training of the teacher-student network, the cross-domain multi-prototype alignment module is used to extract prototypes of various categories in the source and target domains and minimize their feature distances. The pixel-level cross-domain contrastive learning module is used to explicitly model the global semantic relationship of cross-domain pixel samples, and establish an embedding space with high intra-class compactness and high inter-class separability.

[0062] Define the source domain dataset image X S ∈R 3×H×W and label Y S ∈R 3×H×W , and the target domain dataset image X T ∈R 3 ×H×W .like Figure 1 As shown, the specific process of this method is:

[0063] Step S1: Construct a teacher-student network and set up a cross-domain multi-prototype alignment module and a pixel-level cross-domain contrastive learning module.

[0064] In this step, the teacher-student network is initialized. The network structures of the two are exactly the same, both of which are encoder-decoder architectures, where the encoder is used to extract multi-scale features of the input image, and the decoder is used to classify the features pixel by pixel. During the training process, the student network parameters change with the losses at each level, while the teacher network parameters are updated with the student network parameters. In addition, a cross-domain multi-prototype module and a pixel-level cross-domain contrast learning module are set to align the class features and pixel-level features of different domains, respectively, to enhance the cross-domain generalization ability of the model; a discriminator is also set to identify the target domain or source domain.

[0065] Step S2: Input the source domain image to the student network to obtain fusion features And the prediction results, and based on the prediction results, calculate the cross entropy loss for source domain supervision

[0066] This step is the training process of the student network S: Input source domain image X S ∈R 3×H×W After feature extraction by the encoder, 4 feature maps of different levels are obtained, which are sorted from shallow to deep as f1, f2, f3, and f4. In order to use multi-scale features at the same time, f1~f4 are mapped to the same size and dimension, and then connected in series to obtain fusion features. Then the fusion features Send to the decoder to obtain the prediction result S(X S ) and calculate its difference with the source domain label Y S ∈R 3×H×W The cross entropy loss Perform source domain supervision.

[0067]

[0068] Step S3: Input the target domain image to the teacher network to obtain fusion features And the prediction results, calculate the high confidence pseudo label based on the prediction results, and update the parameters based on the teacher network parameters.

[0069] This step is to train the teacher network T: the initial parameters of the teacher network T are the same as those of the student network S. For the input target domain image X T ∈R 3×H×W , the encoder of the teacher network obtains a multi-level feature map, and the fusion feature of the target domain image is obtained by the same operation as step S2 And the prediction result T(X T ), and then use the threshold t to filter out high confidence pseudo labels, denoted as The formula is as follows:

[0070] p T =max CT(X T )

[0071]

[0072] Among them, max C T(·) represents the maximum value of the class prediction probability of each pixel of the image with a total of C classes by the teacher network, denoted as p T . argmax C T(·) represents taking the class corresponding to the maximum value of the class prediction probability of each pixel as the final prediction result. When p T ≥t, the prediction result is considered reliable. When p T <t, the prediction is considered unreliable, denoted as nan, indicating non-numeric and not participating in the loss calculation.

[0073] In addition, the parameter update method of the teacher network T with the number of iterations t is as follows:

[0074] T t+1 =αT t +(1 - α)S t

[0075] Among them, S t represents the parameters of the student network at the number of iterations t, and T t represents the parameters of the teacher network at the number of iterations t, and α represents the set weight threshold.

[0076] Step S4: Use the source domain data (X S , Y S ) and the target domain data to construct mixed data Input the mixed data into the student network to obtain the fused feature and the prediction result, and construct the cross-entropy loss between the prediction result and the pseudo-label

[0077] The specific process of this step is as follows: Randomly select half of the classes in Y S , set these class regions to 1, and the others to 0 to form a mask M. M is multiplied by (X S , Y S ) pixel by pixel. The inverse of M is denoted as M′ and multiplied by , and then the sum of the results is

[0078]

[0079] Subsequently, input the mixed image X Mix into the student network to obtain the fused feature of the mixed domain image And the prediction results, calculate the prediction results and pseudo labels The cross entropy loss between To enhance the domain robustness of the model.

[0080]

[0081] Step S5: Use the cross-domain multi-prototype alignment module to align the source domain features Source domain label Y S and target domain features Target domain label Align and obtain the domain prototypes Proto of the source domain and the target domain S and Proto T , and further obtain the global prototype Proto G ; Computational domain prototype Proto S and Proto T With Proto G The weighted feature distance L between FD .

[0082] The specific process of this step is: Source domain label Y S And the target domain features obtained in step S3 Target domain label Input into the cross-domain multi-prototype alignment module to calculate the class prototype. The method to initialize the prototype is as follows:

[0083]

[0084] Here the indicator function The definition of is that if Y(i,c) = C, it is 1, otherwise it is 0. Therefore, the prototype can be regarded as an approximate representation of the centroid of each category. The above formula is used to calculate the prototype of the current batch in the source domain and the target domain, respectively, denoted as Proto S and Proto T Subsequently, the global prototype Proto is updated by weighting the domain prototypes G , as shown below:

[0085]

[0086] Among them, t represents the current batch and t+1 represents the next batch.

[0087] Since the class prototype is calculated based on the sample mean, it is insensitive to the number of samples, which helps to enhance the model's learning of rare class features. S and proto T With proto GThe weighted feature distance L between FD To enhance the model's ability to identify rare classes, the formula is as follows:

[0088]

[0089] Among them, the weight ω S and ω T According to the source domain label Y S and the target domain label The number of pixel samples in is calculated using the following formula:

[0090]

[0091] Step S6: Calculate the similarity between the feature and the global prototype to determine the category of the feature Minimize calculations and pseudo labels The cross entropy loss between The target domain prototype and the source domain prototype are input into the discriminator, and the discriminator is used to determine whether the prototype extracted in the current batch comes from the source domain or the target domain, and the discriminator loss L is obtained. D ; Further based on the output of the discriminator to the source domain, construct the minimization loss function L Main .

[0092] In the initial stage of training, the model tends to assign higher confidence scores to target data that are similar to the source domain data distribution. However, high confidence does not necessarily equal correct prediction, which introduces noise in the self-training process. The category of a feature is determined by calculating the similarity between the feature and the global prototype, as follows:

[0093]

[0094] Then, minimize and pseudo labels The cross entropy loss between The formula is as follows:

[0095]

[0096] This loss ensures the consistency between the results of prototype-guided feature classification and the pseudo-labels, thus helping to correct erroneous pseudo-labels.

[0097] Finally, a discriminator is used to determine whether the prototype extracted from the current batch is from the source domain or the target domain, thereby enhancing the model's ability to extract domain-invariant features. D The definition is as follows:

[0098]

[0099] Where D(P) represents the output of the discriminator for the prototype, and y represents the corresponding label (1 for the source domain and 0 for the target domain). The goal of this part is to make the prototypes generated for the two domains indistinguishable to the discriminator. Therefore, the following loss function is minimized:

[0100]

[0101] This approach ensures that the features learned by the model are robust across different domains, thereby improving the performance of the model on unseen target domain data.

[0102] Step S7: Use the pixel-level cross-domain contrast learning module to fuse the mixed domain features Compare pixel feature similarity with mixed domain labels and calculate pixel-level contrast loss

[0103] The specific process of this step is: the mixed domain fusion feature obtained in step S4 and mixed domain labels are fed into the pixel-level cross-domain contrastive learning module. For a pixel i with a true semantic label C, the positive samples include all pixels belonging to the same category C, while the negative samples consist of pixels of other categories, excluding C. Therefore, the pixel-level contrastive loss It can be defined as follows:

[0104]

[0105] Among them, v represents the feature vector map obtained from the encoder, where v i represents the feature vector of the i-th pixel. τ is a temperature parameter used to scale the dot product of the feature. D represents the feature similarity of two pixels, which is characterized by the exponential cosine similarity.

[0106] The pixel-level cross-domain contrastive learning module establishes an embedding space with high intra-class compactness and high inter-class separability by explicitly modeling the global semantic relationship of cross-domain pixel samples, thereby significantly improving the adaptability of the model.

[0107] Step S8: Add the losses of each part of the above steps and record them as L total :

[0108]

[0109] Minimize L during training total ,After the training is completed, the model with the best performance ,indicator is selected for the unsupervised domain adaptation task of ,optical remote sensing images.

[0110] The embodiments of the present application establish pixel-level associations between the source domain and the target domain, align intra-class features across domains, and separate inter-class features to constrain feature dispersion and optimize the feature space, thereby effectively solving the common "intra-class variability" and "inter-class similarity" challenges in UDA tasks. In addition, it is proposed to use prototypes to effectively solve the class imbalance problem that is prevalent in remote sensing scenes. The average-based prototype is inherently less sensitive to data imbalance, enabling fair extraction of features for all categories regardless of their occurrence rate. This ensures that each category has a concentrated representation, especially for categories with limited samples in the source or target domain, thereby directly alleviating the adverse effects of data sparsity on model performance.

[0111] The overall network structure described in the above steps is as follows Figure 1 As shown, this method is based on the Transformer architecture and proposes a neural network framework based on contrastive learning and multi-prototype alignment, which includes a pixel-level cross-domain contrastive learning module and a cross-domain multi-prototype alignment module.

[0112] First, the source domain data source domain data (X S ,Y S ) and target domain data Perform image stitching to construct mixed data Acts as a bridge to connect images from different domains.

[0113] Second, the pixel-level cross-domain contrastive learning module applies contrastive learning to attract embeddings of pixels within the same category across different domains while repelling pixels of different categories to promote discriminative feature learning, thus achieving a richer, more compact, and well-separated feature space.

[0114] Thirdly, in order to cope with the complex challenges brought by the sample imbalance problem, this paper proposes a cross-domain multi-prototype alignment method, such as Figure 2 As shown in Figure 1, the encoder extracts multi-scale features from the source domain and the target domain respectively and calculates the feature prototypes of each domain, and then updates the global prototype. The weighted feature distance between the domain prototype and the global prototype is calculated, thereby enhancing the learning of features related to these rare categories.

[0115] At the same time, the consistency between the results of prototype-guided feature classification and the pseudo-labels is calculated to correct the erroneous pseudo-labels that appear during the self-training process.

[0116] Each module proposed in this application specifically addresses the significant domain differences and obvious category imbalance problems in remote sensing image UDA, effectively improving the performance of remote sensing image UDA.

[0117] The present invention provides an unsupervised domain adaptation method for remote sensing images based on the Transformer structure. The algorithm adopts MIT_5 as the backbone network for feature extraction. In order to improve the unsupervised domain adaptation segmentation accuracy under the conditions of significant domain differences and obvious category imbalance, two modules, pixel-level cross-domain contrastive learning and cross-domain multi-prototype alignment, are proposed, which effectively improves the cross-domain adaptation ability of the model.

[0118] In this embodiment, the teacher-student network adopts the MIT_5 backbone network. Figure 1 As shown in the figure, during training, the source domain image and the target domain image are respectively subjected to the MIT_5 backbone network for feature extraction, and two groups of feature maps of four different levels are obtained, which are sorted from shallow to deep in order: f1, f2, f3, and f4. In order to simultaneously utilize multi-scale features, f1~f4 are mapped to the same size and dimension, and then connected in series to obtain fusion features. and Then the fusion features Feed it into the student network decoder to obtain the prediction result and calculate its difference with the label Y S ∈R 3×H×W The cross entropy loss is used for source domain supervision. Feed it into the teacher network decoder to get the pseudo label After the teacher-student network is trained, the image to be judged is input for judgment, and the pseudo label output by the network is the result of the judgment.

[0119] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment, characterized in that: The specific process is: Build a teacher-student network and set up the cross-domain multi-prototype alignment module and pixel-level cross-domain contrast learning module required for training in the teacher-student network; During the teacher-student network training, the cross-domain multi-prototype alignment module is used to extract prototypes of each category in the source and target domains and minimize their feature distances. The pixel-level cross-domain contrastive learning module is used to explicitly model the global semantic relationship of cross-domain pixel samples to establish an embedding space with high intra-class compactness and high inter-class separability.

2. The unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment according to claim 1, characterized in that: Build a teacher-student network and set up the cross-domain multi-prototype alignment module and pixel-level cross-domain contrast learning module required for training in the teacher-student network; Input source domain image X S To the student network to obtain fusion features And the prediction results, and based on the prediction results, construct a cross entropy loss for source domain supervision Input target domain image X T To the teacher network to obtain fusion features and prediction results, and calculating high confidence pseudo labels based on the prediction results Using source domain data (X S ,Y S ) and target domain data Building hybrid data Input mixed data into the student network to obtain fusion features And the prediction result S(X Mix ), construct prediction results and pseudo labels The cross entropy loss between Using the cross-domain multi-prototype alignment module to align source domain features Source domain label Y S and target domain features Target domain label Align and get the source domain prototype Proto S and the target domain prototype Proto T , and further obtain the global prototype Proto G ; Construct a weighted feature distance loss L between domain prototypes FD ; The discriminator is used to discriminate the input target domain prototype and source domain prototype to obtain whether the prototype extracted in the current batch comes from the source domain or the target domain, and the discriminator loss L is constructed. D , and further based on the output of the discriminator to the source domain, construct the minimum loss L Main ; Using pixel-level cross-domain contrast learning module, mixed domain fusion features Compare pixel feature similarity with mixed domain labels and construct pixel-level contrast loss The above losses are minimized during the training process, and after the training is completed, the teacher-student network model with the best performance indicators is selected for the unsupervised domain adaptation task of optical remote sensing images.

3. The unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment according to claim 2, characterized in that: The teacher network is updated based on the student network parameters in the following way: T t+1 =αT t +(1-α)S t Among them, S t represents the parameters of the student network at iteration number t, T t It represents the parameters of the teacher network at the iteration number t, and α represents the set weight threshold.

4. The unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment according to claim 2, characterized in that: The mixed number construction process is: randomly select Y S Set these category regions to 1 and the others to 0 to form a mask M. S ,Y S ) Multiply by pixel, and negate M as M ′ and with Multiply them and then sum the results separately.

5. The unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment according to claim 2, characterized in that: The construction prediction result S(X Mix ) and pseudo labels The cross entropy loss between for:

6. The unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment according to claim 2, characterized in that: The similarity between the feature and the global prototype is also constructed to determine the category of the feature Minimize calculations and pseudo labels The cross entropy loss between 7. The unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment according to claim 2, characterized in that: The weighted feature distance loss L between the constructed domain prototypes FD for: Among them, the weight ω S and ω T According to the source domain label Y S and the target domain label The number of pixel samples in is calculated using the following formula:

8. The unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment according to claim 2, characterized in that: The discriminator loss L D for: Among them, D(P) represents the output of the discriminator for the prototype, and y represents the corresponding label.

9. The unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment according to claim 2, characterized in that: The discriminator outputs the source domain and constructs a method to minimize the loss L Main 10. The unsupervised domain adaptation method for remote sensing images based on contrastive learning and multi-prototype alignment according to claim 2, characterized in that: The pixel-level contrast loss for: Among them, v represents the feature vector map obtained from the encoder, v i represents the feature vector of the i-th pixel, τ is a temperature parameter used to scale the dot product of the feature, and D represents the feature similarity of two pixels, which is characterized by the exponential cosine similarity.

Images