Prototype contrastive learning based cross-domain remote sensing image classification method
By combining prototype contrastive learning and autoencoders, the problem of insufficient category discrimination features in cross-domain remote sensing image classification is solved, the classification accuracy and generalization ability of the model under domain difference conditions are improved, and efficient cross-domain scene classification is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2023-09-26
- Publication Date
- 2026-07-21
AI Technical Summary
Existing cross-domain remote sensing image classification methods lack sufficient learning of category discrimination features, resulting in poor model generalization ability in the presence of domain differences, which affects classification accuracy.
We employ a prototype contrastive learning approach, combining an autoencoder and a prototype contrastive learning module. Through pseudo-label refinement strategies and mask image modeling, we dynamically maintain a prototype library, optimize the domain adaptation network, reduce the distance between prototype features of the same category and increase the distance between features of different categories, and construct a global prototype library to improve category alignment and model generalization ability.
It improves the accuracy and generalization ability of cross-domain remote sensing image classification, reduces the cost of manual annotation, and enhances scene classification performance in the target domain.
Smart Images
Figure CN117274701B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of optical remote sensing image classification technology, and specifically to a cross-domain remote sensing image classification method based on prototype contrastive learning. Background Technology
[0002] Remote sensing image classification has a wide range of applications, with scene classification of remote sensing images being a representative example. Remote sensing image classification involves analyzing pixel information in remote sensing images and categorizing them into different semantic classes. It has broad applications in land use, urban planning, and environmental monitoring, helping researchers and policymakers obtain information about surface features and changes to support resource management, disaster monitoring, and environmental protection. Traditional deep learning-based scene classification methods perform well under the condition of large amounts of labeled data and independent distribution of training and testing data. However, manually labeling large-scale remote sensing images for model training is costly and time-consuming. Furthermore, unlabeled test data can be obtained from various imaging conditions such as different sensors, geographical locations, lighting conditions, and seasonal variations. Significant domain differences in spectral and spatial resolution between training and testing data make it impossible for them to satisfy the independent and identically distributed condition. Therefore, directly applying a model trained on labeled data (source domain) to unlabeled data with domain differences (target domain) will result in severe performance degradation. Domain adaptation methods for cross-domain scene classification can overcome the poor generalization ability of scene classification models under conditions of domain differences. By transferring knowledge from labeled training data in the source domain to unlabeled test data in the target domain, they bridge the domain differences while avoiding the overhead of intensive manual labeling and retraining. In summary, domain adaptation is of great significance in remote sensing image scene classification. It can address data distribution differences, reduce labeling costs, and improve the model's generalization ability, thereby achieving more accurate scene classification in the target data domain. However, existing cross-domain classification domain adaptation methods still have many problems, especially insufficient learning of category-determining features, which directly affects the accuracy of category discrimination. Summary of the Invention
[0003] To address the aforementioned issues, this invention provides a cross-domain remote sensing image classification method based on prototype contrastive learning. This method combines prototype and contrastive learning with domain adaptation, thereby resolving the problem of insufficient learning of category discrimination features in existing cross-domain methods.
[0004] To achieve the objectives of this invention, the following technical solution is adopted:
[0005] This invention provides a cross-domain remote sensing image classification method based on prototype contrastive learning, comprising the following steps: S101: Constructing a domain adaptation network, the domain adaptation network including an autoencoder and a prototype contrastive learning module; S102: Performing unlabeled self-supervised pre-training on the autoencoder to obtain pre-trained autoencoder model parameters, and loading the pre-trained autoencoder model parameters into the autoencoder; wherein, the target domain image is an unlabeled target domain image; S103: Inputting the target domain image into the autoencoder, and using a pseudo-label thinning strategy to obtain thinned pseudo-labels for the target domain image; S104: Encoding the source domain image, and simultaneously processing the target domain image... Masked image modeling; the source domain image includes labels, the labels indicating the category of the source domain image; S105: construct prototype libraries for the source and target domains and align the prototypes of each category in the source and target domains using the prototype contrastive learning module; S106: construct an overall objective loss function based on the data, and optimize the overall objective loss function to iteratively train the domain adaptation network to obtain a domain adaptation network for classification; wherein the data includes the source domain image, the target domain image, and refined pseudo-labels of the target domain image; S107: input the target domain image to be tested into the domain adaptation network for classification to obtain a classification result, the scene classification result indicating the category of the target domain image to be tested.
[0006] Furthermore, the autoencoder is based on Transformer and includes a decoder and an encoder. The encoder adopts a network structure of Transformer Blocks with a first number of layers and a multi-head attention mechanism; the decoder adopts a network structure of Transformer Blocks with a second number of layers and a multi-head attention mechanism.
[0007] Furthermore, the self-supervised pre-training is performed on the target domain image using a masked image modeling method.
[0008] Further, the process of obtaining refined pseudo-labels for the target domain image using the pseudo-label refinement strategy includes: S103.1: Obtaining encoded features from the target domain image through feature encoding, wherein the encoded features include at least a classification token, and inputting the classification token into a classifier to obtain initial pseudo-labels:
[0009]
[0010] in, k = 1, 2, ..., K; k represents the k-th category, and there are K categories in total. t h() represents the classifier token;
[0011] S103.2: Calculate the initial class center for each category based on the initial pseudo-labels. C k S103.3: Determine the initial class center of the k-th class; S103.3: Obtain the distance of each class based on the initial class center. Where d(·,·) is the cosine distance. S103.4: The distance to each class given the j-th target domain sample; The distribution is modeled and defined as follows:
[0012]
[0013] Where p() represents the probability distribution of the Gaussian uniform mixture model, N + () denotes a Gaussian distribution, σ is the variance of the Gaussian distribution, and U(0,δ) k ) is in the interval [0, δ k The uniform distribution defined above, where π represents the amplitude coefficient of the Gaussian distribution;
[0014] S103.5: Using the Gaussian uniform mixture model, label reliable and unreliable target domain data, and calculate the reliability G of the target domain data using the following formula:
[0015]
[0016] S103.6: Based on the reliability G and the threshold ζ, the target domain data is divided into a reliable subset and an unreliable subset, where the reliable subset is represented as... The remaining target domain samples are then classified into the unreliable subset. in Use x t,k Represents all initial pseudo-tags Target domain sample, x t S103.7: For the target domain data in the reliable subset, the initial pseudo-label is directly used. As a refined pseudo-label, reliable class centers, i.e., the initial feature layer prototypes, are calculated based on the classification tokens of the reliable subset samples. Based on the reliable subset samples, calculate the initial output layer prototype at the classifier output.
[0017]
[0018]
[0019] In the formula and These are the feature layer prototype and output layer prototype of the k-th class in the target domain, respectively, where t represents the target domain, |·| represents the number of elements in the set, h() represents the classifier, and c t For the corresponding target domain sample x t Encoded classification tokens;
[0020] S103.8: For the aforementioned unreliable subset The target domain samples in the data are based on the distance from the unreliable subset. The target domain samples are relabeled according to the category of the initial feature layer prototype, resulting in refined pseudo-labels.
[0021]
[0022] In the formula, For the initial pseudo-labels of the predicted reliable subset, d(·,·) is the cosine distance. This is the initial feature layer prototype for the k-th class.
[0023] Furthermore, the masked image modeling of the target domain image includes:
[0024] S104.1: Process the target domain image into a fourth number of 16×16 resolution patch blocks; S104.2: Randomly select a second proportion of the patch blocks for masking, the second proportion being no greater than 20%; S104.3: Remove the masked patch blocks, and perform feature encoding on the remaining visible patch blocks to obtain encoded features; S104.4: Restore the encoded features to the sequential positions of the remaining visible patch blocks; S104.5: Supplement the missing positions with learnable random initialization vectors to obtain completed encoded features; S104.6: Input the completed encoded features into the decoder to reconstruct the normalized pixel values of the input image.
[0025] Further, the normalized pixel values of the reconstructed input image include: calculating the image reconstruction loss between the reconstructed image and the masked patch pixels of the original image in the normalized pixel space, wherein the image reconstruction loss L rec The expression is as follows:
[0026]
[0027] p represents the normalized pixel values of the target domain image. M Let M be the pixel value predicted by the autoencoder for the mask position of the target domain image, M be the set of mask patch pixels, and |·| represent the number of elements in the set.
[0028] Further, constructing prototype libraries for the source and target domains and aligning the prototypes of each category in the source and target domains using the prototype contrastive learning module includes: S105.1: Obtaining the source domain feature layer prototype and the source domain output layer prototype corresponding to each category based on the source domain image and its labels; constructing a source domain feature layer prototype library using the source domain feature layer prototypes corresponding to each category; S105.2: Obtaining the target domain feature layer prototype and the output layer prototype corresponding to each category based on the target domain image and its refined pseudo-labels; constructing a target domain feature layer prototype library using the target domain output layer prototypes corresponding to each category; S105.3: Dynamically maintaining and adaptively updating the prototype library, which includes the source domain feature layer prototype library, the source domain output layer prototype library, the target domain feature layer prototype library, and the target domain output layer prototype library; S105.4: Calculating the prototype alignment loss based on contrastive learning based on the dynamically maintained prototype library.
[0029] Furthermore, the method further includes: S105.5: Aligning the feature distributions of the source and target domains by reducing the prototype feature distance of the same category between domains and increasing the prototype feature distance of different categories between domains, wherein the feature layer prototype alignment loss is calculated as follows:
[0030]
[0031] The prototype alignment loss of the output layer is calculated as follows:
[0032]
[0033] in, P s + Representation of the prototype of the target domain feature layer For source domain feature layers with the same k-th category, τ is a temperature hyperparameter that determines the concentration level. Represents the prototype of the output layer of the target domain. The source and output layer prototypes have the same k-th class, and τ is the temperature hyperparameter that determines the concentration level; S105.6: Summing the prototype alignment losses of the feature layer and the output layer yields the total prototype contrast loss:
[0034]
[0035] Furthermore, during the iterative training of the domain adaptation network, the pseudo-label refinement strategy is executed at regular intervals to update the refined pseudo-labels.
[0036] Furthermore, constructing the overall objective loss function L based on the data includes:
[0037] S106.1: Constructing the source domain classification loss L src The expression is as follows:
[0038]
[0039] In the formula n s Let H be the total number of samples in the source domain, and H(·,·) be the cross-entropy loss function. It is the classification token of the classification layer for the i-th source domain sample. The output predicted probability, Let be the true label of the i-th source domain sample, where s represents the source domain and i is a positive integer;
[0040] S106.2: Constructing the target domain pseudo-label classification loss L pse The expression is as follows:
[0041]
[0042] In the formula n t Let H be the total number of samples in the source domain, and H(·,·) be the cross-entropy loss function. This represents the classification token of the classification layer for the j-th target domain sample. The output predicted probability, It is the refined pseudo-label of the j-th target domain sample, where t represents the target domain and j is a positive integer;
[0043] S106.3: Constructing the target domain conditional entropy loss L ent The expression is as follows:
[0044]
[0045] In the formula n t The total number of samples in the source domain. This represents the classification token of the classification layer for the j-th target domain sample. The predicted probability of the k-th class, where t represents the target domain, j is a positive integer, and k is a positive integer;
[0046] S106.4: Utilize the source domain classification loss L src The target domain pseudo-label classification loss L pse The target domain conditional entropy loss L ent The image reconstruction loss L rec Compared with the prototype, the loss L con The overall objective number L is established as follows:
[0047] L = L src+L pse +λ1L ent +λ2L rec +λ3L con
[0048] In the formula, λ1, λ2, λ 3 is a hyperparameter, representing the conditional entropy loss L of the target domain. ent The prototype contrast loss L con and the image reconstruction loss L rec The weight.
[0049] It should be understood that the content described in this summary section is not intended to limit the key or essential features of the embodiments of the invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0050] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:
[0051] Figure 1 This is a schematic diagram of the method flow of the present invention;
[0052] Figure 2 This is an overall schematic diagram of the domain adaptation network of the present invention;
[0053] Figure 3 This is a schematic diagram of the pseudo-label refinement strategy of the present invention;
[0054] Figure 4 This is a schematic diagram illustrating the dynamic maintenance of the prototype library for this invention. Detailed Implementation
[0055] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.
[0056] In the description of embodiments of the present invention, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.
[0057] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and descriptions are considered exemplary in nature and not restrictive. There is no requirement for the order of the method steps described, as long as they are achievable, they are within the scope of protection of this invention.
[0058] The following is in conjunction with the accompanying drawings in the instruction manual. Figures 1-4 The present invention will be described in detail.
[0059] This invention provides a cross-domain remote sensing image classification method based on prototype contrastive learning, comprising the following steps:
[0060] S101: Construct a domain adaptation network, which includes an autoencoder and a prototype contrastive learning module.
[0061] like Figure 2 As shown, the domain adaptation network includes an autoencoder and a prototype contrastive learning module. The autoencoder is based on Transformer and includes a decoder and an encoder. The encoder adopts a network structure with a first number of layers of Transformer Blocks and a multi-head attention mechanism, while the decoder adopts a network structure with a second number of layers of Transformer Blocks and a multi-head attention mechanism.
[0062] In some implementations, the first number is 12, and in other implementations it may be 24; the second number is 8, and in other implementations it may be 12.
[0063] S102: Perform unlabeled self-supervised pre-training on the autoencoder to obtain the pre-trained autoencoder model parameters, and load the pre-trained autoencoder model parameters into the autoencoder.
[0064] The self-supervised pre-training is performed on a target domain image using a masked image modeling method. The target domain image is an unlabeled target domain image, and the label indicates the category of the target domain image.
[0065] In some implementations, unlabeled self-supervised pre-training of the autoencoder can be carried out in the following ways:
[0066] S102.1: Process the target domain image into a third number of 16×16 resolution patch blocks; the third number can be selected according to the actual situation, such as 196, 256, etc., and is not limited here.
[0067] S102.2: Randomly select a first proportion of the patch blocks for masking; the first proportion is greater than 60%;
[0068] S102.3: Remove the masked patch blocks and perform feature encoding on the remaining visible patch blocks to obtain encoded features;
[0069] S102.4: Restore the encoded features to the sequential positions of the remaining visible patch blocks;
[0070] S102.5: Fill in the missing positions with learnable random initialization vectors to obtain the completed encoded features;
[0071] S102.6: Input the completed coded features into the decoder to reconstruct the normalized pixel values of the input image.
[0072] The normalized pixel values of the input image can be reconstructed in the following way:
[0073] Calculate the mean square error between the reconstructed image and the masked patch pixels of the original image in the normalized pixel space.
[0074] Image reconstruction loss L rec The expression is as follows:
[0075]
[0076] Where L rec This represents the average reconstruction loss for each normalized pixel in the target domain. p represents the normalized pixel values of the target domain image. M Let M be the pixel value predicted by the autoencoder for the mask position of the target domain image, M be the set of mask patch pixels, and |·| represent the number of elements in the set.
[0077] S103: Input the target domain image into the autoencoder and use a pseudo-label thinning strategy to obtain thinned pseudo-labels for the target domain image.
[0078] like Figure 3 As shown, this step can be implemented in the following ways:
[0079] S103.1: The target domain image is encoded to obtain encoded features, wherein the encoded features include at least a classification token. The classification token is input into a classifier to obtain initial pseudo-labels.
[0080]
[0081] in, k = 1, 2, ..., K, where k represents the k-th category, and there are K categories in total. th() represents the classifier token;
[0082] S103.2: Calculate the initial class center for each category based on the initial pseudo-labels.
[0083]
[0084] Among them, C k This represents the initial class center of the k-th class;
[0085] S103.3: Obtain the distance of each category based on the initial class centers.
[0086]
[0087] Where d(·,·) is the cosine distance. It is a classification token given the j-th target domain sample;
[0088] S103.4: The distance d for each category is calculated using a Gaussian uniform mixture model. t j The distribution is modeled and defined as follows:
[0089]
[0090] Where p() represents the probability distribution of the Gaussian uniform mixture model, N + () denotes a Gaussian distribution, σ is the variance of the Gaussian distribution, and U(0,δ) k ) is in the interval [0, δ k The uniform distribution defined above, where π represents the amplitude coefficient of the Gaussian distribution;
[0091] S103.5: Using the Gaussian uniform mixture model, label reliable and unreliable target domain data, and calculate the reliability G of the target domain data using the following formula:
[0092]
[0093] The Gaussian uniform mixture model consists of a Gaussian distribution and a uniform distribution. It is commonly used for outlier detection, assuming that internal values are distributed according to a Gaussian distribution and outliers are distributed uniformly. Therefore, we use Gaussian and uniform components to model the target domain samples of reliable and unreliable pseudo-labels, respectively.
[0094] S103.6: Based on the reliability G and the threshold ζ, the target domain data is divided into a reliable subset and an unreliable subset, where the reliable subset is represented as... The remaining target domain samples are then classified into the unreliable subset. in Use x t,k Represents all initial pseudo-tags Target domain sample, x t Represents the target domain sample;
[0095] S103.7: For the target domain data in the reliable subset, the initial pseudo-label is directly used. As a refined pseudo-label, reliable class centers, i.e., the initial feature layer prototypes, are calculated based on the classification tokens of the reliable subset samples. Calculate the initial output layer prototype based on the output of the classifier using the reliable subset of samples.
[0096]
[0097]
[0098] In the formula and These are the feature layer prototype and output layer prototype of the k-th class in the target domain, respectively, where t represents the target domain, |· represents the number of elements in the set, h() represents the classifier, and c t For the corresponding target domain sample x t Encoded classification tokens;
[0099] S103.8: For the aforementioned unreliable subset The target domain samples in the data are based on the distance from the unreliable subset. The target domain samples are relabeled according to the category of the initial feature layer prototype, resulting in refined pseudo-labels.
[0100]
[0101] In the formula, For the initial pseudo-labels of the predicted reliable subset, d(·,·) is the cosine distance. This is the initial feature layer prototype for the k-th class.
[0102] S104: Encode the source domain image and simultaneously perform mask image modeling on the target domain image; the source domain image includes a label indicating the category of the source domain image.
[0103] Encoding the source domain image can be achieved using conventional methods in existing technologies, and no limitations are specified here.
[0104] Masked image modeling of the target domain image can be implemented in the following ways:
[0105] S104.1: Process the target domain image into a fourth number of 16×16 resolution patch blocks; the fourth number can be selected according to the actual situation, such as 196, 256, etc., and is not limited here.
[0106] S104.2: Randomly select a second proportion of the patch blocks for masking, where the second proportion is no greater than 20%;
[0107] S104.3: Remove the masked patch blocks and perform feature encoding on the remaining visible patch blocks to obtain encoded features;
[0108] S104.4: Restore the encoded features to the sequential positions of the remaining visible patch blocks;
[0109] S104.5: Fill in the missing positions with learnable random initialization vectors to obtain the completed encoded features;
[0110] S104.6: Input the completed coded features into the decoder to reconstruct the normalized pixel values of the input image;
[0111] The normalized pixel values of the input image can be reconstructed in the following way:
[0112] Calculate the image reconstruction loss between the reconstructed image and the masked patch pixels of the original image in the normalized pixel space.
[0113] Image reconstruction loss L rec The expression is as follows:
[0114]
[0115] Where L rec This represents the average reconstruction loss for each normalized pixel in the target domain. p represents the normalized pixel values of the target domain image. M Let M be the pixel value predicted by the autoencoder for the mask position of the target domain image, M be the set of mask patch pixels, and |·| represent the number of elements in the set.
[0116] S105: Construct prototype libraries for the source and target domains and align the prototypes for each category in the source and target domains using the prototype contrastive learning module.
[0117] like Figure 4 As shown, this step can be implemented in the following ways:
[0118] S105.1: Obtain the source domain feature layer prototype for each category based on the source domain image and its label. and the source domain output layer prototype for each category A source domain feature layer prototype library is constructed using the source domain feature layer prototypes corresponding to each category. Construct a source domain output layer prototype library using the source domain output layer prototype corresponding to each category. in:
[0119]
[0120]
[0121] In the formula, and These are the feature layer prototype and output layer prototype of the k-th class in the source domain, respectively, S k It is the set of classification tokens obtained by the encoder from source domain samples labeled as class k, where |·| represents the number of elements in the set;
[0122] S105.2: Based on the target domain image and its refined pseudo-labels, obtain the target domain feature layer prototype corresponding to each refined pseudo-label. and the target domain output layer prototype for each refined pseudo-label Construct a target domain feature layer prototype library using the target domain output layer prototype corresponding to each refined pseudo-label. Construct a target domain output layer prototype library using the target domain output layer prototype corresponding to each refined pseudo-tag.
[0123] S105.3: Dynamically maintain and adaptively update the prototype library, which includes the source domain feature layer prototype library, the source domain output layer prototype library, the target domain feature layer prototype library, and the target domain output layer prototype library;
[0124] The prototypes in the prototype library are adaptively updated by the batch prototypes in each mini-batch of training:
[0125]
[0126]
[0127]
[0128] In the formula, p k(i) For the prototype of the feature layer batch of the k-th class in the i-th mini-batch, c i The classification token is the source or target domain data obtained by the encoder in the i-th mini-batch. Let be the set of classification tokens for samples in the i-th batch whose source domain label or target domain refinement label is of class k, where |·| represents the number of elements in the set, sim(·,·) represents the cosine similarity, and P k(i) To update the feature layer prototype of the k-th class in the prototype library of the source or target domain after i updates, Pk(0) For the initial k-th type feature layer prototype in the prototype library, the adaptive update of the output layer prototype library is similar to that of the feature layer prototype library:
[0129]
[0130] β=sim(o k(i) O k(i-1) )
[0131] O k(i) =o k(i) +(1-β)O k(i-1)
[0132] In the formula, o k(i) For the output layer batch prototype of the k-th class in the i-th mini-batch, c i h() represents the classification token obtained by the encoder from the source or target domain data in the i-th mini-batch, and h() represents the classifier. Let be the set of classification tokens for samples in the i-th batch whose source domain label or target domain refinement label is of class k, where |·| represents the number of elements in the set, sim(·,·) represents the cosine similarity, and O(n) = 1. k(i) To update the feature layer prototype of the k-th class in the prototype library of the source or target domain after i updates, O k(0) This is the k-th output layer prototype in the initial prototype library.
[0133] S105.4: Calculate prototype alignment loss based on contrastive learning based on a dynamically maintained prototype library;
[0134] S105.5: Aligns the feature distributions of the source and target domains by reducing the distance between prototype features of the same class between domains and increasing the distance between prototype features of different classes between domains.
[0135] The feature layer prototype alignment loss is calculated as follows:
[0136]
[0137] The output layer prototype alignment loss is calculated as follows:
[0138]
[0139] in, P s + Represents the prototype P of the feature layer of the target domain. t k For source domain feature layers with the same k-th category, τ is a temperature hyperparameter that determines the concentration level. Represents the prototype of the output layer of the target domain. For source-output layer prototypes with the same k-th category, τ is a temperature hyperparameter that determines the concentration level.
[0140] S105.6: Summing the prototype alignment losses of the feature layer and the output layer yields the total prototype contrast loss:
[0141]
[0142] S106: Construct an overall objective loss function based on the data, and optimize the overall objective loss function to iteratively train the domain adaptation network to obtain a domain adaptation network for classification.
[0143] The data includes the source domain image, the target domain image, and the refined pseudo-labels of the target domain image.
[0144] During the iterative training of the domain adaptation network, the pseudo-label refinement strategy is executed at certain training steps to update the refined pseudo-labels. For example, the pseudo-label refinement strategy can be executed every 500 training steps to update the refined pseudo-labels. The number of training steps can be flexibly adjusted according to the actual situation and is not limited here.
[0145] The overall target loss function, constructed based on the source domain image, the target domain image, and the refined pseudo-labels of the target domain image, can be implemented in the following way:
[0146] S106.1: Constructing the source domain classification loss L src The expression is as follows:
[0147]
[0148] In the formula, n s Let H be the total number of samples in the source domain, and H(·,·) be the cross-entropy loss function. It is the classification token of the classification layer for the i-th source domain sample. The output predicted probability, Let be the true label of the i-th source domain sample, where s represents the source domain and i is a positive integer;
[0149] S106.2: Constructing the target domain pseudo-label classification loss L pse The expression is as follows:
[0150]
[0151] In the formula, n t Let H be the total number of samples in the source domain, and H(·,·) be the cross-entropy loss function. This represents the classification token of the classification layer for the j-th target domain sample. The output predicted probability, It is the refined pseudo-label of the j-th target domain sample, where t represents the target domain and j is a positive integer;
[0152] S106.3: Constructing the target domain conditional entropy loss L ent The expression is as follows:
[0153]
[0154] In the formula n t The total number of samples in the source domain. This represents the classification token of the classification layer for the j-th target domain sample. The predicted probability of the k-th class, where t represents the target domain, j is a positive integer, and k is a positive integer;
[0155] S106.4: Utilize the source domain classification loss L src The target domain pseudo-label classification loss L pse The target domain conditional entropy loss L ent The image reconstruction loss L rec Compared with the prototype, the loss L con The overall objective number L is established as follows:
[0156] L = L src +L pse +λ1L ent +λ2L rec +λ3L con
[0157] In the formula, λ1, λ2, and λ3 are hyperparameters, representing the conditional entropy loss L of the target domain, respectively. ent The prototype contrast loss L con and the image reconstruction loss L rec The weight.
[0158] S107: Input the target domain image to be tested into the domain adaptation network used for classification to obtain a classification result, wherein the scene classification result indicates the category of the target domain image to be tested.
[0159] The following experiment will verify this.
[0160] We first selected three different open-source remote sensing datasets—NWPU-RESISC45, AID, and UC Merced Land-Use—to construct a cross-domain scene classification task. We constructed six cross-scene tasks by manually observing and extracting common categories between pairs of datasets: NWPU→AID, AID→NWPU, NWPU→UCM, UCM→NWPU, AID→UCM, and UCM→AID. The method proposed in this invention is implemented using the PyTorch library in Python; other methods can also be used to implement this scheme, and no limitation is made here. Furthermore, we conducted experimental evaluations on the constructed cross-domain scene classification tasks using three other domain adaptation methods: Domain Adversarial Neural Network (DANN), Joint Adaptive Network (JAN), and Domain-Oriented Transformer (DOT), to compare with the cross-domain remote sensing scene classification method of this invention, which uses masked image modeling-assisted prototype contrastive learning. This invention evaluates each cross-domain scene classification task using overall classification accuracy and calculates the average of the overall classification accuracy of the six tasks as the overall evaluation index. The experimental results are shown in Table 2 below, expressed as percentages.
[0161] Table 2. Evaluation of classification accuracy using different methods
[0162]
[0163]
[0164] As shown in Table 2, the method of this invention achieved the highest accuracy on all cross-domain scene classification tasks. This indicates that the invention can learn domain-specific information of the target domain by performing mask image modeling on the target domain image through an autoencoder, and improve the separability of categories by aligning the prototypes of each category in the source and target domains through a pseudo-label refinement strategy and a prototype contrast learning module. At the same time, it enhances the generalization ability of the model and achieves high-precision scene classification of the target domain image.
[0165] Compared with the prior art, the beneficial effects of the present invention are:
[0166] (1) This invention combines prototype and contrastive learning and closely links them with domain adaptation. Prototypes are calculated by using pseudo-labels of target samples, and contrastive learning is used to reduce the prototype feature distance of the same category between domains and to increase the prototype feature distance of different categories between domains to align the category distribution of the source domain and the target domain. At the same time, mask image modeling is used to learn the domain-specific information of the target domain to assist in cross-domain scene classification, which can improve the quality of pseudo-labels of the target domain, thereby constructing a better prototype library, thereby improving prototype contrast performance and promoting category alignment.
[0167] (2) The present invention uses a pseudo-label refinement strategy to generate more accurate pseudo-labels, and uses prototypes to generate more generalizable representations, which can further suppress the negative impact of erroneous pseudo-label samples.
[0168] (3) The present invention establishes a global prototype library and maintains it dynamically. The global prototype is adaptively updated by the batch prototype in each small batch, which ensures that the prototype has good global semantics and generates more representative category features.
[0169] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A cross-domain remote sensing image classification method based on prototype contrastive learning, characterized in that, Includes the following steps: S101: Construct a domain adaptation network, which includes an autoencoder and a prototype contrastive learning module; S102: Perform unlabeled self-supervised pre-training on the autoencoder to obtain the pre-trained autoencoder model parameters, and load the pre-trained autoencoder model parameters into the autoencoder; S103: Input the target domain image into the autoencoder and use a pseudo-label thinning strategy to obtain a thinned pseudo-label for the target domain image; wherein, the target domain image is an unlabeled target domain image; S104: Encode the source domain image and simultaneously perform mask image modeling on the target domain image; the source domain image includes a label, which indicates the category of the source domain image; S105: Construct prototype libraries for the source and target domains and align the prototypes of each category in the source and target domains using the prototype contrastive learning module; specifically including: S105.1: Obtain the source domain feature layer prototype for each category based on the source domain image and its label. and the source domain output layer prototype for each category A source domain feature layer prototype library is constructed using the source domain feature layer prototypes corresponding to each category. A source domain output layer prototype library is constructed using the source domain output layer prototype corresponding to each category. ,in: ， In the formula, and They are the source domain number The feature layer prototype and output layer prototype of the class It is tagged as the first The set of classification tokens obtained by the autoencoder from the source domain samples of the class. c represents the number of elements in the set. s It is the source domain sample x t The encoded classification token, h() is the classifier; S105.2: Based on the target domain image and its refined pseudo-labels, obtain the target domain feature layer prototype corresponding to each refined pseudo-label. and the target domain output layer prototype for each refined pseudo-label A target domain feature layer prototype library is constructed using the target domain output layer prototype corresponding to each refined pseudo-label. A target domain output layer prototype library is constructed using the target domain output layer prototype corresponding to each refined pseudo-tag. ; S105.3: Dynamically maintain and adaptively update the prototype library, which includes the source domain feature layer prototype library, the source domain output layer prototype library, the target domain feature layer prototype library, and the target domain output layer prototype library; The prototypes in the prototype library are adaptively updated by the batch prototypes in each mini-batch of training: In the formula, For the first The first in the small batch Batch prototype of the class's feature layer For the first The classification tokens obtained by the encoder from a small batch of source or target domain data. For the first In each batch, the source domain label or target domain refined label is the first... A set of classification tokens for samples of a class. Indicates the number of elements in the set. Represents cosine similarity. For updates The next source domain or target domain in the prototype library Class feature layer prototype, For the first prototype library in the initial stage The adaptive update of the output layer prototype library is similar to that of the feature layer prototype library: In the formula, For the first The first in the small batch The output layer of the class is a batch prototype. For the first The classification tokens obtained by the encoder from a small batch of source or target domain data. This refers to the classifier. For the first In each batch, the source domain label or target domain refined label is the first... A set of classification tokens for samples of a class. Indicates the number of elements in the set. Represents cosine similarity. For updates The next source domain or target domain in the prototype library Class feature layer prototype, For the first prototype library in the initial stage Class output layer prototype; S105.4: Calculate prototype alignment loss based on contrastive learning based on a dynamically maintained prototype library; S105.5: Aligns the feature distributions of the source and target domains by reducing the distance between prototype features of the same class between domains and increasing the distance between prototype features of different classes between domains. The feature layer prototype alignment loss is calculated as follows: The output layer prototype alignment loss is calculated as follows: in, , Representation of the prototype of the target domain feature layer Having the same first Prototypes of source domain feature layers for each category, It is the temperature hyperparameter that determines the concentration level. , Represents the prototype of the output layer of the target domain. Having the same first Source domain output layer prototypes for each category, It is the temperature hyperparameter that determines the concentration level; S105.6: Summing the prototype alignment losses of the feature layer and the output layer yields the total prototype contrast loss: ； S106: Construct an overall objective loss function based on the data, and optimize the overall objective loss function to iteratively train the domain adaptation network to obtain a domain adaptation network for classification; wherein, the data includes the source domain image, the target domain image, and the refined pseudo-labels of the target domain image; S107: Input the target domain image to be tested into the domain adaptation network used for classification to obtain a classification result. The scene classification result indicates the category of the target domain image to be tested.
2. The method according to claim 1, characterized in that, The autoencoder is based on Transformer and includes a decoder and an encoder. The encoder adopts a TransformerBlock network structure with a first number of layers and a multi-head attention mechanism; the decoder adopts a TransformerBlock network structure with a second number of layers and a multi-head attention mechanism.
3. The method according to claim 1, characterized in that, The self-supervised pre-training is performed on the target domain image using a masked image modeling method.
4. The method according to claim 1, characterized in that, The method of obtaining thinned pseudo-labels for the target domain image using the pseudo-label thinning strategy includes: S103.1: The target domain image is encoded to obtain encoded features, wherein the encoded features include at least a classification token. The classification token is input into a classifier to obtain initial pseudo-labels. , in, , =1,2,…, ; Indicates the first There are 10 categories, with a total of 100 categories. Categories Indicates a category token. Represents a classifier; S103.2: Calculate the initial class center for each category based on the initial pseudo-labels. , Indicates the first The initial class center; S103.3: Obtain the distance of each category based on the initial class centers. , ,in It is the cosine distance. Is given the first Classification tokens for each target domain sample; S103.4: Distance to each category using a Gaussian uniform mixture model The distribution is modeled and defined as follows: in, This represents the probability distribution of the Gaussian uniform mixture model. Indicates a Gaussian distribution. It is the variance of the Gaussian distribution. It is in the interval The uniform distribution defined above, This represents the amplitude coefficient of the Gaussian distribution; S103.5: Using the Gaussian uniform mixture model, label reliable and unreliable target domain data, and calculate the reliability G of the target domain data using the following formula: ； S103.6: Based on the reliability G and the threshold The target domain data is divided into a reliable subset and an unreliable subset, where the reliable subset is represented as... The remaining target domain samples are then classified into the unreliable subset. ,in ,use Represents all initial pseudo-tags Target domain samples, Represents the target domain sample; S103.7: For the target domain data in the reliable subset, the initial pseudo-label is directly used. As a refined pseudo-label, reliable class centers, i.e., the initial feature layer prototypes, are calculated based on the classification tokens of the reliable subset samples. The initial output layer prototype is calculated at the classifier output based on the reliable subset of samples. : ， ， In the formula and These are the target domains. The feature layer prototype and output layer prototype of the class Represents the target domain. Indicates the number of elements in the set. This refers to the classifier. For the corresponding target domain samples Encoded classification tokens; S103.8: For the aforementioned unreliable subset The target domain samples in the data are based on the distance from the unreliable subset. The target domain samples are relabeled according to the category to which the initial feature layer prototype belongs, resulting in refined pseudo-labels. : ， In the formula The initial pseudo-labels for the predicted reliable subset, It is the cosine distance. For the first The initial feature layer prototype of the class.
5. The method according to claim 2, characterized in that, The masked image modeling of the target domain image includes: S104.1: Process the target domain image into a fourth number of 16×16 resolution patch blocks; S104.2: Randomly select a second proportion of the patch blocks for masking, where the second proportion is no greater than 20%; S104.3: Remove the masked patch blocks and perform feature encoding on the remaining visible patch blocks to obtain encoded features; S104.4: Restore the encoded features to the sequential positions of the remaining visible patch blocks; S104.5: Fill in the missing positions with learnable random initialization vectors to obtain the completed encoded features; S104.6: Input the completed coded features into the decoder to reconstruct the normalized pixel values of the input image.
6. The method according to claim 3 or 5, characterized in that, The normalized pixel values of the reconstructed input image include: Calculate the image reconstruction loss between the reconstructed image and the masked patch pixels of the original image in the normalized pixel space, the image reconstruction loss The expression is as follows: For the normalized pixel values of the target domain image, The pixel value predicted by the autoencoder for the masking position of the target domain image. A collection of pixels for the mask patch block. This indicates the number of elements in the set.
7. The method according to claim 1, characterized in that, During the iterative training of the domain adaptation network, the pseudo-label refinement strategy is executed at regular intervals to update the refined pseudo-labels.
8. The method according to claim 1, characterized in that, The overall objective loss function is constructed based on the data. include: S106.1: Constructing the source domain classification loss The expression is as follows: ， In the formula The total number of samples in the source domain. It is the cross-entropy loss function. The classification layer is the first Classification tokens for each source domain sample The output predicted probability, For the first The true labels of the source domain samples Represents the source domain. It is a positive integer; S106.2: Constructing the target domain pseudo-label classification loss The expression is as follows: In the formula The total number of samples in the source domain. It is the cross-entropy loss function. This indicates that the classification layer represents the first input. Classification tokens for each target domain sample The output predicted probability, It is the first Refined pseudo-labels for each target domain sample. Represents the target domain. It is a positive integer; S106.3: Constructing the target domain conditional entropy loss The expression is as follows: , In the formula The total number of samples in the source domain. This indicates that the classification layer represents the first input. Classification tokens for each target domain sample The predicted first The probability of a class Represents the target domain. It is a positive integer. It is a positive integer; S106.4: Utilize the source domain classification loss The target domain pseudo-label classification loss The target domain conditional entropy loss The image reconstruction loss Loss compared with the prototype Establish the overall target loss The expression is as follows: In the formula, These are hyperparameters, representing the conditional entropy loss of the target domain, respectively. The prototype contrast loss and image reconstruction loss The weight.