A cross-domain remote sensing scene classification method of mask image modeling guided domain adaptation

Abstract

The application discloses a kind of mask image modeling guide domain adaptation cross-domain remote sensing scene classification method, comprising: constructing domain adaptation network, the self-encoding of unlabelled self-supervised pre-training is carried out, and the model parameter of pre-training self-encoding is obtained;Self-encoding model parameters are loaded into self-encoder, data are input into domain adaptation network, and the mask image modeling of source domain image and target domain image is carried out using self-encoder;High-level semantic feature distribution of source domain and target domain is aligned using feature adaptation module;And based on data, overall target loss function is constructed, and the iterative training of domain adaptation network is carried out by optimizing overall target loss function, the decoder part of self-encoder is removed, the encoder of self-encoder and feature adaptation module are used to test target domain image, and good scene classification result is obtained.The application preserves domain-specific features in the process of extracting domain-invariant features, and further improves the classification generalization ability for unlabelled data target domain.

Description

Technical Field

[0001] This invention relates to the field of optical remote sensing image scene classification technology, specifically to a cross-domain remote sensing scene classification method guided by mask image modeling and domain adaptation. Background Technology

[0002] Remote sensing image scene classification refers to the semantic labeling of remote sensing scenes based on their content. This enables the interpretation of scene information from massive amounts of remote sensing data and is widely applied in fields such as land resource surveys, urban planning, agricultural assessment, and forest monitoring. However, in practical applications, images generated by different types of satellites are affected by various factors such as sensor type, geographical location, lighting conditions, and seasonal changes, resulting in significant differences in data distribution across different payloads. For example, different sensors lead to variations in image resolution and saturation, resulting in stylistic differences in the images; areas with good lighting conditions have high image brightness, while areas with poor lighting conditions have low image brightness. Deep learning-based scene classification models trained on one dataset and then tested on another dataset with significantly different distributions show a significant performance drop. Therefore, models trained on labeled data (source domain) are difficult to directly apply to unlabeled data (target domain) with domain differences. While large-scale manual data annotation can ensure the algorithm's classification ability for unlabeled data with significant distribution differences (target domain), large-scale, high-quality manual annotation would be extremely costly in terms of manpower and resources, especially given the explosive growth of remote sensing Earth observation data. At the same time, it inevitably introduces errors from manual annotation, leading to a decline in the performance of the trained model. Therefore, improving the model's generalization performance on unlabeled target domain data with large distribution differences is key to further enhancing the intelligent interpretation capabilities of remote sensing data. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides a cross-domain remote sensing scene classification method guided by mask image modeling and domain adaptation. This method solves the problem that existing metric- or adversarial cross-domain scene classification methods lose domain-specific information when extracting domain-invariant features, resulting in poor classification ability of the learned features for the target domain.

[0004] To achieve the objectives of this invention, the following technical solution is adopted:

[0005] This invention provides a cross-domain remote sensing scene classification method guided by masked image modeling, comprising the following steps: constructing a domain adaptation network, the domain adaptation network including an autoencoder and a feature adaptation module; wherein, the autoencoder includes a decoder and an encoder; performing unlabeled self-supervised pre-training on the autoencoder to obtain pre-trained autoencoder model parameters; wherein, the self-supervised pre-training is performed on source and target domain images using masked image modeling; the autoencoder model parameters include at least weight parameters; the source domain image is an unlabeled source domain image, and the target domain image is an unlabeled target domain image; loading the autoencoder model parameters into the autoencoder, inputting data into the domain adaptation network, using the autoencoder to perform masked image modeling on the source and target domain images, and using the feature adaptation module to align the high-level semantic feature distributions of the source and target domains; wherein, the data includes the source domain image and its annotations and the target domain image; and constructing an overall target loss function based on the data, and optimizing the overall target loss function to iteratively train the domain adaptation network.

[0006] Furthermore, the decoder is removed, and the target domain image is tested using the encoder and the feature adaptation module to obtain a scene classification result, which indicates the scene category of the target domain image.

[0007] Furthermore, the autoencoder is based on Transformer, wherein the encoder of the autoencoder adopts a network structure of Transformer Blocks with a first number of layers and a multi-head attention mechanism; and the decoder of the autoencoder adopts a network structure of Transformer Blocks with a second number of layers and a multi-head attention mechanism.

[0008] Further, the unlabeled self-supervised pre-training includes: processing the source domain image and the target domain image into a third number of 16×16 resolution patches; randomly selecting a first proportion of the patches for masking; removing the masked patches and inputting the remaining visible patches into the encoder to obtain encoded features; restoring the encoded features to the sequential positions of the remaining visible patches; supplementing the missing positions with learnable random initialization vectors to obtain completed encoded features; and inputting the completed encoded features into the decoder to reconstruct the normalized pixel values ​​of the input image.

[0009] Further, the masked image modeling using the autoencoder for the source and target domain images includes: processing the source and target domain images into a fourth number of 16×16 resolution patches; randomly selecting a second proportion of the patches for masking, where the second proportion is no greater than 20%; removing the masked patches and inputting the remaining visible patches into the autoencoder to obtain encoded features; restoring the encoded features to the sequential positions of the remaining visible patches; supplementing the missing positions with learnable random initialization vectors to obtain completed encoded features; and inputting the completed encoded features into the decoder to reconstruct the normalized pixel values ​​of the input image.

[0010] Further, the normalized pixel values ​​of the reconstructed input image include: calculating the mean square error between the masked patch pixels of the reconstructed image and the original image in the normalized pixel space, and the image reconstruction loss. The expression is as follows:

[0011]

[0012] In the formula n represents the average reconstruction loss for each normalized pixel in the source and target domains. s and n t These represent the total number of samples in the source and target domains, respectively. and p represents the normalized pixel values ​​of the i-th source domain image and the j-th target domain image, respectively. s and p t , respectively, are the pixel values ​​predicted by the autoencoder for the source domain image and the target domain image, M is the set of pixels of the mask patch block, |·| represents the number of elements in the set, s represents the source domain, t represents the target domain, i is a positive integer, and j is a positive integer.

[0013] Further, the step of aligning the high-level semantic feature distributions of the source and target domains using the feature adaptation module includes: obtaining a classification token from the encoded features of the source and target domain data through the encoder; inputting the classification token as a high-level semantic feature into the feature adaptation module; wherein the feature adaptation module includes a first feature alignment block and a second feature alignment block, the first feature alignment block being before the classification layer and the second feature alignment block being after the classification layer; and calculating the feature alignment loss based on the source and target domain data. and regularization loss

[0014] Furthermore, the feature alignment loss is calculated based on the source domain data and the target domain data. Includes: the feature alignment loss The sum of the maximum mean differences between the source and target domain features on the first and second feature alignment blocks is expressed as follows:

[0015]

[0016] In the formula n s and n t Let l represent the total number of samples in the source domain and the target domain, respectively, and l = 1, 2. For the i-th source domain sample, the source domain high-level semantic features are aligned to the l-th feature block. It is the high-level semantic feature of the target domain after the j-th target domain sample is aligned with the l-th feature block. It has a characteristic kernel The reproducing kernel Hilbert space, φ(·) is the corresponding eigenmap, and MMD is the maximum mean difference. The source domain high-level semantic features are used to align the target domain sample with the l-th feature block. Let s represent the high-level semantic features of the target domain after aligning the block with the l-th feature, where s represents the source domain, t represents the target domain, i is a positive integer, and j is a positive integer.

[0017] Furthermore, the regularization loss is calculated based on the source domain data and the target domain data. The process includes: selecting a random subset of the source domain data, forcing the random subset to pass through the target domain path of the feature adaptation module; and calculating the sum of the maximum mean differences between each class in the source domain and this random subset on the first feature alignment block and the second feature alignment block, as shown in the following expression:

[0018]

[0019] In the formula, l = 1, 2, It is a set of high-level semantic features of a random subset of samples from the source domain, C n S represents the total number of categories. k This represents the set of high-level semantic features of the k-th class of the source domain samples. It is the source domain high-level semantic feature of the i-th source domain sample through the l-th feature alignment block. It is the source domain high-level semantic feature of the j-th sample in a random subset of the source domain aligned to the target domain path of the block by the l-th feature. It has a characteristic kernel The reproducing kernel Hilbert space, φ(·) is the corresponding eigenmap, |·| represents the number of elements in the set, and MMD is the maximum mean difference. The source domain high-level semantic features are used to align the target domain sample with the l-th feature block. The high-level semantic features are the random subset of the source domain samples that are aligned to the target domain by the l-th feature alignment module. s represents the source domain, t represents the target domain, i is a positive integer, j is a positive integer, and k is a positive integer.

[0020] Furthermore, the overall objective loss function is constructed based on the data. Includes: constructing the source domain classification loss The expression is as follows: In the formula n s Let H be the total number of samples in the source domain, and H(·,·) be the cross-entropy loss function. It is the high-level semantic feature of the i-th source domain sample by the feature adaptation module. The predicted probability Let be the true label of the i-th source domain sample, where s represents the source domain and i is a positive integer;

[0021] Constructing the target domain conditional entropy loss The expression is as follows: In the formula n t The total number of samples in the source domain. This represents the high-level semantic features of the j-th target domain sample input by the feature adaptation module. The predicted probability of the k-th class, where t represents the target domain, j is a positive integer, and k is a positive integer;

[0022] Using the source domain classification loss The target domain conditional entropy loss The image reconstruction loss The feature alignment loss and the regularization loss Establish the overall objective loss function The expression is as follows:

[0023]

[0024] In the formula, α, β, and γ are hyperparameters, representing the feature alignment loss. The regularization loss The target domain conditional entropy loss and the image reconstruction loss The weight.

[0025] This invention introduces random perturbations into the feature mapping distribution of the source and target domain images by using a random masking method with a small masking ratio. This enriches the mapping description space of domain-invariant features. At the same time, through the reconstruction task of the patch blocks of the randomly masked image, the constructed domain-invariant features also contain specific information of the two domains. This allows domain-specific features to be preserved during the extraction of domain-invariant features, thereby further improving the classification and generalization ability for the target domain of unlabeled data.

[0026] It should be understood that the content described in this summary section is not intended to limit the key or essential features of the embodiments of the invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0027] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0028] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0029] Figure 2 This is an overall schematic diagram of the domain adaptation network of the present invention;

[0030] Figure 3 A schematic diagram illustrating the mask image modeling of this invention;

[0031] Figure 4 This is a schematic diagram of the feature adaptation module of the present invention aligning the feature distribution. Detailed Implementation

[0032] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0033] In the description of embodiments of the present invention, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0034] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and descriptions are considered exemplary in nature and not restrictive. There is no requirement for the order of the method steps described, as long as they are achievable, they are within the scope of protection of this invention.

[0035] like Figure 1-4 As shown, a cross-domain remote sensing scene classification method guided by mask image modeling includes: constructing a domain adaptation network, wherein the domain adaptation network includes an autoencoder and a feature adaptation module;

[0036] The autoencoder is subjected to unlabeled self-supervised pre-training to obtain the pre-trained autoencoder model parameters;

[0037] The self-supervised pre-training is performed on the source domain image and the target domain image using a masked image modeling method; the autoencoder model parameters include at least weight parameters; the source domain image is an unlabeled source domain image, and the target domain image is an unlabeled target domain image; the source domain image and the target domain image are acquired from remote sensing satellite images or other images.

[0038] The Transformer-based autoencoder includes an encoder and a decoder. The encoder employs a network structure with a first number of Transformer Blocks having a multi-head attention mechanism; the decoder employs a network structure with a second number of Transformer Blocks having a multi-head attention mechanism. An example of the first number is 12, but in other embodiments it can be 24, and the present invention is not limited in this respect. An example of the second number is 8, but in other embodiments it can be 12, and the present invention is not limited in this respect.

[0039] Load the autoencoder model parameters into the autoencoder;

[0040] The data is input into the domain adaptation network, and the source domain image and the target domain image are modeled using an autoencoder. The high-level semantic feature distributions of the source domain and the target domain are aligned using the feature adaptation module. The data includes the source domain image, as well as the annotations and the target domain image.

[0041] Construct an overall objective loss function based on the data;

[0042] The domain adaptation network is iteratively trained by optimizing the overall objective loss function.

[0043] Remove the decoder, and use the encoder and the feature adaptation module to test the target domain image to obtain the scene classification result, which indicates the scene category of the target domain image.

[0044] In some embodiments, the unlabeled self-supervised pre-training includes: processing the source domain image and the target domain image into a third number of 16×16 resolution patches; randomly selecting a first proportion of the patches for masking; removing the masked patches and inputting the remaining visible patches into the encoder to obtain encoded features; restoring the encoded features to the sequential positions of the remaining visible patches; supplementing the missing positions with learnable random initialization vectors to obtain completed encoded features; and inputting the completed encoded features into the decoder to reconstruct the normalized pixel values ​​of the input image.

[0045] In some embodiments, the source and target domain images are processed into 196 16×16 resolution patch blocks. 147 of these, or 75%, are randomly selected for masking, and the masked patch blocks are removed. The remaining visible patch blocks are input into the encoder of the autoencoder to obtain encoded features. The encoded features are then restored to the corresponding original patch block sequence positions, and learnable random initialization vectors are added to the missing positions. The completed encoded features are input into the decoder of the autoencoder to reconstruct the normalized pixel values ​​of the input image.

[0046] In some embodiments, the source and target domain images can be processed into 196 16×16 resolution patch blocks. 137 of these, or 70%, are randomly selected for masking, and the masked patch blocks are removed. The remaining visible patch blocks are input into the encoder of the autoencoder to obtain encoded features. The encoded features are then restored to the corresponding original patch block sequence positions, and learnable random initialization vectors are added to the missing positions. The completed encoded features are input into the decoder of the autoencoder to reconstruct the normalized pixel values ​​of the input image.

[0047] In some embodiments, the source and target domain images can be processed into 256 16×16 resolution patch blocks. 166 of these, or 65%, are randomly selected for masking, and the masked patch blocks are removed. The remaining visible patch blocks are input into the encoder of the autoencoder to obtain encoded features. The encoded features are then restored to the corresponding original patch block sequence positions, and learnable random initialization vectors are added to the missing positions. The completed encoded features are input into the decoder of the autoencoder to reconstruct the normalized pixel values ​​of the input image.

[0048] It should be understood that the above embodiments are merely exemplary and should not be construed as limiting the present invention.

[0049] In some embodiments, such as Figure 3 As shown, the masked image modeling of the source and target domain images using the autoencoder includes: processing the source and target domain images into a fourth number of 16×16 resolution patches; randomly selecting a second proportion of the patches for masking, where the second proportion is no greater than 20%; removing the masked patches and inputting the remaining visible patches into the autoencoder to obtain encoded features; restoring the encoded features to the sequential positions of the remaining visible patches; supplementing the missing positions with learnable random initialization vectors to obtain completed encoded features; and inputting the completed encoded features into the decoder to reconstruct the normalized pixel values ​​of the input image.

[0050] In some embodiments, the source and target domain images are processed into 196 16×16 resolution patch blocks, and 20 of them, i.e., 10%, are randomly selected for masking. The masked patch blocks are then removed, and the remaining visible patch blocks are input into the encoder of the autoencoder to obtain encoded features. The encoded features are then restored to the corresponding original patch block sequence positions, and learnable random initialization vectors are added to the missing positions. The completed encoded features are input into the decoder of the autoencoder to reconstruct the normalized pixel values ​​of the input image.

[0051] In some embodiments, the source and target domain images can be processed into 196 16×16 resolution patch blocks, and 10 of them, i.e. 5%, are randomly selected for masking. The masked patch blocks are then removed, and the remaining visible patch blocks are input into the encoder of the autoencoder to obtain encoded features. The encoded features are then restored to the corresponding original patch block sequence positions, and learnable random initialization vectors are added to the missing positions. The completed encoded features are then input into the decoder of the autoencoder to reconstruct the normalized pixel values ​​of the input image.

[0052] In some embodiments, the source and target domain images can be processed into 256 16×16 resolution patch blocks, and 38 of them, i.e. 15%, are randomly selected for masking. The masked patch blocks are then removed, and the remaining visible patch blocks are input into the encoder of the autoencoder to obtain encoded features. The encoded features are then restored to the corresponding original patch block sequence positions, and learnable random initialization vectors are added to the missing positions. The completed encoded features are then input into the decoder of the autoencoder to reconstruct the normalized pixel values ​​of the input image.

[0053] By randomly masking the image with a small masking ratio of no more than 20%, random perturbation is added to the feature mapping distribution of the source and target domain images, enriching the mapping description space of domain-invariant features. At the same time, through the reconstruction task of patch blocks of randomly masked images, the constructed domain-invariant features also contain specific information of the two domains, improving the classification ability of the target domain image.

[0054] It should be understood that the above embodiments are merely exemplary and should not be construed as limiting the present invention.

[0055] In some embodiments, the normalized pixel values ​​of the reconstructed input image include: calculating the mean square error between the reconstructed image and the original image in the normalized pixel space, and only calculating the loss of the masked patch pixels, resulting in image reconstruction loss. The expression is as follows:

[0056]

[0057] In formula (1) n represents the average reconstruction loss for each normalized pixel in the source and target domains. s and n t These represent the total number of samples in the source and target domains, respectively. and p represents the normalized pixel values ​​of the i-th source domain image and the j-th target domain image, respectively. s and p t Let be the pixel values ​​predicted by the autoencoder for the source and target domain images, M be the set of mask patch pixels, |·| represent the number of elements in the set, s represent the source domain, t represent the target domain, i is a positive integer, and j is a positive integer.

[0058] In some embodiments, such as Figure 4As shown, aligning the high-level semantic feature distributions of the source and target domains using the feature adaptation module includes: obtaining classification tokens from the encoded features of source and target domain data through an encoder; inputting the classification tokens as high-level semantic features into the feature adaptation module; the feature adaptation module includes a first feature alignment block and a second feature alignment block; the first feature alignment block is deployed before the classification layer, and the second feature alignment block is deployed after the classification layer. The first feature alignment block consists of a fully connected layer, a ReLU activation function, and a fully connected layer in sequence, with residual connections between the two fully connected layers; and only target domain data passes through these additional layers, while source domain data flows directly into the classification layer. The second feature alignment block consists of a fully connected layer, a ReLU activation function, and a fully connected layer in sequence, with residual connections between the two fully connected layers; and only target domain data passes through these additional layers; a softmax layer follows the additional layers. The feature alignment loss is calculated based on the source and target domain data. and regularization loss

[0059] Wherein, the feature alignment loss The sum of the maximum mean differences between the source and target domain features on the first and second feature alignment blocks is expressed as follows:

[0060]

[0061] In equation (2) n s and n t These represent the total number of samples in the source and target domains, respectively. For the i-th source domain sample, the source domain high-level semantic features are aligned to the l-th feature block. It is the high-level semantic feature of the target domain of the j-th target domain sample through the target domain high-level semantic feature of the l-th feature alignment block additional layer. It has a characteristic kernel The reproducing kernel Hilbert space, φ(·) is the corresponding eigenmap, and MMD is the maximum mean difference. The source domain high-level semantic features are used to align the target domain sample with the l-th feature block. Let s represent the high-level semantic features of the target domain after aligning the block with the l-th feature, where s represents the source domain, t represents the target domain, i is a positive integer, and j is a positive integer.

[0062] Among them, the regularization loss is calculated. The specific method includes: to prevent excessive alignment between the source domain data and the target domain data from destroying the domain structure information, a random subset of the source domain data is selected, forced to pass through the target domain path of the additional layer of the feature alignment module, and the sum of the maximum mean differences between each class in the source domain and this random subset on the two feature alignment blocks is calculated, as shown in the following expression:

[0063]

[0064] In equation (3), l = 1, 2, It is a set of high-level semantic features of a random subset of samples from the source domain, C n S represents the total number of categories. k This represents the set of high-level semantic features of the k-th class of the source domain samples. It is the source domain high-level semantic feature of the i-th source domain sample through the l-th feature alignment block. It is the source domain high-level semantic feature of the target domain path of the j-th sample in the random subset of the source domain through the l-th feature alignment block additional layer. It has a characteristic kernel The reproducing kernel Hilbert space, φ(·) is the corresponding eigenmap, |·| represents the number of elements in the set, and MMD is the maximum mean difference. The source domain high-level semantic features are used to align the target domain sample with the l-th feature block. The high-level semantic features are the random subset of the source domain samples that are aligned to the target domain by the l-th feature alignment module. s represents the source domain, t represents the target domain, i is a positive integer, j is a positive integer, and k is a positive integer.

[0065] In some embodiments, constructing the overall objective loss function based on the data includes:

[0066] Construct source domain classification loss The expression is as follows:

[0067]

[0068] In equation (4), n s Let H be the total number of samples in the source domain, and H(·,·) be the cross-entropy loss function. It is the high-level semantic feature of the i-th source domain sample by the feature adaptation module. The predicted probability Let be the true label of the i-th source domain sample, where s represents the source domain and i is a positive integer;

[0069] Constructing the target domain conditional entropy loss The expression is as follows:

[0070]

[0071] In equation (5) n t The total number of samples in the source domain. This represents the high-level semantic features of the j-th target domain sample input by the feature adaptation module. The predicted probability of the k-th class, where t represents the target domain, j is a positive integer, and k is a positive integer.

[0072] Using the source domain classification loss The target domain conditional entropy loss The image reconstruction loss The feature alignment loss and the regularization loss Establish the overall objective loss function The expression is as follows:

[0073]

[0074] In equation (6), α, β, and γ are hyperparameters, representing the feature alignment loss. The regularization loss The target domain conditional entropy loss and the image reconstruction loss The weight.

[0075] The following experiment will verify this.

[0076] We first selected three different open-source remote sensing datasets to construct the remote sensing cross-domain scene classification task: the NWPU-RESISC45 dataset, the AID dataset, and the UC Merced Land-Use dataset. We constructed six cross-scene tasks, named NWPU→AID, AID→NWPU, NWPU→UCM, UCM→NWPU, AID→UCM, and UCM→AID, by visually extracting common categories between each dataset. Table 1 lists the number of labeled source domain images and unlabeled target domain images used for each cross-domain scene classification task, as well as the number of common categories.

[0077] Table 1. Cross-Domain Scenario Classification Task Data Table

[0078] Cross-domain scenario classification task Number of source domain images Number of target domain images Total number of categories NWPU→AID 16100 7740 23 AID→NWPU 7740 16100 23 NWPU→UCM 14000 2000 20 UCM→NWPU 2000 14000 20 AID→UCM 4560 1300 13 UCM→AID 1300 4560 13

[0079] The method proposed in this invention is implemented using the PyTorch library in Python. Other methods can also be used to implement this scheme, and no limitation is made here. Furthermore, we conducted experimental evaluations on the constructed cross-domain scene classification task using other domain adaptation methods, including Deep Domain Adaptation Network (DAN), Adversarial Discriminative Domain Adaptation (ADDA), Conditional Domain Adversarial Network (CDAN), Adversarial Tight Matching (ATM), Attention-Based Multi-Scale Residual Adaptation Network (AMRAN), and Cross-Domain Transformer (CDTrans), to compare it with the cross-domain remote sensing scene classification method based on mask image modeling guided domain adaptation proposed in this invention. This invention evaluates each cross-domain scene classification task using overall classification accuracy and calculates the average of the overall classification accuracy for the six tasks as the overall evaluation index. The experimental results are shown in Table 2 below, with results expressed as percentages.

[0080] Table 2. Evaluation of classification accuracy using different methods

[0081]

[0082] As shown in Table 2, the method of this invention achieves the highest accuracy in every cross-domain scene classification task, and its overall performance is superior to the existing technologies. This indicates that the present invention can learn domain-specific features of low-level textures by performing mask image modeling on source and target domain images through an autoencoder, and extract domain-invariant features by aligning the distribution of high-level semantic features of the source and target domains through a feature adaptation module. The mask image modeling-guided domain adaptation framework constructed by the two can improve the generalization ability and feature discriminativeness of the model, and achieve accurate scene classification of target domain samples.

[0083] Compared with the prior art, the beneficial effects of the present invention are:

[0084] (1) By randomly masking the source domain image and the target domain image and reconstructing the source domain image and the target domain image, the present invention can reveal potential global features and effectively capture the domain-specific feature information contained in the underlying texture and structure.

[0085] (2) This invention constructs an end-to-end masked image modeling-guided domain adaptation framework through the collaboration of an autoencoder and a feature adaptation module. During the domain alignment process, a random masking image reconstruction task is introduced. By randomly masking the image with a small masking ratio, random perturbation is added to the feature mapping distribution of the source and target domain images, which enriches the mapping description space of the domain invariant features. This allows for more robust acquisition of the domain invariant description, enabling data knowledge to be transferred more robustly from the source domain to the target domain. At the same time, through the reconstruction task of the random masked image patch, the constructed domain invariant features also contain specific information of the two domains, improving the classification ability for unlabeled data.

[0086] (3) This invention improves the classification capability of unlabeled remote sensing scene data based on the Transformer model architecture.

[0087] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-domain remote sensing scene classification method guided by masked image modeling and domain adaptation, characterized in that, Includes the following steps: Construct a domain adaptation network, which includes an autoencoder and a feature adaptation module; wherein the autoencoder includes a decoder and an encoder; The autoencoder is subjected to unlabeled self-supervised pre-training to obtain the pre-trained autoencoder model parameters; wherein, the self-supervised pre-training is performed on the source domain image and the target domain image using mask image modeling; the autoencoder model parameters include at least weight parameters; the source domain image is an unlabeled source domain image, and the target domain image is an unlabeled target domain image. The autoencoder model parameters are loaded into the autoencoder, and the data is input into the domain adaptation network. The autoencoder is used to perform mask image modeling on the source domain image and the target domain image. The feature adaptation module is used to align the high-level semantic feature distributions of the source domain and the target domain. The data includes the source domain image and its annotation and the target domain image. as well as Based on the data, an overall objective loss function is constructed, and the overall objective loss function is optimized to iteratively train the domain adaptation network; The method of masking image modeling for the source and target domain images using the autoencoder includes: The source domain image and the target domain image are processed into a fourth number of 16×16 resolution patch blocks; Randomly select a second proportion of the patch blocks for masking, where the second proportion is no greater than 20%; Remove the masked patch blocks and input the remaining visible patch blocks into the autoencoder to obtain the encoded features; Restore the encoded features to the sequential positions of the remaining visible patch blocks; By filling in the missing positions with learnable, randomly initialized vectors, the completed encoded features are obtained; and The completed encoded features are input into the decoder to reconstruct the normalized pixel values ​​of the input image; The normalized pixel values ​​of the reconstructed input image include: Calculate the mean square error between the reconstructed image and the masked patch pixels of the original image in the normalized pixel space. Image reconstruction loss The expression is as follows: in, n represents the average reconstruction loss for each normalized pixel in the source and target domains. s and n t These represent the total number of samples in the source and target domains, respectively. and p represents the normalized pixel values ​​of the i-th source domain image and the j-th target domain image, respectively. s and p t , respectively, are the pixel values ​​predicted by the autoencoder for the source domain image and the target domain image, M is the set of pixels of the mask patch block, |·| represents the number of elements in the set, s represents the source domain, t represents the target domain, i is a positive integer, and j is a positive integer; The high-level semantic feature distributions of the source and target domains aligned using the feature adaptation module include: The source domain data and target domain data are processed by the encoder to obtain the classification token in the encoded features; The classification token is input as a high-level semantic feature into the feature adaptation module; wherein, the feature adaptation module includes a first feature alignment block and a second feature alignment block, the first feature alignment block being before the classification layer and the second feature alignment block being after the classification layer; The first feature alignment block consists of a fully connected layer, a ReLU activation function, and another fully connected layer in sequence. There is a residual connection between the two fully connected layers, and only target domain data passes through these additional layers, while source domain data flows directly into the classification layer. The second feature alignment block consists of a fully connected layer, a ReLU activation function, and another fully connected layer in sequence. There is a residual connection between the two fully connected layers, and only target domain data passes through these additional layers. After the additional layers, there is a softmax layer. as well as Calculate feature alignment loss based on the source domain data and the target domain data. and regularization loss The feature alignment loss is calculated based on the source domain data and the target domain data. include: The feature alignment loss The sum of the maximum mean differences between the source and target domain features on the first and second feature alignment blocks is expressed as follows: Where, n s and n t Let l represent the total number of samples in the source domain and the target domain, respectively, and l = 1, 2. For the i-th source domain sample, the source domain high-level semantic features are aligned to the l-th feature block. It is the high-level semantic feature of the target domain after the j-th target domain sample is aligned with the l-th feature block. It has a characteristic kernel The reproducing kernel Hilbert space, φ(·) is the corresponding eigenmap, and MMD is the maximum mean difference. The source domain high-level semantic features are used to align the target domain sample with the l-th feature block. The target domain high-level semantic features are defined by aligning the block with the l-th feature, where s represents the source domain, t represents the target domain, i is a positive integer, and j is a positive integer. The regularization loss is calculated based on the source domain data and the target domain data. Includes: Select a random subset of the source domain data and force the random subset to pass through the target domain path of the feature adaptation module; The sum of the maximum mean differences between each class in the source domain and this random subset on the first and second feature alignment blocks is calculated as follows: Where l = 1, 2, It is a set of high-level semantic features of a random subset of samples from the source domain, C n S represents the total number of categories. k This represents the set of high-level semantic features of the k-th class of the source domain samples. It is the source domain high-level semantic feature of the i-th source domain sample through the l-th feature alignment block. It is the source domain high-level semantic feature of the j-th sample in a random subset of the source domain aligned to the target domain path of the block by the l-th feature. It has a characteristic kernel The reproducing kernel Hilbert space, φ(·) is the corresponding eigenmap, |·| represents the number of elements in the set, and MMD is the maximum mean difference. The source domain high-level semantic features are used to align the target domain sample with the l-th feature block. The high-level semantic features of the random subset of source domain samples through the target domain path of the l-th feature alignment module are: s represents the source domain, t represents the target domain, i is a positive integer, j is a positive integer, and k is a positive integer. The overall objective loss function is constructed based on the data. include: Construct source domain classification loss The expression is as follows: Where, n s Let H be the total number of samples in the source domain, and H(·,·) be the cross-entropy loss function. It is the high-level semantic feature of the i-th source domain sample by the feature adaptation module. The predicted probability Let be the true label of the i-th source domain sample, where s represents the source domain and i is a positive integer; Constructing the target domain conditional entropy loss The expression is as follows: Where, n t The total number of samples in the source domain. This represents the high-level semantic features of the j-th target domain sample input by the feature adaptation module. The predicted probability of the k-th class, where t represents the target domain, j is a positive integer, and k is a positive integer; Using the source domain classification loss The target domain conditional entropy loss The image reconstruction loss The feature alignment loss and the regularization loss Establish the overall objective loss function The expression is as follows: Where α, β, and γ are hyperparameters, representing the feature alignment loss. The regularization loss The target domain conditional entropy loss and the image reconstruction loss The weight.

2. The method according to claim 1, characterized in that, The method further includes: Remove the decoder, and use the encoder and the feature adaptation module to test the target domain image to obtain a scene classification result, which indicates the scene category of the target domain image.

3. The method according to claim 1, characterized in that, The autoencoder is based on Transformer, wherein the encoder of the autoencoder adopts a first number of layers of TransformerBlock network structure with multi-head attention mechanism; and the decoder of the autoencoder adopts a second number of layers of TransformerBlock network structure with multi-head attention mechanism.

Images