A domain-discriminative attention-enhanced visual language remote sensing scene classification method

Abstract

The present invention relates to a visual language remote sensing scene classification method with domain-discriminative attention enhancement, comprising the following steps: selecting and processing a remote sensing scene classification dataset, constructing a domain-discriminative attention-enhanced spatial state embedding network, training the domain-discriminative attention-enhanced spatial state embedding network, and testing the domain-discriminative attention-enhanced spatial state embedding network. Compared with existing technologies, the domain-discriminative attention enhancement module can dynamically adjust the weights of source domain features based on target domain features, allowing target domain features to focus on the most valuable information in the source domain. The encoder of the CLIP model, which is pre-trained using contrastive language images, can make judgments based on the association with natural language descriptions and the relevant knowledge learned from large-scale data, thereby enhancing the network's generalization ability in different scenarios. At the same time, the spatial state embedding module constructs a richer feature representation, allowing the model to capture the interaction between the two modalities and mine potential information.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image analysis and computer vision technology, and in particular to a domain-discriminative attention-enhanced visual language remote sensing scene classification method. Background Art

[0002] Remote sensing scene classification aims to accurately classify different scenes in remote sensing images (such as farmland, cities, forests, etc.). It is an important basic task in remote sensing image analysis and is applicable to land use monitoring, urban planning, disaster warning and other fields.

[0003] Traditional remote sensing scene classification methods rely heavily on handcrafted features, such as texture and spectral features. These methods require high-level feature engineering and have limited classification effectiveness in complex scenes. With the development of deep learning, methods based on convolutional neural networks (CNNs) have significantly improved classification accuracy by automatically extracting image features. However, existing deep learning methods still suffer from the following issues: 1. Data limitations: labeling high-quality remote sensing image data is expensive, and limited training data results in insufficient model generalization when faced with unseen scenes or shifting data distributions. 2. Insufficient utilization of semantic information: Existing deep learning methods primarily focus on visual image features and struggle to effectively integrate scene-related textual semantic information, limiting their understanding of complex scene semantics. 3. Poor cross-domain adaptability: Remote sensing images acquired from different sensors, at different times, and in different regions exhibit data distribution differences (i.e., domain disparity). Existing models perform poorly in cross-domain scene classification tasks.

[0004] In recent years, contrastive learning and pretraining techniques have achieved remarkable results in natural language processing and computer vision. For example, the CLIP (Contrastive Language-Image Pretraining) model uses contrastive learning to map images and text into a shared feature space, enabling zero-shot learning. However, direct application of these methods to remote sensing scene classification still fails to fully address the unique challenges of remote sensing data, such as effectively processing cross-domain data and accurately modeling the complex semantics of remote sensing scenes. Therefore, a new approach is urgently needed that can integrate the characteristics of remote sensing data, effectively utilize multimodal information and cross-domain knowledge, and improve remote sensing scene classification performance. Summary of the Invention

[0005] The purpose of this invention is to provide a domain-discriminative attention-enhanced visual language remote sensing scene classification method. By introducing a domain-discriminative attention enhancement module, the model's adaptability to remote sensing data from different domains is enhanced. Combined with visual language pre-training, the visual features of remote sensing images and the semantic features of text are effectively integrated, thereby improving the accuracy and generalization ability of remote sensing scene classification, and solving the problems of existing methods in terms of limited data, insufficient semantic utilization, and poor cross-domain adaptability.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] A domain-discriminative attention-enhanced visual language remote sensing scene classification method includes the following steps:

[0008] 11) Select and process a remote sensing scene classification dataset, including the following steps:

[0009] 111) We selected AID, NWPU-RESISC45, and UC Merced LandUse datasets from publicly available remote sensing scene classification datasets, extracted eight common scene categories from the three datasets as training datasets, and selected two datasets from each dataset, one as source domain data and the other as target domain data.

[0010] 112) Source domain data includes source domain images and source domain labels, where the source domain labels exist in the form of text information, and target domain data only contains target domain images without corresponding label information;

[0011] 113) All images were cropped to 224×224 pixels and rotated by 20%, Gaussian blurred, and then preprocessed with CutMix. The CutMix operation randomly selects a region in the image and removes the pixel values ​​in that region. The removed region is then filled with pixel values ​​from the corresponding region of another randomly selected image in the training set. The classification result label is then proportionally assigned to the filled region.

[0012] 12) Construct a domain-discriminative attention-enhanced spatial state embedding network, which includes an image encoder, a text encoder, a domain-discriminative attention-enhanced module, and a spatial state embedding module;

[0013] 13) Training the domain-discriminative attention-enhanced spatial state embedding network includes the following steps:

[0014] 131) First, use the pre-trained weights of the Contrastive Language-Image Pretraining (CLIP) model to initialize the image encoder and text encoder respectively;

[0015] 132) Input the source domain image and the target domain image into the initialized image encoder to obtain source domain image features and target domain image features respectively, and input the source domain label into the initialized text encoder to obtain text features;

[0016] 133) Then the source domain image features and the target domain image features are input into the domain discrimination attention enhancement module, and after processing, the image features of the fusion domain information are obtained;

[0017] 134) Then, the image features and text features of the fusion domain information are input into the spatial state embedding module to obtain the fusion modal features. The similarity between the fusion modal features and the text features is calculated. The category with the highest similarity is the classification result of the target domain image prediction;

[0018] 135) Finally, update the network parameters based on the loss back propagation and save the learned optimal weight parameters;

[0019] 14) Test the domain-discriminative attention-enhanced spatial state embedding network: Input the target domain image and source domain label into the trained domain-discriminative attention-enhanced spatial state embedding network, and obtain the classification result of the target domain image through the proposed image encoder, text encoder, domain-discriminative attention enhancement module and spatial state embedding module.

[0020] The construction of the domain-discriminative attention-enhanced spatial state embedding network comprises the following steps:

[0021] 21) Build an image encoder to extract image features. Use the ViT-B / 16 model in CLIP as the encoder, which includes a 12-layer transformer structure with 12 attention heads and three fully connected linear layers.

[0022] 211) The 16 in ViT-B / 16 indicates that the patch size of the Patch Embedding layer input is 16×16, and B stands for Base;

[0023] 212) Transformer includes two layer normalized LNs, a 12-head self-attention, a multi-layer perceptron MLP, and two residual connections;

[0024] 213) The three Linear layers are used to generate the target domain query Q, source domain key K, and source domain value V matrices respectively;

[0025] 22) Build a text encoder to extract text features. The text encoder uses a 12-layer transformer structure. The transformer structure is the same as the image encoder, with 8 self-attention heads.

[0026] 23) Construct a domain-discriminative attention enhancement module, which includes a cross-domain attention layer, a feature enhancement module, and a gradient reversal layer GRL;

[0027] 231) The feature enhancement module specifically includes: a global average pooling GAP, two fully connected FC layers, a ReLU layer, a Sigmoid activation layer, a 1×3 convolution, a 3×1 convolution, a 1×1 convolution, and an element-by-element multiplication operation;

[0028] 24) Construct a spatial state embedding module, which includes the following operations: point-by-point convolution (PWC), 3×3 convolution, Conv1d convolution, ReLU activation function, SiLU activation function, Linear layer, state space model (SSM), element-wise addition operation, and concatenation operation.

[0029] The training of the domain discriminative attention-enhanced spatial state embedding network comprises the following steps:

[0030] 31) Initialize the domain-discriminative attention-enhanced spatial state embedding network using the ViT-B / 16 encoder pre-trained with the CLIP model for contrastive language image pre-training;

[0031] 32) Input the source domain image and the target domain image into the image encoder to obtain the source domain image features S feature and target domain image features T feature ;

[0032] 321) Execute the Patch Embedding layer once for the source domain image and the target domain image respectively to obtain the feature embedding S with position information embed and T embed ;

[0033] 322) Input the two feature embeddings into the transformer separately, first embed the feature into S embed and T embed Input layer normalization LN to obtain source domain query Q s , source domain key K s , source domain value V s , target domain query Q t , target domain key K t , target domain value V t Six matrices, each of which is calculated through multi-head self-attention to obtain attention scores attn1 and attn2;

[0034] ,

[0035] ,

[0036] in, is the scaling factor, which is 64 in ViT-B / 16. represents the transpose of matrix K, and softmax is the activation function;

[0037] 323) The residual of the attention score is added to the feature embedding, and then the result of the addition is normalized by LN, and then output by the multi-layer perceptron MLP. The output result is added to the residual of the attention score to obtain the two outputs of the first transformer. sand output t ;

[0038] ,

[0039] ,

[0040] in, is the scaling factor, which is 64 in ViT-B / 16. represents the transpose of matrix K;

[0041] 324) The two outputs of the first transformer are passed through 11 transformers with the same structure to obtain a more advanced semantic feature S feature and T feature ;

[0042] 325) S feature Input two fully connected Linear layers to obtain the source domain key K and source domain value V matrix, T feature Input the Linear layer to obtain the target domain query Q matrix;

[0043] 33) Input the source domain label into the text encoder to obtain the text feature z t ;

[0044] 331) First, tokenize the source domain labels, encode them into tokens, add a start word and an end word to each token, map the token to the index corresponding to the CLIP definition vocabulary, and pad with zeros to ensure that all text sequences have the same length.

[0045] 332) Then, the word index is converted into a low-dimensional word vector with a dimension of 512 through a trainable word embedding matrix;

[0046] 333) Then the low-dimensional word vector is input into the transformer model for processing. The calculation process is consistent with the image operation process. The low-dimensional word vector is calculated by 12 layers of transformers in turn to obtain the text feature z t ;

[0047] ,

[0048] Among them, TextEncoder is a text encoder, Tokenize is a word segmentation operation, and SourceLabel represents the source domain label;

[0049] 34) The target domain query Q, source domain key K, and source domain value V matrix output by the image encoder are input into the domain discrimination attention enhancement module. After processing by this module, the image feature z of the fusion domain information is obtained.i ,During the processing, the gradient reversal layer GRL is used to promote the model to learn domain-invariant features;

[0050] 341) First, input the target domain query Q, source domain key K, and source domain value V matrix into the cross-domain attention, and calculate the cross-domain attention score matrix CrossAttn between the target domain query Q and the source domain key value;

[0051] ,

[0052] 342) Split the target domain query Q, source domain key K, and source domain value V into 8 heads along the feature dimension. Each head independently calculates attention and concatenates the results.

[0053] ,

[0054] in, , i represents the number of attention heads 1…8, To output the projection matrix, concat is a channel-by-channel splicing operation;

[0055] 343) The cross-domain attention score CrossAttn is residually connected with the target domain query Q matrix, and the output image feature z is obtained by layer normalization LN stabilization training. i ;

[0056] ,

[0057] in, Optional dimension alignment matrix;

[0058] 344) Output image feature z i After the feature enhancement module, the image feature z of the fusion domain information is obtained j ;

[0059] 3441) First, the image feature z i Perform a global average pooling (GAP) operation to compress the spatial dimension to obtain statistical information for each channel;

[0060] 3442) The results of the GAP operation are then input into two fully connected layers FC in sequence, and ReLU activation is used between the two fully connected layers FC to introduce nonlinear transformation;

[0061] 3443) The output of the second FC layer is activated by Sigmoid to obtain the channel weight vector w;

[0062] ,

[0063] 3444) Next, the image feature zi Perform 1×3 convolution, 3×1 convolution, and 1×1 convolution in sequence to obtain the asymmetric convolution output feature z a ;

[0064] ,

[0065] 3445) Finally, the channel weight vector w is combined with the asymmetric convolution output feature z a Multiply to get the image feature z of the fusion domain information j ;

[0066] ,

[0067] in, Represents element-wise multiplication;

[0068] 345) Use the gradient reversal layer for implicit adversarial learning to promote the model to learn domain-invariant features. GRL acts as an identity mapping during forward propagation and multiplies the gradient by a negative learning rate scaling factor during backpropagation. ;

[0069] ,

[0070] ,

[0071] Where L is the loss function, x is the input feature, and y is the output feature of GRL(x). It is an adjustable hyperparameter, ranging from (0,1);

[0072] 35) The image feature z of the fusion domain information j and text feature z t Input the spatial state embedding module to obtain the fusion modal feature z f ;

[0073] 351) First, the image feature z of the fusion domain information j and text feature z t After a 3×3 convolution, a point-by-point convolution PWC and ReLU, we get out j 、out t ;

[0074] ,

[0075] ,

[0076] 352) will be out j and out tAfter element-wise addition, it goes through a 3×3 convolution and ReLU activation, a linear layer, a Conv1d convolution and SiLU activation to get out1;

[0077] ,

[0078] in, Represents element-by-element addition, and Linear represents a linear layer;

[0079] 353) then out j and out t After channel-wise concatenation, it undergoes a 3×3 convolution and ReLU activation, a linear layer, a Conv1d convolution and SiLU activation, and a state space model SSM to obtain out2;

[0080] ,

[0081] 354) Finally, out1 and out2 are concatenated and passed through a linear layer to obtain the fused modal feature z f ;

[0082] ,

[0083] 36) Calculate the fusion modal feature z f and text feature z t The similarity between them is used to determine the text with the highest similarity in the feature space as the predicted label of the image;

[0084] ,

[0085] in, is the L2 norm, Similarity represents the similarity;

[0086] 37) Calculate the cross entropy loss Loss between the predicted label and the source domain label;

[0087] ,

[0088] Among them, cross_entropy_loss is the encapsulated cross entropy loss function in the pytorch framework, and logit represents the predicted label;

[0089] 38) Use the Loss to perform backpropagation to update the network parameters and determine whether the set number of rounds has been reached or the Loss no longer decreases, then stop training and save the best weights of the training. Otherwise, return to step 32) to continue loading data for training.

[0090] The specific steps of testing the domain-discriminative attention-enhanced spatial state embedding network are as follows:

[0091] 41) Input the single target domain image to be tested and the source domain label into the domain discrimination attention enhancement spatial state embedding network;

[0092] 411) The target domain image is input into the image encoder to obtain the output, and then passes through three linear layers to generate the target domain query Q, target domain key K, and target domain value V matrices, which are input into the domain discrimination attention enhancement module to obtain the image features of the fused domain information;

[0093] 412) The source domain label is input into the text encoder to obtain text features;

[0094] 413) Inputting the image features and text features of the fusion domain information into the spatial state embedding module to obtain the fusion modal features;

[0095] 42) Calculate the similarity between the fused modal features and the text features. The text label with the highest similarity is the category of the test image.

[0096] Beneficial effects

[0097] This invention relates to a domain-discriminative attention-enhanced visual language remote sensing scene classification method. Compared with existing technologies, this method reduces the model's reliance on large-scale annotated data by loading CLIP pre-trained weights, improving the model's generalization ability. Combined with visual language pre-training, it deeply integrates the visual features of remote sensing images with textual semantic features, enabling the model to better understand the semantic information of remote sensing scenes. Classification accuracy is significantly improved for complex scenes and scenes that are difficult to distinguish directly through visual features. The domain-discriminative attention enhancement module effectively captures the feature differences and commonalities of remote sensing data from different domains, enhancing the model's adaptability to cross-domain data and improving classification performance on remote sensing images from different sensors and regions. The spatial state embedding module effectively captures the global perspective and better integrates image and text modal information. This method can be applied to a variety of remote sensing scene classification tasks and has high practical value and promotional significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 This is a flowchart of a domain-discriminative attention-enhanced visual language remote sensing scene classification method, showing the various steps from data preprocessing to final scene classification;

[0099] Figure 2 This is a model structure diagram for building a domain-discriminative attention-enhanced spatial state embedding network, which details the structure of the image encoder, text encoder, domain-discriminative attention enhancement module, and spatial state embedding module, as well as the connection relationship between them;

[0100] Figure 3 It is the structural details of the feature enhancement module;

[0101] Figure 4 are the structural details of the spatial state embedding module;

[0102] Figure 5 This is a test result diagram of a domain-discriminative attention-enhanced visual language remote sensing scene classification method. The class above the image shows the predicted label, and the probability is the possibility of belonging to the class. DETAILED DESCRIPTION

[0103] In order to provide a further understanding and recognition of the structural features and the effects achieved by the present invention, a detailed description is given with reference to preferred embodiments and accompanying drawings as follows:

[0104] like Figure 1 As shown, the domain-discriminative attention-enhanced visual language remote sensing scene classification method of the present invention includes the following steps:

[0105] The first step is to select and process the remote sensing scene classification dataset. The specific steps are as follows:

[0106] (1) We selected AID, NWPU-RESISC45, and UC Merced LandUse datasets from the publicly available remote sensing scene classification datasets, extracted eight scene categories shared by the three datasets as training datasets, and selected two datasets from each dataset, one as the source domain data and the other as the target domain data;

[0107] (2) The source domain data includes the source domain image SourceImage and the source domain label SourceLabel, where the source domain label SourceLabel exists in the form of text information, and the target domain data only contains the target domain image TargetImage without corresponding label information;

[0108] (3) All images were cropped to 224 × 224 pixels and rotated, Gaussian blurred, and CutMix preprocessed at a ratio of 20%. The CutMix operation randomly selected an area in the image and removed the pixel values ​​in that area. The removed area was filled with the pixel values ​​of the corresponding area of ​​other images randomly selected from the training set. At the same time, the classification result labels were proportionally distributed according to the proportion of the filled area.

[0109] The second step is to build a domain-discriminative attention-enhanced spatial state embedding network. The specific steps are as follows:

[0110] (1) Construct an image encoder ImageEncoder to extract image features. Use the ViT-B / 16 model in CLIP, which includes a Patch Embedding layer, a 12-layer transformer structure, and three fully connected Linear layers.

[0111] (1-1) The 16 in ViT-B / 16 indicates that the patch size of the Patch Embedding layer input is 16×16, and B represents Base;

[0112] (1-2) Transformer includes two layer normalized LNs, a 12-head self-attention, a multi-layer perceptron MLP and two residual connections;

[0113] (1-3) The three Linear layers are used to generate the target domain query Q, source domain key K, and source domain value V matrices respectively;

[0114] (2) Construct a text encoder TextEncoder to extract text features. TextEncoder uses a 12-layer transformer structure. The transformer structure is the same as the image encoder ImageEncoder, with 8 self-attention heads;

[0115] (3) Construct a domain-discriminative attention enhancement module, which includes a cross-domain attention layer, a feature enhancement module, and a gradient reversal layer (GRL);

[0116] (3-1) The feature enhancement module specifically includes: a global average pooling (GAP), two fully connected FC layers, a ReLU layer, a Sigmoid activation layer, a 1×3 convolution, a 3×1 convolution, a 1×1 convolution, and an element-wise multiplication operation;

[0117] (4) Construct a spatial state embedding module, which includes the following operations: point-by-point convolution PWC, 3×3 convolution, Conv1d convolution, ReLU activation function, SiLU activation function, Linear layer, state space model SSM, element-by-element addition operation and splicing operation.

[0118] First, the preprocessed source domain image SourceImage and target domain image TargetImage are input into the image encoder ImageEncoder, and then the image features of the fused domain information are obtained through the domain discrimination attention enhancement module. Then the text label SourceLabel is input into TextEncoder to obtain the text features. The image features and text features are fused through the spatial state embedding module. Finally, the similarity between the text features and the fused modal features is calculated to complete the construction of the entire model.

[0119] The third step is to train the domain-discriminative attention-enhanced spatial state embedding network. The specific steps are as follows:

[0120] (1) Initialize the domain-discriminative attention-enhanced spatial state embedding network using the ViT-B / 16 encoder pre-trained with the CLIP model for contrastive language image pre-training;

[0121] (2) Input the source domain image SourceImage and the target domain image TargetImage into the image encoder ImageEncoder to obtain the source domain image features S feature and target domain image features T feature ;

[0122] (2-1) Execute the PatchEmbedding layer once for the source domain image SourceImage and the target domain image TargetImage respectively to obtain the feature embedding S with position information embed and T embed ;

[0123] (2-2) The two features are embedded into the transformer respectively. First, the feature is embedded into S embed and T embed Input layer normalization LN to obtain source domain query Q s , source domain key K s , source domain value V s , target domain query Q t , target domain key K t , target domain value V t Six matrices, each of which is calculated through multi-head self-attention to obtain attention scores attn1 and attn2;

[0124] ,

[0125] ,

[0126] in, is the scaling factor, which is 64 in ViT-B / 16. represents the transpose of matrix K, and softmax is the activation function;

[0127] (2-3) The residual of the attention score is added to the feature embedding, and then the result of the addition is normalized by LN layer, and then output by MLP layer. The output result is added to the residual of the attention score to obtain the two outputs of the first transformer. s and output t ;

[0128] ,

[0129] ,

[0130] in, is the scaling factor, which is 64 in ViT-B / 16. represents the transpose of matrix K;

[0131] (2-4) The two outputs of the first transformer are passed through 11 transformers with the same structure to obtain a more advanced semantic feature S feature and T feature ;

[0132] (2-5) S feature Input two Linear layers to obtain the source domain key K and source domain value V matrix, T feature Input the Linear layer to obtain the target domain query Q matrix;

[0133] (3) Input the source domain label SourceLabel into the text encoder TextEncoder to obtain the text feature z t ;

[0134] (3-1) First, perform a tokenization operation on the source domain label SourceLabel, encode it into tokens for encapsulation, add a start word and an end word to each token, map the token to the index corresponding to the CLIP definition vocabulary, and use zero padding to make the length of all text sequences consistent;

[0135] (3-2) Then, the word index is converted into a low-dimensional word vector with a dimension of 512 through a trainable word embedding matrix;

[0136] (3-3) The low-dimensional word vector is then input into the transformer model for processing. The calculation process is consistent with the image operation process. The low-dimensional word vector is calculated by 12 layers of transformers in turn to obtain the text feature z t ;

[0137] ,

[0138] Among them, Tokenize is a word segmentation operation;

[0139] (4) The target domain query Q, source domain key K, and source domain value V matrix output by the image encoder are input into the domain discrimination attention enhancement module. After processing by this module, the image feature z of the fusion domain information is obtained. i ,During the processing, the gradient reversal layer GRL is used to promote the model to learn domain-invariant features;

[0140] (4-1) First, input the target domain query Q, source domain key K, and source domain value V matrix into the cross-domain attention CrossAttention, and calculate the cross-domain attention score matrix CrossAttn between the target domain query Q and the source domain key value;

[0141] ,

[0142] (4-2) Split the target domain query Q, source domain key K, and source domain value V into 8 heads along the feature dimension. Each head independently calculates attention and concatenates the results.

[0143] ,

[0144] in, , i represents the number of attention heads 1…8, To output the projection matrix, concat is a channel-by-channel splicing operation;

[0145] (4-3) The cross-domain attention score CrossAttn is residually connected with the target domain query Q matrix, and the output image feature z is obtained by layer normalization LN stabilization training. i ;

[0146] ,

[0147] in, Optional dimension alignment matrix;

[0148] (4-4) The output image feature z i After the feature enhancement module, the image feature z of the fusion domain information is obtained j ;

[0149] (4-4-1) First, the image feature z i Perform a global average pooling (GAP) operation to compress the spatial dimension to obtain statistical information for each channel;

[0150] (4-4-2) The results of the GAP operation are then input into two fully connected layers FC in sequence, and ReLU activation is used between the two fully connected layers FC to introduce nonlinear transformation;

[0151] (4-4-3) The output of the second FC layer is activated by Sigmoid to obtain the channel weight vector w;

[0152] ,

[0153] (4-4-4) Next, the image feature z i Perform 1×3 convolution, 3×1 convolution, and 1×1 convolution in sequence to obtain the asymmetric convolution output feature za ;

[0154] ,

[0155] (4-4-5) Finally, the channel weight vector w is combined with the asymmetric convolution output feature z a Multiply to get the image feature z of the fusion domain information j ;

[0156] ,

[0157] in, Represents element-wise multiplication;

[0158] (4-5) Use the gradient reversal layer for implicit adversarial learning to promote the model to learn domain-invariant features. GRL acts as an identity mapping during forward propagation and multiplies the gradient by a negative learning rate scaling factor during backpropagation. ;

[0159] ,

[0160] ,

[0161] Where L is the loss function, x is the input feature, and y is the output feature of GRL(x). It is an adjustable hyperparameter, ranging from (0,1);

[0162] (5) The image feature z of the fusion domain information j and text feature z t Input the spatial state embedding module to obtain the fusion modal feature z f ;

[0163] (5-1) First, the image feature z of the fusion domain information j and text feature z t After a 3×3 convolution, a point-by-point convolution PWC and ReLU, we get out j 、out t ;

[0164] ,

[0165] ,

[0166] (5-2) will be out j and out t After element-wise addition, it goes through a 3×3 convolution and ReLU activation, a linear layer, a Conv1d convolution and SiLU activation to get out1;

[0167] ,

[0168] in, Represents element-by-element addition, and Linear represents a linear layer;

[0169] (5-3) then out j and out t After channel-wise concatenation, it undergoes a 3×3 convolution and ReLU activation, a linear layer, a Conv1d convolution and SiLU activation, and a state space model SSM to obtain out2;

[0170] ,

[0171] (5-4) Finally, out1 and out2 are concatenated and passed through a linear layer to obtain the fused modal feature z f ;

[0172] ,

[0173] (6) Calculate the fusion modal feature z f and text feature z t The similarity between them is used to determine the text with the highest similarity in the feature space as the predicted label logit of the image;

[0174] ,

[0175] in, is the L2 norm, Similarity represents the similarity;

[0176] (7) Calculate the cross entropy loss Loss between the predicted label logit and the source domain label SourceLabel;

[0177] ,

[0178] Among them, cross_entropy_loss is the cross entropy loss function encapsulated in the pytorch framework;

[0179] (8) Use Loss to perform back propagation to update the network parameters, and determine whether the set number of rounds has been reached, or if the Loss no longer decreases, stop training and save the best weights of the training. Otherwise, return to step (2) to continue loading data for training.

[0180] The fourth step is to test the domain-discriminative attention-enhanced spatial state embedding network. The specific steps are as follows:

[0181] (1) Input the single target domain image TargetImage to be tested and eight common category labels SourceLabel into the network;

[0182] (1-1) TargetImage is input into ImageEncoder to get the output, and then passes through three Linear layers to generate the target domain query Q, target domain key K, and target domain value V matrices, which are input into the domain discrimination attention enhancement module to obtain the image features of the fused domain information;

[0183] (1-2) SourceLabel is input into TextEncoder to obtain text features;

[0184] (1-3) Input the image features and text features of the fusion domain information into the spatial state embedding module to obtain the fusion modal features;

[0185] (2) Calculate the similarity between the fused modal features and the text features. The text label with the highest similarity is the category of the test image TargetImage.

[0186] The method proposed in the present invention is described below using NWPU-RESISC45 as the source domain and AID as the target domain dataset as an example:

[0187] A total of 8 categories are selected as the label set, and an image of AID is selected. After the preprocessing operation of the present invention, an image of size 224×224 is obtained. Using the method of the present invention, the predicted image category is as follows: Figure 5 As shown, above the image is the predicted label and the possibility of belonging to the category, and 6 images are listed for display.

[0188] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A domain-discriminative attention-enhanced visual language remote sensing scene classification method, characterized by: The following steps are involved: 11) Selection and processing of remote sensing scene classification datasets: Eight categories shared by the AID, NWPU-RESISC45, and UC Merced Land Use datasets were selected as training data. Two training data sets were selected as source and target domain data. The source domain data consisted of source domain images and source domain labels, while the target domain data consisted only of target domain images. 12) Construct a domain-discriminative attention-enhanced spatial state embedding network: including an image encoder, a text encoder, a domain-discriminative attention-enhanced module, and a spatial state embedding module; 121) Build an image encoder to extract image features. Use the ViT-B / 16 model in CLIP as the encoder, which includes a 12-layer transformer structure with 12 attention heads and three linear layers. 1211) The 16 in ViT-B / 16 indicates that the patch size of the Patch Embedding layer input is 16×16, and B represents Base; 1212) Transformer includes two layer normalized LNs, a 12-head self-attention, a multi-layer perceptron MLP and two residual connections; 1213) The three Linear layers are used to generate the target domain query Q, source domain key K, and source domain value V matrices respectively; 122) Build a text encoder to extract text features. The text encoder uses a 12-layer transformer structure. The transformer structure is the same as the image encoder, with 8 self-attention heads; 123) Construct a domain discrimination attention enhancement module, which includes a cross-domain attention layer, a feature enhancement module and a gradient reversal layer GRL; 1231) The feature enhancement module specifically includes: a global average pooling GAP, two fully connected FC layers, a ReLU layer, a Sigmoid activation layer, a 1×3 convolution, a 3×1 convolution, a 1×1 convolution, and an element-by-element multiplication operation; 124) Construct a spatial state embedding module, which includes the following operations: point-by-point convolution PWC, 3×3 convolution, Conv1d convolution, ReLU activation function, SiLU activation function, Linear layer, state space model SSM, element-by-element addition operation and splicing operation; 13) Training the domain-discriminative attention-enhanced spatial state embedding network: First, use the pre-trained weights of the CLIP model to initialize the parameters of the image and text encoders. Then, input the source domain data and target domain data into the domain-discriminative attention-enhanced spatial state embedding network respectively. Update the network parameters based on the loss backpropagation and save the optimal weight parameters. 14) Test the domain-discriminative attention-enhanced spatial state embedding network: Input the target domain image and source domain label into the trained domain-discriminative attention-enhanced spatial state embedding network to obtain the classification result of the target domain image.

2. The method for visual language remote sensing scene classification with enhanced domain discrimination attention according to claim 1 is characterized in that: The training of the domain discriminative attention-enhanced spatial state embedding network comprises the following steps: 21) Initialize the domain-discriminative attention-enhanced spatial state embedding network using the ViT-B / 16 encoder pre-trained on the CLIP model for contrastive language image pre-training; 22) Input the source domain image and the target domain image into the image encoder to obtain the source domain image features S feature and target domain image features T feature ; 221) Execute the Patch Embedding layer once for the source domain image and the target domain image respectively to obtain the feature embedding S with position information embed and T embed ; 222) The two features are embedded into the transformer respectively. First, the feature is embedded into S embed and T embed Input layer normalization LN to obtain source domain query Q s , source domain key K s , source domain value V s , target domain query Q t , target domain key K t , target domain value V t Six matrices, each of which is calculated through multi-head self-attention to obtain attention scores attn1 and attn2; in, is the scaling factor, which is 64 in ViT-B / 16, K T represents the transpose of matrix K, and softmax is the activation function; 223) The residual of the attention score is added to the feature embedding, and then the result of the addition is normalized by LN, and then output by the multi-layer perceptron MLP. The output result is added to the residual of the attention score to obtain the two outputs of the first transformer. s and output t ; output s =MLP(LN(attn s +S embed ))+(attn s +S embed ), output t =MLP(LN(attn t +T embed ))+(attn t +T embed ), in, is the scaling factor, which is 64 in ViT-B / 16, K T represents the transpose of matrix K; 224) The two outputs of the first transformer are passed through 11 transformers with the same structure to obtain a more advanced semantic feature S feature and T feature ; 225) S feature Input two fully connected Linear layers to obtain the source domain key K and source domain value V matrix, T feature Input the Linear layer to obtain the target domain query Q matrix; 23) Input the source domain label into the text encoder to obtain the text feature z t ; 231) First, tokenize the source domain labels, encode them into tokens, add a start word and an end word to each token, map the token to the index corresponding to the CLIP definition vocabulary, and pad with zeros to ensure the same length of all text sequences; 232) Then, the word index is converted into a low-dimensional word vector with a dimension of 512 through a trainable word embedding matrix; 233) Then the low-dimensional word vector is input into the transformer model for processing. The calculation process is consistent with the image operation process. The low-dimensional word vector is calculated by 12 layers of transformers in turn to obtain the text feature z t ; from t =TextEncoder(Tokenize(SourceLabel)), Among them, TextEncoder is a text encoder, Tokenize is a word segmentation operation, and SourceLabel represents the source domain label; 24) The target domain query Q, source domain key K, and source domain value V matrix output by the image encoder are input into the domain discrimination attention enhancement module. After processing by the module, the image feature z of the fusion domain information is obtained. i ,During the processing, the gradient reversal layer GRL is used to promote the model to learn domain-invariant features; 241) First, input the target domain query Q, source domain key K, and source domain value V matrix into the cross-domain attention, and calculate the cross-domain attention score matrix CrossAttn between the target domain query Q and the source domain key value; 242) Split the target domain query Q, source domain key K, and source domain value V into 8 heads along the feature dimension. Each head independently calculates attention and concatenates the results. MultiHead(Q,K,V)=Concat(Head1,...,Head8)·W o , Among them, Head i =CrossAttn(Q i , K i , V i ), i represents the number of attention heads 1…8, W o To output the projection matrix, concat is a channel-by-channel splicing operation; 243) The cross-domain attention score CrossAttn is residually connected with the target domain query Q matrix, and the output image feature z is obtained by layer normalization LN stabilization training. i ; z i =LN(Q+MultiHead·W o ′), Among them, W o ′ is an optional dimension alignment matrix; 244) Output image feature z i After the feature enhancement module, the image feature z of the fusion domain information is obtained j ; 2441) First, the image feature z i Perform a global average pooling (GAP) operation to compress the spatial dimension to obtain statistical information for each channel; 2442) The results of the GAP operation are then sequentially input into two fully connected layers FC, and ReLU activation is used between the two fully connected layers FC to introduce nonlinear transformation; 2443) The output of the second FC layer is activated by Sigmoid to obtain the channel weight vector w; w=Sigmoid(FC-ReLU-FC(GAP(z i ))), 2444) Next, the image feature z i Perform 1×3 convolution, 3×1 convolution, and 1×1 convolution in sequence to obtain the asymmetric convolution output feature z a ; z a =Conv 1×1 (Conv 3×1 (Conv 1×3 (z i ))), 2445) Finally, the channel weight vector w is combined with the asymmetric convolution output feature z a Multiply to get the image feature z of the fusion domain information j ; in, Represents element-wise multiplication; 245) Use the gradient reversal layer for implicit adversarial learning to promote the model to learn domain-invariant features. GRL acts as an identity mapping during forward propagation and multiplies the gradient by a negative learning rate scaling factor λ during backpropagation. GRL(x)=x, Where L is the loss function, x is the input feature, y is the output feature of GRL(x), and λ is an adjustable hyperparameter ranging from (0, 1). 25) The image feature z of the fusion domain information j and text feature z t Input the spatial state embedding module to obtain the fusion modal feature z f ; 251) First, the image feature z of the fusion domain information j and text feature z t After a 3×3 convolution, a point-by-point convolution PWC and ReLU, we get out j 、out t ; out j =ReLU(PWC(ReLU(Conv 3×3 (z j )))), out t =ReLU(PWC(ReLU(Conv 3×3 (z t )))), 252) will be out j and out t After element-wise addition, it goes through a 3×3 convolution and ReLU activation, a linear layer, a Conv1d convolution and SiLU activation to get out1; in, Represents element-by-element addition, and Linear represents a linear layer; 253) then out j and out t After channel-wise concatenation, it undergoes a 3×3 convolution and ReLU activation, a linear layer, a Conv1d convolution and SiLU activation, and a state space model SSM to obtain out2; out2=SSM[SiLU(Conv1d(Linear(ReLU(Conv 3×3 (concat(out j ,out t ))))))], 254) Finally, out1 and out2 are concatenated and passed through a linear layer to obtain the fused modal feature z f ; z f =Linear(concat(out1,out2)), 26) Calculate the fusion modal feature z f and text feature z t The similarity between them is used to determine the text with the highest similarity in the feature space as the predicted label of the image; Among them, || ||2 is the L2 norm, Similarity represents the similarity; 27) Calculate the cross entropy loss Loss between the predicted label and the source domain label; Loss=cross_entropy_loss(logit,SourceLabel), Among them, cross_entropy_loss is the encapsulated cross entropy loss function in the pytorch framework, and logit represents the predicted label; 28) Use Loss to perform back propagation to update the network parameters, and determine whether the set number of rounds has been reached, or if the Loss no longer decreases, stop training and save the best weights of the training. Otherwise, return to step 22) to continue loading data for training.

3. The method for visual language remote sensing scene classification with enhanced domain discrimination attention according to claim 1, characterized in that: The specific steps of testing the domain-discriminative attention-enhanced spatial state embedding network are as follows: 31) Input the single target domain image to be tested and the source domain label into the domain discrimination attention enhancement spatial state embedding network; 311) The target domain image is input into the image encoder to obtain the output, and then passes through three fully connected linear layers to generate the target domain query Q, target domain key K, and target domain value V matrices, which are input into the domain discrimination attention enhancement module to obtain the image features of the fused domain information; 312) The source domain label is input into the text encoder to obtain text features; 313) inputting the image features and text features of the fusion domain information into the spatial state embedding module to obtain the fusion modal features; 32) Calculate the similarity between the fused modal features and the text features. The text label with the highest similarity is the category of the test image.

Images