Remote sensing scene classification method based on masked attention mechanism

By using the mask attention mechanism method in remote sensing scene classification, combined with ResNet50 and Transformer encoder, image chunking and masking operations are performed, the problem of low accuracy in remote sensing scene image classification in the existing technology is solved, and the further development is achieved. Efficient automation classification effect.

CN117372763BActive Publication Date: 2024-11-29NAVAL AVIATION UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311334853.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-16
Publication Date
2024-11-29
Estimated Expiration
2043-10-16

AI Technical Summary

Technical Problem

The existing remote sensing scene image classification model ignores the potential relationship between categories when extracting feature, resulting in low accuracy of remote sensing scene image classification and unable to effectively reduce labor and time costs.

Method used

Using a remote sensing scene classification method based on mask attention mechanism, by establishing a network model containing ResNet50 and Transformer encoder, image blocking and masking operations are performed, cross-entropy loss and distillation loss are used for network optimization, and multiple features are integrated into Improve classification accuracy.

Benefits of technology

It improves the accuracy of image classification in remote sensing scenes, reduces labor and time costs, and achieves more efficient and automated classification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117372763B_ABST
    Figure CN117372763B_ABST
Patent Text Reader

Abstract

The present invention discloses a remote sensing scene classification method based on a masked attention mechanism, comprising: establishing a network model, constructing a sample data set, training the network model, and using the trained network model for classification. The present invention first divides the image into blocks, and then performs a mask operation on the divided image. For the unmasked image blocks, ResNet50 and Transformer encoders are used to perform preliminary feature extraction respectively, and the features extracted by the former are further feature fused, and the features extracted by the latter are further feature enhanced, and then the cross entropy loss of each of the two paths and the comprehensive distillation cross entropy loss are calculated, and then the total loss is calculated, and finally the total loss is used to reversely optimize the network model, thereby improving the accuracy of remote sensing scene classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to an image classification technology, and specifically relates to a remote sensing scene classification method. Background Art

[0002] Deep learning technology has been widely used in the field of remote sensing. The application of deep learning technology in the field of remote sensing can reduce the high labor costs caused by traditional repetitive labor performed by humans.

[0003] Traditional remote sensing scene classification requires manual observation of scene categories and manual classification. However, various remote sensing technologies generate tens of thousands of scene images, which leads to a sharp increase in labor costs. Therefore, people began to consider using deep learning technology to automatically classify remote sensing scene images, thereby screening out the desired categories, thereby greatly reducing time and money costs.

[0004] In recent years, some researchers have applied classification models to the classification task of remote sensing scene images, such as AlexNet and VGG. However, such network models only perform feature extraction and ignore the potential relationship between categories, making it difficult to achieve good results, and the accuracy of remote sensing scene image classification is low. Summary of the invention

[0005] The present invention proposes a remote sensing scene classification method based on a masked attention mechanism, the purpose of which is to improve the accuracy of remote sensing scene image classification.

[0006] The technical solution of the present invention is as follows:

[0007] A remote sensing scene classification method based on a masked attention mechanism, comprising the following steps:

[0008] Step 1: Establish a network model, wherein the network model includes a first classification module, a second classification module and a result fusion module;

[0009] The first classification module includes a first preliminary feature extraction module and a feature fusion module, wherein the feature fusion module is used to fuse multiple groups of features extracted by the first preliminary feature extraction module;

[0010] The second classification module includes a second preliminary feature extraction module and a channel attention module, wherein the channel attention module is used to perform feature enhancement on the features extracted by the second preliminary feature extraction module;

[0011] The result fusion module is used to fuse the first classification result obtained by the first classification module and the second classification result obtained by the second classification module into a final classification result;

[0012] Step 2: Construct a sample data set;

[0013] Step 3, training the first classification module and the second classification model in the network model: dividing the images in the sample data set into blocks and performing random mask operations to obtain unmasked image blocks, and inputting the unmasked image blocks into the first classification module and the second classification module respectively; after all images in the sample data set are processed, two cross entropy losses are calculated according to all the first classification results and all the second classification results, and a distillation loss is calculated based on all the first classification results and all the second classification results, and then a total loss is calculated based on the two cross entropy losses and one distillation loss, and the network model is reversely optimized based on the total loss;

[0014] Step 4: Divide the image to be classified into blocks, and then input them into the trained network model to obtain the final classification result output by the network model, thereby realizing the scene classification of the image to be classified.

[0015] As a further improvement of the remote sensing scene classification method based on the masked attention mechanism: the first preliminary feature extraction module is a ResNet50 network;

[0016] In the first classification module, for the i-th image X to be input i , X i ∈R C×H×W , C represents the number of image channels, H represents the image height, and W represents the image width: Let the input sequence consisting of the image blocks obtained based on the image and to be input into the network model be in Represents image X i The jth image block to be input into the network model, k is the total number of image blocks input into the network model;

[0017] The processing process of the first classification module includes:

[0018] Step A1, ResNet50 network obtains the following feature sequence according to the input sequence: (x i,1 ,x i,2 ,…,x i,k );in According to the image block The features obtained by the convolution operation with a convolution kernel size of 1x1, the convolution operation with a convolution kernel size of 3x3, and the convolution operation with a convolution kernel size of 1x1 are:

[0019] As a further improvement of the remote sensing scene classification method based on the mask attention mechanism: the number of the feature fusion modules is k-1; each of the feature fusion modules includes a Relu activation function, a fully connected layer, a Relu activation function, a fully connected layer and a global pooling layer connected in sequence;

[0020] Then the processing process of the first classification module also includes:

[0021] Step A2: Perform feature fusion through k-1 feature fusion modules:

[0022] Step A2-1: For the first feature fusion module, the input it accepts is x i,1 and x i,2 , the initial attention score is: S f,i,1 =Relu(FC(Relu(Fc(MaxPool(x i,1 +x i,2 )))));in, MaxPool(·) represents the maximum pooling operation, FC(·) represents the full connection, and Relu(·) represents the Relu activation function;

[0023] Set two learnable parameters Used to enhance feature x i,1 and x i,2 , and add the enhanced features as the output of the first feature fusion module:

[0024] x f,i,1 =(S f,i,1 ·W1·x i,1 )+(S f,i,1 ·W2·x i,2 );

[0025] Step A2-2: Starting from the second feature fusion module, the jth feature fusion module receives the output x of the j-1th feature fusion module. f,i,j-1 And feature x i,j+1 , then the process of the j-th feature fusion module is:

[0026] S f,i,1 =Relu(FC(Relu(Fc(MaxPool(x i,1 +x i,2 )))));

[0027] The output of the jth feature fusion module is:

[0028] x f,i,j =(S f,i,j ·W1·x i,j+1 )+(S f,i,j·W2·x f,i,j-1 );

[0029] For any feature fusion module, The output of the k-1th feature fusion module corresponding to the i-th image is x f,i,k-1 .

[0030] As a further improvement of the remote sensing scene classification method based on the mask attention mechanism: the processing process of the first classification module also includes:

[0031] Step A3: For image X i , the first classification result output by the first classification module is:

[0032]

[0033] in, Represents the first classification module for X i The predicted category, S represents the number of categories in the remote sensing scene dataset; Linear(·) represents the linear classification function, which converts the feature x f,i,k-1 The dimension of is mapped to S dimension; softmax(·) is a normalized exponential function.

[0034] As a further improvement of the remote sensing scene classification method based on the masked attention mechanism: the second preliminary feature extraction module includes L layers of Transformer encoders connected in sequence, and each Transformer encoder includes a normalization layer LN, a multi-head attention MHSA and a multi-layer perceptron MLP.

[0035] As a further improvement of the remote sensing scene classification method based on the mask attention mechanism: in the second classification module, for the i-th image X to be input i , X i ∈R C×H×W , C represents the number of image channels, H represents the image height, and W represents the image width: Let the input sequence consisting of the image blocks obtained based on the image and to be input into the network model be in Represents image X i The jth image block to be input into the network model, k is the total number of image blocks input into the network model, and the dimension of the image block is p×p;

[0036] The processing of the second classification module includes:

[0037] Step B1: Get sequence F based on the input series i,0 =(x t,i,1 ,x t,i,2 ,…,x t,i,k), F i,0 ∈R D×k , where any x t,i,j , j=1,2,…,k, and its calculation method is: Conv p×p (·) represents a convolution operation with a kernel size of p×p, x t,i,j ∈R D×1 ; e∈R D×1 , represents the position code;

[0038] Step B2: Sequence F i,0 Input to the Transformer encoder and get the image X as follows i The corresponding feature F i,L :

[0039] F i,u ′=MHSA(LN(F i,u-1 ))+F i,u-1 ,u=1,···,L;

[0040] F i,u =MLP(LN(F i,u ′))+F i,u ′,u=1,···,L;

[0041] Among them, LN(·) represents the normalization operation, MLP(·) represents the multi-layer perception processing; MHSA(LN(F i,u-1 The operation process of )) is as follows:

[0042] Step B2-1, LN(F i,u-1 )∈R D×k Segment along the channel dimension to obtain feature groups (L1, L2, ..., L h ), d k =D / h, z=1,…,h, h is a hyperparameter;

[0043] Step B2-2, calculate (H1, H2, ···, H h ):

[0044]

[0045] Among them, softmax(·) is the normalized exponential function, and are all learnable parameter matrices;

[0046] Step B2-3: (H1, H2, ···, H h) is concatenated along the channel dimension to obtain the multi-head attention calculation MHSA(LN(F i,u-1 )) The result of Concat(H1,H2,···,H h )W o , where Concat(·) means stacking along the channel dimension, W o ∈R k×k represents the learnable parameter matrix;

[0047] F i,L ∈R D×k For image X i The feature results obtained by the second preliminary feature extraction module.

[0048] As a further improvement of the remote sensing scene classification method based on the mask attention mechanism: the processing process of the second classification module also includes:

[0049] Step B3: F i,L Input into the dimensionality-raising module Rise to perform three-dimensional attention calculation:

[0050] F R,i =Interpolate(Reshape(F i,L ));

[0051] Among them, Interpolate(·) represents the interpolation operation; Reshape(·) represents the dimension conversion operation;

[0052] Step B4: Set feature F R,i The input is sent to the channel attention module for processing as follows;

[0053] S c,i =Conv(MaxPool c (F R,i ));

[0054] S H,i =Conv(MaxPool H (F R,i ));

[0055] S W,i =Conv(MaxPool W (F R,i ));

[0056] F all,i =(S c,i ·F R,i )+(S H,i ·F R,i )+(S W,i ·F R,i );

[0057] in, MaxPool c (·), MaxPool H (·), MaxPool W (·) represents the global pooling process along the three dimensions of channel, length, and width, respectively, and Conv(·) represents the convolution operation;

[0058] F all,i For image X i The output after processing by the channel attention module.

[0059] As a further improvement of the remote sensing scene classification method based on the mask attention mechanism: the processing process of the second classification module also includes:

[0060] Step B5: For image X i , the second classification result output by the second classification module is:

[0061]

[0062] Represents the second classification module for X i The prediction category of S represents the number of categories in the remote sensing scene dataset; Linear(·) represents the linear classification function, which converts the feature x f,i,k-1 The dimension of is mapped to S dimension; softmax(·) is a normalized exponential function.

[0063] As a further improvement of the remote sensing scene classification method based on the mask attention mechanism: the process of calculating the total loss in step 3 is:

[0064] Step C1: Based on all images X i The corresponding first classification result Calculate the first cross entropy loss L1:

[0065]

[0066] N is the total number of samples input into the network model during training; Y i Represents sample X i The real category of

[0067] Step C2: Based on all images X i The corresponding second classification results Calculate the second cross entropy loss L2:

[0068]

[0069] Step C3: Based on all images X iThe corresponding first classification result And the second classification results Calculate the distillation cross entropy loss L3:

[0070]

[0071] Indicates that The value of the largest element in is set to 1, and the other elements are set to 0;

[0072] Step C4, calculate the total loss:

[0073] L total =αL1+βL2+γL3;

[0074] α, β, and γ are all hyperparameters.

[0075] As a further improvement of the remote sensing scene classification method based on the mask attention mechanism: the result fusion module is used for the input image X i The final classification result is calculated as follows:

[0076]

[0077] For image X i The first classification result obtained in the first classification module, For image X i A second classification result obtained in a second classification module; Indicates that The value of the largest element in is set to 1, and the other elements are set to 0, that is The category corresponding to the largest element in is the final classification result.

[0078] Compared with the prior art, the present invention has the following beneficial effects: the present invention first divides the image into blocks, and then performs a mask operation on the divided image. For the unmasked image blocks, ResNet50 and Transformer encoders are used to perform preliminary feature extraction respectively. The features extracted by the former are further subjected to feature fusion, and the features extracted by the latter are further subjected to feature enhancement. Then, the cross entropy losses of the two paths and the comprehensive distillation cross entropy loss are calculated, and then the total loss is calculated. Finally, the total loss is used to reversely optimize the network model, thereby improving the accuracy of remote sensing scene classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0080] The technical solution of the present invention is described in detail below with reference to the accompanying drawings:

[0081] like Figure 1 , a remote sensing scene classification method based on mask attention mechanism, the steps include:

[0082] Step 1: Establish a network model, which includes a first classification module, a second classification module and a result fusion module.

[0083] The first classification module includes a first preliminary feature extraction module and a feature fusion module, and the feature fusion module is used to fuse multiple groups of features extracted by the first preliminary feature extraction module.

[0084] The second classification module includes a second preliminary feature extraction module and a channel attention module, and the channel attention module is used to enhance the features extracted by the second preliminary feature extraction module.

[0085] The result fusion module is used to fuse the first classification result obtained by the first classification module and the second classification result obtained by the second classification module into a final classification result.

[0086] Step 2: Construct the sample data set required for training. Each image and its corresponding category are recorded as Among them, X i ∈R C×H×W represents the i-th scene image, C represents the number of image channels, H represents the image height, W represents the image width, and Y i Represents the scene image X i Corresponding categories; i = 1, 2, ..., N; N represents the number of scene images.

[0087] Step three, train the first classification module and the second classification model in the network model: divide the images in the sample data set into blocks and perform random mask operations to obtain unmasked image blocks. The purpose of masking is to enable the network model to extract representation information when receiving a small number of features. The unmasked image blocks are input into the first classification module and the second classification module respectively; after all images in the sample data set are processed, two cross entropy losses are calculated according to all the first classification results and all the second classification results, and a distillation loss is calculated based on all the first classification results and all the second classification results, and then the total loss is calculated based on the two cross entropy losses and one distillation loss, and the network model is reversely optimized based on the total loss.

[0088] In this embodiment, the Adam optimizer is used to optimize and solve the loss.

[0089] Step 4: Divide the image to be classified into blocks, and then input them into the trained network model to obtain the final classification result output by the network model, thereby realizing the scene classification of the image to be classified.

[0090] The specific architecture of the network model is as follows:

[0091] The first preliminary feature extraction module is a ResNet50 network.

[0092] Specifically, in the first classification module, for the i-th image X to be input i , X i ∈R C×H×W , C represents the number of image channels, H represents the image height, and W represents the image width: Let the input sequence consisting of the image blocks obtained based on the image and to be input into the network model be in Represents image X i The jth image block to be input into the network model, k is the total number of image blocks input into the network model.

[0093] The processing process of the first classification module includes:

[0094] Step A1, ResNet50 network obtains the following feature sequence according to the input sequence: (x i,1 ,x i,2 ,…,x i,k );in According to the image block The features obtained by the convolution operation with a convolution kernel size of 1x1, the convolution operation with a convolution kernel size of 3x3, and the convolution operation with a convolution kernel size of 1x1 are:

[0095] Furthermore, there are k-1 feature fusion modules; each of the feature fusion modules includes 1 Relu activation function, 1 fully connected layer, 1 Relu activation function, 1 fully connected layer and 1 global pooling layer connected in sequence.

[0096] Step A2: Perform feature fusion through k-1 feature fusion modules:

[0097] Step A2-1: For the first feature fusion module, the input it accepts is x i,1 and x i,2 , the initial attention score is: S f,i,1 =Relu(FC(Relu(Fc(MaxPool(x i,1 +x i,2 )))));in, MaxPool(·) represents the maximum pooling operation, FC(·) represents the fully connected, and Relu(·) represents the Relu activation function.

[0098] Set two learnable parameters Used to enhance feature x i,1 and x i,2 , and add the enhanced features as the output of the first feature fusion module:

[0099] x f,i,1 =(S f,i,1 ·W1·x i,1 )+(S f,i,1 ·W2·x i,2 );

[0100] Step A2-2: Starting from the second feature fusion module, the jth feature fusion module receives the output x of the j-1th feature fusion module. f,i,j-1 And feature x i,j+1 , then the process of the j-th feature fusion module is:

[0101] S f,i,1 =Relu(FC(Relu(Fc(MaxPool(x i,1 +x i,2 )))));

[0102] The output of the jth feature fusion module is:

[0103] x f,i,j =(S f,i,j ·W1·x i,j+1 )+(S f,i,j ·W2·x f,i,j-1 );

[0104] For any feature fusion module, The output of the k-1th feature fusion module corresponding to the i-th image is x f,i,k-1 .

[0105] Step A3: For image X i , the first classification result output by the first classification module is:

[0106]

[0107] in, Represents the first classification module for X i The predicted category, S represents the number of categories in the remote sensing scene dataset; Linear(·) represents the linear classification function, which converts the feature x f,i,k-1 The dimension of is mapped to S dimension; softmax(·) is a normalized exponential function.

[0108] The second preliminary feature extraction module includes L layers of sequentially connected Transformer encoders, each of which includes a normalization layer LN, a multi-head attention MHSA, and a multi-layer perceptron MLP.

[0109] In the specific second classification module, for the i-th image X to be input i , X i ∈R C×H×W , C represents the number of image channels, H represents the image height, and W represents the image width: Let the input sequence consisting of the image blocks obtained based on the image and to be input into the network model be in Represents the image X i The jth image block to be input into the network model, k is the total number of image blocks input into the network model, and the dimension of the image block is p×p.

[0110] The processing of the second classification module includes:

[0111] Step B1: Get sequence F based on the input series i,0 =(x t,i,1 ,x t,i,2 ,…,x t,i,k ), F i,0 ∈R D×k , where any x t,i,j , j=1,2,…,k, and its calculation method is: Conv p×p (·) represents a convolution operation with a kernel size of p×p, x t,i,j ∈R D×1 ; e∈R D×1 , represents the positional encoding.

[0112] Step B2: Sequence F i,0 Input to the Transformer encoder and get the image X as follows i The corresponding feature F i,L :

[0113] F i,u ′=MHSA(LN(F i,u-1 ))+F i,u-1 ,u=1,···,L;

[0114] F i,u =MLP(LN(F i,u ′))+F i,u ′,u=1,···,L;

[0115] Among them, LN(·) represents the normalization operation, MLP(·) represents the multi-layer perception processing; MHSA(LN(F i,u-1 The operation process of )) is as follows:

[0116] Step B2-1, LN(F i,u-1 )∈R D×k Segment along the channel dimension to obtain feature groups (L1, L2, ..., L h ), d k =D / h, z=1,…,h, h is a hyperparameter.

[0117] Step B2-2, calculate (H1, H2, ···, H h ):

[0118]

[0119] Among them, softmax(·) is the normalized exponential function, and are all learnable parameter matrices.

[0120] Step B2-3: (H1, H2, ···, H h ) is concatenated along the channel dimension to obtain the multi-head attention calculation MHSA(LN(F i,u-1 )) The result of Concat(H1,H2,···,H h )W o , where Concat(·) means stacking along the channel dimension, W o ∈R k×k Represents the learnable parameter matrix.

[0121] F i,L ∈R D×k For image X i The feature results obtained by the second preliminary feature extraction module.

[0122] Step B3: F i,L Input into the dimensionality-raising module Rise to perform three-dimensional attention calculation:

[0123] F R,i =Interpolate(Reshape(F i,L ));

[0124] Among them, Interpolate(·) represents the interpolation operation; Reshape(·) represents the dimension conversion operation.

[0125] Step B4: Set feature F R,iThe input is sent to the channel attention module for processing as follows;

[0126] S c,i =Conv(MaxPool c (F R,i ));

[0127] S H,i =Conv(maxPool H (F R,i ));

[0128] S W,i =Conv(MaxPool W (F R,i ));

[0129] F all,i =(S c,i ·F R,i )+(S H,i ·F R,i )+(S W,i ·F R,i );

[0130] in, MaxPool c (·), MaxPool H (·), MaxPool W (·) represents the global pooling processing along the three dimensions of channel, length and width, respectively, and Conv(·) represents the convolution operation.

[0131] F all,i For image X i The output after processing by the channel attention module.

[0132] Step B5: For image X i , the second classification result output by the second classification module is:

[0133]

[0134] Represents the second classification module for X i The prediction category of S represents the number of categories in the remote sensing scene dataset; Linear(·) represents the linear classification function, which converts the feature x f,i,k-1 The dimension of is mapped to S dimension; softmax(·) is a normalized exponential function.

[0135] Furthermore, the process of calculating the total loss in step 3 is:

[0136] Step C1: Based on all images X i The corresponding first classification result Calculate the first cross entropy loss L1:

[0137]

[0138] N is the total number of samples input into the network model during training; Y i Represents sample X i The real category.

[0139] Step C2: Based on all images X i The corresponding second classification results Calculate the second cross entropy loss L2:

[0140]

[0141] Step C3: Based on all images X i The corresponding first classification result And the second classification results Calculate the distillation cross entropy loss L3:

[0142]

[0143] Indicates that The value of the largest element in is set to 1, and the other elements are set to 0.

[0144] Step C4, calculate the total loss:

[0145] L total =αL1+βL2+γL3;

[0146] α, β, and γ are all hyperparameters. In this embodiment, α is set to 0.45, β is set to 0.45, and γ is set to 0.1.

[0147] Furthermore, the result fusion module is used for the input image X i The final classification result is calculated as follows:

[0148]

[0149] For image X i The first classification result obtained in the first classification module, For images A second classification result obtained in a second classification module; Indicates that The value of the largest element in is set to 1, and the other elements are set to 0, that is The category corresponding to the largest element in is the final classification result.

[0150] It should be noted that in steps 3 and 4, for the i-th image X to be input i , the source sequence composed of m image blocks is in Represents image X i The j-th image block; p×p represents the dimension of each image block, the length of the image block is p pixels, the width is p pixels, and m = (H×W) / p 2 .

[0151] In step 3, the image blocks in the source sequence are randomly masked, and the unmasked image blocks are combined into a new sequence as the input sequence. At this time, k <m。

[0152] In step 4, the source sequence is directly used as the input sequence, and k=m at this time.

Claims

1. A remote sensing scene classification method based on masked attention mechanism, characterized by the steps include: Step 1: Establish a network model, wherein the network model includes a first classification module, a second classification module and a result fusion module; The first classification module includes a first preliminary feature extraction module and a feature fusion module, wherein the feature fusion module is used to fuse multiple groups of features extracted by the first preliminary feature extraction module; the second classification module includes a second preliminary feature extraction module and a channel attention module, wherein the channel attention module is used to enhance the features extracted by the second preliminary feature extraction module; The result fusion module is used to fuse the first classification result obtained by the first classification module and the second classification result obtained by the second classification module into a final classification result; Step 2: Construct a sample data set; Step 3: training the first classification module and the second classification model in the network model: performing block division and random masking operations on the images in the sample data set to obtain unmasked image blocks, and inputting the unmasked image blocks into the first classification module and the second classification module respectively; After all images in the sample data set are processed, two cross entropy losses are calculated based on all the first classification results and all the second classification results, and a distillation loss is calculated based on all the first classification results and all the second classification results. The total loss is then calculated based on the two cross entropy losses and one distillation loss, and the network model is reversely optimized based on the total loss; Step 4: Divide the image to be classified into blocks, and then input them into the trained network model to obtain the final classification result output by the network model, so as to realize the scene classification of the image to be classified; the first preliminary feature extraction module is a ResNet50 network; In the first classification module, for the i-th image X to be input i , X i ∈R C×H×W , C represents the number of image channels, H represents the image height, and W represents the image width: Let the input sequence consisting of the image blocks obtained based on the image and to be input into the network model be in Represents the image X i The jth image block to be input into the network model, k is the total number of image blocks input into the network model; The processing process of the first classification module includes: Step A1, ResNet50 network obtains the following feature sequence according to the input sequence: (x i,1 ,x i,2 ,…,x i,k );in According to the image block The features obtained by the convolution operation with a convolution kernel size of 1x1, the convolution operation with a convolution kernel size of 3x3, and the convolution operation with a convolution kernel size of 1x1 are: The number of the feature fusion modules is k-1; each of the feature fusion modules comprises a Relu activation function, a fully connected layer, a Relu activation function, a fully connected layer and a global pooling layer connected in sequence; Then the processing process of the first classification module also includes: Step A2: Perform feature fusion through k-1 feature fusion modules: Step A2-1: For the first feature fusion module, the input it accepts is x i,1 and x i,2 , the initial attention score is: S f,i,1 =Relu(FC(Relu(Fc(MaxPool(x i,1 +x i,2 )))));in, MaxPool(·) represents the maximum pooling operation, FC(·) represents the full connection, and Relu(·) represents the Relu activation function; Set two learnable parameters Used to enhance feature x i,1 and x i,2 , and add the enhanced features as the output of the first feature fusion module: x f,i,1 =(S f,i,1 ·W1·x i,1 )+(S f,i,1 ·W2·x i,2 ); Step A2-2: Starting from the second feature fusion module, the jth feature fusion module receives the output x of the j-1th feature fusion module. f,i,j-1 And feature x i,j+1 , then the process of the j-th feature fusion module is: S f,i,1 =Relu(FC(Relu(Fc(MaxPool(x). i,1 +x i,2 )))))) The output of the jth feature fusion module is: x f,i,j =(S f,i,j ·W1·x i,j+1 )+(S f,i,j ·W2·x f,i,j-1 ); For any feature fusion module, The output of the k-1th feature fusion module corresponding to the i-th image is x f,i,k-1 ; The processing of the first classification module also includes: Step A3: For image X i , the first classification result output by the first classification module is: in, Represents the first classification module for X i The predicted category, S represents the number of categories in the remote sensing scene dataset; Linear(·) represents the linear classification function, which converts the feature x f,i,k-1 The dimension of is mapped to S dimension; sovtmax(·) is the normalized exponential function; The second preliminary feature extraction module includes L layers of sequentially connected Transformer encoders, each of which includes a normalization layer LN, a multi-head attention MHSA, and a multi-layer perceptron MLP; In the second classification module, for the i-th image X to be input i , X i ∈R C×H×W , C represents the number of image channels, H represents the image height, and W represents the image width: Let the input sequence consisting of the image blocks obtained based on the image and to be input into the network model be in Represents image X i The jth image block to be input into the network model, k is the total number of image blocks input into the network model, and the dimension of the image block is p×p; The processing of the second classification module includes: Step B1: Get sequence F based on the input series i,0 =(x t,i,1 , x t,i,2 , ..., x t,i,k ), F i,0 ∈R D×k , where any x t,i,j , j = 1, 2, ..., k, and the calculation method is: Conv p×p (·) represents a convolution operation with a kernel size of p×p, x t,i,j ∈R D×1 ; e∈R D×1 , represents the position code; Step B2: Sequence F i,0 Input to the Transformer encoder and get the image X as follows i The corresponding feature F i,L : F i,u ′=MHSA(LN(F i,u-1 ))+F i,u -1,u=1,…,L; F i,u =MLP(LN(F i,u ′))+F i,u ′,u=1,…,L; Among them, LN(·) represents the normalization operation, MLP(·) represents the multi-layer perception processing; MHSA(LN(F i,u-1 The operation process of )) is as follows: Step B2-1, LN(F i,u-1 )∈R D×k Segment along the channel dimension to obtain feature groups (L1, L2, ..., L h ), d k =D / h, z=1,...,h, h is a hyperparameter; Step B2-2, calculate (H1, H2, ..., H h ): Among them, softmax(·) is the normalized exponential function, and are all learnable parameter matrices; Step B2-3: (H1, H2, ..., H h ) is concatenated along the channel dimension to obtain the multi-head attention calculation MHSA(LN(F i,u-1 )) The result of Concat(H1, H2, ..., H h )W o , where Concat(·) means stacking along the channel dimension, W o ∈R k×k represents the learnable parameter matrix; F i,L ∈R D×k For image X i Feature results obtained by the second preliminary feature extraction module; The processing of the second classification module also includes: Step B3: F i,L Input into the dimensionality-raising module Rise to perform three-dimensional attention calculation: F R,i =Interpolate(Reshape(F i,L )); Among them, Interpolate(·) represents the interpolation operation; Reshape(·) represents the dimension conversion operation; Step B4: Set feature F R,i The input is sent to the channel attention module for processing as follows; S c,i =Conv(MaxPoof c (F R,i )); S H,i =Conv(MaxPoof H (F R,i )); S W,i =Conv(MaxPoof W (F R,i )); F aff,i =(S c,i ·F R,i )+( H,i ·F R,i )+(S W,i ·F R,i ); in, MaxPool c (·), MaxPool H (·), MaxPool W (·) represents the global pooling process along the three dimensions of channel, length, and width, respectively, and Conv(·) represents the convolution operation; F afl,i For image X i Output result after processing by the channel attention module; The processing of the second classification module also includes: Step B5: For image X i , the second classification result output by the second classification module is: Represents the second classification module for X i The prediction category of S represents the number of categories in the remote sensing scene dataset; Linear(·) represents the linear classification function, which converts the feature x f,i,k-1 The dimension of is mapped to S dimension; sovtmax(·) is the normalized exponential function.

2. The remote sensing scene classification method based on masked attention mechanism according to claim 1, characterized in that: The process of calculating the total loss in step 3 is: Step C1: Based on all images X i The corresponding first classification result Calculate the first cross entropy loss L1: N is the total number of samples input into the network model during training; Y i Represents sample X i The real category of Step C2: Based on all images X i The corresponding second classification results Calculate the second cross entropy loss L2: Step C3: Based on all images X i The corresponding first classification result And the second classification results Calculate the distillation cross entropy loss L3: Indicates that The value of the largest element in is set to 1, and the other elements are set to 0; Step C4, calculate the total loss: L total =αL1+βL2+γL3; α, β, and γ are all hyperparameters.

3. The remote sensing scene classification method based on masked attention mechanism according to claim 1, characterized in that: The result fusion module is for the input image X i The final classification result is calculated as follows: For image X i The first classification result obtained in the first classification module, For image X i A second classification result obtained in a second classification module; Indicates that The value of the largest element in is set to 1, and the other elements are set to 0, that is The category corresponding to the largest element in is the final classification result.