A remote sensing image scene classification method

Abstract

The application relates to a remote sensing image scene classification method and belongs to the technical field of remote sensing image processing. The application first extracts feature maps of different scales of an image by using a deep convolutional neural network; then residual attention mechanism is used to enhance semantic information of the feature maps of different scales and suppress redundant noise information; finally, global mean pooling is used to obtain global information of the feature maps of different scales, a feature vector is constructed, feature vectors of different scales are fused, the feature vectors of different scales and the fused feature vector are classified by using independent classifiers respectively; during training, a joint loss is used to optimize the model, and during prediction, a multi-classifier decision level fusion mode is adopted, so that the robustness and classification precision of classification are improved. Experiments are conducted on UC Merced, AID and NWPU-RESISC45 data sets, the feasibility and effectiveness of the application are verified, and compared with other same type methods, the classification precision of the application is significantly improved.

Description

Technical Field

[0001] This invention relates to a method for classifying scenes in remote sensing images, belonging to the field of remote sensing image processing technology. Background Technology

[0002] With the development of Earth observation technology, the image data acquired using various remote sensing platforms is growing explosively. Remote sensing imagery has entered the era of big data, and how to efficiently organize, analyze, and utilize massive amounts of remote sensing imagery has become a pressing research issue. Remote sensing image scene classification is one of the important methods of remote sensing image interpretation, aiming to determine and identify the categories of remote sensing images. Therefore, remote sensing image scene classification has important and wide-ranging applications in land use surveys, urban planning, target detection, map updating, and geological hazard detection and assessment.

[0003] Feature fusion-based and attention-based methods are currently the two most widely used methods that significantly improve the accuracy of remote sensing image scene classification. However, with the increase in remote sensing image resolution and the refinement of image categories, problems such as scale differences, inter-class similarity, and intra-class differences in remote sensing images are becoming increasingly prominent. General convolutional neural networks still have room for improvement in their ability to distinguish similar but different categories of images. Many methods, when using feature fusion for image classification, do not distinguish the importance or differences of features of different types or scales, but only use simple addition or concatenation. If one feature becomes dominant, other features may not be able to play their role. When using attention mechanisms for feature enhancement, while reducing redundant noise information, useful information for classification may be lost, resulting in relatively low classification accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a remote sensing image scene classification method to solve the problem of low classification accuracy caused by the loss of useful information in current classification methods.

[0005] To solve the above-mentioned technical problems, this invention provides a remote sensing image scene classification method, which includes the following steps:

[0006] 1) Construct a scene classification model, which includes a multi-feature extraction network, a residual attention module, and a classifier. The multi-feature extraction network is used to extract feature maps at different scales using a deep convolutional neural network. The residual attention module is used to enhance the feature maps at different scales and use pooling to convert the enhanced feature maps at each scale into feature vectors corresponding to each scale. The feature vectors are then fused to obtain a multi-scale fused feature vector. The classifier includes multiple classifiers, which are used to classify the feature vectors at each scale and the fused feature vectors respectively.

[0007] 2) Jointly train the constructed scene classification model;

[0008] 3) Acquire the remote sensing image to be classified and input it into the trained scene classification model. Perform decision-level fusion based on the classification results of the feature vectors at each scale and the fused feature vectors, and use the fused result as the final classification result.

[0009] This invention first utilizes a deep convolutional neural network to extract feature maps at different scales of images; then, it employs a residual attention mechanism to enhance the semantic information of feature maps at different scales and suppress redundant noise information; finally, it uses global mean pooling to obtain global information from feature maps at different scales, constructs feature vectors, and fuses feature vectors at different scales. Each feature vector at a different scale and the fused feature vector are classified using independent classifiers. During training, a joint loss optimization model is used, and during prediction, a multi-classifier decision-level fusion approach is adopted, which improves the robustness and accuracy of classification.

[0010] Furthermore, the multi-feature extraction network adopts a ResNet network or a RegNet network.

[0011] This invention utilizes existing ResNet or RegNet networks to extract features at different scales, ensuring that the extracted feature maps contain rich semantic information and providing reliable support for subsequent classification.

[0012] Furthermore, the residual attention module performs dimensionality transformation and feature enhancement as follows:

[0013] a. Unify the number of channels in feature maps of different scales to the same number and input them into the weight branch and feature extraction branch;

[0014] b. The input feature map is downsampled, convolved, and upsampled through the weight branch, and features are extracted through the feature extraction branch by performing convolution operations on the input feature map.

[0015] c. The results of the weight branch processing and the features extracted by the feature extraction branch are superimposed, and the superimposed results are processed by the convolution module and activation function to obtain the corresponding normalized weights;

[0016] d. Multiply the obtained normalized weights by the features extracted by the feature extraction branch, and then add the multiplication result to the features extracted by the feature extraction branch to achieve feature enhancement.

[0017] This invention replaces the residual module with a convolutional module, reducing computational load. Furthermore, during weight determination, the weight branch is added to the feature extraction branch, establishing a connection between them. The residual attention module of this invention not only amplifies the difference between the information of the interest target and redundant noise information but also ensures that the information contained in the attention-enhanced new feature map is no less than that in the original feature map.

[0018] Furthermore, the feature extraction branch uses two 1×1 convolutional modules to extract features from the feature map.

[0019] This invention uses two 1×1 convolutional modules to extract features from the feature map, ensuring that the extracted features retain the original feature information.

[0020] Furthermore, the residual attention module uses global mean pooling to transform the enhanced feature map into a feature vector.

[0021] This invention uses global mean pooling to transform feature maps, quickly converting them into corresponding feature vectors.

[0022] Furthermore, the residual attention module is used to connect feature vectors at each scale along the channel dimension to achieve the fusion of features at each scale.

[0023] This invention uses a channel dimension overlay method to achieve the fusion of features at different scales.

[0024] Furthermore, the scene classification model is trained separately for each classifier during training, and the loss function used for each classifier is:

[0025]

[0026] q′(k|x)=(1-ε)q(k|x)+ε / K

[0027] Where x represents a sample, k represents the class of the sample, K represents the number of classes, p(k|x) represents the probability that the sample x output by the i-th classifier belongs to class k, q′(k|x) represents the label after smoothing, and ε is the smoothing parameter.

[0028] During the training process, this invention employs the Label Smoothing loss function to improve the generalization of the model, thus avoiding the situation where fully connected layers require output values ​​of negative infinity.

[0029] Furthermore, the joint loss used during training is a weighted sum of the losses of each classifier.

[0030] Furthermore, the decision-level fusion in step 3) is to add up the probability outputs predicted by each classifier. Attached Figure Description

[0031] Figure 1 This is a schematic diagram illustrating the principle of remote sensing image scene classification methods.

[0032] Figure 2a This is a schematic diagram of the existing residual attention module structure;

[0033] Figure 2b This is a schematic diagram of the residual attention module structure used in this invention. Detailed Implementation

[0034] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0035] This invention first utilizes a deep convolutional neural network to extract feature maps at different scales of images; then, it employs a residual attention mechanism to enhance the semantic information of feature maps at different scales and suppress redundant noise; finally, it uses global mean pooling to obtain global information from feature maps at different scales, constructs feature vectors, and fuses feature vectors at different scales. Each feature vector at a different scale, as well as the fused feature vector, is classified using an independent classifier. During training, a joint loss optimization model is used, and during prediction, a multi-classifier decision-level fusion approach is adopted, improving the robustness and accuracy of classification. The specific implementation process is described in detail below.

[0036] 1. Construct a scene classification model

[0037] like Figure 1 As shown, the scene classification model constructed in this invention includes a multi-feature extraction network, a residual attention module, and classifiers. The multi-feature extraction network is used to extract feature maps at different scales using a deep convolutional neural network; the residual attention module is used to perform dimensionality transformation and feature enhancement on the feature maps at different scales; global mean pooling is used to convert the enhanced feature maps at each scale into feature vectors corresponding to each scale, and the feature vectors are fused to obtain a multi-scale fused feature vector; the classifier includes multiple classifiers, which are used to classify the feature vectors at each scale and the fused feature vectors respectively. The following is a detailed description of each of the above components.

[0038] 1) Multi-feature extraction network

[0039] In scene classification, convolutional neural networks (CNNs) are typically used to extract global semantic information from images. Commonly used CNNs like VGGNet and ResNet can transform images into 512 or 2048-dimensional feature vectors, which represent the global semantic information of the image. To reduce the number of parameters and computational cost, CNNs use pooling layers multiple times. After multiple pooling operations, the output feature map is only 1 / 32 the size of the original image. For example, if the input image is 256×256×3 pixels, the output feature map will be 8×8×512 pixels. This means that targets smaller than a certain pixel size are difficult to pass to the final classification layer after multiple pooling operations, and may even be completely lost. This is detrimental to both target detection and image classification. Remote sensing images have a top-down imaging perspective and often contain multiple targets of different sizes. Using only the final feature map from a CNN for scene classification can easily overlook small targets and the contextual information between them. Therefore, to improve the accuracy of remote sensing image scene classification, extracting multi-scale information with different receptive fields from the image is crucial.

[0040] Therefore, to effectively extract multi-scale information from images, this invention employs two types of convolutional neural networks, ResNet and RegNet, as backbone networks to extract features at different scales. ResNet consists of several residual modules, each employing a feature cross-layer connection method, which can transmit the main features of the image to the bottom of the network, overcoming to some extent the problems of gradient vanishing and model overfitting that occur during the training of deep convolutional neural networks. RegNet inherits the basic ideas of ResNet and further explores the structural design principles of convolutional neural networks, using a quantization linear function to determine the width and depth of the network, thus improving the interpretability of the convolutional neural network structure design. ResNet and RegNet are not single convolutional neural networks, but rather each contains a series of convolutional neural networks of different depths. The depth of the network is determined by the number of residual modules (the more residual modules, the stronger the network performance, but the greater the number of parameters and computational cost).

[0041] Based on the size of the output feature map, both ResNet and RegNet are divided into 5 stages (e.g., Figure 1 As shown on the left), after each stage of processing, the number of channels in the input image or feature map is expanded, while the width and height are reduced to half their original size. For example, when the input image size is 256×256×3 pixels, the feature maps output at different stages are denoted as C1, C2, C3, C4, and C5, with sizes of 128×128×n1, 64×64×n2, 32×32×n3, 16×16×n4, and 8×8×n5, respectively. iThe number of channels is represented by i∈{1,2,3,4,5}. Feature maps located in the shallow layers of convolutional neural networks are large in size, high in resolution, and small in receptive field, with rich spatial detail information but weak semantic information. Feature maps located in the deep layers of convolutional neural networks are small in size, low in resolution, and large in receptive field, with severe loss of spatial detail information but rich in global semantic information. The feature maps (C1 and C2) in the first two stages of ResNet and RegNet do not have enough semantic information and are rarely used for image classification and object detection tasks. Therefore, this invention extracts the feature maps C3, C4, and C5 from the last three stages (referred to as low-level (small-scale) feature map, mid-level feature map (medium-scale) and high-level feature map (large-scale) respectively) for remote sensing image scene classification tasks.

[0042] 2) Residual Attention Module

[0043] Attention mechanisms can quickly filter out interesting and high-value information from a large amount of data, and effectively integrate the acquired information from various parts to form a holistic perception of the target. The core idea of ​​attention mechanisms in convolutional neural networks is to adaptively learn weights for different channels or spatial locations on the feature map, multiplying the learned weights by the feature map to highlight information beneficial to target recognition and suppress redundant noise. These weights are essentially a normalized feature map (weight values ​​between 0 and 1). Although multiplying weights by the feature map amplifies the difference between the information of the target of interest and redundant noise, most of the information on the feature map is weakened. In extreme cases, when most weight values ​​are 0, most of the information in the original feature map will be lost. Therefore, when using attention mechanisms, it is necessary to ensure that the main information on the original feature map is not eliminated as much as possible. The structure of existing residual attention modules is as follows: Figure 2a As shown, it contains two branches. The first branch is used to generate weights, and the second branch is used to extract and preserve the original features. The two branches are independent of each other. The important information on the feature map is further enhanced by the residual connection, which avoids the problem of the weights generated by the ordinary attention mechanism weakening the information of the feature map. However, it requires multiple residual modules (ResBlock) and the computation is relatively large.

[0044] Therefore, this invention further improves and simplifies the existing residual attention module, such as... Figure 2b As shown, this residual attention module also includes two branches: a weight branch and a feature extraction branch. Before the weight branch generates weights, a feature fusion method is used to add the weight branch and the feature extraction branch, establishing a connection between the weight branch and the feature extraction branch. Figure 2aThe residual attention module (ResBlock) containing multiple convolutional layers used in this invention has been replaced with a convolutional module (ConvBlock) containing only a single convolutional layer, reducing the number of parameters and computational cost. Based on this invention, the aforementioned residual attention module is proposed. The following describes the processing of feature maps C3, C4, and C5 extracted by ResNet and RegNet in this invention using the residual attention module.

[0045] Since the extracted feature maps C3, C4, and C5 have different scales and channel counts, to facilitate subsequent feature map fusion, this invention first uses a 1×1 convolution module to unify the number of channels in the extracted feature maps C3, C4, and C5 of different scales to the same number (in this embodiment, it is set to 512). Then, the transformed feature maps are input into the weight branch and the feature extraction branch: the weight branch downsamples the feature maps, and then restores them to their original size through two 1×1 convolution modules and upsampling (using the "pooling-convolution-upsampling" method to effectively expand the receptive field of the feature maps, preserving and enhancing them). The feature extraction branch uses two 1×1 convolutional modules to extract features from the feature map, preserving the original feature information. The processing results of the two branches are added together, and the result is passed through two 1×1 convolutional modules and a sigmoid activation function to determine the normalized weights. The features extracted by the feature extraction branch are multiplied by the obtained weights to enhance the features. Finally, the enhanced features of the feature extraction branch are added together with the features extracted by the feature extraction branch, taking into account both the original features and the significant features enhanced by attention. Even when the attention weight is close to 0, the original feature information can still be preserved.

[0046] Therefore, the residual attention module of this invention not only expands the difference between the information of the interest target and the redundant noise information, but also ensures that the information contained in the new feature map after attention enhancement is no less than that in the original feature map. For a given feature map X∈w×h×c, its feature map XRA after residual attention enhancement can be expressed as:

[0047] X0 = CBR(X) (1)

[0048] X1=Upsample(CBR(CBR(Maxpool(X0)))) (2)

[0049] X2 = CBR(CBR(X0)) (3)

[0050]

[0051] In the formula, CBR(·) indicates that the feature map has been processed by convolution, batch normalization, and modified linear unit activation function; CB(·) indicates that the feature map has been processed by convolution and batch normalization; σ is the Sigmoid activation function.

[0052] In addition to enhancing the obtained feature maps at different scales, the residual attention module also needs to convert the enhanced feature maps at each scale into feature vectors and then fuse them. This invention uses global mean pooling to convert the feature maps into feature vectors, and then fuses the feature vectors by concatenating them along the channel dimension. In this embodiment, the enhanced features at each scale are converted into 512-dimensional feature vectors. This can be expressed using the following formula.

[0053] V1 = F GAP (F AM (C3)) (5)

[0054] V2 = F GAP (F AM (C4)) (6)

[0055] V3 = F GAP (F AM (C5)) (7)

[0056] V4 = concat(V1, V2, V3) (8)

[0057] In the formula F AM (·) represents the mathematical expression corresponding to residual attention, F GAP (·) is the global mean pooling function, V1, V2, and V3 are feature vectors generated from feature maps at different scales, V4 is the feature vector fused by concatenation, and concat(·) means concatenating the feature maps along the channel dimension.

[0058] 3) Classifier

[0059] In this embodiment, since three feature maps of different scales are extracted, and the fused features are added, the classifier of this invention includes four classes, which are used to classify the three feature vectors obtained from the three different scales and the fused feature vector, respectively. For this invention, the classifier is implemented using fully connected layers, that is, four fully connected layers are used to classify the feature vectors V1, V2, V3 and V4 respectively.

[0060] 2. Train the constructed classification model.

[0061] Since the classification model of this invention uses multiple classifiers, multi-loss joint training is adopted when training the classification model. That is, the losses of multiple classifiers are added together during training, and the model is trained according to the total loss value.

[0062] In this invention, the obtained multi-scale feature vectors and fused feature vectors {V1, V2, V3, V4} are respectively input into four parallel fully connected layer classifiers {cls1, cls2, cls3, cls 4} Calculate the probability of its corresponding category k (k∈{1,2,…,K}, where K is the total number of categories).

[0063] p i (k|x)=F softmax (cls i (V i (9)

[0064] In the formula cls i p represents the classifier of the i-th fully connected layer. i (k|x) represents the probability that the sample x output by the i-th classifier belongs to class k, and the function F softmax (·) can be represented as

[0065]

[0066] The true class of sample x is y. One-hot encoding is used when creating the training labels, with positive examples set to 1 and negative examples set to 0, i.e., q(k=y|x)=1, q(k≠y|x)=0. The cross-entropy loss function L... CE It can be defined as:

[0067]

[0068] As shown in equation (10), for the predicted value p(k|x), ideally, to achieve a loss value of 0, the output value of the fully connected layer corresponding to positive examples should be constant, while the output value corresponding to negative examples should be negative infinity (it is difficult for convolutional neural networks to produce output results of negative infinity). During training, if the output values ​​of the correctly predicted class and the incorrect class are too far apart, the model will focus more on the correct class samples. When there is insufficient training data, imbalanced data, or mislabels, the model is prone to overfitting. Therefore, this invention uses the label smoothing loss function (Label Smoothing) to improve the generalization of the model during training. In the label smoothing function, the true negative label is set to a non-zero decimal to soften the one-hot label and avoid the situation where the fully connected layer needs an output value of negative infinity. The smoothed label is:

[0069] q′(k|x)=(1-ε)q(k|x)+ε / K (12)

[0070] Then, for a single fully connected layer classifier, the loss L after smoothing the labels of sample x is... LS It can be represented as:

[0071]

[0072] In the formula, ε is the smoothing parameter, which is generally taken as 0.1. The eigenvector V is calculated according to equations (12) and (13). i The corresponding loss value and its joint loss value L

[0073]

[0074] In the formula λ i Represents the eigenvector V i The weight of the corresponding loss value in the joint loss is i∈{1,2,3,4}. Four feature vectors at different scales employ independent loss functions, and joint training with multiple losses achieves multi-objective optimization. Different classifiers are used for feature vectors at different scales, ensuring that features at different scales can independently predict the category of the input image. This avoids the problem of low feature utilization efficiency when different features are directly added or concatenated, improving the model's ability to identify images or targets at different scales.

[0075] 3. Use the trained classification model to classify the remote sensing images to be classified.

[0076] The remote sensing image to be classified is acquired and input into a trained scene classification model. Decision-level fusion is performed based on the classification results of the classifiers for feature vectors at different scales and the fused feature vectors. The fused result is used as the final classification result. Specifically, when using convolutional neural networks for image classification, generally only one classifier (the last fully connected layer) is used to output the prediction result: when predicting the input image, the category corresponding to the highest probability in the prediction value p is the predicted category of the image, i.e., P = arg max(p). For the multiple classifiers in this invention, the classification results of multiple classifiers need to be combined. Utilizing the ideas of ensemble learning and decision-level fusion can effectively improve the model's performance. Therefore, when predicting the image category, this invention adds the probability outputs of the four classifiers (p1, p2, p3, p4) to obtain the final prediction result Ps, i.e.,

[0077] Ps=arg max(p1+p2+p3+p4) (15)

[0078] Experimental verification

[0079] To verify the performance of this invention on remote sensing image scene classification tasks, experiments were conducted on three widely used benchmark datasets for remote sensing image scene classification: UC Merce (hereinafter referred to as UCM), AID, and NWPU-RESISC4 (hereinafter referred to as NWPU). The UCM dataset contains 21 classes and 2100 images, with an image size of 256×256 pixels and a resolution of approximately 0.3m; the AID dataset contains 30 classes and 10000 images, with an image size of 600×600 pixels and a resolution of approximately 0.3m-8m; and the NWPU dataset contains 45 classes and 31500 images, with an image size of 256×256 pixels and a resolution of approximately 0.2m-30m. During the experiments, a certain proportion of data from each dataset was selected as training data, and the remaining data was used as test data. When dividing the training and test data, for each dataset and at each training ratio (TR), the images in the training set were randomly selected from all images, and the mean of five training and test results was calculated as the final result.

[0080] This experiment utilizes the PyTorch deep learning framework to construct a convolutional neural network model, and conducts experiments on an RTX 2080 Ti hardware device with 11GB of VRAM. The network structure parameters are set as follows: pre-trained weights of the backbone network are loaded as initial parameters, with an initial learning rate of 0.001 for the backbone network and 0.01 for all other parts of the model; a total of 60 epochs are trained, using exponential decay to reduce the learning rate, with a learning rate decay parameter set to 0.96. The model is optimized using stochastic gradient descent, with a weight decay factor of 0.0005 and a momentum value of 0.9; the batch size is 32. Overall accuracy (OA) is used as the evaluation metric, which is the proportion of correctly classified images out of all test images. In the experiment, eight convolutional neural networks of different depths, RegNet and ResNet, were selected as the backbone networks of the invented method. The eight convolutional neural networks are RegNetX-3.2G (reg32), RegNetX-4.0G (reg40), RegNetX-6.4G (reg64), RegNetX-8.0G (reg80), ResNet18 (res18), ResNet34 (res34), ResNet50 (res50) and ResNet101 (res101).

[0081] Experiment 1: The impact of residual attention modules on the performance of classification models

[0082] To further explore the impact of attention mechanisms on model performance, different attention mechanisms such as SE (Squeeze-and-Excitation), CBAM (Convolutional Block Attention Module), and others were tried in multi-feature joint loss training. Figure 2a The residual attention module (referred to as the residual mechanism) and the impact of different attention methods on model performance are shown in Table 1. SE is a channel attention method, while CBAM uses both channel and spatial attention. Compared with the JLT method without attention, the classification accuracy of the two datasets was not significantly improved. The reason for this phenomenon may be that SE and CBAM attention are usually nested in the residual module and stacked multiple times throughout the network, while this invention only uses them before feature fusion and classification, thus their impact on network performance is limited. In addition, SE and CBAM finally use the Sigmoid activation function, and the output weights are between 0 and 1, which may weaken the overall information of the feature map. Residual attention, by introducing residual connections into the attention mechanism, can effectively solve the problem of feature map information weakening. Therefore, it can be used in different backbone networks. Figure 2a The residual attention method has a better impact on model performance than the SE and CBAM methods. The residual attention module RAM designed in this invention further enhances the model's ability to extract and express semantic information, thereby improving the model's classification accuracy.

[0083] Table 1

[0084]

[0085] Note: × indicates the multi-feature joint loss training method JLT that does not use an attention mechanism.

[0086] Experiment 2: Performance Comparison of Different Classification Methods

[0087] Augmenting the training data during model training can further improve the model's generalization performance. Therefore, after expanding the training data by rotating and flipping the images, the overall classification accuracy of the method of this invention compared with other remote sensing image scene classification methods based on convolutional neural networks is shown in Table 2. From the classification results, most methods achieve an overall accuracy of 99% or higher on the UCM dataset (TR = 80%). The advantage of the method of this invention is not obvious, mainly because: 80% of the images are used for training and 20% for testing, resulting in a large proportion of training images, making it difficult to accurately judge the model's generalization performance; the UCM dataset was released in 2010, with a relatively small number of image categories and low image complexity. Furthermore, methods such as HABFNet, MBCNet, SCCov16, and ACR-MLFF, which perform well on the UCM dataset, show significantly lower generalization ability than the method of this invention when facing the AID and NWPU datasets, which have more categories and larger data volumes.

[0088] Based on the classification results across the three datasets, it is evident that the performance of this invention significantly outperforms other existing methods. From a classification methodology perspective, the classification accuracy of attention-based methods (RADC-Net), multi-scale feature fusion methods (MBCNet employs a multi-branch feature addition fusion approach), and methods combining attention and multi-scale feature fusion (ACNet, ACR-MLFF) are all significantly lower than that of this invention. This verifies the feasibility and effectiveness of the attention-guided feature fusion and multi-feature joint loss training method employed in this invention. This invention provides a novel approach for remote sensing image scene classification tasks.

[0089] This experiment used two types of deep convolutional neural networks, ResNet and RegNet, to extract feature maps at different scales. Table 2 compares the performance of different backbone networks, showing that RegNet significantly outperforms ResNet. Furthermore, when using the lightweight res18 as the backbone network, the proposed classification performance significantly surpasses many res50-based methods (such as HABFNet and SAFF), demonstrating the superior performance of this invention and its potential in lightweight scene classification tasks. In addition, the performance of the backbone network directly affects the overall network performance; different backbone networks also have significantly different parameter counts and computational costs. Therefore, for specific scene classification tasks, it is crucial to select an appropriate backbone network, attention mechanism, and joint loss weights.

[0090] Table 2

[0091]

[0092] Note: Bold text indicates the optimal value, and underlined text indicates the suboptimal value.

[0093] Compared to other fusion methods that add or connect features at different scales, this invention employs residual attention to enhance information in multi-scale feature maps, designs a multi-objective optimization method to enable features of different sizes to independently predict image categories, and uses decision-level fusion to obtain stable and reliable prediction results. Experimental results on the UC Merced, AID, and NWPU-RESISC45 datasets verify the feasibility and effectiveness of this invention. Compared to other similar methods, the classification accuracy of this invention is significantly improved.

Claims

1. A method for classifying scenes in remote sensing images, characterized in that, This classification method includes the following steps: 1) Construct a scene classification model, which includes a multi-feature extraction network, a residual attention module, and a classifier. The multi-feature extraction network is used to extract feature maps of different scales using a deep convolutional neural network. The residual attention module is used to unify the number of channels of feature maps of different scales to the same number and input them into the weight branch and the feature extraction branch. The weight branch performs downsampling, convolution, and upsampling on the input feature map, and the feature extraction branch performs convolution on the input feature map to extract features. The results of the weight branch processing and the features extracted by the feature extraction branch are superimposed. The superposition result is processed by the convolution module and the activation function to obtain the corresponding normalized weights. The obtained normalized weights are multiplied by the features extracted by the feature extraction branch, and the multiplication result is added to the features extracted by the feature extraction branch to achieve feature enhancement. Pooling is used to convert the enhanced feature maps at each scale into feature vectors corresponding to each scale, and the feature vectors are fused to obtain a multi-scale fused feature vector; the classifier includes multiple classifiers, which are used to classify the feature vectors at each scale and the fused feature vectors respectively. 2) Jointly train the constructed scene classification model; 3) Acquire the remote sensing image to be classified and input it into the trained scene classification model. Perform decision-level fusion based on the classification results of the feature vectors at each scale and the fused feature vectors, and use the fused result as the final classification result.

2. The remote sensing image scene classification method according to claim 1, characterized in that, The multi-feature extraction network described above uses either ResNet or RegNet.

3. The remote sensing image scene classification method according to claim 1, characterized in that, The feature extraction branch uses two 1×1 convolutional modules to extract features from the feature map.

4. The remote sensing image scene classification method according to claim 1, characterized in that, The residual attention module uses global mean pooling to transform the enhanced feature map into a feature vector.

5. The remote sensing image scene classification method according to claim 1, characterized in that, The residual attention module is used to connect feature vectors at each scale along the channel dimension to achieve the fusion of features at each scale.

6. The remote sensing image scene classification method according to claim 1, characterized in that, The scene classification model described above trains each classifier separately during training, and the loss function used for each classifier is as follows: in Indicates a sample, Indicates the category of the sample. Let p(k|x) represent the number of categories, and p(k|x) represent the probability that the sample x output by the i-th classifier belongs to category k. This indicates the label after smoothing. For smoothing parameters.

7. The remote sensing image scene classification method according to claim 6, characterized in that, The joint loss used during training is a weighted sum of the losses of each classifier.

8. The remote sensing image scene classification method according to claim 1, characterized in that, The decision-level fusion in step 3) is to add up the probability outputs predicted by each classifier.

Images