A remote sensing image scene classification method based on a multi-channel self-attention network

Abstract

The application provides a remote sensing image scene classification method based on a multi-channel self-attention network. A traditional method does not effectively combine local information and global features in an image, and the relationship between features in each layer in the channel dimension is not further explored. To solve this problem, a scene classification model fusing local-global features is provided. The method comprises the following steps: multi-scale features of a remote sensing image are extracted through a double-branch convolutional network; then, a feature fusion unit is used to establish the relationship between the multi-scale features, a Transformer Encoder based on a multi-head attention mechanism is used to model the fused features in the channel dimension, the relationship between the features in the channels is derived, the global receptive field is further expanded, and the semantic structure information is captured, so that the classification accuracy of the network is effectively improved. Experiments prove that the method has good practicability and can be widely applied to accurate identification of remote sensing image scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the image recognition problem in the field of computer vision, and in particular to a scene classification method based on remote sensing images. BACKGROUND

[0002] Compared with traditional natural images, remote sensing images have the characteristics of macro, objective, comprehensive, real-time, dynamic, fast, etc., and have higher spatial resolution and spectral resolution, and the description of ground objects is more detailed. The rich local texture features, color features and global structure features of remote sensing images make remote sensing images widely used in land use and cover, environmental protection, natural disaster detection, urban planning and carbon emission detection and many other fields.

[0003] For remote sensing image scene classification, although the machine learning-based algorithm can extract some low-level features in the image, it cannot well perceive high-level semantic features, and the generalization ability is insufficient for multi-classification problems. Although the convolutional neural network (CNN) based on deep learning effectively expands the feature receptive field through a deep network, the representation of semantic features has been significantly improved, but CNN is often limited to the perception of local features and lacks the expression of global structure information. The Visual Transformer based on Transformer effectively establishes long-distance feature dependency through multi-head attention and multi-layer perception, and enhances the representation of global information, but lacks local texture information for accurate image segmentation. Therefore, the present application effectively fuses the double-branch neural network based on convolution and the Inter-Channel Transformer (ICT) model based on self-attention mechanism, which is of great significance for scene classification of high-resolution remote sensing images. SUMMARY

[0004] The purpose of the present application is to provide a remote sensing image scene classification method based on a multi-channel self-attention network. The parallel double-branch neural network is used to extract multi-resolution features for describing the spatial texture information of remote sensing images. A large amount of spatial information and channel information is contained in the above-mentioned features, so the channel attention and spatial attention (CBAM) module is introduced to reduce the dimension of the features in the channel and spatial dimensions, realizing the fine expression of the features while reducing the computational complexity. Secondly, a multi-scale feature matrix is constructed by a multi-scale feature generator, and the feature fusion module in ICT is used to effectively establish the relationship between the features of each scale. Finally, the Transformer Encoder based on the self-attention mechanism in ICT is used to derive the relationship between the features of each layer in the channel dimension, and the visual features used for remote sensing image scene classification are abstracted.

[0005] For convenience of explanation, the following concepts are first introduced:

[0006] Residual Neural Network: The core idea of residual network is to simplify the problem of learning identity mapping function, which is to transform learning identity mapping into learning residual term approximation 0, optimize the learning difficulty of network, and solve the problems of gradient disappearance / explosion in deep network.

[0007] Channel and Spatial Attention (CBAM): MaxPool and AvgPool are used to aggregate feature information in channel and spatial dimensions, thereby enhancing the expression ability of features.

[0008] Transformer Encoder: composed of multi-head attention mechanism, LayerNorm and multi-layer perceptron (MLP), multi-head attention is used to weight and fuse the input patches, thereby aggregating important information, LayerNorm is used to normalize the features of each layer, and multi-layer perceptron is used to extract nonlinear features of patches, thereby capturing long-distance feature dependency.

[0009] The application specifically adopts the following technical solutions:

[0010] A remote sensing image scene classification method based on a multi-channel self-attention network, characterized by:

[0011] a. extracting initial features of remote sensing images through a double-branch neural network, and using spatial attention and channel attention (CBAM) modules to aggregate spatial information and channel information in the initial features;

[0012] b. further mining the initial features through a multi-scale feature generator to construct multi-scale features;

[0013] c. fusing multi-scale features through a feature fusion unit in an inter-channel Transformer (ICT) based on self-attention, and encoding the features using an encoder of the ICT to achieve accurate classification of scenes in remote sensing images;

[0014] The method mainly includes the following steps:

[0015] (1) obtaining a high-resolution remote sensing image dataset for scene classification;

[0016] (2) The low-dimensional local features of the remote sensing image are extracted through a shallow bottleneck block in a parallel double-branch neural network, then CBAM is embedded in a deep basic residual block to realize fine representation of the features and reduce the operation amount, and finally multi-scale feature generators are used to generate multi-scale features through feature splicing and fusion;

[0017] (3) The multi-scale feature weights are obtained through the Sigmoid activation function of the feature fusion part in the ICT, the weight matrix is multiplied with the multi-scale feature matrix to obtain the fused feature map, and multi-channel high-level semantic features are generated through linear mapping; the encoder in the ICT divides the high-level semantic features into multiple patches according to the channel, encodes each patch in the channel dimension, and finally generates global visual features for classification.

[0018] The beneficial effects of the present application are:

[0019] (1) The powerful local feature extraction capability of the convolutional neural network is fully utilized to automatically extract image features by the computer, which effectively avoids the low precision, slow time efficiency and high cost of the features designed by experts, and has a wide coverage and strong adaptability.

[0020] (2) According to the characteristics of large scale change of ground objects in remote sensing images, a multi-scale feature generator is designed to effectively extract features of different scales, and a feature fusion module in the ICT is used to establish the relationship between different scale features and fully fuse the information of features of different scales.

[0021] (3) The encoder in the ICT models the features in the channel dimension and establishes a global receptive field to realize the leap from local features to global features, and effectively extracts global visual features for accurate classification. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 It is a remote sensing image scene classification method framework based on a multi-channel self-attention network.

[0023] Figure 2 It is a multi-scale feature generation network.

[0024] Figure 3 It is an inter-channel transformer (ICT) structure diagram.

[0025] Figure 4 It is a confusion matrix of the present application on the data set AISC.

[0026] Figure 5 It is a confusion matrix of the present application on the data set SIRI_WHU. DETAILED DESCRIPTION

[0027] The application will be further described below in conjunction with the accompanying drawings and embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the application and cannot be understood as limiting the protection scope of the application. Those skilled in the art can make some non-essential improvements and adjustments to the specific implementation of the application according to the above description, which should still belong to the protection scope of the application.

[0028] Remote sensing image scene classification method based on multi-channel self-attention network

[0029] (1) The AISC dataset is divided into a training set, a validation set and a test set according to a ratio of 6:2:2, and the SIRI_WHU dataset is divided into a training set and a test set according to a ratio of 8:2. Through two 3x3 convolutions in the double-branch neural network, the initial image is down-sampled with the same smoothing step, the number of feature channels is expanded, information is fully extracted, two Bottleneck residual blocks are used for feature dimension reduction to reduce the amount of calculation, and then parallel convolution is used to down-sample the features with different steps to generate multi-resolution features containing spatial texture information;

[0030] (2) The CBAM module is embedded in the Basicblock residual block to fully utilize its ability to aggregate spatial information and channel information, so that the multi-resolution features can be expressed more finely, and the residual connection ensures that the original data can be transmitted losslessly while the model extracts multi-dimensional complex features;

[0031] (3) In order to avoid the influence of the change of the target scale in the image on the classification accuracy, a multi-scale feature generator is designed by combining convolution and deconvolution with different sampling steps and matrix splicing methods. In the multi-scale feature generator, the receptive field and resolution of the features are changed by different sliding steps, thereby generating three groups of multi-scale feature matrices;

[0032] (4) In order to establish the connection between different scale features, the Sigmoid activation function is used to obtain the weight corresponding to the multi-scale feature matrix in the feature fusion part of the inter-channel Transformer (ICT) based on self-attention, then the feature matrix and the weight matrix are multiplied and fused, and the point convolution is used to map the feature information in the channel, effectively fusing the feature information of different scales to obtain high-dimensional semantic features;

[0033] (5) In the encoder part of ICT, the feature map is divided into multiple patches by channel, and the interaction relationship of each layer feature is inferred by modeling each patch in the channel dimension through multi-head attention mechanism. A multi-layer perception (MLP) is used to establish a global receptive field to capture long-distance feature dependency relationship. Finally, a global visual feature vector is generated to comprehensively analyze the scene type in the current image.

[0034] (6) The AISC dataset covers more than 20 cities in China, and a total of six categories, which are: residential, administration, industrial, road, commercial and green space. The dataset has a total of 17831 high-resolution remote sensing images, about 2900 images for each category, with a spatial resolution of 0.6m, and each image is 200x200 pixels. The SIRI_WHU dataset designed by the RE-IDEA team of Wuhan University and released in 2016 includes 12 scene categories such as agriculture, commercial, and industrial, a total of 2400 images, 200 images for each category, and each image has a pixel size of 200x200, with a spatial resolution of 2m. The experiment uses 64-bit Ubuntu, Intel(R) Core(TM) i7-9700@3.00GHz CPU, 64GB memory, and an NVIDIA RTX2080Ti graphics card with 11GB memory, and the deep learning framework uses PyTorch 1.7.1. The cross-entropy loss function is used to calculate the loss value between the predicted value and the true label, and the Adam optimizer is used to optimize the model. The input quantity of each batch of samples (batch size) is set to 32, the total number of training times (epoch) is 300 rounds, the initial learning rate is 0.0001, and the learning rate is multiplied by 0.5 every 150 rounds of training. The experimental results of the present application are shown in Tables 1 and 2, respectively, wherein the overall accuracy (OA) on the AISC dataset and the SIRI_WHU dataset is 95.70% and 94.00%, respectively.

[0035] Table 1 Test results of different algorithms on AISC dataset

[0036]

[0037] Table 2 Test results of different algorithms on SIRI_WHU dataset

[0038]

Claims

1. A remote sensing image scene classification method based on a multi-channel self-attention network, characterized in that: a. Initial features of remote sensing images are extracted using a dual-branch neural network, and spatial and channel information is aggregated using spatial attention and channel attention (CBAM) modules; b. Further mine the initial features using a multi-scale feature generator to construct multi-scale features; c. Multi-scale features are fused by the feature fusion unit in the self-attention-based Inter-Channel Transformer (ICT), and the features are encoded by the encoder of the ICT to achieve accurate scene classification in remote sensing images. This method mainly includes the following steps: (1) Obtain a high-resolution remote sensing image dataset for scene classification; (2) Low-dimensional local features of remote sensing images are extracted by shallow bottleneck blocks in parallel dual-branch neural networks. Then, CBAM is embedded in deep basic residual blocks to achieve refined representation of features while reducing computational load. Finally, a multi-scale feature generator is used to stitch and fuse features to generate multi-scale features. (3) Obtain multi-scale feature weights through the Sigmoid activation function of the feature fusion part in ICT, multiply the weight matrix and the multi-scale feature matrix to obtain the fused feature map, and generate multi-channel high-level semantic features through linear mapping; the encoder in ICT divides the high-level semantic features into multiple patches according to the channel, and encodes each patch in the channel dimension, and finally generates global visual features for classification.

2. The remote sensing image scene classification method based on a multi-channel self-attention network as described in claim 1, characterized in that... The parallel dual-branch neural network constructed in step (2) first uses a convolutional filter to extract the initial local features of the remote sensing image, and then uses Bottleneck to reduce the dimensionality of the features while keeping the feature scale unchanged; secondly, CBAM is introduced into the parallel dual-branch Basicblock to aggregate the channel information and spatial information in the features while reducing the amount of computation and abstracting the multidimensional features. Finally, considering the large scale variation of ground objects in remote sensing images, a multi-scale feature generator was designed to further mine fine-grained features and construct a multi-scale feature matrix.

3. The remote sensing image scene classification method based on a multi-channel self-attention network as described in claim 1, characterized in that... In step (3), the ICT module first constructs the connection between multi-scale features through the feature fusion part. The Sigmoid activation function is used to obtain the weight matrix of each scale feature. The weight matrix is ​​multiplied with the multi-scale feature matrix and linearly mapped to a high-dimensional space to obtain multi-channel high-level semantic features. Secondly, the features are divided into multiple patches according to the channel. The ICT encoder (Transformer Encoder) based on the self-attention mechanism models each patch in the channel dimension to capture long-distance dependencies and finally extract global visual features for classification.

Images