A remote sensing scene classification method based on multi-layer feature fusion

Abstract

In view of the problems existing in the prior art, the purpose of the present invention is to propose a remote sensing scene classification method based on multi-layer feature fusion, which is used to construct a more discriminative feature representation of the remote sensing scene, thereby improving the classification performance. The purpose of the present invention is achieved through the following technical steps: Step 1) extracting multi-level convolutional features through the CNN basic network; Step 2) using residual spatial attention to spatially calibrate the multi-level convolutional features; Step 3) using a gating mechanism to fuse the spatially calibrated multi-level convolutional features; Step 4) classifying the fused features based on a classifier.

Description

Technical Field

[0001] The present invention designs a remote sensing scene classification method based on multi-layer feature fusion, which can better address the problems of complex background of remote sensing scenes and large scale changes of targets of interest. It has high practical value and can be effectively used for remote sensing scene classification. It can also be used as a feature aggregation method in other related fields. Background Art

[0002] With the rapid development of satellite imaging technology, the resolution of remote sensing has been continuously improved to sub-meter level. Correspondingly, the remote sensing image analysis mode has gradually changed from pixel level and object level to semantic level, such as scene classification. Remote sensing scene classification has received increasing attention in remote sensing applications, such as land cover classification and change detection, object detection, image description, semantic segmentation, etc.

[0003] Remote sensing scene classification can be summarized as data preprocessing, feature extraction, and classification. In recent years, most of the research has focused on feature extraction, which aims to construct a more discriminative scene feature representation. According to feature extraction, there are two types of remote sensing image scene classification: handcrafted feature-based methods and deep feature-based methods. On the one hand, the former mainly utilizes features such as color, texture, and shape, such as color moment, gray-level co-occurrence matrix, etc. On the other hand, with the development of artificial intelligence, feature learning-based methods have also been introduced into remote sensing image scene classification, such as deep belief networks (DBNs) and convolutional neural networks (CNNs). Due to the unprecedented learning ability of convolutional neural networks, deep feature-based methods have brought the performance of remote sensing scene classification to a new level and gradually become the dominant paradigm compared with handcrafted feature-based methods.

[0004] The diversity of remote sensing image acquisition platforms and sensors leads to large variations in the scale of targets in remote sensing scenes. In addition, remote sensing scenes contain complex ground objects, especially urban scenes. Due to the large scale variation and complex background, the target area only occupies a small part of the entire image and is easily disturbed by useless objects contained in the complex background.

[0005] Therefore, although the convolutional neural network-based methods have greatly improved the performance of remote sensing scene classification, the classification performance is still affected by large scale differences and complex backgrounds. In particular, the continuous pooling operation in the convolutional neural network leads to a reduction in the size of the feature map, which easily leads to the neglect of key areas and details of the scene.

[0006] The present invention combines spatial attention and gating mechanisms to construct a multi-layer feature fusion strategy to solve the above problems by extracting more discriminative features. Summary of the invention

[0007] In view of the problems existing in the prior art, the purpose of the present invention is to propose a remote sensing scene classification method based on multi-layer feature fusion, which improves the classification effect by constructing more discriminative representation features of the remote sensing scene.

[0008] The purpose of the present invention is achieved through the following technical steps:

[0009] Step 1) Extract multi-level convolutional features through the CNN basic network.

[0010] Step 2) Use residual spatial attention to spatially calibrate multi-level convolutional features.

[0011] Step 3) Use the gating mechanism to fuse the multi-level convolutional features after spatial calibration.

[0012] Step 4) Classify the fused features based on the classifier.

[0013] Furthermore, the specific method of step 2) is: using the Softmax function to generate a spatial descriptor of the feature map based on the channel dimension, generating a spatial weight s based on a nonlinear module composed of two fully connected layers, Relu, and Softmax, and then using s to recalibrate the original feature x in the form of a residual to obtain the final output O.

[0014] Furthermore, the specific method of step 3) is: by using global average pooling for each input feature, compressing it into one dimension based on concatenation and convolution operations, and then using multiple fully connected layers to generate importance weights for different channels of multi-layer features, and fusing them according to the weights. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Overall structure diagram;

[0016] Figure 2 Residual spatial attention map;

[0017] Figure 3 Gated features and plots.

[0018] Figure 4 Classification confusion matrix based on the AID dataset (a is the result when the training ratio is 20%, and b is the result when the training ratio is 50%) DETAILED DESCRIPTION

[0019] The present invention "a remote sensing scene classification method based on multi-layer feature fusion" is further explained below in conjunction with the accompanying drawings.

[0020] As attached Figure 1As shown, the multi-layer feature fusion network proposed in the present invention consists of three parts: a convolutional neural network backbone, a multi-layer feature fusion part and a classification part. The multi-layer feature fusion part includes a residual attention module (RSAM) and a gated feature fusion module (GFM). The convolutional neural network extracts multi-layer convolutional features. This time, ResNet50 is used to obtain multi-layer features in the convolution layer. Then, the convolutional features are input into RSAM for spatial feature calibration, so that the model focuses on key areas and ignores background information. GFM aims to select and fuse multi-layer features after spatial calibration, and reduce feature redundancy and mutual exclusion. Finally, the final features are obtained through a global average pooling operation, and the remote sensing scene is classified through a Softmax classifier.

[0021] (I) Convolutional Neural Network Backbone

[0022] In this example, ResNet50 will be used as the base network. The output features of its 2nd, 3rd, and 4th layers will be used as low-level features, mid-level features, and high-level features, respectively. Other convolutional neural networks can also be used to extract multi-level features.

[0023] 2. Residual Space Attention

[0024] Due to the large scale differences and complex backgrounds of remote sensing images, convolutional neural networks usually have difficulty in obtaining good spatial representation. Therefore, the present invention designs a novel spatial attention module to generate corresponding spatial attention weights and improve spatial representation, as shown in the attached figure. Figure 2 As shown in Figure 3, the classification performance can be improved more effectively by increasing the weights of important regions and suppressing background regions to adaptively calibrate features in the spatial dimension.

[0025] First, the global channel information is compressed into a positional descriptor. We generate the spatial descriptor based on the Softmax weighting method in the channel dimension. Formally, the statistic z∈R H×W is the contraction x∈R by the channel dimension C C×H×W Each element of is generated. z(i, j) is calculated as follows:

[0026]

[0027] Then, based on the location descriptor, a nonlinear function is designed to generate the spatial weight s as follows:

[0028] z′=flatten(z)

[0029]

[0030] s = reshape(s′)

[0031] Where δ represents the Relu function, represents the softmax function, z′∈R 1×M , s∈R H×W To reduce the dimensionality of the model, two fully connected layers, a dimensionality reduction layer, a ReLU, and then a dimensionality increase layer are used to return to the original dimension with a dimensionality reduction ratio of r1. The final output O is obtained by recalibrating x in residual form:

[0032]

[0033] in Represents element-by-element multiplication. This module essentially introduces a dynamic adaptive strategy based on the input, which can be regarded as a self-attention function in the spatial dimension. This relationship is not limited to the convolution kernel corresponding to the local receptive field, but acts on the entire feature map.

[0034] (III) Gated feature fusion

[0035] In order to effectively utilize the hierarchical information of the convolutional structure, widely used fusion methods include element-by-element addition and channel cascading. However, these methods easily introduce some interference information, such as feature redundancy and mutual exclusion. In order to consider the importance of features at different layers, this study plans to design a multi-layer feature fusion module based on a gating mechanism, as shown in the attached Figure 3 As shown. O1, O2, O3 represent multi-layer feature maps after spatial enhancement, which contain features from different layers. gp represents the concatenation of the global average pooling results of O1, O2, O3. p, q and l represent the three sets of weights of O1, O2, O3. fc1, fc2, fc3 and fc4 are fully connected layers, and I represents the final feature map after fusion. The following is a detailed introduction to the multi-layer feature fusion module based on the gating mechanism.

[0036] First, the results of the spatial attention module are fused. By using global average pooling, the feature map O∈R C×H×W is compressed to generate channel statistics gp o ∈R C .gp o The cth element of is obtained by shrinking the cth channel of O by the spatial dimension H×W, gp o The cth element of is calculated by:

[0037]

[0038] gp = cat[gp1, gp2, gp3]

[0039] gp′=conv(gp)

[0040] Its crepresents the cth channel of O. gp1, gp2, gp3 represent the global average pooling results of O1, O2, and O3 respectively. cat and conv represent the concatenation and convolution operations. gp represents the combined result of gp1, gp2, gp3 using concatenation, and gp′ represents the dimensionality reduction result of the convolution operation on gp.

[0041] To improve efficiency and generalization ability, the compact features z∈R are then obtained through a simple fully connected layer with dimensionality reduction d .

[0042] z=fc1(gp′) In addition, three fully connected layers and the softmax function are used to generate three sets of weights p, q and l by the following formula:

[0043] p′=fc2(z)

[0044] q′=fc3(z)

[0045] l′=fc4(z)

[0046]

[0047] Among them, fc2, fc3 and fc4 have the same structure but do not share parameters. p′, q′ and l′ are three sets of values ​​generated by fc2, fc3 and fc4, indicating the importance of information in different channels in the input feature map. It means to perform softmax operation on the corresponding positions of p′, q′ and l′, so that the elements of p, q, and l are in the range of 0 to 1, and the corresponding positions are summed to 1. The final feature map I can be obtained by:

[0048]

[0049] The gating mechanism adaptively selects effective information from multiple layers of features for fusion, making the fused features complementary to the multiple layers of features and avoiding feature redundancy and mutual exclusion. By using global average pooling for each input feature, it is compressed into one dimension based on concatenation and convolution operations. Then three fully connected layers are used to generate importance weights for different channels of the multiple layers of features.

[0050] (IV) Classification

[0051] After the final fusion feature map is generated, in order to train the network more conveniently, a feature vector with a dimension of 1×1×C is obtained through global average pooling to reduce the parameters of the fully connected layer. In this way, the backbone, spatial attention module and gating mechanism can be optimized as a whole. After calculating the loss of the forward propagation, the entire model can be optimized according to it in the backward propagation.

[0052] We choose the cross entropy loss function to optimize the entire framework. The cross entropy loss function is calculated as follows:

[0053]

[0054] Where N is the number of samples in the batch, M is the number of categories, y is the category label, x is the final feature, and w is the parameter of the softmax classifier. n =m|x,w) represents the probability of predicting the sample category as m, 1{·} is the indicator function (if y n If it is equal to m, it returns 1; otherwise it returns 0. )

[0055] The present invention has been verified on the AID dataset, and the classification results are shown in the attached Figure 4 As shown, better results can be achieved.

Claims

1. A remote sensing scene classification method based on multi-layer feature fusion, the method comprising the following steps: Step 1) Extract multi-level convolution features through the CNN basic network; Step 2) Use residual spatial attention to spatially calibrate multi-level convolutional features; Step 3) Using the gating mechanism to fuse the multi-level convolutional features after spatial calibration; Step 4) classifying the fused features based on the classifier; As described in step 2), it is characterized in that Residual spatial attention first compresses the global channel information into a position descriptor z: Among them, x a represents the ath band of the input feature map x, i and j represent the position of the feature map, and c represents the channel dimension; Then, based on the location descriptor, a nonlinear function is designed to generate the spatial weight s as follows: z′=flatten(z) s = reshape(s′) Among them, flatten represents the flattening operation, δ represents the Relu function, represents the softmax function, z′∈R 1×M , s∈R H×W , r1 represents the dimensionality reduction ratio, reshape represents the deformation operation, and x is further recalibrated in the form of residual to obtain the final output O: in, stands for element-wise multiplication; As described in step 3), it is characterized in that the gating mechanism performs global average pooling on the multi-layer feature map after spatial enhancement, concatenates the pooling results of each layer, and performs a convolution operation on the concatenated pooling results to generate a dimensionality reduction result: gp = cat[gp1, gp2, gp3] gp′=conv(gp) Among them, O c represents the cth channel of the input feature O, gp1, gp2 and gp3 represent the global average pooling results of O1, O2 and O3 respectively, O1, O2 and O3 represent the results of low-level feature, medium-level feature and high-level feature space calibration respectively, cat and conv represent the concatenation and convolution operations, gp represents the combination result of gp1, gp2 and gp3 using concatenation, and gp′ represents the dimensionality reduction result of the convolution operation on gp; And use multiple fully connected layers fc1, fc2, fc3, fc4 and Softmax function to generate channel importance weights of multi-layer features to adaptively select effective information of each layer, as follows: z=fc1(gp′) p′=fc2(z) q′=fc3(z) l′=fc4(z) Among them, fc2, fc3 and fc4 have the same structure but do not share parameters. p′, q′ and l′ are three sets of values ​​generated by fc2, fc3 and fc4, indicating the importance of information in different channels in the input feature map. It means to perform softmax operation on the corresponding positions of p′, q′ and l′, so that the elements of p, q, and l are in the range of 0 to 1, and the corresponding positions are summed to 1; the features are weighted fused element by element according to the weights.

Images