A Remote Sensing Image Classification Method Based on Multi-Scale Pyramid Spatial Independent Convolution

Abstract

The present invention relates to the technical field of remote sensing image classification, and relates to a remote sensing image classification method based on multi-scale pyramid spatial independent convolution, including: S1, performing preprocessing on the remote sensing image, and selecting representative map sheets to produce a classification sample data set by manually annotating each pixel; S2, performing image enhancement on the classification sample data set obtained in S1, and the image enhancement method includes horizontal and vertical flipping, random resampling of size, and Gaussian blur; S3, using a deep residual network to extract the features of the image after image enhancement; S4, further constructing a multi-scale feature pyramid based on the features of the image in S3, and outputting multi-resolution multi-scale features; S5, designing a spatial independent convolution to fuse the multi-scale features in S4 according to the principle of decoupling and outputting fused features; S6, using a linear layer to map the fused features in S5 to categories and outputting a classification result. The present invention realizes the dynamic fusion of multi-scale features, is simple and effective, has less computational complexity, and further improves the classification accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image classification, and specifically, it relates to a remote sensing image classification method based on multi-scale pyramid spatial independent convolution. Background Art

[0002] Remote sensing image classification is a basic and key task for carrying out land use research, and it is also a basic problem that needs to be solved first in land resource surveys and evaluations of different scales and levels. Multi-scale ground objects widely exist in various remote sensing images. For example, large-scale buildings, farmlands and small-scale trees, vehicles, etc. Their occupied areas and ground object characteristics are quite different. Therefore, constructing a remote sensing classification model that can simultaneously identify multi-scale ground objects has great practical significance.

[0003] The classification model is the core of remote sensing image classification. Currently, the mainstream classification models include traditional machine learning and deep learning methods. Due to the development of big data, parallel computing and neural networks, the deep learning method based on convolutional neural network (CNN) has developed rapidly and achieved fruitful results, such as fully convolutional network (FCN), PSPNet, DeepLab, etc. However, the current CNN classification model mainly based on downsampling still has poor recognition ability for multi-scale ground objects. A large amount of downsampling destroys the features of small-scale ground objects, making it difficult to identify them, which limits the application in actual scenarios. Summary of the Invention

[0004] Aiming at the problem that the current remote sensing image classification model based on CNN has poor recognition of multi-scale ground objects, the present invention provides a remote sensing image classification solution based on multi-scale pyramid spatial independent convolution. The specific steps include:

[0005] A remote sensing image classification method based on multi-scale pyramid spatial independent convolution, including:

[0006] S1. Perform preprocessing on the remote sensing image, and select representative image frames to make a classification sample data set by manually annotating each pixel.

[0007] S2. Perform image enhancement on the classification sample data set obtained in S1. The image enhancement methods include horizontal and vertical flipping, random resampling of size, and Gaussian blur.

[0008] S3. Use a deep residual network to extract the features of the image after image enhancement in S2.

[0009] S4. Further construct a multi-scale feature pyramid based on the features of the image in S3, and output multi-resolution multi-scale features.

[0010] S5. Design spatial independent convolution, and fuse the multi-scale features in S4 according to the principle of decoupling and output the fused features.

[0011] S6, use the linear layer to map the fusion features of S5 to categories and output the classification results.

[0012] Preferably, the deep residual network in S3 is composed of a plurality of convolutional residual blocks stacked together, and the residual block divides the convolution into a direct mapping part and a residual part, specifically: i+1 =x i +F c (x i , w i ), x i For the direct mapping part, f c (x i , w i ) is the residual part;

[0013] If x i with x i+1 If the number of channels is different, you need to use 1×1 convolution on x i Perform dimension scaling, and the residual block is represented as: x i+1 =H(x i , w h )+F c (x i , w i ), H represents the 1×1 convolution operation with adjusted dimensions.

[0014] Preferably, in S4, the feature pyramid is modified directly on the deep residual network, and the feature map of each grayscale is added point by point with the feature map of the next resolution scaled twice, and finally the scale features are aligned and output.

[0015] Preferably, in S5, the output fusion feature is x∈R (C×G)×H×W , H and W represent the length and width of multi-scale features, C is the number of channels of each scale feature, G represents the number of different scales, and the spatial independent convolution is:

[0016]

[0017] Represented by the convolution kernel W∈R (O×(C×G)×Ω×Ω)×(C×G)×K×K The generated spatial independent convolution kernel, F represents the spatial independent convolution kernel Generated fusion features.

[0018] Preferably, grouped convolution is used to generate The features from the same branch are regarded as a group, and the Softmax function is added after the group convolution to constrain the sum of the weights of features of different resolutions to 1. The improved spatial independent convolution is: W∈R(O ×G)×G×K×K ,

[0019] Preferably, the linear layer in S6 is the fully connected layer, and the expression is: y i，j = W T x i，j + b, which performs operations in units of the points outputting the fused features in S5, converts the dimension of the features into the dimension of the categories, and realizes the classification of ground objects.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] (1) The present invention proposes a multi-scale pyramid feature extraction method for the problem of difficult multi-scale ground object recognition in the CNN-based remote sensing classification method, which can better take into account both large and small ground objects.

[0022] (2) The spatially independent convolution proposed by the present invention generates customized convolution kernels for different positions, realizing the dynamic fusion of multi-scale features. Compared with the standard convolution, the spatially independent convolution has the following advantages: 1) Decouples features of different scales, with stronger interpretability. 2) Simple and effective, with less computational complexity and further improved classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is the overall structure diagram of the remote sensing image classification method based on multi-scale pyramid spatially independent convolution of the present invention;

[0024] Figure 2 is the structure diagram of the deep residual network of the present invention;

[0025] Figure 3 is the structure diagram of the multi-scale feature pyramid of the present invention;

[0026] Figure 4 is the schematic diagram of the spatially independent convolution of the present invention;

[0027] Figure 5 is the schematic diagram of large-scale building classification in the embodiment of the present invention;

[0028] Figure 6 is the schematic diagram of large-scale wetland classification in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] Hereinafter, the present invention will be specifically described by way of exemplary embodiments. However, it should be understood that, without further elaboration, elements, structures, and features in one embodiment can also be beneficially incorporated into other embodiments.

[0030] The current CNN classification model mainly based on sampling has poor recognition ability for multi-scale ground objects. For example, small ground objects often become too small or even disappear after a large amount of downsampling, which limits the application of its classification model in real scenes.

[0031] To solve the above problems, an embodiment of the present invention provides a remote sensing image classification method based on multi-scale pyramid spatial independent convolution. A feature pyramid is constructed on the basis of a deep residual network to output multi-scale features; and spatial independent convolution is used to output and fuse multi-scale features to achieve multi-scale ground object classification.

[0032] Example: Realize the classification of coastal buildings and wetlands based on the GF-2 remote sensing image in the Qingdao area. The specific steps are as follows:

[0033] S1. Orthorectify and radiometrically calibrate the GF-2 multispectral image, and orthorectify the panchromatic image. Then, fuse the multispectral image and the panchromatic image, and perform atmospheric correction on the fused image. Finally, output the fused high-resolution multispectral image with a resolution of about 1 meter. Then, crop representative image tiles on the fused high-resolution image according to the principle of full ground object coverage and low redundancy, and perform detailed pixel-level annotation on them. The annotated sample data constitutes a classification data set.

[0034] S2. Before reading the training sample image obtained in S1 into the model, perform image enhancement to indirectly increase the number of samples. The enhancement methods include horizontal and vertical flipping, random resampling of size (0.5 times - 2.0 times), and Gaussian blur. After processing, the image needs to be regularly cropped to a size of 512×512 to meet the input conditions of the model.

[0035] S3. Use a deep residual network (ResNet) to extract the features of the enhanced image in step S2. The deep residual network is stacked by multiple convolutional residual blocks, as Figure 2 shown. Let the current feature be x i , and the ordinary convolution operation directly takes the convolved feature as the output result, as shown in formula (1):

[0036] x i+1 =F c (x i , w i ) (1)

[0037] In the formula, F c represents the convolution operation, and w i is the convolution kernel parameter. The residual block divides the convolution process into a direct mapping part and a residual part, as shown in formula (2):

[0038] x i+1 =x i +F c (xi , w i ) (2)

[0039] where x i is the direct mapping part, and F c (x i , w i ) is the residual part. In addition, if the number of channels of x i is different from that of x i+1 , then a 1×1 convolution needs to be used to scale the dimension of x i . At this time, the residual block is expressed as:

[0040] x i+1 = H(x i , w h ) + F c (x i , w i ) (3)

[0041] where H represents the 1×1 convolution operation for adjusting the dimension.

[0042] Compared with ordinary convolution, the convolutional residual block is easier to optimize, which can improve the training speed and accuracy of the model; moreover, as the model deepens, its accuracy will not degenerate. In addition, introducing the residual mechanism does not add extra computational load to the network.

[0043] S4. Further construct a multi-scale feature pyramid based on the features of S3 to output multi-resolution and multi-scale features. The feature pyramid is used to solve the problem that the downsampling amplitude of ResNet is too large, resulting in small ground objects being unrecognizable, so as to achieve the classification of multi-scale ground objects. The feature pyramid is directly modified on ResNet. The feature map of each resolution is added point by point to the feature map of the next resolution scaled by two times, and finally the aligned scale features are output, as Figure 3 shown. Through this operation, the feature map of each scale integrates features of different scales and different semantic intensities, ensuring that each scale has the ability to recognize ground objects of the corresponding resolution. Moreover, the pyramid method only adds additional cross-layer connections on the basis of ResNet, hardly affecting the efficiency of the model.

[0044] S5. Design space-independent convolution to fuse the multi-scale features of S4 according to the principle of decoupling and output the fused features. Let the output multi-scale feature be x ∈ R (C×G)×H×W, where H and W represent the length and width of the multi-scale features, C is the number of channels of each scale feature, and G represents the number of different scales. The objective of the present invention is to design an adaptive fusion method to fuse the aligned multi-scale features generated by the pyramid backbone, allowing feature points at different positions to automatically adjust the weights of different scales, that is, having spatial independence. However, due to the translational invariance of the standard convolutional layer, all positions share the same convolutional kernel, and the weights of the convolutional kernel are fixed after training, so it cannot achieve the effect of dynamically adjusting the weights of different resolutions according to the spatial position. The standard convolution can be expressed as the following formula:

[0045]

[0046] where \(W\in R\) O×(C×G)×K×K is the convolutional kernel parameter, and F is the feature generated after convolution. The spatially independent convolution is designed to solve the above problems of the standard convolution. Its main idea is to use convolution to generate the convolutional kernel, and it is realized to dynamically generate an independent multi-scale feature fusion convolutional kernel for each spatial position, as shown in formula (5):

[0047]

[0048] where represents the spatially independent convolutional kernel generated by the convolutional kernel \(W\in R\) (O×(C×G)×Ω×Ω)×(C×G)×K×K , and F represents the fused feature generated by the spatially independent convolutional kernel . However, generating a complete convolutional kernel for each point will consume a large amount of memory. Therefore, for the aligned multi-scale features, the present invention makes an assumption to simplify the calculation, that is, features from the same resolution have the same importance and share the same weight. Based on this assumption, the present invention uses grouped convolution to generate and regards the features from the same branch as a group, and adds the Softmax function after the grouped convolution to constrain the sum of the weights of different scale features to 1, as Figure 4 shown. The improved spatially independent convolution is shown in formula (6):

[0049]

[0050] where \(W\in R\) (O×G)×G×K×K ,

[0051] The spatially independent convolution can be used to adaptively fuse the aligned multi-scale features generated by the feature pyramid, allowing feature points at different positions to automatically adjust the weights of different scales, realizing the decoupling of multi-scale features, and improving the accuracy under the condition of higher interpretability.

[0052] S6. Map the fused features of S5 to categories through a linear layer and output the classification results. The linear layer is the fully connected layer, which can be expressed as the following formula:

[0053] y i，j =W T x i，j +b (7)

[0054] It performs operations in units of the points where S5 outputs the fused features, converts the dimension of the features to the dimension of the categories, and realizes the classification of ground objects.

[0055] S7. Model accuracy verification. In this example, the dataset is divided into a training set and a test set according to a ratio of 6:4, and the overall accuracy (OA), F1 score, and mean Intersection over Union (mIoU) are used to test the model accuracy on the test set. OA, F1 score, and mIoU are shown in the following formula:

[0056]

[0057]

[0058]

[0059] Among them, TP, TN, FP, and FN represent true positives, true negatives, false positives, and false negatives in the confusion matrix, N is the total number of pixel points, and PR and RE are the accuracy and recall rates calculated from the confusion matrix. After calculation on the test set, the accuracy of this model is OA: 95.12, F1 score: 79.63, mIoU: 66.17.

[0060] S8. Real-scene classification. The classification targets are buildings and wetlands in the GF-2 image of Qingdao area. Before reading into the model, the image needs to be cropped according to the rules to meet the input conditions. After classification by the model, it is spliced back to the original image according to the cropping rules to complete the classification. Figure 5 And Figure 6 are the large-scale classification result maps of buildings and wetlands.

[0061] It can be seen from the classification results that the present invention can adapt to multi-scale ground objects of different sizes and realize multi-scale ground object classification under fixed model conditions.

[0062] The above-mentioned embodiments are only used for conveniently illustrating the present invention and are not limitations on the protection scope of the present invention. Within the scope of the technical solution of the present invention, various simple deformations and modifications made by those skilled in the art should be included in the scope of the above-mentioned patent application.

[0063] Of course, the above description is not a limitation on the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions, or substitutions made by those skilled in the art within the essence of the present invention should also belong to the protection scope of the present invention.

Claims

1. A remote sensing image classification method based on multi-scale pyramid spatial independent convolution, characterized in that, Including: S1. Perform preprocessing on the remote sensing image, and select representative map sheets to produce a classification sample data set by manually annotating each pixel; S2. Perform image enhancement on the classification sample data set obtained in S1. The image enhancement methods include horizontal and vertical flipping, random resampling of size, and Gaussian blur; S3. Use a deep residual network to extract the features of the image after image enhancement in S2; S4. Further construct a multi-scale feature pyramid based on the features of the image in S3, and output multi-resolution and multi-scale features; S5. Design a spatially independent convolution to fuse the multi-scale features of S4 according to the principle of decoupling and output the fused features; S6. Use a linear layer to map the fused features of S5 to categories and output the classification results; The deep residual network in S3 is stacked by multiple convolutional residual blocks. The residual block divides the convolution into a direct mapping part and a residual part, specifically: x i+1 = x i + F c (x i , w i ), x i is the direct mapping part, and F c (x i , w i ) is the residual part; If x i has a different number of channels from x i+1 , then a 1×1 convolution needs to be used to scale the dimensions of x i . At this time, the residual block is expressed as: x i+1 = H(x i , w h ) + F c (x i , w i ), where H represents the 1×1 convolution operation for adjusting dimensions; In S4, the feature pyramid is directly modified on the deep residual network. The feature map of each gray level is added to the feature map with twice the resolution of the latter one point by point, and finally the scale features are aligned and output; In S5, let the output fused feature be \(x\in\mathbb{R}\). (C×G)×H×W , where \(H\) and \(W\) represent the length and width of the multi-scale features, \(C\) is the number of channels of each scale feature, \(G\) represents the number of different scales, and the spatially independent convolution is: represents the spatially independent convolution kernel generated by the convolution kernel W ∈ R (O×(C×G)×Ω×Ω)×(C×G)×K×K and F represents the fused feature generated by the spatially independent convolution kernel ​ Use grouped convolution to generate and regard the features from the same branch as a group. After grouped convolution, add the Softmax function to constrain the sum of the weights of features with different resolutions to 1. The improved spatially independent convolution is as follows: The linear layer described in S6 is the fully connected layer, and its expression is: y i,j = W T x i,j + b. It performs operations in units of the points outputting the fused features in S5, converts the dimension of the features into the dimension of categories, and realizes the classification of ground objects.

Images