Remote sensing image scene classification method based on graph convolution network

Abstract

The application provides a remote sensing image scene classification method based on a graph convolution network, and relates to the technical field of remote sensing, which comprises the following steps: constructing a scene classification model and training the model; performing superpixel segmentation on training samples to obtain image blocks; extracting the image blocks by using a deep convolution network to obtain a first feature map, constructing a neighborhood graph based on the image blocks and the first feature map, and obtaining a first feature matrix and a neighborhood matrix; establishing a graph convolution network to obtain a second feature; performing position coding and attention feature extraction on the image blocks according to a transformer module to obtain a third feature; fusing the second feature and the third feature, inputting the fused feature into a classification layer, and obtaining a classification result; obtaining a remote sensing image to be classified, inputting the remote sensing image into the scene classification model, and obtaining a scene classification result. The scene classification model has high classification precision and can describe semantic information of the remote sensing image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing, and particularly relates to a remote sensing image scene classification method based on a graph convolution network. BACKGROUND

[0002] The earth is the common home of human beings, and with the continuous progress of human civilization, using technical means to continuously discover and understand the unknown world has become a powerful driving force for the progress of human civilization. Due to the vastness and vastness of the earth's surface, although humans have evolved on the earth for millions of years, the understanding of the environment in which they live is very limited from the local to the whole. Until the mid-20th century, with the advent of satellite remote sensing technology, humans obtained image data of the earth's surface through the "sky eye", which truly opened the curtain of the relative continuous cognition of the earth as a whole. Especially in the early 21st century, with the rapid progress of technology and the vigorous development of remote sensing technology, the number of remote sensing satellites is increasing, and the observation data of the earth's surface obtained by humans is rapidly accumulating, and the data scale has reached EB level. Subsequently, the big data technology that emerged as the times require provides technical support for the processing and information mining of massive data.

[0003] As one of the current research hotspots in the field of remote sensing earth observation technology, remote sensing image scene classification aims to automatically classify images into a specific semantic label according to the content of the remote sensing scene image, and provides auxiliary reference for image understanding. Remote sensing image scene classification needs to distinguish semantic categories by means of visual features and spatial context information of the image scene, and it assumes that scenes of the same type have more similar features, so the key to scene classification is the extraction of image features. The traditional manual feature extraction method can only obtain medium and low-level feature representation of the image, and this method lacks generalization ability and is difficult to accurately describe the image semantics. The hierarchical abstract expression ability of deep learning technology can automatically learn high-level visual features about the image scene, effectively improving the performance of scene classification. In related research, deep convolutional neural networks are mostly used for feature learning and combined with classifiers to complete the classification task. However, most of the existing deep learning methods are based on convolutional neural networks, and the classification accuracy of the classification model used by these methods needs to be further improved. SUMMARY

[0004] Based on the above technical problems, the present application provides a remote sensing image scene classification method based on a graph convolution network, which obtains features with embedding space topological information by establishing a graph convolution network during training, and extracts features with embedding position information by a transformer. The two kinds of features are fused, and then classified. The classification accuracy of the scene classification model in this method is high, and it can describe the semantic information of the image.

[0005] The present application provides a remote sensing image scene classification method based on a graph convolution network, which comprises:

[0006] S1 constructing a scene classification model and training the scene classification model, the scene classification model comprising a deep convolutional network, a graph convolutional network, a transformer module and a classification layer, wherein the graph convolutional network is established when the scene classification model is trained, and the training step comprises:

[0007] S11 performing superpixel segmentation on a training sample to obtain a plurality of image blocks, wherein the training sample is represented as an image block sequence;

[0008] S12 inputting the training sample into the deep convolutional network to obtain a first feature map corresponding to the image block;

[0009] S13 constructing a region adjacency graph based on the image block and the first feature map, obtaining a first feature matrix and an adjacency matrix;

[0010] S14 establishing the graph convolutional network by taking the first feature matrix and the adjacency matrix as graph data, and learning a second feature according to a message passing mechanism of the graph convolutional network;

[0011] S15 inputting the training sample into the transformer module, the transformer module comprising an input layer and a transformer layer, performing position encoding on the image block by using the input layer to obtain a position vector sequence, embedding the position vector sequence into a representation of the corresponding image block sequence, and then inputting the representation into the transformer layer to obtain a third feature;

[0012] S16 fusing the second feature and the third feature, and inputting the fused feature into the classification layer to obtain a classification result;

[0013] S2 obtaining a remote sensing image to be classified, inputting the remote sensing image into the scene classification model, and obtaining a scene classification result.

[0014] In an embodiment of the present application, step S13 comprises:

[0015] upsampling the first feature map, processing the upsampled first feature map according to a max-pooling method to obtain a first feature matrix corresponding to each first feature map;

[0016] establishing a spatial 4-neighborhood relationship for the image block, and constructing a neighborhood matrix, wherein the neighborhood matrix describes the spatial topological structure between each image block.

[0017] In an embodiment of the present application, the method for upsampling the first feature map is:

[0018] using a nearest-neighbor interpolation method to interpolate each pixel of the first feature map, and enlarging the first feature map to the same size as the corresponding image block.

[0019] In an embodiment of the present application, the second feature comprises a representation of the spatial topology.

[0020] In an embodiment of the present application, the image patches are non-overlapping image regions.

[0021] In an embodiment of the present application, step S15 comprises:

[0022] The transformer module comprises an input layer and a transformer layer, the input layer is a position encoding layer, the image patches are position encoded through the position encoding layer to obtain a sequence of position vectors, and then the sequence of position vectors is embedded into the sequence of image patches to obtain a sequence of image patches embedded with position vector information;

[0023] The transformer layer comprises four encoders, each encoder comprises a multi-head attention layer and a multi-layer perceptron, the sequence of image patches embedded with position vector information is input into the transformer layer for encoding, and the third feature is output.

[0024] In an embodiment of the present application, the multi-layer perceptron comprises an activation function.

[0025] In an embodiment of the present application, the third feature is the output feature of the last encoder.

[0026] In an embodiment of the present application, the activation function is one of a sigmoid function, a tanh function and a relu function.

[0027] In an embodiment of the present application, the second feature and the third feature have the same feature scale.

[0028] The beneficial effects of the present application are: the present application provides a remote sensing image scene classification method based on a graph convolution network, first, the image is superpixel segmented, and the image block is taken as the basic processing unit, then the image block is processed in two branches, the first branch is: the first feature map is extracted by using the deep convolution network, and the first feature map is up-sampled to make the scale transform to the size of the original image block, then the neighborhood graph is constructed for the image block, the first feature matrix and the neighborhood matrix are taken as the graph data, and the graph convolution network is established combined with the spatial topological relationship, according to the message passing mechanism of the graph convolution network, the second feature embedded with the spatial topological relationship can be obtained, the second feature can effectively learn the spatial relationship between the features and the targets in the remote sensing image scene, has strong representation ability, and is beneficial to improve the classification precision; the second branch is: the position encoding and attention feature extraction of the image block are carried out according to the transformer module, the image block sequence and the position vector information are encoded based on the multi-head attention mechanism and the multilayer perceptron, and the third feature embedded with the position information is obtained, in the calculation, the multi-head attention mechanism can expand the receptive field area and improve the performance of the model, and the step can be calculated in parallel, so that the calculation amount is effectively reduced; since the second feature and the third feature have the same scale, the feature fusion can be more easily carried out, and the fused feature can be more comprehensive and more representative, and the remote sensing image which is easy to confuse and the complex remote sensing image also have high classification accuracy, and the problem of target loss in classification is effectively avoided. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0030] Figure 1 The method flowchart of the embodiments of the present application is shown in the following.

[0031] Figure 2 The training flowchart of the embodiments of the present application is shown in the following. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art also belong to the scope of protection of the present application.

[0033] Please refer to Figure 1 and Figure 2The application provides a remote sensing image scene classification method based on a graph convolution network, which comprises the following steps:

[0034] S1, a scene classification model is constructed and trained, the scene classification model comprising a deep convolution network, a graph convolution network, a transformer module and a classification layer, wherein the graph convolution network is established during the training of the scene classification model, and the training steps comprise:

[0035] S11, superpixel segmentation is performed on a training sample to obtain a plurality of image blocks, wherein the training sample is represented as an image block sequence;

[0036] S12, the training sample is input into the deep convolution network to obtain a first feature map corresponding to the image block;

[0037] S13, a region adjacency graph is constructed based on the image block and the first feature map, a first feature matrix and an adjacency matrix are obtained;

[0038] S14, the first feature matrix and the adjacency matrix are taken as graph data to establish the graph convolution network, and a second feature is learned according to the message passing mechanism of the graph convolution network;

[0039] S15, the training sample is input into the transformer module, the transformer module comprises an input layer and a transformer layer, the position encoding of the image block is performed by using the input layer to obtain a position vector sequence, the position vector sequence is embedded into the representation of the corresponding image block sequence, and then the position vector sequence is input into the transformer layer to obtain a third feature;

[0040] S16, the second feature and the third feature are fused, and the fused feature is input into the classification layer to obtain a classification result;

[0041] S2, a remote sensing image to be classified is obtained and input into the scene classification model to obtain a scene classification result.

[0042] A group of remote sensing images are obtained, which are taken as training samples to train the scene classification model, and the training sample is denoted as The fth training sample is denoted as C, H and W represent the channel number, height and width of the training sample, respectively, y i The scene category corresponding to the training sample is denoted as N is the number of training samples.

[0043] The SLIC algorithm is used for superpixel segmentation of the training sample to obtain a plurality of image blocks, the image blocks are mutually non-overlapping image regions, and the training sample can be represented as an image block sequence, such as The ith training sample is represented as a sequence comprising m image blocks, and each image block where p represents the dimension of each image block, and m = HW / p 2 The specific process of superpixel segmentation is as follows:

[0044] (1) Initialize seed points, i.e. clustering centers

[0045] According to the set number of superpixels, the seed points are evenly distributed in the training sample. If the total number of pixels in each training sample is N, the pre-segmentation is m superpixels of the same size, and the size of each superpixel is N / m. Then the distance (step length) of adjacent seed points is approximately S = sqrt(N / m).

[0046] (2) Re-select seed points in the neighborhood

[0047] Re-select seed points in the 3*3 neighborhood of the seed points. Specifically, calculate the gradient values of all pixel points in the neighborhood, and move the seed point to the place with the smallest gradient in the neighborhood. The purpose of this is to avoid the seed point falling on the contour boundary with large gradient, so as to avoid affecting the subsequent clustering effect.

[0048] (3) Assign a class label to each pixel point in the neighborhood around each seed point, i.e. which clustering center it belongs to

[0049] The search range of SLIC is limited to 2S*2S, which can accelerate the convergence of the algorithm.

[0050] (4) Distance measurement

[0051] It includes color distance and spatial distance. For each searched pixel point, the distance between it and the seed point is calculated respectively. The distance calculation method is as follows:

[0052]

[0053]

[0054]

[0055] where d c represents the color distance, d s represents the spatial distance, x and y represent the coordinate values of the pixel point, L represents the brightness value of the pixel point, a represents the range of the pixel point from magenta to green, b represents the range of the pixel point from yellow to blue, N s is the maximum spatial distance in the class, N s =S=sqrt(N / m), which is applicable to each cluster, and N c is the maximum color distance, which varies with the picture and the cluster. Therefore, a fixed constant k is used instead in this embodiment. The value range of k is 1-40, and formula (3) is changed to:

[0056]

[0057] The result of formula (4) is the final distance metric. Since each pixel point is searched by multiple seed points, each pixel point will have a distance to the surrounding seed points, and the minimum value corresponding to the seed point is taken as the clustering center of the pixel point.

[0058] It should be noted that the above color distance is related to the Lab color space. The Lab color model is composed of three elements: brightness (L) and related colors a and b. L represents brightness, and the value range of L is from 0 (black) to 100 (white). a represents the range from magenta to green (a negative value indicates green and a positive value indicates magenta), and b represents the range from yellow to blue (a negative value indicates blue and a positive value indicates yellow). The advantages of Lab color space: 1) Unlike RGB and CMYK color spaces, Lab color is designed to approximate human physiological vision. It strives for perceptual uniformity, and its L component closely matches human brightness perception. Therefore, it can be used to make precise color balance by modifying the output color scale of a and b components, or to adjust the brightness contrast using the L component. These transformations are difficult or impossible in RGB or CMYK. 2) Because Lab describes the way colors are displayed, not the specific amount of colorant required by a device (such as a display, printer, or digital camera) to generate a color, Lab is considered a device-independent color model. 3) Wide color gamut. It not only contains all the color gamut of RGB and CMYK, but also can represent the colors that they cannot represent. The colors that can be perceived by the human eye can be represented by the Lab model. In addition, the Lab color model makes up for the uneven distribution of colors in the RGB color model, as the RGB model has too many transition colors between blue and green, and lacks yellow and other colors between green and red.

[0059] (5) Iterative optimization

[0060] Iterate steps (1)-(4) until the clustering center of each pixel point no longer changes.

[0061] (6) Enhance connectivity

[0062] After the above iterative optimization, the following defects may occur: multiple connectivity, superpixel size too small, a single superpixel is cut into multiple discontinuous superpixels, etc. These situations can be solved by enhancing connectivity. The main idea is: a new label table is created, and all elements in the table are -1. According to the "Z" type (from left to right, from top to bottom in order) The discontinuous superpixels, superpixels with too small size are redistributed to adjacent superpixels, and the traversed pixel points are assigned to the corresponding label until all points are traversed.

[0063] After the training samples are superpixel segmented to obtain the image blocks, two processing procedures are then used for feature extraction, and the specific procedure steps are as follows:

[0064] The first processing procedure is: the training samples are input into a deep convolutional network to obtain a first feature map corresponding to the image blocks. The deep convolutional network in this embodiment is ResNet-50. The image blocks are input into the ResNet-50 to obtain four layers of feature maps, and the last layer, i.e., the fourth layer of feature maps, is taken as the first feature map. ResNet is a deep residual neural network. The network adds a jump connection in each convolutional layer to realize feature identity mapping, which can avoid the problem of loss of details when the convolutional layer extracts features by compressing features, and the last layer of feature maps contains rich semantic information of the image, which can more facilitate the improvement of the subsequent classification accuracy.

[0065] Then, a nearest neighbor interpolation method is used to interpolate each pixel of the first feature map, and the first feature map is enlarged to the same size as the corresponding image block. According to the max-pooling method, the first feature map after upsampling is processed to obtain a first feature matrix corresponding to each first feature map, denoted as m is the number of image blocks, and t is the first feature dimension corresponding to the image blocks; a spatial 4-neighbor relationship is established for the image blocks to construct a neighborhood matrix Thus, the neighborhood matrix describes the spatial topological structure between each image block. Upsampling the first feature map can make it become high-resolution while retaining high-level features, thereby retaining image information to a certain extent.

[0066] The first feature matrix is taken as the node of the graph structure, and a graph convolutional network is established in combination with the adjacency matrix and the spatial topological relationship. Then, the graph convolutional network is trained. According to the hierarchical propagation rule of the graph convolutional network, the spatial topological relationship and the first feature matrix are used for message passing to obtain a feature embedded with the spatial topological relationship, i.e., a second feature Since the first feature map is upsampled, it is transformed to the original scale after convolution, and the second feature obtained thereafter is also of the same scale, which can avoid additional calculation caused by different feature scales, and simplify the subsequent calculation and processing procedure.

[0067] The second processing flow is: inputting a training sample into a transformer module, the training sample is represented by an image block sequence, that is, each training sample is an image block sequence, the transformer module includes an input layer and a transformer layer, the input layer is a position encoding layer, the image block includes a position vector, the position vector can be represented by position encoding, the position vector sequence is obtained by using the position encoding layer to perform position encoding on the image block, and then the position vector sequence is embedded into the image block sequence to obtain an image block sequence embedded with position vector information, which is represented as follows:

[0068]

[0069] In the formula, E represents a learnable embedding matrix used for projecting the image block into an mxt-dimensional embedding representation, E pos is used to represent the spatial position of each image block in the training sample image and is encoded into the embedding representation. In the embodiment, the position encoding is performed in the following form:

[0070]

[0071]

[0072] In the formula, p is the position of the image block in the corresponding image block sequence, d model is the vector length of the position encoding information, which is the same as the vector length of the second feature, that is, the dimensions are the same, and f represents the position of the image block in the training sample. The formula adds a sin variable at the even position of each image block and a cos variable at the odd position, so as to generate a spatial position vector with the same dimension as the second feature. Then, the spatial position vector is embedded into the image block sequence according to formula (5), and the result of formula (5) is the input of the transformer layer.

[0073] The transformer layer includes four encoders, and each encoder includes a multi-head attention layer and a multi-layer perceptron. The image block sequence embedded with the position vector information is input into the transformer layer, and the image block sequence is encoded to obtain the third feature. When the transformer layer is trained, three matrices Z Q , Z K and Z V in the transformer layer are trained, and the three matrices are multiplied with the input sequence to obtain a query matrix, a key matrix and a value matrix. The self-attention mechanism is as follows:

[0074]

[0075] In the formula, Q, K and V are respectively a query matrix, a key matrix and a value matrix, dk is the input dimension, and softmax() represents a softmax function. In this embodiment, a multi-head attention mechanism is used to improve the performance of the transformer layer, i.e., multiple Z Q , Z K , Z V The matrix generates multiple query matrices, key matrices, and value matrices, outputs multiple feature values according to formula (7), and then splices the multiple feature values and multiplies them by a matrix parameter to output a feature. The multi-layer perceptron is a forward structure artificial neural network, which includes two linear connection layers and one activation function. The activation function is one of a sigmoid function, a tanh function, and a relu function. In this embodiment, a relu activation function is used. The multi-layer perceptron can be regarded as a directed graph composed of multiple node layers. Each layer is fully connected to the next layer. The features output by the multi-head attention layer are input into the multi-layer perceptron to obtain an output feature of an encoder. The above steps are repeated three times. The output feature of the first encoder is the input of the second encoder, and the output feature of the last encoder is the third feature In the above processing flow, during calculation, the multi-head attention mechanism can expand the receptive field area and improve the performance of the model. Moreover, this step can be calculated in parallel, effectively reducing the amount of calculation. In addition, the multi-head attention mechanism performs attention weighting on the feature representation, and the third feature obtained can describe more rich semantic information.

[0076] After obtaining the second feature and the third feature, the two kinds of features are fused to obtain a fused feature. The fusion formula is as follows:

[0077] W = concat(B', X') (8)

[0078] W is the fused feature, B' is the second feature, X' is the third feature, and concat() represents merging the two features. The fused feature is input into a classification layer to obtain a classification result. The fused feature can be more comprehensive and more representative, and has a high classification accuracy for images that are easy to confuse. The classification layer can be an SVM classifier or other classifiers. During training, the classification accuracy, recall rate, and F1 score are used to evaluate the classification result, and the parameters of the scene classification model are adjusted according to the evaluation result.

[0079] After the scene classification model is trained, the scene classification of a to-be-classified remote sensing image can be performed.

[0080] The application has the beneficial effects that the application provides a remote sensing image scene classification method based on a graph convolution network, first performs superpixel segmentation on an image, takes an image block as a basic processing unit, then performs a two-branch processing procedure on the image block, the first branch is: a first feature map is extracted by using a deep convolution network, and upsampling is performed on the first feature map to make the scale of the first feature map be transformed into the size of the original image block, then a neighborhood graph is constructed for the image block, a graph convolution network is established by taking the first feature matrix and the neighborhood matrix as graph data and combining spatial topological relations, and a second feature embedded with spatial topological relations can be obtained according to the message passing mechanism of the graph convolution network, the second feature can effectively learn the spatial relations between features and targets in a remote sensing image scene, has strong representation ability, and is beneficial to improving the classification accuracy; the second branch is: position encoding and attention feature extraction are performed on the image block according to a transformer module, the image block sequence and position vector information are encoded based on a multi-head attention mechanism and a multilayer perceptron to obtain a third feature embedded with position information, in the calculation, the multi-head attention mechanism can expand the receptive field area and improve the performance of the model, and the step can be calculated in parallel, effectively reducing the calculation amount; since the second feature and the third feature have the same scale, the feature fusion can be easier to perform, and the fused feature can be more comprehensive and more representative, and also has a high classification accuracy for remote sensing images that are easy to confuse and complex remote sensing images, effectively avoiding the problem of target loss in classification.

Claims

1. A method for remote sensing image scene classification based on graph convolution network, characterized in that, The method comprises: S1, constructing a scene classification model and training the scene classification model, the scene classification model comprising a deep convolutional network, a graph convolutional network, a transformer module and a classification layer, wherein the graph convolutional network is established when the scene classification model is trained, and the training step comprises: S11, performing superpixel segmentation on a training sample to obtain a plurality of image blocks, wherein the training sample is represented as an image block sequence; S12, inputting the training sample into the deep convolutional network to obtain a first feature map corresponding to the image block; S13, constructing a region adjacency graph based on the image block and the first feature map, obtaining a first feature matrix and an adjacency matrix; S14, establishing the graph convolutional network by taking the first feature matrix and the adjacency matrix as graph data, and learning the second feature according to the message passing mechanism of the graph convolutional network; S15, inputting the training sample into the transformer module, the transformer module comprising an input layer and a transformer layer, performing position encoding on the image block by using the input layer to obtain a position vector sequence, embedding the position vector sequence into the representation of the corresponding image block sequence, and then inputting it into the transformer layer to obtain the third feature; S16, fusing the second feature and the third feature, and inputting the fused feature into the classification layer to obtain a classification result; S2, obtaining a to-be-classified remote sensing image, inputting it into the scene classification model, and obtaining a scene classification result; The step S13 comprises: performing up-sampling on the first feature map, processing the up-sampled first feature map according to a max-pooling method to obtain a first feature matrix corresponding to each first feature map; establishing a spatial 4-neighborhood relationship for the image block to construct a neighborhood matrix, the neighborhood matrix describing the spatial topological structure between each image block.

2. The method of claim 1, wherein the graph convolutional network-based remote sensing image scene classification method is characterized by, The method for up-sampling the first feature map is: using a nearest-neighbor interpolation method to interpolate each pixel of the first feature map, and enlarging the first feature map to the same size as the corresponding image block.

3. The method of claim 1, wherein the graph convolutional network-based remote sensing image scene classification method is characterized by, The second feature contains a representation of the spatial topological structure.

4. The method of claim 1, wherein the graph convolutional network-based remote sensing image scene classification method is characterized by, The image block is a non-overlapping image region.

5. The method of claim 1, wherein the graph convolutional network-based remote sensing image scene classification method is characterized by, Step S15 comprises: The transformer module comprises an input layer and a transformer layer, the input layer being a position encoding layer, the position encoding layer being used to perform position encoding on the image block to obtain a position vector sequence, and then the position vector sequence is embedded into the image block sequence to obtain an image block sequence embedded with position vector information; The transformer layer comprises four encoders, each encoder comprising a multi-head attention layer and a multi-layer perceptron, the image block sequence embedded with position vector information is input into the transformer layer for encoding, and the output is the third feature.

6. The method of claim 5, wherein the graph convolutional network-based remote sensing image scene classification method is characterized by, The multi-layer perceptron comprises an activation function.

7. The method of claim 6, wherein the graph convolutional network-based remote sensing image scene classification method is characterized by, The third feature is the output feature of the last encoder.

8. The method of claim 6, wherein the graph convolutional network-based remote sensing image scene classification method is characterized by, The activation function is one of a sigmoid function, a tanh function and a relu function.

9. The method of claim 1, wherein the graph convolutional network-based remote sensing image scene classification method is characterized by, The second feature and the third feature have the same feature scale.

Images