Hyperspectral Image Classification Method Based on Multi-Scale Superpixel Node Aggregation Graph Convolution Residual Network

Abstract

The present invention discloses a hyperspectral image classification method based on a multi-scale superpixel node aggregation graph convolutional residual network, belonging to the field of image classification technology. The method constructs a multi-scale superpixel module and uses node aggregation operations to construct different graphs at multiple scales. This method can not only reduce the computational burden brought by large graphs, but also describe the category edge areas from different angles, improving the accuracy of edge classification; an end-to-end graph convolutional residual network module is constructed to more deeply mine the potential features of the data; a minibatch graph convolutional residual network is designed to further consider the computational cost and accelerate network convergence. The hyperspectral image classification method based on the multi-scale superpixel node aggregation graph convolutional residual network explores the characteristic structure of different regions according to the superpixel information of different scales, enhances the extraction of the model's spatial spectrum features, and has higher accuracy and more robust performance than other methods.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and in particular relates to a hyperspectral image classification method based on a multi-scale superpixel node aggregation graph convolutional residual network. Background Art

[0002] In recent years, image convolutional networks (GCNNs) have attracted increasing attention in the field of hyperspectral image classification, aiming to better utilize the relationship graphs between data. However, most existing methods based on graph convolutions have two major drawbacks: on the one hand, using the structural graph constructed from all pixel nodes results in a smoothing of pixel-level node differences and a loss of potential spatial information; and on the other hand, they incur high computational costs due to the catastrophic size of the GCNNs. On the other hand, no good adjacency features have been explored to represent the spatial-spectral relationships hidden in hyperspectral images. To this end, a hyperspectral image classification method based on a multi-scale superpixel node aggregation graph convolution residual network is proposed. Summary of the Invention

[0003] To solve the above problems, the present invention provides a hyperspectral image classification method based on a multi-scale superpixel node aggregation graph convolutional residual network, the method comprising the steps of:

[0004] (1) Constructing a multi-scale superpixel module

[0005] (1a) Principal component analysis is performed on the original data set to generate experimental data. The first dimension features of the experimental data are extracted to perform multi-scale superpixel segmentation, generating superpixel information at multiple scales. The larger the scale, the more superpixels there are after segmentation.

[0006] (1b) Divide the experimental data and labeled data into training set and test set;

[0007] (2) Constructing a node aggregation module

[0008] (2a) Input the training set and aggregate the pixel nodes in the same superpixel block at the same scale to generate a node-level feature. The same operation is performed on other superpixel blocks, and the number of node-level features generated is the same as the number of superpixel blocks. The aggregation operation is also the same for other scales. In other words, superpixel information at multiple scales corresponds to node-level features at multiple scales.

[0009] (2b) Construct an adjacency matrix based on the node-level features and prepare the node-level features and adjacency matrix of each scale as network input;

[0010] (3) Constructing a multi-scale graph convolutional residual network module

[0011] (3a) A graph convolution residual network branch is constructed for each scale. The network consists of three layers of GCRN. The GCRN consists of a graph convolution module and a residual term. The graph convolution module is composed of a BN layer, a GCN layer, a BN layer, and a RELU layer. The residual term is the concatenation of the input of the first BN layer and the output of the RELU layer in the feature dimension. The largest scale is different from other scales. The minibatch method is used to construct the graph convolution residual network. The small batch strategy is used to input only part of the samples and the corresponding sub-adjacency matrix each time. The node-level features and adjacency matrices of different scales are input into the corresponding network branches to obtain the network output features.

[0012] (3b) Perform a diffusion operation on the network output features to restore them to the dimension size before aggregation, generate pixel-level features, and then merge the pixel-level features of each scale into a feature that is finally used for classification;

[0013] (3c) The features used for classification are sent to the softmax classification module for classification to obtain the predicted label;

[0014] (4) Construct a loss function based on the cross entropy function to calculate the loss value and perform network training until the loss value is minimized and the network converges;

[0015] (5) The test set is passed into the node aggregation module, and then passed through the graph convolutional residual network module to obtain the classification results and calculate the classification accuracy;

[0016] Preferably, the principal component analysis step is:

[0017] (1) For the data set X = {x1, x2, ..., x n Decentralization means subtracting the mean value of each dimension from the feature.

[0018] (2) Calculate the covariance matrix C = (1 / n-1)X T X.

[0019] (3) The eigenvalue decomposition method is used to find the eigenvalues and eigenvectors of the covariance matrix.

[0020] (4) Arrange the eigenvectors according to their eigenvalues to form the eigenvector matrix P.

[0021] (5) Feature extraction X pca =PX.

[0022] Preferably, the multi-scale superpixel segmentation is to segment the data using the ERS superpixel segmentation algorithm, and generate superpixels of multiple scales by changing the number of generated superpixels.

[0023] Preferably, the node aggregation module is to aggregate the pixels in the homogeneous area divided by the superpixel into a node-level feature, and its aggregation operation expression is:

[0024]

[0025] Among them, X pca is the data after principal component analysis. i Indicates that in the pth superpixel region r p The weight of the i-th pixel in , that is, X pca (r p The weight of (i)) is expressed as:

[0026]

[0027] in, represents the sample mean within the p-th superpixel area.

[0028] Preferably, the adjacency matrix is expressed as:

[0029]

[0030] Among them, v i represents the i-th node, and τ is the control parameter.

[0031] Preferably, the convolutional residual network module consists of three layers of GCRN, and the expression of GCRN is:

[0032]

[0033] Among them, RELU(·) and BN(·) represent activation function and batch normalization, W (l) and b (l) Represents the learnable parameters of layer l. R (l) Represents the residual term of the lth layer, and its expression is:

[0034]

[0035] Where d(·) represents the potential feature dimension, W R (l) and b R (l) represents the learnable parameters.

[0036] Preferably, the minibatch method constructs a graph convolutional residual network, which differs from the above-mentioned convolutional residual network module in that, first, the training method adopts the minibatch method, and each time some sample features and corresponding subgraphs are taken for batch training until all samples are taken out, and the expression is:

[0037]

[0038]

[0039] Among them, H s (l) Represents the input of the first layer s batch, A (mb)s represents the adjacency matrix of the sth batch of subgraphs, H (l+1) It represents the final output after all batches are trained, N / m represents the number of subgraphs, and m is the batch size.

[0040] Second: The branch adjacency matrix construction expression is:

[0041]

[0042] Among them, U Vi Represents the set adjacent to the sample feature Vi.

[0043] Preferably, the diffusion operation is the reverse operation of node aggregation, which copies the features in the same superpixel area into corresponding node-level features to generate pixel-level features. The diffusion operation expression is:

[0044] X pca (r p (i))=V p

[0045] Preferably, the pixel-level feature merging operation expression is:

[0046]

[0047] Where M represents the number of scales.

[0048] Preferably, the loss function expression is:

[0049]

[0050] Among them, N represents the number of samples, C represents the number of categories, and y i,c and p i,c Represents the label value and predicted value of the sample. M represents the number of scales, and l represents the number of network layers.

[0051] The hyperspectral image classification method based on a multi-scale superpixel node aggregation graph convolutional residual network provided by the present invention has the following advantages:

[0052] (1) Under the multi-scale superpixel module, different graphs are constructed at multiple scales through the node aggregation module, which not only reduces the computational burden brought by large images, but also describes the category edge areas from different angles and improves the accuracy of edge classification;

[0053] (2) To make the network modules stackable and solve the problem of network degradation, we introduced the idea of residual networks and constructed an end-to-end graph convolutional residual network module, which avoids the degradation of the objective function and can more deeply explore the potential features of the data;

[0054] (3) Based on the superpixel information of different scales, the characteristic structures of different regions are explored. In the large-scale superpixel division branch, a minibatch scheme is designed to further consider the computational cost and accelerate the network convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 This is a flow chart of a hyperspectral image classification method based on a multi-scale superpixel node aggregation graph convolutional residual network provided by the present invention;

[0057] Figure 2 This is a structural diagram of a hyperspectral image classification method based on a multi-scale superpixel node aggregation graph convolutional residual network provided by the present invention; DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0059] The present invention provides a hyperspectral image classification method based on a multi-scale superpixel node aggregation graph convolutional residual network. Taking the Indian pine dataset as an example, the method steps are as follows:

[0060] Step S1: Get the Indian pine dataset, which includes original data X and label Y, and the image size is (145, 145, 200).

[0061] Step S2: Perform principal component analysis on the original data X without changing the original data dimension to generate experimental data X pca .

[0062] Step S3: Extract experimental data X pcaThe first dimension features are segmented into multi-scale superpixels, the number of superpixels is set to (145×145) / k, and k is set to 100, 300, 500, and 900 to generate superpixels of four scales.

[0063] Step S4: Divide the experimental data and label data into a training set (695, 200) and a test set (9671, 200).

[0064] Step S5: Input the training set into the node aggregation module to generate node-level features of four scales. The size of each scale feature is (sn, 200), where sn is the number of superpixels at the same scale.

[0065] Step S6: Construct adjacency matrices of features at scales of 300, 500, and 900, with a size of (sn,sn). Input the features and adjacency matrices corresponding to the three scales into the graph convolutional residual network branch for training. The three scales correspond to three graph convolutional residual network branches, each consisting of three layers of GCRN. The dimensions of the three-layer GCRN network are set to (200, 128), (256, 128), and (256, 16), respectively. The dimensions of the first and third residual structures are set to (200, 128) and (128, 16), respectively. The momentum of the batch normalization layer is set to 0.9, and the learning rate is set to 0.001.

[0066] Step S7: Use the minibatch method to construct the adjacency matrix of features at scale 100. Set the batch size to 32 and randomly input 32 features and their corresponding submatrices into the graph convolutional residual network each time. This training cycle is completed after all samples have been extracted. The graph convolutional residual network construction is the same as in step 6.

[0067] Step S8: Diffusion operation is performed on the outputs of the four branches to restore them to the shape and size before aggregation (695, 16). That is, all samples in the same superpixel area are replaced with the corresponding node features. Then, the feature dimensions are merged to obtain the final classification feature of size (695, 16*4).

[0068] Step S9: The final classification features are passed to the softmax classifier for classification, resulting in a prediction result of size (695, 16). The loss value is calculated based on the training labels. The network is trained using the gradient descent algorithm with the default Adam configuration. Training ends when the loss value reaches the minimum.

[0069] Step S10: Construct the node-level features and adjacency matrix of the test set, pass them into the network for prediction, and calculate the classification accuracy based on the prediction results.

[0070] Detailed description of the invention Results

[0071] This implementation uses three publicly available datasets. The details of the datasets are as follows:

[0072] The Indian pine dataset has 200 bands, a total of 145*145 pixels, and a total of 16 categories.

[0073] The University of Pavia dataset has 103 bands, a total of 610*340 pixels, and a total of 9 categories.

[0074] The Houston 2013 dataset has 144 bands, a total of 349*1905 pixels, and a total of 15 categories.

[0075] To demonstrate the superiority of our implementation, we compared it with several existing state-of-the-art methods, including MGCN, VIT, SF, and SSFTT. The accuracy of these methods on the classification of the three public datasets is compared. The specific data comparisons are shown in the following tables. Table 1 shows the results on the Indian Pine dataset; Table 2 shows the results on the University of Pavia dataset; and Table 3 shows the results on the Houston 2013 dataset.

[0076] Table 1 Classification results of Indian pine dataset

[0077]

[0078] Table 2 Classification results of the University of Pavia dataset

[0079]

[0080] Table 3 Classification results of Houston 2013 dataset

[0081]

[0082]

[0083] By comparing the data in the above table, it can be clearly seen that this method achieves very good performance and significantly improves the classification performance.

[0084] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or illustrative of the principles of the present invention and do not constitute limitations of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included within the scope of protection of the present invention. In addition, the appended claims are intended to cover all variations and modifications that fall within the scope and metes and bounds of the appended claims, or equivalents thereof.

Claims

1. A hyperspectral image classification method based on multi-scale superpixel node aggregation graph convolutional residual network, characterized by: The method comprises the steps of: (1) Constructing a multi-scale superpixel module (1a) Principal component analysis is performed on the original data set to generate experimental data. The first dimension features of the experimental data are extracted to perform multi-scale superpixel segmentation, generating superpixel information at multiple scales. The larger the scale, the more superpixels there are after segmentation. (1b) Divide the experimental data and labeled data into training set and test set; (2) Constructing a node aggregation module (2a) Input the training set, aggregate the pixel nodes in the same superpixel block at the same scale, and generate a node-level feature; The operations on other superpixel blocks are the same, and the number of node-level features generated is consistent with the number of superpixel blocks. The aggregation operations on other scales are also the same. In other words, superpixel information at multiple scales generates node-level features at multiple scales. (2b) Construct an adjacency matrix based on the node-level features and prepare the node-level features and adjacency matrix of each scale as network input; (3) Constructing a multi-scale graph convolutional residual network module (3a) A graph convolution residual network branch is constructed for each scale. The network consists of three layers of GCRN. The GCRN consists of a graph convolution module and a residual term. The graph convolution module is composed of a BN layer, a GCN layer, a BN layer, and a RELU layer. The residual term is the concatenation of the input of the first BN layer and the output of the RELU layer in the feature dimension. The largest scale is different from other scales. The minibatch method is used to construct the graph convolution residual network. The small batch strategy is used to input only part of the samples and the corresponding sub-adjacency matrix each time. The node-level features and adjacency matrices of different scales are input into the corresponding network branches to obtain the network output features. (3b) Perform a diffusion operation on the network output features to restore them to the dimension size before aggregation, generate pixel-level features, and then merge the pixel-level features of each scale into a feature that is finally used for classification; (3c) The features used for classification are sent to the softmax classification module for classification to obtain the predicted label; (4) Construct a loss function based on the cross entropy function to calculate the loss value and perform network training until the loss value is minimized and the network converges; (5) The test set is passed into the node aggregation module, and then passed through the graph convolution residual network module to obtain the classification results and calculate the classification accuracy.

2. The hyperspectral image classification method based on multi-scale superpixel node aggregation graph convolutional residual network according to claim 1 is characterized in that The principal component analysis steps are: (1) For the data set X = {x1, x2, ..., x n Decentralization, that is, subtracting the mean value of each dimension feature; (2) Calculate the covariance matrix C = (1 / n-1)X T X; (3) Eigenvalue decomposition method to find the eigenvalues and eigenvectors of the covariance matrix; (4) Arrange the eigenvectors according to their eigenvalues to form an eigenvector matrix P; (5) Feature extraction X pca =PX.

3. The hyperspectral image classification method based on multi-scale superpixel node aggregation graph convolutional residual network according to claim 1 is characterized in that The multi-scale superpixel segmentation is to segment data using the ERS superpixel segmentation algorithm, and generate superpixels of multiple scales by changing the number of generated superpixels.

4. The hyperspectral image classification method based on multi-scale superpixel node aggregation graph convolutional residual network according to claim 1 is characterized in that The node aggregation operation refers to aggregating the pixels in the homogeneous area divided by the superpixel into a node-level feature. The aggregation operation expression is: Among them, X pca is the data after principal component analysis, λ i Indicates that in the pth superpixel region r p The weight of the i-th pixel in , that is, X pca (r p The weight of (i)) is expressed as: in, represents the sample mean within the p-th superpixel area.

5. The hyperspectral image classification method based on multi-scale superpixel node aggregation graph convolutional residual network according to claim 1 is characterized in that The expression of the adjacency matrix is: Among them, v i represents the i-th node, and τ is the control parameter.

6. The hyperspectral image classification method based on multi-scale superpixel node aggregation graph convolutional residual network according to claim 1 is characterized in that The graph convolutional residual network consists of three layers of GCRN, and the expression of GCRN is: Among them, RELU(·) and BN(·) represent activation function and batch normalization, Indicates splicing by dimension, W (l) and b (l) represents the learnable parameters of the lth layer, A represents the adjacency matrix, R( l) Represents the residual term of the lth layer, and its expression is: Where d(·) represents the potential feature dimension, W R (l) and b R (l) represents the learnable parameters.

7. The hyperspectral image classification method based on multi-scale superpixel node aggregation graph convolutional residual network according to claim 1 is characterized in that The minibatch method constructs a graph convolutional residual network. First, the training method adopts the minibatch method, and each time some sample features and the corresponding subgraphs are taken for batch training until all samples are taken out. The expression is: Among them, H s (l) Represents the input of the first layer s batch, A (mb)s represents the adjacency matrix of the sth batch of subgraphs, H (l+1) represents the final output after all batches of training are completed, N / m represents the number of subgraphs, and m is the batch size; Second: The branch adjacency matrix construction expression is: Among them, U Vi Represents the set adjacent to the sample feature Vi.

8. The hyperspectral image classification method based on multi-scale superpixel node aggregation graph convolutional residual network according to claim 1 is characterized in that: The diffusion operation is the reverse operation of node aggregation, which copies the features within the same superpixel area into corresponding node-level features to generate pixel-level features. The diffusion operation expression is: X pca (r p (i))=V p Among them, X pca is the data after principal component analysis, X pca (r p (i)) represents the pth superpixel region r p The i-th pixel feature in V p Represents the network output features.

9. The hyperspectral image classification method based on multi-scale superpixel node aggregation graph convolutional residual network according to claim 1 is characterized in that The pixel-level feature merging operation expression is: Where M represents the number of scales.

10. The hyperspectral image classification method based on multi-scale superpixel node aggregation graph convolutional residual network according to claim 1 is characterized in that: The loss function expression is: Among them, N represents the number of samples, C represents the number of categories, and y i,c and p i,c Represents the label value and predicted value of the sample, M represents the number of scales, and l represents the number of network layers.

Images