Hyperspectral image classification method based on cascade residual GCN and hybrid convolution

Abstract

The invention discloses a hyperspectral image classification method based on cascade residual GCN and hybrid convolution. The method comprises the following steps: firstly, removing redundant spectral bands in an original hyperspectral image; pixel points with similar space-spectrum information are gathered by using a simple linear iterative algorithm to form superpixel points, and a relation matrix is established and converted into superpixel-level graph structure data through a graph encoder; the method comprises the following steps: obtaining a feature map containing a similarity relationship by using two residual image convolution modules, remodeling the feature map into a three-dimensional form, inputting the feature map into a residual 3D convolution module to extract preliminary spatial-spectral features, performing channel weighting, inputting the feature map into a densely connected depth separable convolution module, further extracting spatial features, and obtaining the spatial-spectral features; and finally, after space attention weighting adjustment, inputting into a classification module to obtain a category prediction probability. Based on limited samples, the spatial features and the spectral features of the hyperspectral image can be fully extracted, and the classification precision is improved under the conditions that the network parameter quantity is not large and the structure is not complex.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, relates to the classification of hyperspectral images, and specifically relates to a hyperspectral image classification method based on cascaded residual GCN and hybrid convolution Background Art

[0002] Hyperspectral image technology combines imaging and fine spectral analysis, providing a multi-dimensional data acquisition method. For a specific surface area, hyperspectral imaging equipment can obtain hundreds of narrow and continuous spectral channels from the visible light to the near-infrared band, and its spectral resolution can reach the nanometer level. This means that each pixel in the image corresponds to a continuous spectral curve, and this curve, as the unique spectral identifier of the object at that position, reveals the characteristics of the substance. With its fine spectral resolution ability, hyperspectral remote sensing makes it easy to identify some ground object types that are difficult to distinguish in traditional visible light remote sensing images, becoming an important research hotspot in the field of remote sensing science, and being widely applied to many key fields such as precision agriculture, urban planning, environmental protection, resource exploration, and military reconnaissance, providing decision-making support information for various industries

[0003] The main challenges faced by hyperspectral image classification include the high-dimensional characteristics of data, limited labeled samples, and the uneven distribution of ground objects. When dealing with such tasks, traditional machine learning methods usually focus on spectral information and use techniques such as the K-nearest neighbor algorithm, support vector machine (SVM), and sparse representation for research and processing. However, these methods rely on the manually designed feature extraction process, highly depend on the experience of domain experts, and are mainly limited to the acquisition of shallow features, making it difficult to capture deep and more discriminative abstract features. In addition, they often ignore the spatial dimension information, which is crucial for improving the accuracy of pixel classification in hyperspectral images

[0004] Deep learning, with its excellent feature extraction ability, has become the preferred tool for researchers to handle hyperspectral image classification tasks. Convolutional Neural Networks (CNNs) are famous for their automatic feature learning and parameter sharing advantages. However, although one-dimensional convolutional neural networks (1D-CNNs) can extract spectral information from hyperspectral data, they lack the understanding of spatial context; two-dimensional convolutional neural networks (2D-CNNs) perform well in capturing the spatial structure of images but fail to fully utilize the rich spectral dimension. To comprehensively consider spectral and spatial information, three-dimensional convolutional neural networks (3D-CNNs) emerged, which can provide more comprehensive feature extraction when processing spatial and spectral dimensions simultaneously. In view of this, many researchers have proposed and developed classification methods based on 3D-CNNs, aiming to better utilize the multi-dimensional information in hyperspectral images, thereby improving the accuracy and reliability of classification

[0005] However, the convolutional neural network (CNN) uses a fixed-window sliding method for feature extraction in hyperspectral image classification. This method limits its receptive field to a predefined local area, causing the convolution operation to introduce pixel points that do not belong to the same class, especially near the class boundaries, thus leading to classification errors. In addition, since CNN mainly focuses on features within a local area, it is difficult for it to effectively capture the similarity and long-range dependence relationships between pixel points. In contrast, the graph convolutional network (GCN) provides a new perspective for processing non-Euclidean data and can model the similarity between points. By constructing the topological structure between pixels, GCN can capture the similarity relationships and long-distance dependencies of pixel points, which helps to improve the classification accuracy. However, directly constructing the graph structure of the entire hyperspectral image with each pixel as a node will lead to a sharp increase in computational cost, making this method impractical in actual applications. At the same time, as the number of GCN layers increases, the feature propagation through neighborhood aggregation may lead to the "over-smoothing" phenomenon, that is, the features of different nodes tend to be the same, reducing the discriminative ability of the features. Summary of the Invention

[0006] Aiming at the deficiencies of the prior art, the present invention proposes a hyperspectral image classification method based on cascaded residual GCN and hybrid convolution, which combines the advantages of the GCN network and the CNN network, uses the residual and dense connection structures as the connection methods, and fuses features with weights through the attention mechanism to solve the problem of poor accuracy in hyperspectral image classification tasks.

[0007] The hyperspectral image classification method based on cascaded residual GCN and hybrid convolution specifically includes the following steps:

[0008] S1. Input the original hyperspectral image X ∈ R H×W×C , and use the principal component analysis method for preprocessing to obtain the dimension-reduced image X 1 ∈ R H×W×B , where H and W respectively represent the height and width of the spatial dimension of the hyperspectral image, C is the number of spectral bands, and B is the spectral dimension after dimension reduction.

[0009] S2. Use the Simple Linear Iterative Clustering (SLIC) algorithm to obtain superpixel nodes for the image output after preprocessing in S1, calculate the association matrix Q ∈ R HW×K of each pixel point in the superpixel nodes, construct the adjacency matrix A ∈ R K×K based on the superpixel nodes, and obtain the graph data G = (V, E) according to the association matrix Q and the adjacency matrix A, where K is the number of superpixels, V represents the set of superpixel nodes, and E represents the set of edges between superpixel nodes.

[0010] Aggregate features through two cascaded residual graph convolution modules to obtain a superpixel-level feature map O GCN . Each residual graph convolution module includes two graph convolutional neural networks connected by a residual connection.

[0011] S3. Restore the superpixel-level feature map O GCN to the feature map Ax of the three-dimensional shape through a decoder, then enter the residual 3D convolution module, repeat 3D-CNN, batch normalization (BN), and ReLU activation 3 times, and then perform a skip connection between the feature map Ax and the output of the last ReLU activation, and then enter the channel attention module to integrate the channel information and output the feature map Bx.

[0012] Among them, the convolution kernel size of the first 3D-CNN is (3, 3, 7), the second is (5, 5, 7), and the third is (7, 7, 7).

[0013] S4. Input the feature map Bx output by the residual 3D convolution module into a densely connected depthwise separable module. The densely connected depthwise separable module connects three depthwise separable convolutional layers in a densely connected manner. The output of each layer will be passed to all subsequent layers as input, thus allowing for more smooth feature reuse and gradient flow. And perform spatial attention enhancement on the output of the last depthwise separable convolutional layer. Each depthwise separable convolutional layer sequentially performs depthwise separable convolution, batch normalization, and ReLU activation on the input features. The convolution kernel size of the depthwise convolution in the depthwise separable convolution is 3*3, and the convolution kernel size of the pointwise convolution is 1*1.

[0014] S5. Send the output feature map of S4 into the classification module to output the class prediction probability at the pixel level of the hyperspectral image.

[0015] The present invention has the following beneficial effects:

[0016] 1. The present invention uses the method of cascading residual graph convolution and hybrid convolution CNN, and simultaneously utilizes the ability of GCN to process irregular data and the powerful feature extraction ability of the CNN network. The class boundary information and similarity relationship extracted by GCN make up for the class boundary classification error and local receptive field problems caused by the fixed window of the convolutional network to extract information, making the extracted spatial-spectral information more detailed and conducive to the final accurate classification.

[0017] 2. The present invention uses the SLIC algorithm to divide the image into superpixel nodes and constructs an adjacency matrix, which reduces the computational complexity of graph convolution. At the same time, the design of the residual structure also solves the problem that the feature aggregation of graph convolution tends to be smooth. Similarly, the residual connection of the 3D convolution module also prevents network degradation and reduces the overfitting pressure. At the same time, the densely connected depthwise separable layer can also reuse features from shallow to deep and reduce the computational complexity, and the expressive ability of the model is also improved.

[0018] 3. The present invention introduces a spatial attention mechanism and a channel attention mechanism, and adds them at different feature extraction stages. By weighted integration of the information at spatial and channel positions at different stages, important features are captured, unimportant features are suppressed, the interference caused by redundant information is reduced, and the classification efficiency is improved. Brief Description of the Drawings

[0019] Figure 1 is a flowchart of a hyperspectral image classification method based on cascaded residual GCN and hybrid convolution;

[0020] Figure 2 is a schematic structural diagram of a residual graph convolution module;

[0021] Figure 3 is a schematic structural diagram of a residual 3D convolution module;

[0022] Figure 4 is a schematic structural diagram of a channel attention module;

[0023] Figure 5 is a schematic structural diagram of a densely connected depthwise separable module;

[0024] Figure 6 is a schematic structural diagram of a spatial attention module;

[0025] Figure 7 is a schematic diagram of hyperspectral images of different datasets in the embodiment;

[0026] Figure 8 is a standard classification map corresponding to different datasets in the embodiment;

[0027] Figure 9 is the classification result of different classification methods for the Indian pines dataset in the embodiment;

[0028] Figure 10 is the classification result of different classification methods for the PaviaU dataset in the embodiment;

[0029] Figure 11 is the classification result of different classification methods for the Salinas dataset in the embodiment. Detailed Embodiment

[0030] The present invention will be further explained below with reference to the accompanying drawings;

[0031] As Figure 1 shown, a hyperspectral image classification method based on cascaded residual graph convolution and hybrid convolutional attention mechanism includes the following steps:

[0032] Step 1: Input the original hyperspectral image X ∈ R H×W×C , and use the PCA algorithm to perform feature dimensionality reduction on the redundant dimensions of the original hyperspectral image.

[0033] Step 2: For the image X 1 ∈ R H×W×B preprocessed in Step 1, use the SLIC algorithm to obtain superpixel nodes, calculate the association matrix Q ∈ R m between each pixel point x HW×K , construct the adjacency matrix A ∈ R K ×K based on the superpixel nodes, obtain the graph data G = (V, E) according to the association matrix Q and the adjacency matrix A, and then use two cascaded residual graph convolution modules to aggregate features to obtain the superpixel-level output feature O GCN :

[0034] Step 2.1: To reduce the computational complexity and at the same time retain the local structure of the hyperspectral image, use the SLIC algorithm to divide the entire image into many spatially connected superpixel nodes, aggregate the pixel points into the superpixel nodes according to the spatial-spectral similarity, and establish the adjacency relationship between the superpixel nodes, converting the hyperspectral image into an undirected graph G = (V, E). Where V ∈ R K×B is the node set composed of superpixels, K is the number of superpixel nodes, the i-th superpixel node i ∈ [1, K], N is the number of pixel points included in a single superpixel. E is the edge set, represented by the adjacency matrix A, and A i,j represents the connection relationship between the superpixel nodes V i , V j :

[0035]

[0036] where σ is the bandwidth parameter, used to control the width of the Gaussian kernel.

[0037] Step 2.2: To construct a graph encoder and a graph decoder to realize the mutual transformation between the superpixel-level graph data and the original pixel-level image, define the association matrix introduced by the SLIC algorithm as Q ∈ R HW×K as:

[0038]

[0039] Step 2.3: Use the association matrix Q to encode the hyperspectral image into graph nodes in the form of matrix multiplication:

[0040]

[0041] where Flatten(·) means to expand X along the spatial dimension 1 for expansion.

[0042] Step 2.4: Aggregate features through two cascaded residual graph convolution modules to obtain the superpixel-level feature map O GCN . As Figure 2 shown, each residual graph convolution module includes two graph convolutional neural networks connected by residual connections, and a BN layer and a ReLU activation function layer are added after each graph convolutional neural network.

[0043] In a single graph convolutional neural network, the input node features H (l+1) of the (l + 1)-th layer are:

[0044]

[0045] where f is the activation function, H (l) is the input node features of the l-th layer, W (l) and b (l) are the learnable weight parameters and biases of the l-th layer respectively, is the adjacency matrix A with self-loops introduced, I is the identity matrix,

[0046] After propagation through two layers of graph convolutional neural networks, a skip connection is added to add the input features of the residual graph convolution module to the output of the aggregation of the two-layer features, effectively alleviating the over-smoothing problem caused by neighborhood aggregation in feature propagation.

[0047] Step Three: Restore the obtained superpixel-level feature map O GCN back to the original three-dimensional shape through the decoder to obtain the feature map Ax, and then enter the residual 3D convolution module to output the feature map Bx:

[0048] Step 3.1: The role of the graph decoder is to convert the superpixel-level features output by the graph convolutional network into pixel-level features for subsequent convolutional network feature extraction. The calculation formula for the decoding operation is:

[0049]

[0050] where is the final output O of the graph convolutional network GCNThe vertex set and the incidence matrix Q are used as the input of the decoder, and are restored to the feature map Ax of the three-dimensional shape through the Reshape operation.

[0051] Step 3.2, as Figure 3 shown, in the residual 3D convolution module, the feature map Ax is repeatedly subjected to 3D-CNN, batch normalization (BN), and ReLU activation 3 times. The size of the first 3D-CNN convolution kernel is (3, 3, 7), the second is (5, 5, 7), and the third is (7, 7, 7). Convolution kernels of different sizes are used to extract feature information of multiple scales. The feature map Ax is skip-connected to the output of the last ReLU activation to prevent overfitting.

[0052] Step 3.3, the channel attention module can adaptively re-weight the features of each channel, enabling the network to pay more attention to the features useful for the current task and suppressing the useless channel information. As Figure 4 shown, for the input feature of size h*w*c, first perform average pooling and max pooling, and then use 2D convolution with a convolution kernel size of 1*1 for 2 layers to calculate the channel position weights, respectively obtaining the average output and the max output. After adding the two, use the Sigmoid activation function for integration to finally obtain the c-dimensional channel weight vector μ:

[0053] μ(x) = Sigmoid(avgout(x) + maxout(x))

[0054] avgout(x) = Conv2d(ReLU(Conv2d(AdaptiveAvgPool2d(x))))

[0055] maxout(x) = Conv2d(ReLU(Conv2d(AdaptiveMaxPool2d(x))))

[0056] where h and w represent the size of the input feature space dimension, and c is the number of channels.

[0057] Multiply the corresponding channels of each pixel point in the spatial position by the channel weight vector μ to implement the channel weighting adjustment of each pixel point of the feature map, and output the feature map Bx.

[0058] Step Four: Input the feature map Bx extracted by the 3D convolution module into the densely connected depthwise separable module:

[0059] Step 4.1, as Figure 5As shown, the densely connected depthwise separable module includes three densely connected depthwise separable convolutional layers. Each depthwise separable convolutional layer includes a depthwise convolution with a convolution kernel size of 3*3 and a pointwise convolution with a convolution kernel size of 1*1. After the depthwise convolution and the pointwise convolution, a BN layer and a ReLU activation function layer are connected respectively.

[0060] The depthwise convolution convolves each channel of the input feature map separately to capture the spatial features of each channel. The pointwise convolution integrates all the extracted spatial features and learns the channel-related information of the input feature map, and performs a channel fusion operation similar to that of a normal convolution on the obtained feature map. Thus, the number of parameters and the computational amount are significantly reduced, and overfitting of the model is prevented. The densely connected structure can reuse the features extracted by each layer, combine the features from shallow to deep, enhance the feature extraction ability of the network, and also prevent overfitting.

[0061] Step 4.2: Perform spatial attention weighting on the output features of the last depthwise separable convolutional layer, as Figure 6 shown. For an input feature map with a size of H1*W1*C1, the spatial attention module first calculates the average value and the maximum value using max pooling and average pooling respectively, then concatenates the average value and the maximum value features into 2 channels, and then calculates the spatial position weights using a two-dimensional convolution with a convolution kernel size of 7*7, and integrates them using the Sigmoid activation function. Finally, a spatial weight matrix γ with a size of H1*W1 is obtained, which is used to multiply the corresponding spatial position elements of all channels of the input features.

[0062] Step Five: Feed the features output by the spatial attention module in Step Four into a classification module composed of 1 layer of fully connected layer and a Softmax layer. The parameter of the fully connected layer is 128, and a dropout layer with a dropout rate of 0.5 is connected behind to prevent overfitting. Finally, the final class prediction probability is obtained through the Softmax layer, and the cross-entropy function L is used as the loss function for model training:

[0063]

[0064] where Y label represents the sample label set, Y sc represents that the s-th pixel is the label of the c-th class, P sc represents the probability that the s-th pixel output by the classification module belongs to the c-th class, and C represents the total number of pixel classes.

[0065] The hardware device used in the simulation experiment of this embodiment is a PC equipped with a 15vCPU Intel(R)Xeon(R)Platinum 8474C and an NVIDIA GeForce RTX 4090D GPU, with a memory of 24GB. The compilation environment is TensorFlow 2.9.0Python 3.8(ubuntu20.04)Cuda 11.2.

[0066] To verify the performance of this method in the hyperspectral image classification task, Support Vector Machine (SVM) and 3D Convolutional Neural Network (3D-CNN) are selected as the comparison benchmarks. To ensure the fairness of the experiment, the hyperparameter settings of all methods are kept consistent, and the data is reduced to 30 bands using the principal component analysis method in the preprocessing step.

[0067] Figure 7 The sample images used in this embodiment are from the publicly available hyperspectral datasets Indian Pines (IP), Pavia University (PU), and Salinas (SA), Figure 8 which are the standard classification maps corresponding to the three datasets.

[0068] For the IP dataset, 5% of the data is randomly selected as the training samples; for the PU and SA datasets, 0.3% of the data is randomly selected as the training samples. The number of training iterations is set to 200, the learning rate is 0.001, and the batch size is 32. The Adam optimizer is used to accelerate the training process, and the cross-entropy loss function is adopted to guide the model learning. To prevent overfitting, a dropout layer with an inactivation probability of 0.5 is added. For different model methods, ten experiments are conducted respectively and the average value is taken as the experimental result.

[0069] Figures 9 to 11The classification results comparison of different classification methods on three hyperspectral datasets, namely Indian Pines (IP), Pavia University (PU), and Salinas (SA), is shown respectively. From left to right, they are the support vector machine, 3D convolutional neural network, and the proposed method. It can be seen that there are a large number of misclassifications in the classification results of the support vector machine, with a low classification accuracy, presenting a distribution pattern similar to snowflakes. In contrast, the 3D convolutional neural network and the proposed method have significantly improved the classification effect, and the overall image is clearer and smoother. However, the fixed window size of the 3D convolutional neural network limits the flexibility of feature capture and fails to fully consider the similarity relationship between pixels. The proposed method shows fewer cases of mixed land cover, almost eliminates noise points and contaminated areas, greatly improves the classification quality, demonstrates the best smoothness and accuracy, and its classification map is highly consistent with the standard classification map, reflecting the excellent classification performance of the model.

[0070] The applicant believes that the above experimental results can illustrate that the proposed method not only overcomes the limitations of traditional methods but also achieves more accurate and smooth classification results on all test datasets. The above simulation analysis proves the correctness and effectiveness of the proposed method.

[0071] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Obviously, for professionals in the field, after understanding the content and principle of the present invention, various modifications and changes in form and details may be made without departing from the principle and structure of the present invention. However, these modifications and changes based on the idea of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A hyperspectral image classification method based on cascaded residual GCN and hybrid convolution, characterized by: The specific steps include: Step 1: Use simple linear iterative clustering to obtain superpixel nodes for the hyperspectral image, and calculate the correlation matrix Q∈R between each pixel in the superpixel node HW×K , construct the adjacency matrix A∈R based on superpixel nodes K×K , according to the correlation matrix Q and the adjacency matrix A, we get the graph data G = (V, E), where K is the number of superpixels, V represents the set of superpixel nodes, and E represents the set of edges between superpixel nodes; The features are aggregated by two serially connected residual graph convolution modules to obtain a super-pixel feature map O GCN ; Each residual graph convolution module consists of two graph convolutional neural networks connected by residuals; Step 2: Transform the super-pixel feature map O GCN The decoder recovers the feature map Ax of the three-dimensional shape, and then enters the residual 3D convolution module, repeats multiple 3D-CNN, batch normalization (BN) and ReLU activations, and then jump-connects the feature map Ax with the output of the last ReLU activation, and then enters the channel attention module to integrate the channel information and output the feature map Bx; Step 3: Input the feature map Bx output by the residual 3D convolution module into a densely connected depth-separable module. The densely connected depth-separable module connects multiple depth-separable convolutional layers in a densely connected manner. The output of each layer is passed to all subsequent layers as input, and the output of the last depth-separable convolutional layer is spatially enhanced. Step 4: Send the output feature map of the densely connected deep separable module to the classification module to output the category prediction probability at the pixel level of the hyperspectral image.

2. The hyperspectral image classification method based on cascaded residual GCN and hybrid convolution as claimed in claim 1, characterized in that: The principal component analysis method is used to preprocess the hyperspectral images and filter out redundant bands.

3. The hyperspectral image classification method based on cascaded residual GCN and hybrid convolution as claimed in claim 1, characterized in that: The adjacency matrix A records the relationship between superpixel nodes: Among them, A i,j Represents a superpixel node V i 、V j , σ is the bandwidth parameter, which is used to control the width of the Gaussian kernel.

4. The hyperspectral image classification method based on cascaded residual GCN and hybrid convolution as claimed in claim 1, characterized in that: In a single graph convolutional neural network, the input node feature H of the l+1th layer (l+1) for: Among them, f is the activation function, H (l) is the input node feature of the lth layer, W (l) and b (l) are the learnable weight parameters and bias of the lth layer, respectively. To introduce the self-loop adjacency matrix A, I is the unit matrix, 5. The hyperspectral image classification method based on cascaded residual GCN and hybrid convolution as claimed in claim 1, characterized in that: The residual 3D convolution module includes three repetitions of 3D-CNN, batch normalization (BN) and ReLU activation, and the sizes of the three 3D-CNN convolution kernels are (3, 3, 7), (5, 5, 7), and (7, 7, 7), respectively.

6. The hyperspectral image classification method based on cascaded residual GCN and hybrid convolution as claimed in claim 1, characterized in that: The densely connected depth-wise separable module includes three densely connected depth-wise separable convolutional layers, each of which includes a depth-wise convolution with a convolution kernel size of 3*3 and a point-wise convolution with a convolution kernel size of 1*1. The depth-wise convolution and the point-by-point convolution are both connected to a BN layer and a ReLU activation function layer.

7. The hyperspectral image classification method based on cascaded residual GCN and hybrid convolution as claimed in claim 1, characterized in that: The channel attention module first performs average pooling and maximum pooling on the input features, and then uses 2D convolution with a 2-layer convolution kernel size of 1*1 to calculate the channel position weights, respectively obtaining the average output and the maximum output, and then adds the two together and integrates them using the Sigmoid activation function to finally obtain the c-dimensional channel weight vector μ; The spatial attention module first uses maximum pooling and average pooling to calculate the average and maximum values ​​of the input features respectively, then concatenates the average and maximum features into 2 channels, and then uses a two-dimensional convolution with a convolution kernel size of 7*7 to calculate the spatial position weight, and integrates it using the Sigmoid activation function to finally obtain a spatial weight matrix γ of size H1*W1.

8. The hyperspectral image classification method based on cascaded residual GCN and hybrid convolution as claimed in claim 1, characterized in that: The classification module includes a fully connected layer, a Softmax layer and a dropout layer.

9. The hyperspectral image classification method based on cascaded residual GCN and hybrid convolution as claimed in claim 8, characterized in that: The parameter of the fully connected layer is 128, and the deactivation rate of the dropout layer is 0.

5.

10. The hyperspectral image classification method based on cascaded residual GCN and hybrid convolution as claimed in any one of claims 1, 4 to 9, characterized in that: Use the cross entropy function L as the loss function: where Y label represents the sample label set, Y sc Indicates that the sth pixel is the label of the cth class, P sc It represents the probability that the s-th pixel output by the classification module belongs to the c-th category, and C represents the total number of pixel categories.

Images