Hyperspectral image classification method fusing mixed multi-hop graph convolutional network

Abstract

The invention provides a hyperspectral image classification method fusing a hybrid multi-hop graph convolutional network, and belongs to the technical field of hyperspectral image processing, and the method comprises the steps: carrying out the dimension reduction processing of original HSI image data, and obtaining the HSI image data after the dimension reduction; segmenting the HSI image data after dimension reduction to obtain a superpixel segmentation result; inputting the original HSI image data into a convolutional feature pyramid network to obtain a first feature representation, and inputting the superpixel segmentation result into a hybrid multi-hop graph convolutional neural network to obtain a second feature representation; splicing the first feature representation and the second feature representation, and inputting the spliced feature representation into a mixed feature fusion module to obtain a fused feature; and inputting the features into a softmax classifier to obtain a final classification result. Local spatial-spectral features in the superpixel can be learned, long-distance associated information of the captured image can be captured, and rich multi-scale features and semantic information in the image can be effectively extracted; and higher classification precision is realized with less parameter quantity.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral image processing, and in particular relates to a hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network. Background Art

[0002] Hyperspectral Image (HSI) is a large amount of image data with continuous spectra and high spectral resolution obtained by an imaging spectrometer in the ultraviolet, visible, near-infrared, and mid-infrared regions of the electromagnetic spectrum, containing rich spatial, radiometric, and spectral information. The spectral-spatial integration characteristic of hyperspectral images can improve the ability to identify ground objects, so it is widely used in precision agriculture, urban planning, ecological environment monitoring, geological resource exploration, military and national defense detection, and other fields. Hyperspectral image classification is to assign a specific category to each pixel point in the image according to the spectral and spatial information contained in the hyperspectral image, so as to achieve the purpose of distinguishing ground object targets. Hyperspectral image classification is one of the key technologies for remote sensing to solve practical application problems, and its accuracy is crucial for subsequent image analysis and applications.

[0003] In recent decades, a large number of hyperspectral image classification methods have emerged and made great progress. Traditional machine learning-based methods usually extract features manually and then use algorithms such as support vector machines and random forests to achieve hyperspectral image classification. Due to the limited feature expression ability of manual feature extraction, its performance is restricted. Benefiting from the powerful feature extraction ability of neural networks, hyperspectral image classification methods based on neural networks have become a research hotspot. From the initial 1D-CNN that only extracts the spectral information of hyperspectral images, to the 2D-CNN that simultaneously uses spatial and spectral information, and the 3D-CNN that directly extracts the spatio-spectral features of images, the accuracy of hyperspectral image classification tasks is continuously increasing. In addition, there are also the hybrid CNN architecture HybridSN that combines 2D-CNN and 3D-CNN, and the methods CTMixer and CTA-net that combine CNN and Transformer.

[0004] Traditional neural networks have made great progress in the field of hyperspectral image classification. However, these methods still have insufficient extraction of spatio-spectral features of hyperspectral images and inefficient feature fusion problems, which lead to unsatisfactory classification results. To address these problems, the present invention provides a hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network. Summary of the Invention

[0005] The present invention provides a hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network, so as to at least solve the problems of insufficient extraction of spatio-spectral features of hyperspectral images and inefficient feature fusion in the prior art, which lead to unsatisfactory classification results.

[0006] The method includes: Step S1: Perform dimensionality reduction on the original HSI image data using principal component analysis to obtain the dimensionality-reduced HSI image data , denotes the height of the original HSI image data, denotes the width of the original HSI image data, denotes the number of bands of the original HSI image data; Step S2: Use the linear iterative clustering method to segment the dimensionality-reduced HSI image data to obtain the superpixel segmentation result ; Step S3: Input the original HSI image data into the convolutional feature pyramid network CFPN to obtain the first feature representation , and input the superpixel segmentation result into the hybrid multi-hop graph convolutional neural network MMGCN to obtain the second feature representation ; Step S4: Concatenate the first feature representation and the second feature representation to obtain the concatenated feature representation, and input the concatenated feature representation into the hybrid feature fusion module HFFM to obtain the fused feature F; Step S5: Input the feature F into the softmax classifier to obtain the final classification result Y.

[0007] Furthermore, in step S2, use the SLIC superpixel segmentation method to segment the original HSI image data to obtain the association matrix between pixels and superpixels , denotes the number of superpixels; Using the association matrix, the conversion between pixel-level features and superpixel-level features can be realized through a graph encoder and a graph decoder, and the formula is as follows:

[0008]

[0009] In the formula, denotes the superpixel-level feature, i.e., the graph node feature, denotes the restored pixel-level feature, denotes the graph encoder that converts pixel-level features to superpixel-level features, denotes the graph encoder that converts superpixel-level features to pixel-level features, denotes the column-normalized association matrix, denotes flattening in the spatial dimension, Represents restoring the spatial dimension of the flattened data; According to the superpixel segmentation result, construct the adjacency matrix of the segmented graph, and its expression is:

[0010] In the formula, Represents the Euclidean distance between two superpixel-level features, Represents the bandwidth of the Gaussian kernel, , Represents the superpixel The set of adjacent superpixels.

[0011] Furthermore, in step S3, the mixed multi-hop graph convolutional neural network MMGCN includes three cascaded mixed 1-hop graph convolutional neural networks Mix 1-hop GCN, mixed 2-hop graph convolutional neural networks Mix 2-hop GCN, and mixed 3-hop graph convolutional neural networks Mix 3-hop GCN; The mixed 1-hop graph convolutional neural network includes a parallel 0-hop graph convolutional neural network and a mixed 1-hop graph convolutional neural network. The 0-hop graph convolutional neural network and the mixed 1-hop graph convolutional neural network use two graph convolutional layers and the LeakyReLU activation function to respectively learn the local information within the superpixels and the global information between the superpixels to obtain a richer feature map. The expression of the whole process is:

[0012] In the formula, Represents the output of the mixed hop graph convolutional neural network, Represents the input of the mixed hop graph convolutional neural network, Represents the learnable weight parameter matrix, Represents the identity matrix, Represents the normalized hop adjacency matrix; Using the adjacency matrix obtained from the superpixel segmentation result to calculate the

[0013] In the formula, Represents the intermediate node of the path; Concatenate the feature maps output by the mixed 1-hop graph convolutional neural network Mix 1-hop GCN, the mixed 2-hop graph convolutional neural network Mix 2-hop GCN, and the mixed 3-hop graph convolutional neural network Mix 3-hop GCN to obtain the output of the mixed multi-hop graph convolutional neural network, that is, the second feature representation 。

[0014] Furthermore, in step S3, the convolutional feature pyramid network CFPN uses two convolutional layers to extract the spectral information of the original HSI image data; Batch normalization BN is added before the convolutional layer, and the LReLU activation function is added after the convolutional layer to obtain the feature map ; The obtained feature map is used as the input of the atrous feature pyramid module AFPM.

[0015] Furthermore, in step S3, the convolutional feature pyramid network CFPN includes two cascaded atrous feature pyramid modules AFPM.

[0016] Furthermore, the atrous feature pyramid module AFPM includes three parallel atrous blueprint separable convolution blocks ABSConv Block with different atrous rates; The output feature maps of the three atrous blueprint separable convolution blocks are added together for feature fusion, and its expression is:

[0017] In the formula, represents the atrous blueprint separable convolution block with an atrous rate of 1, represents the atrous blueprint separable convolution block with an atrous rate of 3, represents the atrous blueprint separable convolution block with an atrous rate of 5; In the second atrous feature pyramid module AFPM, the feature maps output by the three atrous blueprint separable convolution blocks are concatenated to obtain the first feature extracted by the convolutional feature pyramid network CFPN.

[0018] Furthermore, in step S4, the first feature representation and the second feature representation are concatenated to obtain the concatenated feature representation, and the concatenated feature representation is input into the hybrid feature fusion module HFFM to obtain the fused feature F, which specifically includes: The first feature and the second feature are concatenated in the channel dimension; the channel shuffle strategy is used to uniformly shuffle the concatenated feature map; One convolutional layer is used to refine the shuffled feature map; The expression of feature fusion is:

[0019] In the formula, Denotes a concatenation operation along the channel dimension, Denotes a channel shuffle operation, Denotes a grouped convolution operation with a convolution size of .

[0020] Furthermore, the method further includes: using the cross-entropy loss commonly used in classification tasks as the loss function to train the network, and its expression is:

[0021] In the formula, Denotes the -th element of the true label of pixel . When pixel belongs to class , otherwise, ; Denotes the predicted probability that pixel belongs to class , Denotes the number of training samples, Denotes the total number of classes.

[0022] Furthermore, the expression of the loss function is:

[0023] In the formula, Denotes the loss of the overall network, Denotes the loss of the MMGCN branch, Denotes the loss of the CFPN branch.

[0024] From the above technical solutions, it can be seen that the present invention has the following advantages: In the hyperspectral image classification method of the fusion hybrid multi-hop graph convolutional network provided in this application, the hybrid multi-hop graph convolutional neural network MMGCN learns the local spatial-spectral features inside the superpixels and captures the long-range correlation information of the image. The convolutional feature pyramid network CFPN can effectively extract rich multi-scale features and semantic information in the image. The hybrid feature fusion module HFFM fully fuses the superpixel-level features and pixel-level features from the two branches of the hybrid multi-hop graph convolutional neural network MMGCN and the convolutional feature pyramid network CFPN, and can achieve higher classification accuracy with fewer parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] To more clearly illustrate the technical solutions of this application, the accompanying drawings required for the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0026] Figure 1 This is the flowchart of the hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network of the present invention.

[0027] Figure 2 This is the specific structure diagram of the hybrid multi-hop graph convolutional neural network of the hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network.

[0028] Figure 3 This is the specific structure diagram of the dilated blueprint separable convolution block of the hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network.

[0029] Figure 4 This is the classification result diagram of the IP dataset of the hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network.

[0030] Figure 5 This is the classification accuracy of the hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network at different superpixel scales on the IP dataset.

[0031] Figure 6 This is the classification accuracy of the hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network at different training set sample ratios on the IP dataset. Specific Embodiments

[0032] To make the application purpose, features, and advantages of this application more obvious and understandable, specific embodiments and accompanying drawings will be used below to clearly and completely describe the technical solutions protected by this application. Obviously, the embodiments described below are only some embodiments of this application, not all embodiments. Based on the embodiments in this patent, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this patent.

[0033] Currently, methods based on convolutional neural networks have improved the classification accuracy of hyperspectral classification tasks. However, such methods require a large number of labeled samples, and the annotation cost of hyperspectral images is expensive. Moreover, convolutional neural networks can only process regular grid data, making it difficult to capture the complex topological structure of images and insufficient in extracting global image information. To address these issues, a semi-supervised learning method based on graph neural networks has been proposed. This graph neural network-based method improves the learning of global image information and alleviates the impact of a small number of labeled samples on classification accuracy to a certain extent. Since traditional graph neural network methods cannot directly extract multi-scale spatial information of images, MDGCN indirectly extracts multi-scale information by using multiple sub-networks, but this results in a high computational complexity of the model. Additionally, methods such as CEGCN, WFCG, and AMGCFN attempt to fuse convolutional neural networks and graph neural networks, leveraging the advantages of both networks to extract global and local features of images respectively. Experimental results have proven that this hybrid network architecture can further improve the classification accuracy. However, existing methods still do not extract the spatio-spectral features of images sufficiently. Moreover, regarding the feature fusion strategy, existing methods either adopt simple concatenation or addition operations, resulting in insufficient feature fusion; or use complex attention mechanisms, which have good feature fusion effects but increase the number of model parameters.

[0034] To alleviate the problems of insufficient spatio-spectral feature extraction of hyperspectral images and inefficient feature fusion, which lead to unsatisfactory classification results, the embodiments of this application provide a hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network. Specifically, to fully extract the global spatio-spectral features of images and reduce the complexity of graph convolutional neural networks, a graph structure is constructed based on superpixels, and a Mixed Multi-hop Graph Convolutional Network (MMGCN) is proposed. This network can not only obtain the local spatio-spectral features within superpixels but also capture the long-range dependence information between superpixels using multi-hop graph convolution. At the same time, a Convolutional Feature Pyramid Network (CFPN) is proposed to extract multi-scale pixel-level features and rich semantic information of images. This network is mainly implemented using dilated convolution, which can not only expand the receptive field of convolution operations but also effectively reduce the number of network parameters. To effectively fuse the features extracted by the MMGCN and CFPN networks, a high-efficiency Hybrid Feature Fusion Module (HFFM) is designed using channel shuffle operations and grouped convolution. Compared with existing algorithms, the hybrid network proposed in this application can effectively extract the spatio-spectral features of hyperspectral images and achieve higher classification accuracy on multiple public datasets.

[0035] The technical solutions proposed in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0036] Figure 1 It is a flowchart of a hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network provided for an embodiment of the present application. As Figure 1 shown, a hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network provided for an embodiment of the present application specifically includes the following steps: The method includes: First, since the original HSI image data has high-dimensional characteristics, with hundreds of bands, containing a large amount of redundant information and noise, principal component analysis (PCA) is used to perform dimensionality reduction processing on the original HSI image data to obtain the dimensionality-reduced HSI image data. denotes the dimensionality-reduced HSI image data, and the number of channels of the dimensionality-reduced HSI image data becomes 3.

[0037] denotes the original HSI image data, denotes the height of the original HSI image data, denotes the width of the original HSI image data, denotes the number of bands of the original HSI image data.

[0038] Then, the linear iterative clustering method (Simple Linear Iterative Clustering, SLIC) is used to segment the dimensionality-reduced HSI image data into superpixels that are spatially connected and spectrally similar. denotes the superpixel segmentation result.

[0039] Next, the original HSI image data is input into the convolutional feature pyramid network CFPN to obtain the first feature representation , and the superpixel segmentation result is input into the hybrid multi-hop graph convolutional neural network MMGCN to obtain the second feature representation .

[0040] Then, the first feature representation and the second feature representation are concatenated to obtain the concatenated feature representation, and the concatenated feature representation is input into the hybrid feature fusion module (HFFM) to obtain the fused feature F.

[0041] Finally, the feature F is input into the softmax classifier to obtain the final classification result Y.

[0042] The graph convolutional neural network can process irregular data with non-Euclidean structures to preserve the irregular class boundary information in hyperspectral images. Considering that directly using pixel points as graph nodes to construct a graph results in a large graph scale, which makes the subsequent processing complexity high. To reduce the number of graph nodes, the original HSI image data after dimensionality reduction is segmented into superpixels, and then a graph structure is constructed based on the superpixels; A superpixel is a region composed of adjacent pixels with similar features. Compared with pixels, superpixels not only contain more structural information of the image but also effectively reduce the number of primitives representing the image.

[0043] Therefore, first, the original HSI image data is segmented by the SLIC superpixel segmentation method to obtain the association matrix between pixels and superpixels , representing the number of superpixels; Using the association matrix, the conversion between pixel-level features and superpixel-level features can be achieved through a graph encoder and a graph decoder. The formula is as follows:

[0044]

[0045] In the formula, represents the superpixel-level feature, that is, the graph node feature, represents the restored pixel-level feature, represents the graph encoder that converts pixel-level features to superpixel-level features, represents the graph encoder that converts superpixel-level features to pixel-level features, represents the column-normalized association matrix, represents flattening in the spatial dimension, represents restoring the spatial dimension of the flattened data; According to the superpixel segmentation result, the adjacency matrix of the segmented graph is constructed. Considering that different neighbor nodes have different influences on the central node, the Gaussian kernel function is used to calculate the similarity between different node features, and the adjacency matrix of the graph is established according to the similarity. Its expression is:

[0046] In the formula, represents the Euclidean distance between two superpixel-level features, represents the bandwidth of the Gaussian kernel, , represents the set of superpixels adjacent to superpixel ; In this way, the original HSI image data is converted into undirected graph structure data , Represents the set of nodes in the graph, Represents the set of edges in the graph, which is determined by the graph node feature matrix and the adjacency matrix Indicates.

[0047] To obtain a more refined classification result, a 0-hop graph convolutional neural network is introduced based on the multi-hop graph convolutional neural network. The 0-hop graph convolutional neural network runs in parallel with the multi-hop graph convolutional neural network and is used to learn the internal information of each node, that is, the empty spectral information within each superpixel. This makes the features extracted by the graph convolutional neural network branch richer, containing both local information of the image and capturing its global information, and can better fuse the pixel-level features of the convolutional neural network branch. In summary, as Figure 1 shown, the Mixed Multi-hop Graph Convolutional Network (MMGCN) contains three cascaded Mixed 1-hop Graph Convolutional Networks, Mixed 2-hop Graph Convolutional Networks, and Mixed 3-hop Graph Convolutional Networks. The specific structure is as Figure 2 shown. Taking the Mixed 1-hop Graph Convolutional Network (Mix 1-hop GCN) as an example, the Mixed 1-hop Graph Convolutional Network includes a parallel 0-hop graph convolutional neural network and a Mixed 1-hop graph convolutional neural network. Both the 0-hop graph convolutional neural network and the Mixed 1-hop graph convolutional neural network use two graph convolutional layers and the LeakyReLU activation function to learn the local information within the superpixels and the global information between the superpixels respectively to obtain a richer feature map. The expression for the entire process is:

[0048] In the formula, Represents the output of the Mixed hop graph convolutional neural network, Represents the input of the Mixed hop graph convolutional neural network, and Both represent learnable weight parameter matrices, Represents the identity matrix, Represents the normalized hop adjacency matrix; Using the adjacency matrix obtained from the superpixel segmentation result, calculate the

[0049] In the formula, Represents the intermediate node of the path; Concatenate the feature maps output by the hybrid 1-hop graph convolutional neural network, the hybrid 2-hop graph convolutional neural network, and the hybrid 3-hop graph convolutional neural network to obtain the output of the hybrid multi-hop graph convolutional neural network.

[0050] The graph convolutional neural network can well model the long-range correlation information in the hyperspectral image, but it is insufficient in extracting local information in the image. Therefore, the present invention designs an auxiliary lightweight convolutional neural network branch, which is a Convolutional Feature Pyramid Network (CFPN), to fully extract the local information in the image, and fully fuse the features extracted by the Convolutional Feature Pyramid Network (CFPN) and the hybrid multi-hop graph convolutional neural network (MMGCN) to achieve more accurate classification.

[0051] First, use two convolutional layers to extract the spectral information of the original HSI image data; as Figure 1 shown, add batch normalization BN before the convolutional layer and add the LReLU activation function after the convolutional layer to obtain the feature map ; used to accelerate network convergence and improve the non-linear expression ability of the network. Then, use the obtained feature map as the input of the Atrous Feature Pyramid Module (AFPM).

[0052] To avoid overfitting caused by too many parameters in the convolutional neural network branch, this embodiment uses atrous blueprint separable convolution to replace the standard convolution to reduce the number of parameters and expand the receptive field of the network, and proposes an Atrous Blueprint Separable Convolution Block (ABSConv Block), whose structure is as Figure 3 shown. The atrous blueprint separable convolution, that is, the reverse depthwise separable convolution, consists of a pointwise convolutional layer and a depthwise convolutional layer. The depthwise convolution here is implemented in the form of atrous convolution to obtain a larger receptive field with fewer parameters. Each convolutional layer is followed by a Mish activation function to increase the non-linear representation ability of the network. In addition, an average pooling layer AvgPool is added between the two convolutional layers to further reduce the interference of redundant information. The batch normalization operation BN is used to accelerate network convergence.

[0053] To fully extract the features of the original HSI image data, two cascaded Atrous Feature Pyramid Modules (AFPMs) are used, as Figure 1 shown, and each Atrous Feature Pyramid Module (AFPM) includes three parallel different atrous rates The Atrous Blueprint Separable Convolution Block (ABSConv Block) has a hollow blueprint. To enhance the feature expression ability, the output feature maps of three Atrous Blueprint Separable Convolution Blocks are added together for feature fusion, and its expression is:

[0054] In the formula, represents the Atrous Blueprint Separable Convolution Block with a dilation rate of 1, represents the Atrous Blueprint Separable Convolution Block with a dilation rate of 3, represents the Atrous Blueprint Separable Convolution Block with a dilation rate of 5; In the second Atrous Feature Pyramid Module (AFPM), the feature maps output by three Atrous Blueprint Separable Convolution Blocks are concatenated to obtain the first feature extracted by the Convolutional Feature Pyramid Network (CFPN) .

[0055] To comprehensively utilize the global and local information of the hyperspectral image to achieve more accurate classification, in this embodiment, a simple and lightweight Hybrid Feature Fusion Module (HFFM) is used to efficiently and fully fuse the first feature from the Convolutional Feature Pyramid Network (CFPN) and the second feature from the Mixed Multi-Hop Graph Convolutional Neural Network (MMGCN) .

[0056] First, the first feature and the second feature are concatenated in the channel dimension; then, to enhance the communication efficiency between the two feature channels, a channel shuffle strategy is used to uniformly shuffle the concatenated feature map; finally, a convolutional layer is used to refine the shuffled feature map; further enhancing the feature representation ability of the network. To reduce the number of parameters of the network, grouped convolution is used instead of the standard convolution operation.

[0057] The expression of feature fusion is:

[0058] In the formula, represents the concatenation operation according to the channel dimension, represents the channel shuffle operation, represents the convolution size of Group convolution operation. Through the hybrid feature fusion module, not only the convolutional feature pyramid network (CFPN) and the output feature maps from the two network branches of the hybrid multi-hop graph convolutional neural network (MMGCN) are fully fused, but also the feature maps are dimensionally reduced in the channel dimension, which reduces the number of parameters required for the final classification layer to a certain extent. Finally, a softmax classifier is used to perform pixel-by-pixel classification on the fused features to obtain the final classification result.

[0059] In this embodiment, the cross-entropy loss commonly used in classification tasks is used as the loss function to train the network, and the formula is as follows:

[0060] In the formula, represents the -th element of the true label of pixel . When pixel belongs to class , , otherwise, ; represents the predicted probability that pixel belongs to class , represents the number of training samples, represents the total number of classes; To better balance the training of the two network branches, not only the loss of the overall network is calculated, but also the losses of the two individual sub-network branches are calculated to optimize the network parameters. Therefore, the final loss function expression is:

[0061] In the formula, represents the loss of the overall network, represents the loss of the MMGCN branch, represents the loss of the CFPN branch.

[0062] As an example, the Indian Pines (IP) dataset is adopted.

[0063] The Indian Pines (IP) dataset was collected by the AVIRIS sensor at the Indian Pine test site in northwestern Indiana. The image size is , the spatial resolution is 20 meters. After removing the bands covering the water absorption areas, there are a total of 200 bands, including 16 land cover classes with 10,366 samples, such as wheat, corn, and soybean crops and non-crop classes such as buildings. During the experiment, 2% of the data samples of each class are randomly selected as the training set, 2% of the data samples of each class are randomly selected as the validation set, and 96% of the data samples of each class are randomly selected as the test set.

[0064] Table 1 Ground object categories of the Indian Pines (IP) dataset, and the number of training set samples, validation set samples, and test set samples for each category

[0065] In this embodiment, the PyTorch framework is used to implement. The Adam optimizer is used to train the model, and the learning rate is , and the weight decay is , and the number of training epochs is 300. The number of superpixels is uniformly set to 1 / 100 of the number of pixels.

[0066] To quantitatively evaluate the classification performance of all methods, the commonly used evaluation metrics of per-Class Accuracy (CA), Overall Accuracy (OA), Average Accuracy (AA), and Kappa Coefficient (KC) are used in the experiment. The higher the value of all evaluation metrics, the better the classification effect. In addition, all experiments are repeated 5 times and the average value is taken to minimize the error caused by randomness, and the corresponding standard deviation values are also calculated.

[0067] The classification result graphs of all comparison methods on the IP dataset are respectively as Figure 4 shown, where Figure 4 (a) is a false color image, Figure 4 (b) is a label, Figure 4 (c) is SVM (OA = 66.54%), Figure 4 (d) is CDCNN (OA = 66.68%), Figure 4 (e) is SSRN (OA = 81.24%), Figure 4 (f) is HybridSN (OA = 84.54%), Figure 4 (g) is CTMixer (OA = 91.62%), Figure 4 (h) is GCN (OA = 78.70%), Figure 4 (i) is CEGCN (OA = 91.33%), Figure 4 (j) is WFCG (OA = 95.98%), Figure 4 (k) is AMGCFN (OA = 94.47%), Figure 4(l) This is the model of the present invention (OA = 96.21%). It can be seen that the misclassified pixels in the classification map obtained by the present invention are the fewest. Especially for the class of "Soybean-notill", other methods are prone to misclassify it with "Corn-notill" and "Soybean-mintill" because their spectral characteristics are relatively similar. Table 2 shows the quantitative comparison results of different methods on the IP dataset, including the average value and standard deviation value of each evaluation index. According to the results in the table, it can be observed that in the case of a small number of training samples, the classification results of the traditional machine learning method SVM and the method based on convolutional neural network are relatively poor because these methods require a large number of labeled samples during training to ensure classification accuracy. The CTMixer method can achieve an overall classification accuracy of 90.64%, mainly because it combines the two network structures of CNN and Transformer and can capture the long-range dependencies in the image. The methods based on graph neural network and convolutional neural network usually obtain better classification results. CEGCN and WFCG achieve overall classification accuracies of 93.97% and 93.40% respectively because these two methods utilize the advantages of the GCN and CNN networks. In contrast, the present invention achieves the best classification result, and the three evaluation indexes of OA, AA, and KC all reach the highest values because the hybrid multi-hop graph convolutional neural network (MMGCN) in the present invention can simultaneously learn the local information inside the superpixels and the long-range information between the superpixels, and fully fuse with the multi-scale pixel-level features extracted by the convolutional feature pyramid network (CFPN) to obtain the final semantically rich feature map.

[0068] Table 2 Comparison of Classification Results on IP Dataset

[0069] To evaluate the model size and running efficiency of the present invention, four methods adopting hybrid model architectures are selected, including CTMixer, CEGCN, WFCG, and AMGCFN, and they are compared with the present invention in terms of the number of parameters, training time, and testing time. Table 3 shows the number of parameters, training time, and testing time of different methods. The number of parameters is calculated based on the IP dataset, and the training time and testing time are the average values of five repeated experiments. It can be seen that the number of parameters of the model of the present invention is far less than that of CTMixer and AMGCFN, and is almost the same as that of the WFCG model with the fewest number of parameters. The training time of the present invention on the IP dataset is very little different from that of the fastest method. In addition, the testing time required by the present invention is very little different from that of other methods based on graph neural network and convolutional neural network, but the present invention achieves a higher classification accuracy.

[0070] Table 3 Comparison of the number of parameters, training time, and testing time of different methods

[0071] The size of the superpixel segmentation scale is closely related to the MMGCN branch. The smaller the superpixel segmentation scale, the more superpixels are generated, and correspondingly, the graph data constructed contains more nodes. To explore the influence of the superpixel segmentation scale on the classification result, the present invention sets the superpixel segmentation scale to 50, 100, 150, 200, 250, and 300 respectively, and tests the classification accuracy of the network on the IP dataset. Each experiment is repeated five times and the average value is taken. The results are as Figure 5 shown. It can be seen that as the superpixel segmentation scale increases, the classification accuracy gradually decreases.

[0072] To evaluate the influence of the number of training samples on the classification performance, the four hybrid model architecture methods of CTMixer, CEGCN, WFCG, and AMGCFN are also selected to compare with the present invention. On the IP dataset, the classification accuracy results under different training set sample ratios are as Figure 6 shown. It can be seen from the figure that as the number of training samples increases, the classification accuracy of all methods has improved. In contrast, the present invention has achieved higher classification accuracy than other methods, and the advantage is more obvious when the sample number is relatively small. This proves that the present invention can also obtain good classification results when the number of training samples is small. This is because the dependence on labeled samples is reduced by adopting the graph-based semi-supervised learning method.

[0073] To explore the respective contributions of the two network branches of MMGCN and CFPN, the classification accuracy of each branch is separately tested on the IP dataset. The experimental results are shown in Table 4. It can be seen that when only the MMGCN branch is available, the classification accuracy of the network is poor because the pixel-level local detail information is missing. The hybrid model combining MMGCN and CFPN has better performance than the single model, which proves that the features extracted by the two network branches are complementary. Here, the feature maps of the MMGCN and CFPN branches are directly concatenated. In contrast, using the designed HFFM to fuse the feature maps of the MMGCN and CFPN branches can improve the classification accuracy by about 1 percentage point.

[0074] Table 4 Comparison of ablation experiment results of different modules on three datasets

[0075] The present invention proposes a hybrid dual-branch network for hyperspectral image classification, which includes three main parts: MMGCN, CFPN, and HFFM. The parallel 0-hop graph convolutional network and multi-hop graph convolutional network in MMGCN are used to learn the local spatio-spectral features within superpixels and capture the long-range correlation information of the image, respectively. CFPN can effectively extract rich multi-scale features and semantic information in the image by using the designed dilated feature pyramid module. HFFM fully fuses the superpixel-level features and pixel-level features from the two branches of MMGCN and CFPN by adopting channel shuffling and lightweight convolutional operations. A large number of experimental results compared with other methods on three publicly available datasets confirm the effectiveness of the present invention, which can achieve higher classification accuracy with fewer parameters.

[0076] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined in the present invention can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in the present invention, but will be accorded the widest scope consistent with the principles and novel features disclosed in the present invention.

[0077] For those of ordinary skill in the art, according to the teachings of the present invention, it does not require creative labor to design different forms of control circuits. These changes, modifications, substitutions, and variations to the embodiments still fall within the protection scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A hyperspectral image classification method integrating a hybrid multi-hop graph convolutional network, characterized in that: The method comprises: Step S1: Principal component analysis is used to analyze the original HSI image data Perform dimensionality reduction processing to obtain the reduced dimensionality HSI image data , Indicates the height of the original HSI image data. Indicates the width of the original HSI image data. Indicates the number of bands of the original HSI image data; Step S2: Use the linear iterative clustering method to segment the HSI image data after dimensionality reduction to obtain the superpixel segmentation result ; Step S3: Convert the original HSI image data Input to the convolutional feature pyramid network CFPN to get the first feature representation , and the superpixel segmentation result Input to the hybrid multi-hop graph convolutional neural network MMGCN to get the second feature representation ; Step S4: Represent the first feature and the second feature representation Perform splicing to obtain a spliced ​​feature representation, and input the spliced ​​feature representation into the hybrid feature fusion module HFFM to obtain a fused feature F; Step S5: Input feature F into the softmax classifier to obtain the final classification result Y.

2. The hyperspectral image classification method fused with a hybrid multi-hop graph convolutional network as claimed in claim 1, characterized in that: In step S2, the original HSI image data is segmented using the SLIC superpixel segmentation method to obtain the correlation matrix between pixels and superpixels , represents the number of superpixels; Using the correlation matrix, the conversion between pixel-level features and super-pixel-level features can be achieved through the graph encoder and the graph decoder. The formula is as follows: In the formula, represents super-pixel-level features, i.e., graph node features, represents the restored pixel-level features, represents a graph encoder that converts pixel-level features into superpixel-level features. represents a graph encoder that converts superpixel-level features into pixel-level features. represents the column-normalized incidence matrix, represents flattening in the spatial dimension, Represents the spatial dimension of the restored flattened data; According to the superpixel segmentation results, the adjacency matrix of the segmented graph is constructed, and its expression is: In the formula, represents the Euclidean distance between two superpixel-level features, represents the bandwidth of the Gaussian kernel, , Representation and Superpixels A collection of adjacent superpixels.

3. The hyperspectral image classification method fused with a hybrid multi-hop graph convolutional network as claimed in claim 2, characterized in that: In step S3, the hybrid multi-hop graph convolutional neural network MMGCN includes three cascaded hybrid 1-hop graph convolutional neural networks Mix 1-hop GCN, hybrid 2-hop graph convolutional neural networks Mix 2-hop GCN and hybrid 3-hop graph convolutional neural networks Mix 3-hopGCN; The hybrid 1-hop graph convolutional neural network includes a parallel 0-hop graph convolutional neural network and a hybrid 1-hop graph convolutional neural network. The 0-hop graph convolutional neural network and the hybrid 1-hop graph convolutional neural network use two graph convolutional layers and LeakyReLU activation functions to learn the local information within the superpixel and the global information between superpixels respectively to obtain a richer feature map. The expression of the whole process is: In the formula, Indicates mixed Output of skip graph convolutional neural network, Indicates mixed The input of the skip graph convolutional neural network, and Both represent learnable weight parameter matrices, represents the identity matrix, Represents the normalized Skip adjacency matrix; Using the adjacency matrix obtained from the superpixel segmentation result ,calculate The jump adjacency matrix is ​​expressed as: In the formula, Represents the intermediate node of the path; The feature maps output by the Mix 1-hop GCN, Mix 2-hop GCN, and Mix 3-hop GCN are concatenated to obtain the output of the Mixed Multi-hop GCN, i.e., the second feature representation. .

4. The hyperspectral image classification method fused with a hybrid multi-hop graph convolutional network as claimed in claim 3, characterized in that: In step S3, the convolutional feature pyramid network CFPN uses two Convolutional layer to extract spectral information of raw HSI image data; Add batch normalization BN before the convolution layer and add LReLU activation function after the convolution layer to get the feature map ; The feature map obtained As the input of the Atrous Feature Pyramid Module AFPM.

5. The hyperspectral image classification method fused with a hybrid multi-hop graph convolutional network as claimed in claim 4, characterized in that: In step S3, the convolutional feature pyramid network CFPN includes two cascaded hole feature pyramid modules AFPM.

6. The hyperspectral image classification method fused with a hybrid multi-hop graph convolutional network as claimed in claim 5, characterized in that: The atrous feature pyramid module AFPM consists of three parallel atrous blueprint separable convolution blocks ABSConvBlock with different atrous rates; The output feature maps of the three dilated blueprint separable convolutional blocks are added together for feature fusion, and the expression is: In the formula, represents a dilated blueprint separable convolutional block with a dilation rate of 1, Represents a dilated blueprint separable convolutional block with a dilation rate of 3. Represents a dilated blueprint separable convolutional block with a dilation rate of 5; In the second hole feature pyramid module AFPM, the feature maps output by the three hole blueprint separable convolution blocks are concatenated to obtain the first feature map extracted by the convolutional feature pyramid network CFPN. .

7. The hyperspectral image classification method fused with a hybrid multi-hop graph convolutional network as claimed in claim 6, characterized in that: In step S4, the first feature is represented as and the second feature representation The splicing is performed to obtain the spliced ​​feature representation, and the spliced ​​feature representation is input into the hybrid feature fusion module HFFM to obtain the fused feature F, which specifically includes: The first feature And the second feature Splicing is performed in the channel dimension; the channel shuffling strategy is used to evenly shuffle the spliced ​​feature maps; Use one The convolutional layer refines the shuffled feature map; The expression of feature fusion is: In the formula, Represents the splicing operation according to the channel dimension, Represents a channel shuffle operation, Indicates that the convolution size is The grouped convolution operation.

8. The hyperspectral image classification method fused with a hybrid multi-hop graph convolutional network as claimed in claim 7, characterized in that: The method further includes: using the cross entropy loss commonly used in classification tasks as a loss function to train the network, which is expressed as: In the formula, Represents pixels The true label of elements, when pixels Belongs to category hour, ,otherwise, ; Represents pixels Belongs to category The predicted probability of represents the number of training samples, Indicates the total number of categories.

9. The hyperspectral image classification method fused with a hybrid multi-hop graph convolutional network as claimed in claim 8, characterized in that: The loss function expression is: In the formula, represents the loss of the overall network, represents the loss of the MMGCN branch, represents the loss of the CFPN branch.

Images