A Hyperspectral Image Classification Method Based on Superpixel Semantic Guidance

Abstract

A hyperspectral image classification method based on superpixel semantic guidance, which belongs to the technical field of hyperspectral image classification. The present invention solves the problem of low accuracy of the existing hyperspectral image classification method based on GCN. The present invention uses the vision large model SAM to process the hyperspectral image, and divides the hyperspectral image into multiple homogeneous superpixels, which can significantly improve the accuracy of superpixel segmentation. Then, each superpixel is used as a node of the graph, and a hyperspectral image classification model including a three-branch graph convolutional network is constructed based on the graph learning method. The three-branch graph convolutional network is respectively used to extract the spatial features, spectral features and spatial-spectral features in the hyperspectral image, and the features of different branches are fused and then classified using AMSoftmax to achieve the purpose of improving the hyperspectral image classification accuracy. The method of the present invention can be applied to hyperspectral image classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral image classification, and particularly relates to a hyperspectral image classification method based on superpixel semantic guidance. Background Art

[0002] Hyperspectral images are an important branch of remote sensing imaging science, and the images obtained contain hundreds of continuous narrow spectral bands spanning along the electromagnetic spectrum (ES). Object information and its physical and chemical properties are identified through the discriminative spatial-spectral features collected simultaneously by hyperspectral imaging. The hyperspectral classification task is to assign each pixel in the image to a set of predefined classes / labels, and the same labels have common features. Hyperspectral image (HSI) classification has a variety of applications, such as precision agriculture, crop monitoring, land cover mapping, and land resource management. In terms of environmental protection, hyperspectral has been applied to detect gases, oil spills, water quality, and vegetation cover. The rich spatial-spectral information obtained from hyperspectral data promotes the rapid development of remote sensing technology in these fields.

[0003] Due to factors such as the high spectral dimension of hyperspectral remote sensing images, high correlation between bands, and mixed pixels, classifying hyperspectral remote sensing images poses a huge challenge. On the one hand, due to the high-dimensional characteristics, uncertainty, information redundancy of the signals, and the phenomena of same object with different spectra and different objects with the same spectrum in surface cover, the hyperspectral data structure is highly non-linear, making it difficult for some classification models based on statistical pattern recognition to directly classify and identify the original hyperspectral data. On the other hand, in the field of image supervised classification, the prior samples are limited and of uneven quality, and the parameters of the classifier model cannot be estimated or are estimated inaccurately. Compared with the number of high-dimensional features in hyperspectral remote sensing images, the training samples involved in supervised classification are often very limited. When the number of samples is insufficient, there will be the so-called Hughes phenomenon or Curse of dimensionality. Traditional classification methods using CNN have problems of overfitting caused by small samples and the inability to establish remote information connections. GCN can explicitly utilize the correlation between adjacent land covers, perform flexible convolution on arbitrarily irregular image regions, and can better model the spatial context structure of HSI. To reduce the computational complexity and promote the learning of the semantic structure of surface cover, GCN usually works on superpixel-based nodes rather than pixel-based nodes, and cannot capture pixel-level spectral-spatial features. Moreover, most of the existing superpixel segmentations adopt the method of simple linear iterative clustering (SLIC), which is mainly implemented by using KMeans for clustering. If the number of superpixels is set too large, it will lead to an increase in computational volume and sensitivity to noise. If the number of superpixels is set too small, it will lead to pixel misclassification. Therefore, the accuracy of the existing superpixel segmentation methods is still relatively limited. And since the existing GCN cannot capture pixel-level spectral-spatial features, the accuracy of the existing hyperspectral image classification methods based on GCN is still low. It is very necessary to propose a new method to solve the above problems. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem of low accuracy of the existing hyperspectral image classification method based on GCN, and a hyperspectral image classification method based on superpixel semantic guidance is proposed.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a hyperspectral image classification method based on superpixel semantic guidance, and the method specifically includes the following steps:

[0006] Step 1: Extract the data of the R, G, and B bands from the hyperspectral image, synthesize an RGB image using the data of the R, G, and B bands, use the RGB image as the input of the SAM model, and combine the output of the SAM model and the SLIC algorithm to obtain the final superpixel segmentation result of the RGB image;

[0007] Step 2: Calculate the initial features of each superpixel in the hyperspectral image based on the superpixel segmentation result in Step 1.

[0008] Step 3: Take each superpixel as a node, and construct a spatial adjacency matrix, a spectral adjacency matrix, and a spatial-spectral adjacency matrix according to the initial features of each superpixel node.

[0009] Step 4: Construct a hyperspectral image classification model, which includes an encoder, a CNN network, a decoder, a fully connected layer, and an AM-SoftMax layer.

[0010] The encoder includes a first GCN graph convolutional network, a second GCN graph convolutional network, and a third GCN graph convolutional network.

[0011] Use the initial features of each superpixel to form an initial feature matrix, and take the initial feature matrix and the spatial adjacency matrix as the input of the first GCN graph convolutional network.

[0012] Take the initial feature matrix and the spectral adjacency matrix as the input of the second GCN graph convolutional network.

[0013] Take the initial feature matrix and the spatial-spectral adjacency matrix as the input of the third GCN graph convolutional network.

[0014] Take the hyperspectral image as the input of the CNN network.

[0015] Concatenate the features output by the first GCN graph convolutional network, the second GCN graph convolutional network, and the third GCN graph convolutional network, and take the concatenation result as the input of the decoder.

[0016] Concatenate the output of the decoder and the output of the CNN network, take the concatenation result as the input of the fully connected layer, then take the output of the fully connected layer as the input of the AM-SoftMax layer, and output the classification result of each pixel point in the hyperspectral image through the AM-SoftMax layer.

[0017] Furthermore, in Step 1, take the RGB image as the input of the SAM model, and combine the output of the SAM model and the SLIC algorithm to obtain the final superpixel segmentation result of the RGB image; the specific process is as follows:

[0018] Step 1-1: Take the RGB image as the input of the SAM model, and output Z binary mask matrices through the SAM model. The number of rows of each binary mask matrix is the same as the number of rows of the RGB image, and the number of columns of each binary mask matrix is the same as the number of columns of the RGB image.

[0019] Each binary mask matrix corresponds to a superpixel respectively, that is, for any binary mask matrix, the pixel points corresponding to the elements with the value of 1 in the binary mask matrix in the RGB image form an initial superpixel, and a total of Z initial superpixels are obtained;

[0020] Step 2: After processing the Z initial superpixels obtained in Step 1, the final superpixel segmentation result is obtained.

[0021] Furthermore, the specific process of processing the obtained Z initial superpixels is as follows:

[0022] If a pixel point belongs to multiple initial superpixels at the same time, then the multiple initial superpixels containing this pixel point are sorted in descending order of the stability score, and this pixel point is assigned to the initial superpixel with the highest stability score in the sorting result;

[0023] If a pixel point does not belong to any initial superpixel, the SLIC algorithm is used to assign the initial superpixel to which this pixel point belongs.

[0024] Furthermore, the specific process of Step 2 is as follows:

[0025]

[0026] where s i is the initial feature of the i-th superpixel, i = 1, 2,..., Z, N i represents the number of pixels in the i-th superpixel, represents the original feature of the j-th pixel in the i-th superpixel.

[0027] Furthermore, the specific spatial adjacency matrix is as follows:

[0028]

[0029] where A s(ik) is the element in the i-th row and k-th column of the spatial adjacency matrix A s s k is the initial feature of the k-th superpixel node, ||·|| represents calculating the 2-norm, e is the base of the natural logarithm, and λ represents a hyperparameter.

[0030] Furthermore, the specific spectral adjacency matrix is as follows:

[0031]

[0032] where A f(ik) represents the element in the i-th row and k-th column of the spectral adjacency matrix A f N k (V i) represents the \(i\)-th superpixel node \(V\) i The set composed of the neighboring superpixel nodes of k is denoted as \(N\) k (V i ), and \(V\) i

[0033] Furthermore, the empty-spectrum adjacency matrix is specifically:

[0034]

[0035] where \(A\) c(ij) represents the element in the \(i\)-th row and \(k\)-th column of the empty-spectrum adjacency matrix \(A\) c .

[0036] Furthermore, the initial features of each superpixel are used to form an initial feature matrix, and the initial feature matrix and the spatial adjacency matrix are used as the inputs of the first GCN graph convolutional network. Among them, the working process of the first GCN graph convolutional network is as follows:

[0037] Step 1: The initial features of each superpixel form an initial feature matrix with a dimension of \(Z\times d\), where \(d\) is the initial feature dimension of each superpixel;

[0038] Step 2: The initial feature matrix and the spatial adjacency matrix are used as the inputs of the first GCN graph convolutional network. The output of the \(l\)-th layer of the first GCN graph convolutional network is:

[0039]

[0040] where represents the adjacency matrix after adding self-loops to the spatial adjacency matrix, \(I\) represents the identity matrix, \(W\) (l) represents the weight matrix of the \(l\)-th layer of the first GCN graph convolutional network, \(\sigma\) represents the activation function, is the degree matrix, and the \(i\)-th diagonal element in the degree matrix \(H\) (l) represents the feature matrix output by the \(l\)-th layer of the first GCN graph convolutional network, \(H\) (l-1) represents the feature matrix output by the \((l - 1)\)-th layer of the first GCN graph convolutional network, \(b\) (l) represents the bias output by the \(l\)-th layer of the first GCN graph convolutional network;

[0041] The output of the last layer in the first GCN graph convolutional network is the output of the first GCN graph convolutional network.

[0042] Furthermore, the working process of the decoder is:

[0043]

[0044] Among them, S represents the concatenation result of the features output by the first GCN graph convolutional network, the second GCN graph convolutional network, and the third GCN graph convolutional network, Q represents the correlation matrix between pixels and superpixels, the dimension of Q is HW*Z, and HW represents the number of pixels in the hyperspectral image. represents the feature vector transformed from superpixels to pixels;

[0045]

[0046] Among them, Q j,i represents the element in the j-th row and the i-th column of the matrix Q, and x j represents the j-th pixel in the hyperspectral image.

[0047] Furthermore, the working process of the CNN network is as follows:

[0048] Inside the CNN network, the input of the CNN network is used as the input of the first three-dimensional convolutional layer, and then the output of the first three-dimensional convolutional layer is used as the input of the first LeakyReLU layer;

[0049] The output of the first LeakyReLU layer is used as the input of the second three-dimensional convolutional layer, and then the output of the second three-dimensional convolutional layer is used as the input of the second LeakyReLU layer;

[0050] The output of the second LeakyReLU layer is used as the input of the third three-dimensional convolutional layer, and then the output of the third three-dimensional convolutional layer is used as the input of the third LeakyReLU layer;

[0051] The output of the third LeakyReLU layer is used as the input of the first Flatten layer, and then the output of the first Flatten layer is used as the input of the first CBAM module;

[0052] The output of the first CBAM module is used as the input of the first two-dimensional convolutional layer, and then the output of the first two-dimensional convolutional layer is used as the input of the fourth LeakyReLU layer;

[0053] The output of the fourth LeakyReLU layer is used as the input of the second Flatten layer, and then the output of the second Flatten layer is used as the input of the first Linear layer;

[0054] The output of the first Linear layer is used as the input of the fifth LeakyReLU layer, and then the output of the fifth LeakyReLU layer is used as the input of the first Dropout layer;

[0055] Use the output of the first Dropout layer as the input of the second Linear layer, and then use the output of the second Linear layer as the input of the sixth LeakyReLU layer;

[0056] Use the output of the sixth LeakyReLU layer as the input of the second Dropout layer, and then use the output of the second Dropout layer as the output of the CNN network.

[0057] The beneficial effects of the present invention are:

[0058] The present invention uses the large vision model SAM to process hyperspectral images, divides the hyperspectral images into multiple homogeneous superpixels, which can significantly improve the accuracy of superpixel segmentation. Then, each superpixel is used as a node of the graph, and a hyperspectral image classification model including a three-branch graph convolutional network is constructed based on the graph learning method. The three-branch graph convolutional network is respectively used to extract the spatial features, spectral features and spatio-spectral features in the hyperspectral images, fuse the features of different branches and then use AMSoftmax to classify them, so as to achieve the purpose of improving the classification accuracy of hyperspectral images.

[0059] Moreover, the superpixel segmentation method of the present invention can reduce the number of superpixels while achieving accurate segmentation, thereby improving the time efficiency. Description of the Drawings

[0060] Figure 1 is a flowchart of the superpixel segmentation method based on SAM and SLIC of the present invention;

[0061] In the figure, Grid Sampling represents grid sampling; Prompt Encoder represents prompt encoding; Multiple Binary Masks represents binary masks; Final Superpixel represents the final superpixel segmentation result;

[0062] Figure 2 is a flowchart of the conversion between superpixels and pixel features;

[0063] Figure 3 is a structural diagram of a three-branch graph convolutional network;

[0064] Figure 4 is a classification result graph of different methods on the Indian Pines dataset;

[0065] (a) represents the False-color image; (b) represents the classification result of the Ground truth; (c) represents the classification result of the SVM; (d) represents the classification result of the Random forest; (e) represents the classification result of the SSRN; (f) represents the classification result of the HybridSN; (g) represents the classification result of the CNCMN; (h) represents the classification result of the CEGCN; (i) represents the classification result of the AMGCFN; (j) represents the classification result of the method of the present invention;

[0066] Figure 5 is the classification result graph of different methods on the Salinas dataset;

[0067] (a) represents the False-color image; (b) represents the classification result of the Ground truth; (c) represents the classification result of the SVM; (d) represents the classification result of the Random forest; (e) represents the classification result of the SSRN; (f) represents the classification result of the HybridSN; (g) represents the classification result of the CNCMN; (h) represents the classification result of the CEGCN; (i) represents the classification result of the AMGCFN; (j) represents the classification result of the method of the present invention. Detailed implementation manner

[0068] Detailed implementation manner one: Combine Figure 3 to illustrate this implementation manner. A hyperspectral image classification method based on superpixel semantic guidance described in this implementation manner, the method is specifically as follows:

[0069] Step one: Extract the data of the R, G, and B bands from the hyperspectral image, and use the data of the R, G, and B bands to synthesize an RGB image, take the RGB image as the input of the SAM (SegmentAnything Model) model, and combine the output of the SAM model and the SLIC algorithm to obtain the final superpixel segmentation result of the RGB image;

[0070] Step two: According to the superpixel segmentation result of step one, calculate the initial features of each superpixel of the hyperspectral image respectively (that is, use the superpixel segmentation result of the RGB image as the superpixel segmentation result of the hyperspectral image);

[0071] Step three: Take each superpixel as a node, and construct a spatial adjacency matrix, a spectral adjacency matrix, and a spatial-spectral adjacency matrix according to the initial features of each superpixel node;

[0072] Step four: Construct a hyperspectral image classification model, and the hyperspectral image classification model includes an encoder, a CNN network, a decoder, a fully connected layer, and an AM-SoftMax layer;

[0073] The encoder includes a first GCN graph convolutional network, a second GCN graph convolutional network, and a third GCN graph convolutional network;

[0074] The initial feature matrix is composed of the initial features of each superpixel, and the initial feature matrix and the spatial adjacency matrix are used as the input of the first GCN graph convolutional network;

[0075] The initial feature matrix and the spectral adjacency matrix are used as the input of the second GCN graph convolutional network;

[0076] The initial feature matrix and the spatial-spectral adjacency matrix are used as the input of the third GCN graph convolutional network;

[0077] The hyperspectral image is used as the input of the CNN network (the CNN network is used to extract the pixel-level features of the hyperspectral image);

[0078] The features output by the first GCN graph convolutional network, the second GCN graph convolutional network, and the third GCN graph convolutional network are concatenated, and the concatenation result is used as the input of the decoder;

[0079] The output of the decoder and the output of the CNN network are concatenated, and the concatenation result is used as the input of the fully connected layer. Then, the output of the fully connected layer is used as the input of the AM-SoftMax layer, and the classification result of each pixel point in the hyperspectral image is output through the AM-SoftMax layer.

[0080] The AM-SoftMax layer adds a margin term to the original SoftMax layer loss function to increase the angular margin between different class features (the angular margin is calculated based on the cosine similarity between different class features). Adding the margin term can prompt the model to learn more discriminative features, thus obtaining better performance in the hyperspectral classification task.

[0081] The class (label) of a pixel can be regarded as a kind of semantic information. Since the superpixel segmentation itself is a clustering based on pixel information, pixels with similar information are segmented into one superpixel. Therefore, the hyperspectral image classification method based on superpixels in the present invention is regarded as a hyperspectral image classification method guided by superpixel semantics.

[0082] Specific Embodiment 2: Combined with Figure 1 This embodiment is described. The difference between this embodiment and Specific Embodiment 1 is that in Step 1, the RGB image is used as the input of the SAM model, and the final superpixel segmentation result of the RGB image is obtained by combining the output of the SAM model and the SLIC algorithm; the specific process is as follows:

[0083] Step 1: Use the RGB image as the input of the SAM model, and output Z binary mask matrices through the SAM model. The number of rows of each binary mask matrix is the same as that of the RGB image, and the number of columns of each binary mask matrix is the same as that of the RGB image;

[0084] Each binary mask matrix corresponds to a superpixel. That is, for any binary mask matrix, the pixel points corresponding to the elements with a value of 1 in the binary mask matrix in the RGB image form an initial superpixel, and a total of Z initial superpixels are obtained;

[0085] Step 2: After processing the Z initial superpixels obtained in Step 1, obtain the final superpixel segmentation result.

[0086] Other steps and parameters are the same as those in the first specific implementation manner.

[0087] The input RGB image is sampled in a grid in the Everything mode to generate 32 * 32 anchor points as the prompt encoding. The SAM model generates multiple binary prediction mask matrices according to the prompt encoding, and each binary mask matrix corresponds to information such as area, intersection over union score, and stability score.

[0088] Specific implementation manner three: The difference between this implementation manner and the first or second specific implementation manner is that the specific process of processing the obtained Z initial superpixels is as follows:

[0089] If a pixel point belongs to multiple initial superpixels at the same time (that is, belongs to two or more superpixels at the same time), then sort the multiple initial superpixels containing this pixel point in descending order according to the stability score, and assign this pixel point to the initial superpixel with the highest stability score in the sorting result;

[0090] If a pixel point does not belong to any initial superpixel, the SLIC algorithm is used to assign the initial superpixel to which this pixel point belongs.

[0091] Other steps and parameters are the same as those in the first or second specific implementation manner.

[0092] Due to problems with the SAM mechanism itself, a pixel may belong to multiple superpixels or not belong to any superpixel. Therefore, a superpixel matching mechanism is used to process each pixel. For pixels that belong to multiple superpixels, a sorting mechanism is used to select the superpixel with the highest stability score. For a pixel point a that does not belong to any superpixel, the result obtained by simple linear iterative clustering (SLIC) is used for matching. That is, for a pixel point a that does not belong to any superpixel, first, simple linear iterative clustering is used to segment the image to obtain all the pixels that belong to the same superpixel as pixel point a. Then, the distribution of these pixels in the initial superpixels output by the SAM model is statistically analyzed, and the number of occurrences of these pixels in each initial superpixel is obtained. The initial superpixel with the most occurrences of these pixels is found, and then pixel point a is assigned to this initial superpixel. If in the initial superpixel segmentation result, a pixel belongs to only one superpixel, then in the final superpixel segmentation result, this pixel belongs to this superpixel.

[0093] Specific Embodiment 4: The difference between this embodiment and one of Embodiments 1 to 3 is that the specific process of step 2 is as follows:

[0094]

[0095] where s i is the initial feature of the i-th superpixel, i = 1, 2, …, Z, and N i represents the number of pixels in the i-th superpixel, represents the original feature of the j-th pixel in the i-th superpixel (a vector composed of the values of the j-th pixel in each band of the hyperspectral image).

[0096] Other steps and parameters are the same as those in one of Embodiments 1 to 3.

[0097] The present invention uses the average feature value of all pixels within a superpixel as the feature of the superpixel, and performs subsequent calculations on the features of the superpixel, which can effectively reduce the computational complexity.

[0098] Specific Embodiment 5: The difference between this embodiment and one of Embodiments 1 to 4 is that the spatial adjacency matrix is specifically:

[0099]

[0100] where A s(ik) is the element in the i-th row and k-th column of the spatial adjacency matrix A s , and the dimension of the spatial adjacency matrix A s is Z×Z, and s kis the initial feature of the k-th superpixel node, ||·|| represents the calculation of the 2-norm, e is the base of the natural logarithm, and λ represents a hyperparameter (the specific value can be set according to the actual situation).

[0101] Other steps and parameters are the same as those in any one of the first to fourth specific embodiments.

[0102] In the present invention, if the pixel point a in the k-th superpixel node is adjacent to the pixel point b in the i-th superpixel node (adjacent in the horizontal, vertical, or diagonal direction), when there is at least one pair of such adjacent nodes between the k-th superpixel node and the i-th superpixel node, the k-th superpixel node is considered adjacent to the i-th superpixel node. More spatial information can be captured by constructing a spatial adjacency matrix.

[0103] Specific embodiment six: The difference between this embodiment and any one of the first to fifth specific embodiments is that the spectral adjacency matrix is specifically:

[0104]

[0105] where A f(ik) represents the element in the i-th row and k-th column of the spectral adjacency matrix A f , and the dimension of the spectral adjacency matrix A f is Z×Z, N k (V i ) represents the set composed of the neighboring superpixel nodes of the i-th superpixel node V i , and V k represents the k-th superpixel node in the set N k (V i ).

[0106] Other steps and parameters are the same as those in any one of the first to fifth specific embodiments.

[0107] The way to obtain the set N k (V i ) is as follows: For the i-th superpixel node V i , calculate the cosine similarity between the initial feature of the i-th superpixel node and the initial features of each other superpixel node respectively, then sort the calculated cosine similarities in descending order, and take the superpixel nodes corresponding to the top K (the specific value can be set according to the actual situation) cosine similarities as the neighboring superpixel nodes of the i-th superpixel node V i , and the K neighboring superpixel nodes of the i-th superpixel node V i form the set N k (V i ).

[0108] Specific Embodiment Seven: The difference between this embodiment and any one of Embodiments One to Six is that the empty spectrum adjacency matrix is specifically:

[0109]

[0110] where A c(ij) represents the element in the i-th row and k-th column of the empty spectrum adjacency matrix A c in the empty spectrum adjacency matrix A.

[0111] Other steps and parameters are the same as any one of Embodiments One to Six.

[0112] Specific Embodiment Eight: The difference between this embodiment and any one of Embodiments One to Seven is that the initial features of each superpixel are used to form an initial feature matrix, and the initial feature matrix and the spatial adjacency matrix are used as the inputs of the first GCN graph convolutional network. The working process of the first GCN graph convolutional network is as follows:

[0113] Step 1: The initial features of each superpixel form an initial feature matrix with a dimension of Z×d, where d is the initial feature dimension of each superpixel;

[0114] Step 2: The initial feature matrix and the spatial adjacency matrix are used as the inputs of the first GCN graph convolutional network. The output of the l-th layer of the first GCN graph convolutional network is:

[0115]

[0116] where represents the adjacency matrix after adding self-loops to the spatial adjacency matrix, I represents the identity matrix, W (l) represents the weight matrix of the l-th layer of the first GCN graph convolutional network, σ represents the activation function, is the degree matrix (the non-diagonal elements in the degree matrix are 0), and the i-th diagonal element in the degree matrix H (l) represents the feature matrix output by the l-th layer of the first GCN graph convolutional network, H (l-1) represents the feature matrix output by the (l-1)-th layer of the first GCN graph convolutional network, H (0) is the initial feature matrix composed of the initial features of each superpixel, and b (l) represents the bias output by the l-th layer of the first GCN graph convolutional network;

[0117] The output of the last layer in the first GCN graph convolutional network is the output of the first GCN graph convolutional network.

[0118] Other steps and parameters are the same as any one of Embodiments One to Seven.

[0119] The working principles of the second GCN graph convolutional network and the third GCN graph convolutional network are the same as those of the first GCN graph convolutional network. The three GCN graph convolutional networks are respectively used to extract spatial features, spectral features, and spatial-spectral common features in the hyperspectral image.

[0120] Specific Embodiment Nine: Combine Figure 2 to illustrate this embodiment. The difference between this embodiment and one of the first to eighth specific embodiments is that the working process of the decoder is as follows:

[0121]

[0122] Among them, S represents the concatenation result of the features output by the first GCN graph convolutional network, the second GCN graph convolutional network, and the third GCN graph convolutional network, Q represents the correlation matrix between pixels and superpixels, the dimension of Q is HW*Z, HW represents the number of pixels in the hyperspectral image, represents the feature vector converted from the superpixel to the pixel, and QS represents the multiplication of the matrix Q and S;

[0123]

[0124] Among them, Q j,i represents the element in the j-th row and i-th column of the matrix Q, and x j represents the j-th pixel in the hyperspectral image.

[0125] Other steps and parameters are the same as those in one of the first to eighth specific embodiments.

[0126] Specific Embodiment Ten: The difference between this embodiment and one of the first to ninth specific embodiments is that the working process of the CNN network is as follows:

[0127] Inside the CNN network, the input of the CNN network is used as the input of the first three-dimensional convolutional layer, and then the output of the first three-dimensional convolutional layer is used as the input of the first LeakyReLU layer;

[0128] The output of the first LeakyReLU layer is used as the input of the second three-dimensional convolutional layer, and then the output of the second three-dimensional convolutional layer is used as the input of the second LeakyReLU layer;

[0129] The output of the second LeakyReLU layer is used as the input of the third three-dimensional convolutional layer, and then the output of the third three-dimensional convolutional layer is used as the input of the third LeakyReLU layer;

[0130] The output of the third LeakyReLU layer is used as the input of the first Flatten layer, and then the output of the first Flatten layer is used as the input of the first CBAM module;

[0131] Use the output of the first CBAM (Convolutional Block Attention Module) module as the input of the first two-dimensional convolutional layer, and then use the output of the first two-dimensional convolutional layer as the input of the fourth LeakyReLU layer;

[0132] Use the output of the fourth LeakyReLU layer as the input of the second Flatten layer, and then use the output of the second Flatten layer as the input of the first Linear layer;

[0133] Use the output of the first Linear layer as the input of the fifth LeakyReLU layer, and then use the output of the fifth LeakyReLU layer as the input of the first Dropout layer;

[0134] Use the output of the first Dropout layer as the input of the second Linear layer, and then use the output of the second Linear layer as the input of the sixth LeakyReLU layer;

[0135] Use the output of the sixth LeakyReLU layer as the input of the second Dropout layer, and then use the output of the second Dropout layer as the output of the CNN network.

[0136] Other steps and parameters are the same as those in any one of the first to ninth specific embodiments.

[0137] Experimental part

[0138] To comprehensively evaluate the performance of the algorithm of the present invention, the algorithm of the present invention is compared and analyzed with traditional machine learning algorithms such as support vector machine (SVM) and random forest (RF), as well as classical deep learning algorithms for hyperspectral classification such as SSRN (Spectral–Spatial Residual Network for Hyperspectral Image Classification: A 3-D Deep Learning Framework) and HybridSN (Exploring 3D-2D CNN Feature Hierarchy for Hyperspectral Image Classification). In addition, the performance of three popular graph-based classification algorithms, namely CEGCN (CNN-Enhanced Graph Convolutional Network With Pixel-and Superpixel-Level Feature Fusion for Hyperspectral Image Classification), CNCMN (Composite Neighbor-Aware Convolutional Metric Networks for Hyperspectral Image Classification), and AMGCFN (Attention Multihop Graph and Multiscale Convolutional Fusion Network for Hyperspectral Image Classification), is also studied. This comprehensive evaluation framework ensures a comprehensive evaluation of the capabilities and effectiveness of the method of the present invention. Two publicly available datasets are used to evaluate the algorithm accuracy: 1) Indian Pines Dataset: The Indian Pines dataset is a hyperspectral remote sensing image collected by the Airborne Visible / Infrared Imaging Spectrometer (AVIRIS) in Indiana, USA in 1992. This dataset contains 220 spectral bands, with a spatial resolution of 20 m and an image size of 145×145 pixels. After removing 20 bands affected by water vapor absorption and low signal-to-noise ratio (104 - 108, 150 - 163, 220), a total of 200 bands are selected for analysis. This dataset includes 16 known land cover classes, with a total of 10,366 samples. Additionally, 1%, 1%, and 98% of the samples from each class are selected to form the training set, validation set, and test set, respectively.2) Salinas Dataset: The Salinas Dataset is a hyperspectral remote sensing image collected by the Airborne Visible / Infrared Imaging Spectrometer (AVIRIS) in the Salinas Valley, California, USA. This high-resolution image consists of 224 spectral bands, with a spatial resolution of 3.7 m and an image size of 512×217 pixels. After removing 20 bands affected by water vapor absorption, a total of 204 bands are used for classification research, representing 16 known land cover classes, with a total of 54,129 samples. 0.1%, 0.1%, and 99.8% of the samples are selected from each class to form the training set, validation set, and test set.

[0139] Table 1. Classification accuracies of single class, OA, AA, and Kpp on the Indian Pines dataset (1% training samples for all)

[0140]

[0141]

[0142] The classification results of all comparison methods on the Indian Pines dataset are shown in Table 1. The method we proposed achieved the highest accuracy compared with other methods. Compared with the sub-optimal CNCMN method, the overall accuracy increased by 2 percentage points, and our method was superior to the CNCMN method in terms of AA and kappa coefficient. In the training datasets of classes 1, 7, 9, and 16, our method achieved accuracies of 85.91%, 82.7%, 100%, and 100% respectively. From Figure 4 the classification result graph, it can be seen that the misclassification phenomenon of the method of the present invention is very few, and the pixel classification of the same plot is more continuous and smooth.

[0143] Table 2. Classification accuracies of single class, OA, AA, and Kpp on the Salinas dataset (0.1% training samples for all)

[0144]

[0145]

[0146]

[0147] As can be seen from Table 2, our method achieved the highest classification accuracy. Some superpixel-based GCN methods, namely CNCMN, CEGCN, and AMGCFN, achieved satisfactory performance. This is mainly attributed to the higher similarity between samples in the Salinas dataset and their spatial clustering, resulting in a purer internal region within superpixels. Therefore, the accuracies of CNCMN, CEGCN, and AMGCFN are 93.56%, 95.03%, and 93.9% respectively. Our method achieved an accuracy of 98.42%, significantly higher than the accuracies of other methods, demonstrating the effectiveness of the algorithm of the present invention. From Figure 5 it can be seen that the classification results of our algorithm are more accurate and there are few misclassification phenomena.

[0148] The above results demonstrate the effectiveness of the hyperspectral classification method based on superpixel semantic guidance of the present invention, thus leading to a significant improvement in classification performance. The superpixel segmentation results obtained using the SAM model are more reasonable, and there are fewer misclassified data points within superpixels. In terms of large object segmentation, the method proposed by us shows obvious advantages in terms of visual continuity and smoothness of object edges. Moreover, using the SAM-based superpixel segmentation method, due to precise segmentation, the number of superpixels is reduced, thereby improving the time efficiency and ultimately achieving a significantly better hyperspectral classification accuracy than other excellent methods.

[0149] The above examples of the present invention are only to illustrate in detail the calculation model and calculation process of the present invention, rather than to limit the implementation manner of the present invention. For those of ordinary skill in the art, other different forms of changes or variations can be made based on the above description. It is impossible to enumerate all the implementation manners here. Any obvious changes or variations derived from the technical solutions of the present invention still fall within the protection scope of the present invention.

Claims

1. A hyperspectral image classification method based on superpixel semantic guidance, characterized in that The method specifically includes the following steps: Step 1: Extract the data of the R, G, and B bands from the hyperspectral image, synthesize the RGB image using the data of the R, G, and B bands, use the RGB image as the input of the SAM model, and combine the output of the SAM model and the SLIC algorithm to obtain the final superpixel segmentation result of the RGB image; Step 2: Calculate the initial features of each superpixel of the hyperspectral image according to the superpixel segmentation result of Step 1; Step 3: Take each superpixel as a node, and construct a spatial adjacency matrix, a spectral adjacency matrix, and a spatial-spectral adjacency matrix according to the initial features of each superpixel node; Step 4: Construct a hyperspectral image classification model, which includes an encoder, a CNN network, a decoder, a fully connected layer, and an AM-SoftMax layer; The encoder includes a first GCN graph convolutional network, a second GCN graph convolutional network, and a third GCN graph convolutional network; Use the initial features of each superpixel to form an initial feature matrix, and use the initial feature matrix and the spatial adjacency matrix as the input of the first GCN graph convolutional network; Use the initial feature matrix and the spectral adjacency matrix as the input of the second GCN graph convolutional network; Use the initial feature matrix and the spatial-spectral adjacency matrix as the input of the third GCN graph convolutional network; Use the hyperspectral image as the input of the CNN network; Concatenate the features output by the first GCN graph convolutional network, the second GCN graph convolutional network, and the third GCN graph convolutional network, and use the concatenation result as the input of the decoder; Concatenate the output of the decoder and the output of the CNN network, use the concatenation result as the input of the fully connected layer, and then use the output of the fully connected layer as the input of the AM-SoftMax layer, and output the classification result of each pixel point in the hyperspectral image through the AM-SoftMax layer.

2. A hyperspectral image classification method based on superpixel semantic guidance according to claim 1, characterized in that In Step 1, use the RGB image as the input of the SAM model, and combine the output of the SAM model and the SLIC algorithm to obtain the final superpixel segmentation result of the RGB image; The specific process is as follows: Step 1-1: Use the RGB image as the input of the SAM model, and output Z binary mask matrices through the SAM model. The number of rows of each binary mask matrix is the same as the number of rows of the RGB image, and the number of columns of each binary mask matrix is the same as the number of columns of the RGB image; Each binary mask matrix corresponds to a superpixel, that is, for any binary mask matrix, the pixel points corresponding to the elements with a value of 1 in the binary mask matrix in the RGB image form an initial superpixel, and a total of Z initial superpixels are obtained; Step 1-2: After processing the Z initial superpixels obtained in Step 1-1, obtain the final superpixel segmentation result.

3. A hyperspectral image classification method based on superpixel semantic guidance according to claim 2, characterized in that, The specific process of processing the obtained Z initial superpixels is as follows: If a pixel point belongs to multiple initial superpixels at the same time, then sort the multiple initial superpixels containing the pixel point in descending order of the stable score, and assign the pixel point to the initial superpixel with the highest stable score in the sorting result; If a pixel does not belong to any initial superpixel, the SLIC algorithm is used to assign the initial superpixel to which the pixel belongs.

4. A hyperspectral image classification method based on superpixel semantic guidance according to claim 3, characterized in that, The specific process of the second step is as follows: where s i is the initial feature of the i-th superpixel, i = 1, 2, ..., Z, N i represents the number of pixels in the i-th superpixel, represents the original feature of the j-th pixel in the i-th superpixel.

5. A hyperspectral image classification method based on superpixel semantic guidance according to claim 4, characterized in that The specific spatial adjacency matrix is as follows: Among them, A s(ik) is the element in the i-th row and k-th column of the spatial adjacency matrix A s , s k is the initial feature of the k-th superpixel node, ||·|| represents the calculation of the 2-norm, e is the base of the natural logarithm, and λ represents the hyperparameter.

6. A hyperspectral image classification method based on superpixel semantic guidance according to claim 5, characterized in that The specific spectral adjacency matrix is as follows: Among them, A f(ik) represents the element at the i-th row and k-th column in the spectral adjacency matrix A f , N k (V i ) represents the set composed of the neighboring superpixel nodes of the i-th superpixel node V i , V k represents the k-th superpixel node in the set N k (V i ).

7. A hyperspectral image classification method based on superpixel semantic guidance according to claim 6, characterized in that The specific spatial-spectral adjacency matrix is as follows: Among them, A c(ik) represents the element in the i-th row and k-th column of the empty spectral adjacency matrix A c where i and k are the row and column indices of the matrix A respectively.

8. A hyperspectral image classification method based on superpixel semantic guidance according to claim 7, characterized in that Using the initial features of each superpixel to form an initial feature matrix, and taking the initial feature matrix and the spatial adjacency matrix as the inputs of the first GCN graph convolutional network. The working process of the first GCN graph convolutional network is as follows: Step 1: The initial features of each superpixel form an initial feature matrix with a dimension of Z×d, where d is the dimension of the initial features of each superpixel; Step 2: Taking the initial feature matrix and the spatial adjacency matrix as the inputs of the first GCN graph convolutional network, the output of the l-th layer of the first GCN graph convolutional network is: Among them, represents the adjacency matrix after adding self-loops to the spatial adjacency matrix, I represents the identity matrix, and W (l) represents the weight matrix of the l-th layer of the first GCN graph convolutional network, and σ represents the activation function, is the degree matrix, and the i-th diagonal element in the degree matrix H (l) represents the feature matrix output by the l-th layer of the first GCN graph convolutional network, and H (l-1) represents the feature matrix output by the (l - 1)-th layer of the first GCN graph convolutional network, and b (l) represents the bias output by the l-th layer of the first GCN graph convolutional network; The output of the last layer in the first GCN graph convolutional network is the output of the first GCN graph convolutional network.

9. A hyperspectral image classification method based on superpixel semantic guidance according to claim 8, characterized in that, The working process of the decoder is as follows: Among them, S represents the concatenation result of the features output by the first GCN graph convolutional network, the second GCN graph convolutional network, and the third GCN graph convolutional network, Q represents the correlation matrix between pixels and superpixels, the dimension of Q is HW*Z, and HW represents the number of pixels in the hyperspectral image. represents the feature vector converted from superpixels to pixels; where Q j，i represents the element in the \(j\)-th row and \(i\)-th column of matrix \(Q\), and \(x\) j represents the \(j\)-th pixel in the hyperspectral image.

10. A hyperspectral image classification method based on superpixel semantic guidance according to claim 9, characterized in that, The working process of the CNN network is as follows: Inside the CNN network, taking the input of the CNN network as the input of the first three-dimensional convolutional layer, and then taking the output of the first three-dimensional convolutional layer as the input of the first LeakyReLU layer; Taking the output of the first LeakyReLU layer as the input of the second three-dimensional convolutional layer, and then taking the output of the second three-dimensional convolutional layer as the input of the second LeakyReLU layer; Taking the output of the second LeakyReLU layer as the input of the third three-dimensional convolutional layer, and then taking the output of the third three-dimensional convolutional layer as the input of the third LeakyReLU layer; Taking the output of the third LeakyReLU layer as the input of the first Flatten layer, and then taking the output of the first Flatten layer as the input of the first CBAM module; Taking the output of the first CBAM module as the input of the first two-dimensional convolutional layer, and then taking the output of the first two-dimensional convolutional layer as the input of the fourth LeakyReLU layer; Taking the output of the fourth LeakyReLU layer as the input of the second Flatten layer, and then taking the output of the second Flatten layer as the input of the first Linear layer; Taking the output of the first Linear layer as the input of the fifth LeakyReLU layer, and then taking the output of the fifth LeakyReLU layer as the input of the first Dropout layer; Taking the output of the first Dropout layer as the input of the second Linear layer, and then taking the output of the second Linear layer as the input of the sixth LeakyReLU layer; Taking the output of the sixth LeakyReLU layer as the input of the second Dropout layer, and then taking the output of the second Dropout layer as the output of the CNN network.

Images