Hyperspectral image classification method based on superpixel semantic guidance

Abstract

The invention discloses a hyperspectral image classification method based on superpixel semantic guidance, and belongs to the technical field of hyperspectral image classification. According to the invention, the problem of low precision of the existing hyperspectral image classification method based on GCN is solved. According to the method, the hyperspectral image is processed by adopting the visual large model SAM, and the hyperspectral image is segmented into a plurality of homogeneous superpixels, so that the accuracy of superpixel segmentation can be remarkably improved. Taking each superpixel as a node of a graph, constructing a hyperspectral image classification model comprising a three-branch graph convolutional network based on a graph learning method, the three-branch graph convolutional network being respectively used for extracting spatial features, spectral features and spatial-spectral features in the hyperspectral image; and features of different branches are fused and then are classified by using AMSoftmax, so that the purpose of improving the classification precision of the hyperspectral image is achieved. The method can be applied to hyperspectral image classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral image classification, and in particular relates to a hyperspectral image classification method based on superpixel semantic guidance. Background Art

[0002] Hyperspectral imaging is an important branch of remote sensing imaging science. The images obtained contain hundreds of continuous narrow spectral bands along the electromagnetic spectrum (ES). The discriminative spatial-spectral features collected simultaneously by hyperspectral imaging are used to identify object information and their physical and chemical properties. The hyperspectral classification task is to assign each pixel in the image to a set of predefined categories / labels, and the same labels have common characteristics. Hyperspectral image (HSI) classification has a variety of applications, such as precision agriculture, crop monitoring, land cover mapping, and land resource management. In terms of environmental protection, hyperspectral has been used to detect gas, oil pollution, water quality, and vegetation cover. The rich spatial-spectral information obtained from hyperspectral data has promoted the rapid development of remote sensing technology in these fields.

[0003] Hyperspectral remote sensing image classification faces great challenges due to factors such as high spectral dimension, high correlation between bands and mixed pixels. On the one hand, due to the high-dimensional characteristics of the signal, uncertainty, information redundancy and the phenomenon of different spectra of the same object and different objects of the same spectrum in the surface cover, the structure of hyperspectral data is highly nonlinear, and some classification models based on statistical pattern recognition are difficult to directly classify and identify the original hyperspectral data. On the other hand, in the field of image supervised classification, the prior samples are limited and the quality is uneven, and the parameters of the classifier model cannot be estimated or estimated inaccurately. Compared with the number of high-dimensional features in hyperspectral remote sensing images, the training samples involved in supervised classification are often very limited. When the number of samples is insufficient, the so-called Hughes phenomenon or the curse of dimensionality will appear. The traditional classification method using CNN has the problem of overfitting caused by small samples and the problem of being unable to establish remote information connections. GCN can explicitly utilize the correlation between adjacent land covers and perform flexible convolution on any irregular image area, which can better model the spatial context structure of HSI. In order to reduce computational complexity and promote the learning of semantic structure of land cover, GCN usually works on superpixel-based nodes instead of pixel-based nodes, and cannot capture pixel-level spectral-spatial features. Moreover, most of the existing superpixel segmentation methods use the simple linear iterative clustering (SLIC) method, which is mainly implemented by clustering using KMeans. If the number of superpixels is set too large, the amount of computation will increase and it will be sensitive to noise. If the number of superpixels is set too small, it will lead to pixel misclassification. Therefore, the accuracy of the existing superpixel segmentation method is still relatively limited. Moreover, since the existing GCN cannot capture pixel-level spectral-spatial features, the accuracy of the existing hyperspectral image classification method based on GCN is still low. It is very necessary to propose a new method to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to solve the problem of low accuracy of the existing hyperspectral image classification method based on GCN, and propose a hyperspectral image classification method based on superpixel semantic guidance.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a hyperspectral image classification method based on superpixel semantic guidance, the method specifically comprises the following steps:

[0006] Step 1: Extract the data of the three bands R, G, and B from the hyperspectral image, and use the data of the three bands R, G, and B to synthesize the RGB image. The RGB image is used as the input of the SAM model, and the final superpixel segmentation result of the RGB image is obtained by combining the output of the SAM model with the SLIC algorithm.

[0007] Step 2: According to the superpixel segmentation result of step 1, the initial features of each superpixel of the hyperspectral image are calculated respectively;

[0008] Step 3: Take each superpixel as a node, and construct a spatial adjacency matrix, a spectral adjacency matrix, and a space-spectral adjacency matrix according to the initial features of each superpixel node;

[0009] Step 4: construct a hyperspectral image classification model, which includes an encoder, a CNN network, a decoder, a fully connected layer and an AM-SoftMax layer;

[0010] The encoder includes a first GCN graph convolutional network, a second GCN graph convolutional network and a third GCN graph convolutional network;

[0011] The initial features of each superpixel are used to form an initial feature matrix, and the initial feature matrix and the spatial adjacency matrix are used as the input of the first GCN graph convolutional network;

[0012] The initial feature matrix and spectral adjacency matrix are used as input to the second GCN graph convolutional network;

[0013] The initial feature matrix and the empty spectral adjacency matrix are used as input to the third GCN graph convolutional network;

[0014] Use the hyperspectral image as the input of the CNN network;

[0015] Concatenate the features output by the first GCN graph convolution network, the second GCN graph convolution network, and the third GCN graph convolution network, and use the concatenated result as the input of the decoder;

[0016] The output of the decoder and the output of the CNN network are spliced, and the splicing result is used as the input of the fully connected layer. The output of the fully connected layer is then used as the input of the AM-SoftMax layer, and the classification result of each pixel in the hyperspectral image is output through the AM-SoftMax layer.

[0017] Furthermore, in step 1, the RGB image is used as the input of the SAM model, and the output of the SAM model and the SLIC algorithm are combined to obtain the final superpixel segmentation result of the RGB image; the specific process is:

[0018] Step 1: Use the RGB image as the input of the SAM model, and output Z binary mask matrices through the SAM model, and the number of rows of each binary mask matrix is ​​the same as the number of rows of the RGB image, and the number of columns of each binary mask matrix is ​​the same as the number of columns of the RGB image;

[0019] Each binary mask matrix corresponds to a superpixel, that is, for any binary mask matrix, the pixel points corresponding to the elements with a value of 1 in the binary mask matrix in the RGB image constitute an initial superpixel, and a total of Z initial superpixels are obtained;

[0020] Step 12: After processing the Z initial superpixels obtained in step 11, the final superpixel segmentation result is obtained.

[0021] Furthermore, the specific process of processing the obtained Z initial superpixels is:

[0022] If a pixel belongs to multiple initial superpixels at the same time, the multiple initial superpixels containing the pixel are sorted in descending order of stability scores, and the pixel is assigned to the initial superpixel with the highest stability score in the sorted results;

[0023] If a pixel does not belong to any initial superpixel, the SLIC algorithm is used to assign the initial superpixel to the pixel.

[0024] Furthermore, the specific process of step 2 is as follows:

[0025]

[0026] Among them, s i is the initial feature of the i-th superpixel, i = 1, 2, ..., Z, N i represents the number of pixels in the i-th superpixel, represents the original feature of the jth pixel in the i-th superpixel.

[0027] Furthermore, the spatial adjacency matrix is ​​specifically:

[0028]

[0029] Among them, A s(ik) is the spatial adjacency matrix A s The element in row i and column k in s k is the initial feature of the kth superpixel node, ||·|| indicates the calculation of the 2-norm, e is the base of the natural logarithm, and λ represents the hyperparameter.

[0030] Furthermore, the spectral adjacency matrix is ​​specifically:

[0031]

[0032] Among them, A f(ik) Represents the spectral adjacency matrix A f The element in the i-th row and k-th column, N k (V i) represents the i-th superpixel node V i The set of neighboring superpixel nodes, V k Represents the set N k (V i ) is the kth superpixel node in .

[0033] Furthermore, the empty spectral adjacency matrix is ​​specifically:

[0034]

[0035] Among them, A c(ij) Represents the empty spectral adjacency matrix A c The element at row i and column k in .

[0036] Furthermore, the initial features of each superpixel are used to form an initial feature matrix, and the initial feature matrix and the spatial adjacency matrix are used as inputs of the first GCN graph convolutional network, wherein the working process of the first GCN graph convolutional network is:

[0037] Step 1: The initial features of each superpixel form an initial feature matrix with a dimension of Z×d, where d is the initial feature dimension of each superpixel;

[0038] Step 2: The initial feature matrix and the spatial adjacency matrix are used as the input of the first GCN graph convolutional network. The output of the first layer of the GCN graph convolutional network is:

[0039]

[0040] in, represents the adjacency matrix after adding self-rings to the spatial adjacency matrix, I represents the identity matrix, W (l) represents the weight matrix of the lth layer of the first GCN graph convolutional network, σ represents the activation function, is the degree matrix, the i-th diagonal element in the degree matrix H (l) H represents the feature matrix of the lth layer output of the first GCN graph convolutional network, (l-1) represents the feature matrix of the output of the l-1th layer of the first GCN graph convolutional network, b (l) Represents the bias of the lth layer output of the first GCN graph convolutional network;

[0041] The output of the last layer in the first GCN graph convolutional network is the output of the first GCN graph convolutional network.

[0042] Furthermore, the working process of the decoder is:

[0043]

[0044] Among them, S represents the concatenation result of the features output by the first GCN graph convolution network, the second GCN graph convolution network and the third GCN graph convolution network, Q represents the correlation matrix between pixels and superpixels, the dimension of Q is HW*Z, HW represents the number of pixels in the hyperspectral image, Represents the feature vector converted from superpixel to pixel;

[0045]

[0046] Among them, Q j,i represents the element in the jth row and ith column of the matrix Q, x j represents the jth pixel in the hyperspectral image.

[0047] Furthermore, the working process of the CNN network is:

[0048] In the CNN network, the input of the CNN network is used as the input of the first 3D convolutional layer, and the output of the first 3D convolutional layer is used as the input of the first LeakyReLU layer;

[0049] Use the output of the first LeakyReLU layer as the input of the second 3D convolutional layer, and then use the output of the second 3D convolutional layer as the input of the second LeakyReLU layer;

[0050] The output of the second LeakyReLU layer is used as the input of the third 3D convolutional layer, and the output of the third 3D convolutional layer is used as the input of the third LeakyReLU layer.

[0051] The output of the third LeakyReLU layer is used as the input of the first Flatten layer, and then the output of the first Flatten layer is used as the input of the first CBAM module;

[0052] The output of the first CBAM module is used as the input of the first two-dimensional convolutional layer, and then the output of the first two-dimensional convolutional layer is used as the input of the fourth LeakyReLU layer;

[0053] Use the output of the fourth LeakyReLU layer as the input of the second Flatten layer, and then use the output of the second Flatten layer as the input of the first Linear layer;

[0054] The output of the first Linear layer is used as the input of the fifth LeakyReLU layer, and the output of the fifth LeakyReLU layer is used as the input of the first Dropout layer;

[0055] The output of the first Dropout layer is used as the input of the second Linear layer, and the output of the second Linear layer is used as the input of the sixth LeakyReLU layer;

[0056] The output of the sixth LeakyReLU layer is used as the input of the second Dropout layer, and the output of the second Dropout layer is used as the output of the CNN network.

[0057] The beneficial effects of the present invention are:

[0058] The present invention uses a large visual model SAM to process hyperspectral images and divides the hyperspectral images into multiple homogeneous superpixels, which can significantly improve the accuracy of superpixel segmentation. Each superpixel is then used as a node in the graph, and a hyperspectral image classification model including a three-branch graph convolutional network is constructed based on the graph learning method. The three-branch graph convolutional network is used to extract spatial features, spectral features, and spatial-spectral features in the hyperspectral image, and the features of different branches are fused and then classified using AMSoftmax, so as to achieve the purpose of improving the classification accuracy of hyperspectral images.

[0059] Moreover, the superpixel segmentation method of the present invention can reduce the number of superpixels while achieving accurate segmentation, thereby improving time efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of a superpixel segmentation method based on SAM and SLIC of the present invention;

[0061] In the figure, Grid Sampling represents grid sampling; Prompt Encoder represents prompt encoding; Multiple Binary Masks represents binary mask; Final Superpixel represents the final superpixel segmentation result;

[0062] Figure 2 It is a flow chart of superpixel and pixel feature conversion;

[0063] Figure 3 It is a structural diagram of a three-branch graph convolutional network;

[0064] Figure 4 This is the classification result diagram of different methods on the Indian Pines dataset;

[0065] (a) represents False-color image; (b) represents the classification result of Ground truth; (c) represents the classification result of SVM; (d) represents the classification result of Random forest; (e) represents the classification result of SSRN; (f) represents the classification result of HybridSN; (g) represents the classification result of CNCMN; (h) represents the classification result of CEGCN; (i) represents the classification result of AMGCFN; (j) represents the classification result of the method of the present invention;

[0066] Figure 5 This is the classification result diagram of different methods on the Salinas dataset;

[0067] (a) represents False-color image; (b) represents the classification result of Ground truth; (c) represents the classification result of SVM; (d) represents the classification result of Random forest; (e) represents the classification result of SSRN; (f) represents the classification result of HybridSN; (g) represents the classification result of CNCMN; (h) represents the classification result of CEGCN; (i) represents the classification result of AMGCFN; (j) represents the classification result of the method of the present invention. DETAILED DESCRIPTION

[0068] Specific implementation method 1: Combination Figure 3 The present embodiment is described as follows. The present embodiment describes a hyperspectral image classification method based on superpixel semantic guidance, and the method is specifically as follows:

[0069] Step 1: Extract the data of the three bands R, G, and B from the hyperspectral image, and use the data of the three bands R, G, and B to synthesize the RGB image. The RGB image is used as the input of the SAM (SegmentAnything Model) model, and the final superpixel segmentation result of the RGB image is obtained by combining the output of the SAM model with the SLIC algorithm;

[0070] Step 2: According to the superpixel segmentation result of step 1, the initial features of each superpixel of the hyperspectral image are calculated respectively (i.e., the superpixel segmentation result of the RGB image is used as the superpixel segmentation result of the hyperspectral image);

[0071] Step 3: Take each superpixel as a node, and construct a spatial adjacency matrix, a spectral adjacency matrix, and a space-spectral adjacency matrix according to the initial features of each superpixel node;

[0072] Step 4: construct a hyperspectral image classification model, which includes an encoder, a CNN network, a decoder, a fully connected layer and an AM-SoftMax layer;

[0073] The encoder includes a first GCN graph convolutional network, a second GCN graph convolutional network and a third GCN graph convolutional network;

[0074] The initial features of each superpixel are used to form an initial feature matrix, and the initial feature matrix and the spatial adjacency matrix are used as the input of the first GCN graph convolutional network;

[0075] The initial feature matrix and spectral adjacency matrix are used as input to the second GCN graph convolutional network;

[0076] The initial feature matrix and the empty spectral adjacency matrix are used as input to the third GCN graph convolutional network;

[0077] The hyperspectral image is used as the input of the CNN network (the CNN network is used to extract pixel-level features of the hyperspectral image);

[0078] Concatenate the features output by the first GCN graph convolution network, the second GCN graph convolution network, and the third GCN graph convolution network, and use the concatenated result as the input of the decoder;

[0079] The output of the decoder and the output of the CNN network are spliced, and the splicing result is used as the input of the fully connected layer. The output of the fully connected layer is then used as the input of the AM-SoftMax layer, and the classification result of each pixel in the hyperspectral image is output through the AM-SoftMax layer.

[0080] The AM-SoftMax layer adds an edge term to the original SoftMax layer loss function to increase the angular interval between features of different categories (the angular interval is calculated based on the cosine similarity between features of different categories). Adding the edge term can prompt the model to learn more discriminative features, thereby achieving better performance on hyperspectral classification tasks.

[0081] The category (label) of a pixel can be regarded as a kind of semantic information. Since superpixel segmentation itself is clustering based on pixel information, pixels with similar information are segmented into a superpixel. Therefore, the present invention regards the hyperspectral image classification method based on superpixels as a hyperspectral image classification method based on superpixel semantic guidance.

[0082] Specific implementation method 2: Combination Figure 1 This embodiment is different from the first embodiment in that, in step 1, the RGB image is used as the input of the SAM model, and the final superpixel segmentation result of the RGB image is obtained by combining the output of the SAM model and the SLIC algorithm; the specific process is as follows:

[0083] Step 1: Use the RGB image as the input of the SAM model, and output Z binary mask matrices through the SAM model, and the number of rows of each binary mask matrix is ​​the same as the number of rows of the RGB image, and the number of columns of each binary mask matrix is ​​the same as the number of columns of the RGB image;

[0084] Each binary mask matrix corresponds to a superpixel, that is, for any binary mask matrix, the pixel points corresponding to the elements with a value of 1 in the binary mask matrix in the RGB image constitute an initial superpixel, and a total of Z initial superpixels are obtained;

[0085] Step 12: After processing the Z initial superpixels obtained in step 11, the final superpixel segmentation result is obtained.

[0086] The other steps and parameters are the same as those in the first embodiment.

[0087] The input RGB image is grid sampled in Everything mode to generate 32*32 anchor points as prompt codes. The SAM model generates multiple binary prediction mask matrices based on the prompt codes, and each binary mask matrix corresponds to information such as area, intersection-over-union score, and stability score.

[0088] Specific implementation method three: This implementation method is different from specific implementation method one or two in that the specific process of processing the obtained Z initial superpixels is as follows:

[0089] If a pixel belongs to multiple initial superpixels at the same time (i.e., it belongs to two or more superpixels at the same time), the multiple initial superpixels containing the pixel are sorted in descending order of stability scores, and the pixel is assigned to the initial superpixel with the highest stability score in the sorted results;

[0090] If a pixel does not belong to any initial superpixel, the SLIC algorithm is used to assign the initial superpixel to the pixel.

[0091] The other steps and parameters are the same as those in the first or second embodiment.

[0092] Because of the problem of SAM mechanism itself, a pixel may belong to multiple superpixels or not belong to any superpixel. Therefore, the superpixel matching mechanism is used to process each pixel. For the case of belonging to multiple superpixels, the sorting mechanism is used to select the superpixel with the highest stability score; for pixel a that does not belong to any superpixel, the result obtained by simple linear iterative clustering (SLIC) is matched, that is, for a pixel a that does not belong to any superpixel, the image is first segmented by simple linear iterative clustering to obtain all the pixels belonging to the same superpixel as pixel a, and the distribution of these pixels in the initial superpixels output by the SAM model is counted, and the number of these pixels appearing in each initial superpixel is obtained, and the initial superpixel with the largest number of these pixels is obtained, and then the pixel a is attributed to this initial superpixel. If a pixel belongs to only one superpixel in the initial superpixel segmentation result, then in the final superpixel segmentation result, the pixel belongs to this superpixel.

[0093] Specific implementation method 4: This implementation method is different from the specific implementation methods 1 to 3 in that the specific process of step 2 is as follows:

[0094]

[0095] Among them, s i is the initial feature of the i-th superpixel, i = 1, 2, ..., Z, N i represents the number of pixels in the i-th superpixel, Represents the original feature of the jth pixel in the i-th superpixel (a vector composed of the values ​​of the jth pixel in each band in the hyperspectral image).

[0096] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0097] The present invention uses the average characteristic value of all pixels in a superpixel as the feature of the superpixel, and performs subsequent calculations on the features of the superpixel, which can effectively reduce the computational complexity.

[0098] Specific implementation method 5: This implementation method is different from the specific implementation methods 1 to 4 in that the spatial adjacency matrix is ​​specifically:

[0099]

[0100] Among them, A s(ik) is the spatial adjacency matrix A s The element in the i-th row and k-th column of the spatial adjacency matrix A s The dimension is Z×Z, s kis the initial feature of the kth superpixel node, ||·|| indicates the calculation of the 2-norm, e is the base of the natural logarithm, and λ indicates a hyperparameter (the specific value can be set according to the actual situation).

[0101] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0102] In the present invention, if pixel a in the kth superpixel node is adjacent to pixel b in the ith superpixel node (adjacent in the horizontal direction, vertical direction or diagonal direction), when there is at least one pair of such adjacent nodes between the kth superpixel node and the ith superpixel node, the kth superpixel node is considered to be adjacent to the ith superpixel node. More spatial information can be captured by constructing a spatial adjacency matrix.

[0103] Specific implementation method 6: This implementation method is different from any one of the specific implementation methods 1 to 5 in that the spectral adjacency matrix is ​​specifically:

[0104]

[0105] Among them, A f(ik) Represents the spectral adjacency matrix A f The element in the i-th row and k-th column of the spectral adjacency matrix A f The dimension is Z×Z, N k (V i ) represents the i-th superpixel node V i The set of neighboring superpixel nodes, V k Represents the set N k (V i ) is the kth superpixel node in .

[0106] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0107] Set N k (V i ) is obtained as follows: for the i-th superpixel node V i , respectively calculate the cosine similarity between the initial feature of the i-th superpixel node and the initial feature of each other superpixel node, and then sort the calculated cosine similarities in descending order, and take the superpixel nodes corresponding to the cosine similarities in the first K (the specific value can be set according to the actual situation) as the i-th superpixel node V i The neighboring superpixel node, the i-th superpixel node V i The K nearest superpixel nodes form a set N k (V i ).

[0108] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the space-spectrum adjacency matrix is ​​specifically:

[0109]

[0110] Among them, A c(ij) Represents the empty spectral adjacency matrix A c The element at row i and column k in .

[0111] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0112] Specific implementation eight: This implementation is different from any one of specific implementations one to seven in that the initial feature matrix is ​​formed by using the initial features of each superpixel, and the initial feature matrix and the spatial adjacency matrix are used as inputs of the first GCN graph convolutional network, wherein the working process of the first GCN graph convolutional network is:

[0113] Step 1: The initial features of each superpixel form an initial feature matrix with a dimension of Z×d, where d is the initial feature dimension of each superpixel;

[0114] Step 2: The initial feature matrix and the spatial adjacency matrix are used as the input of the first GCN graph convolutional network. The output of the first layer of the GCN graph convolutional network is:

[0115]

[0116] in, represents the adjacency matrix after adding self-rings to the spatial adjacency matrix, I represents the identity matrix, W (l) represents the weight matrix of the lth layer of the first GCN graph convolutional network, σ represents the activation function, is the degree matrix (the off-diagonal elements in the degree matrix are 0), the i-th diagonal element in the degree matrix H (l) H represents the feature matrix of the lth layer output of the first GCN graph convolutional network, (l-1) H represents the feature matrix of the output of the l-1th layer of the first GCN graph convolutional network, (0) is the initial feature matrix composed of the initial features of each superpixel, b (l) Represents the bias of the lth layer output of the first GCN graph convolutional network;

[0117] The output of the last layer in the first GCN graph convolutional network is the output of the first GCN graph convolutional network.

[0118] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0119] The second GCN graph convolution network and the third GCN graph convolution network work in the same way as the first GCN graph convolution network. The three GCN graph convolution networks are used to extract spatial features, spectral features, and spatial-spectral common features in hyperspectral images, respectively.

[0120] Specific implementation method nine: Combination Figure 2 This embodiment is different from the first to eighth embodiments in that the working process of the decoder is as follows:

[0121]

[0122] Among them, S represents the concatenation result of the features output by the first GCN graph convolution network, the second GCN graph convolution network and the third GCN graph convolution network, Q represents the correlation matrix between pixels and superpixels, the dimension of Q is HW*Z, HW represents the number of pixels in the hyperspectral image, represents the feature vector converted from superpixel to pixel, QS represents the multiplication of matrix Q and S;

[0123]

[0124] Among them, Q j,i represents the element in the jth row and ith column of the matrix Q, x j represents the jth pixel in the hyperspectral image.

[0125] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0126] Specific implementation method 10: This implementation method is different from the specific implementation methods 1 to 9 in that the working process of the CNN network is as follows:

[0127] In the CNN network, the input of the CNN network is used as the input of the first 3D convolutional layer, and the output of the first 3D convolutional layer is used as the input of the first LeakyReLU layer;

[0128] Use the output of the first LeakyReLU layer as the input of the second 3D convolutional layer, and then use the output of the second 3D convolutional layer as the input of the second LeakyReLU layer;

[0129] The output of the second LeakyReLU layer is used as the input of the third 3D convolutional layer, and the output of the third 3D convolutional layer is used as the input of the third LeakyReLU layer.

[0130] The output of the third LeakyReLU layer is used as the input of the first Flatten layer, and then the output of the first Flatten layer is used as the input of the first CBAM module;

[0131] The output of the first CBAM (Convolutional Block Attention Module) module is used as the input of the first two-dimensional convolutional layer, and then the output of the first two-dimensional convolutional layer is used as the input of the fourth LeakyReLU layer;

[0132] Use the output of the fourth LeakyReLU layer as the input of the second Flatten layer, and then use the output of the second Flatten layer as the input of the first Linear layer;

[0133] The output of the first Linear layer is used as the input of the fifth LeakyReLU layer, and the output of the fifth LeakyReLU layer is used as the input of the first Dropout layer;

[0134] The output of the first Dropout layer is used as the input of the second Linear layer, and the output of the second Linear layer is used as the input of the sixth LeakyReLU layer;

[0135] The output of the sixth LeakyReLU layer is used as the input of the second Dropout layer, and the output of the second Dropout layer is used as the output of the CNN network.

[0136] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0137] Experimental Section

[0138] In order to comprehensively evaluate the performance of the algorithm of the present invention, the algorithm of the present invention is compared with traditional machine learning algorithms such as support vector machine (SVM) and random forest (RF), as well as classic hyperspectral classification deep learning algorithms such as SSRN (Spectral–Spatial Residual Network for Hyperspectral Image Classification: A3-D Deep Learning Framework) and HybridSN (Exploring 3D-2D CNN Feature Hierarchy for Hyperspectral Image Classification). In addition, the performance of three popular graph-based classification algorithms, CEGCN (CNN-Enhanced Graph Convolutional Network With Pixel-and Superpixel-Level Feature Fusion for Hyperspectral Image Classification), CNCMN (Composite Neighbor-Aware Convolutional Metric Networks for Hyperspectral Image Classification) and AMGCFN (Attention Multihop Graph and Multiscale Convolutional Fusion Network for Hyperspectral Image Classification), is also studied. This comprehensive evaluation framework ensures a comprehensive evaluation of the capabilities and effectiveness of the method of the present invention. Two public data are used to evaluate the accuracy of the algorithm. 1) Indian Pines Dataset: Indian Pines dataset is a hyperspectral remote sensing image collected by the Airborne Visible / Infrared Imaging Spectrometer (AVIRIS) in Indiana, USA in 1992. The dataset contains 220 spectral bands, with a spatial resolution of 20m and an image size of 145×145 pixels. After removing 20 bands affected by water vapor absorption and low signal-to-noise ratio (104-108,150-163,220), a total of 200 bands were selected for analysis. The dataset includes 16 known land cover classes and a total of 10,366 samples. In addition, 1%, 1%, and 98% of the samples were selected from each class to form the training set, validation set, and test set.2) Salinas Dataset: Salinas Dataset is a hyperspectral remote sensing image collected by the Airborne Visible / Infrared Imaging Spectrometer (AVIRIS) in the Salinas Valley, California, USA. The high-resolution image consists of 224 spectral bands, with a spatial resolution of 3.7m and an image size of 512×217 pixels. After excluding 20 bands affected by water vapor absorption, a total of 204 bands were used for classification research, representing 16 known land cover categories, with a total of 54,129 samples. 0.1%, 0.1%, and 99.8% of the samples were selected from each class to form the training set, validation set, and test set.

[0139] Table 1. Classification accuracy of single class, OA, AA and Kpp on Indian pine dataset (all with 1% training samples)

[0140]

[0141]

[0142] The classification results of all the compared methods on the Indian Pines dataset are shown in Table 1. Our proposed method achieves the highest accuracy compared with other methods. The overall accuracy is improved by 2 percentage points compared with the second-best CNCMN method, and it outperforms the CNCMN method in terms of AA and kappa coefficient. In the training datasets of categories 1, 7, 9, and 16, our method achieves 85.91%, 82.7%, 100%, and 100% accuracy, respectively. Figure 4 It can be seen from the classification result diagram that the method of the present invention has very few misclassification phenomena, and the pixel classification of the same plot is more continuous and smooth.

[0143] Table 2. Classification accuracy of single class, OA, AA and Kpp on the Salinas dataset (all with 0.1% training samples)

[0144]

[0145]

[0146]

[0147] It can be seen from Table 2 that our method achieved the highest classification accuracy. Some superpixel-based GCN methods, namely CNCMN, CEGCN and AMGCFN, achieved satisfactory performance. This is mainly attributed to the higher similarity between samples of the Salinas dataset and their spatial clustering, which leads to purer areas inside superpixels. Therefore, the accuracies of CNCMN, CEGCN and AMGCFN are 93.56%, 95.03% and 93.9%, respectively. Our method achieved an accuracy of 98.42%, which is significantly higher than the accuracy of other methods, proving the effectiveness of the algorithm of the present invention. Figure 5 It can be seen that the classification results of our algorithm are more accurate and there are few misclassifications.

[0148] The above results demonstrate the effectiveness of the hyperspectral classification method based on superpixel semantic guidance of the present invention, thereby significantly improving the classification performance. The superpixel segmentation results obtained using the SAM model are more reasonable, and there are fewer misclassified data points within the superpixel. In terms of large object segmentation, our proposed method shows obvious advantages in visual continuity and smoothness of object edges. Moreover, the use of the SAM-based superpixel segmentation method reduces the number of superpixels due to precise segmentation, thereby improving time efficiency, and ultimately achieving hyperspectral classification accuracy that is significantly better than other excellent methods.

[0149] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A hyperspectral image classification method based on superpixel semantic guidance, characterized in that: The method specifically comprises the following steps: Step 1: Extract the data of the three bands R, G, and B from the hyperspectral image, and use the data of the three bands R, G, and B to synthesize the RGB image. The RGB image is used as the input of the SAM model, and the final superpixel segmentation result of the RGB image is obtained by combining the output of the SAM model with the SLIC algorithm. Step 2: According to the superpixel segmentation result of step 1, the initial features of each superpixel of the hyperspectral image are calculated respectively; Step 3: Take each superpixel as a node, and construct a spatial adjacency matrix, a spectral adjacency matrix, and a space-spectral adjacency matrix according to the initial features of each superpixel node; Step 4: construct a hyperspectral image classification model, which includes an encoder, a CNN network, a decoder, a fully connected layer and an AM-SoftMax layer; The encoder includes a first GCN graph convolutional network, a second GCN graph convolutional network and a third GCN graph convolutional network; The initial features of each superpixel are used to form an initial feature matrix, and the initial feature matrix and the spatial adjacency matrix are used as the input of the first GCN graph convolutional network; The initial feature matrix and spectral adjacency matrix are used as input to the second GCN graph convolutional network; The initial feature matrix and the empty spectral adjacency matrix are used as input to the third GCN graph convolutional network; Use the hyperspectral image as the input of the CNN network; Concatenate the features output by the first GCN graph convolution network, the second GCN graph convolution network, and the third GCN graph convolution network, and use the concatenated result as the input of the decoder; The output of the decoder and the output of the CNN network are spliced, and the splicing result is used as the input of the fully connected layer. The output of the fully connected layer is then used as the input of the AM-SoftMax layer, and the classification result of each pixel in the hyperspectral image is output through the AM-SoftMax layer.

2. The hyperspectral image classification method based on superpixel semantic guidance according to claim 1 is characterized in that: In the step 1, the RGB image is used as the input of the SAM model, and the output of the SAM model is combined with the SLIC algorithm to obtain the final superpixel segmentation result of the RGB image; The specific process is: Step 1: Use the RGB image as the input of the SAM model, and output Z binary mask matrices through the SAM model, and the number of rows of each binary mask matrix is ​​the same as the number of rows of the RGB image, and the number of columns of each binary mask matrix is ​​the same as the number of columns of the RGB image; Each binary mask matrix corresponds to a superpixel, that is, for any binary mask matrix, the pixel points corresponding to the elements with a value of 1 in the binary mask matrix in the RGB image constitute an initial superpixel, and a total of Z initial superpixels are obtained; Step 12: After processing the Z initial superpixels obtained in step 11, the final superpixel segmentation result is obtained.

3. The hyperspectral image classification method based on superpixel semantic guidance according to claim 2 is characterized in that: The specific process of processing the obtained Z initial superpixels is as follows: If a pixel belongs to multiple initial superpixels at the same time, the multiple initial superpixels containing the pixel are sorted in descending order of stability scores, and the pixel is assigned to the initial superpixel with the highest stability score in the sorted results; If a pixel does not belong to any initial superpixel, the SLIC algorithm is used to assign the initial superpixel to the pixel.

4. The hyperspectral image classification method based on superpixel semantic guidance according to claim 3 is characterized in that: The specific process of step 2 is as follows: Among them, s i is the initial feature of the i-th superpixel, i = 1, 2, ..., Z, N i represents the number of pixels in the i-th superpixel, represents the original feature of the jth pixel in the i-th superpixel.

5. The hyperspectral image classification method based on superpixel semantic guidance according to claim 4 is characterized in that: The spatial adjacency matrix is ​​specifically: Among them, A s(ik) is the spatial adjacency matrix A s The element in row i and column k in s k is the initial feature of the kth superpixel node, ||·|| indicates the calculation of the 2-norm, e is the base of the natural logarithm, and λ represents the hyperparameter.

6. The hyperspectral image classification method based on superpixel semantic guidance according to claim 5 is characterized in that: The spectral adjacency matrix is ​​specifically: Among them, A f(ik) Represents the spectral adjacency matrix A f The element in the i-th row and k-th column, N k (V i ) represents the i-th superpixel node V i The set of neighboring superpixel nodes, V k Represents the set N k (V i ) is the kth superpixel node in .

7. The hyperspectral image classification method based on superpixel semantic guidance according to claim 6 is characterized in that: The empty spectral adjacency matrix is ​​specifically: Among them, A c(ik) Represents the empty spectral adjacency matrix A c The element at row i and column k in .

8. The hyperspectral image classification method based on superpixel semantic guidance according to claim 7 is characterized in that: The initial feature matrix is ​​formed by using the initial features of each superpixel, and the initial feature matrix and the spatial adjacency matrix are used as inputs of the first GCN graph convolutional network, wherein the working process of the first GCN graph convolutional network is: Step 1: The initial features of each superpixel form an initial feature matrix with a dimension of Z×d, where d is the initial feature dimension of each superpixel; Step 2: The initial feature matrix and the spatial adjacency matrix are used as the input of the first GCN graph convolutional network. The output of the first layer of the GCN graph convolutional network is: in, represents the adjacency matrix after adding self-rings to the spatial adjacency matrix, I represents the identity matrix, W (l) represents the weight matrix of the lth layer of the first GCN graph convolutional network, σ represents the activation function, is the degree matrix, the i-th diagonal element in the degree matrix H (l) H represents the feature matrix of the lth layer output of the first GCN graph convolutional network, (l-1) represents the feature matrix of the l-1th layer output of the first GCN graph convolutional network, b (l) Represents the bias of the lth layer output of the first GCN graph convolutional network; The output of the last layer in the first GCN graph convolutional network is the output of the first GCN graph convolutional network.

9. The hyperspectral image classification method based on superpixel semantic guidance according to claim 8, characterized in that: The working process of the decoder is: Among them, S represents the concatenation result of the features output by the first GCN graph convolution network, the second GCN graph convolution network and the third GCN graph convolution network, Q represents the correlation matrix between pixels and superpixels, the dimension of Q is HW*Z, HW represents the number of pixels in the hyperspectral image, Represents the feature vector converted from superpixel to pixel; Among them, Q j，i represents the element in the jth row and ith column of the matrix Q, x j represents the jth pixel in the hyperspectral image.

10. The hyperspectral image classification method based on superpixel semantic guidance according to claim 9, characterized in that: The working process of the CNN network is: In the CNN network, the input of the CNN network is used as the input of the first 3D convolutional layer, and the output of the first 3D convolutional layer is used as the input of the first LeakyReLU layer; Use the output of the first LeakyReLU layer as the input of the second 3D convolutional layer, and then use the output of the second 3D convolutional layer as the input of the second LeakyReLU layer; The output of the second LeakyReLU layer is used as the input of the third 3D convolutional layer, and the output of the third 3D convolutional layer is used as the input of the third LeakyReLU layer. The output of the third LeakyReLU layer is used as the input of the first Flatten layer, and then the output of the first Flatten layer is used as the input of the first CBAM module; The output of the first CBAM module is used as the input of the first two-dimensional convolutional layer, and then the output of the first two-dimensional convolutional layer is used as the input of the fourth LeakyReLU layer; Use the output of the fourth LeakyReLU layer as the input of the second Flatten layer, and then use the output of the second Flatten layer as the input of the first Linear layer; The output of the first Linear layer is used as the input of the fifth LeakyReLU layer, and the output of the fifth LeakyReLU layer is used as the input of the first Dropout layer; The output of the first Dropout layer is used as the input of the second Linear layer, and the output of the second Linear layer is used as the input of the sixth LeakyReLU layer; The output of the sixth LeakyReLU layer is used as the input of the second Dropout layer, and the output of the second Dropout layer is used as the output of the CNN network.

Images