Hyperspectral image classification method and system based on dynamic hypergraph convolutional network

Abstract

The present application provides a hyperspectral image classification method and system based on a dynamic hypergraph convolutional network, which belongs to the field of deep learning and image processing technology. The present application obtains a hyperspectral image, performs dimensionality reduction and superpixel segmentation on the hyperspectral image, and realizes dynamic feature extraction through two hypergraph constructions. Since the entire hyperspectral image is used as input, the superpixel segmentation technology is used to divide the original pixels into superpixel points, thereby greatly reducing the amount of calculation. In addition, the present application can determine the feature labels of some unknown original pixel points through the feature labels of superpixel points, which is equivalent to implicitly increasing the number of samples. This enables the model of the present application to achieve higher classification accuracy with a small number of training samples.

Description

Technical Field

[0001] The present application relates to the field of deep learning and image processing technology, and in particular to a hyperspectral image classification method and system based on a dynamic hypergraph convolutional network. Background Art

[0002] As the technology of hyperspectral real-time classification gradually matures, its application areas have gradually expanded from aerospace telemetry, agriculture, food safety, medical diagnosis and other fields to product quality inspection and classification in the industrial field.

[0003] Hyperspectral remote sensing is an imaging spectral remote sensing technology that organically combines imaging and subdivided spectral techniques. Hyperspectral images are continuous, hundreds-channel images obtained by a hyperspectral remote sensing system by acquiring and recording solar radiation signals that carry ground feature information. Hyperspectral images contain rich spectral and spatial information. Because different components absorb the spectrum to varying degrees, the image at a specific wavelength will more significantly reflect a particular defect. This allows hyperspectral images to not only reflect external quality characteristics of a sample, such as size, shape, and defects, but also fully reflect differences in the sample's internal physical structure and chemical composition.

[0004] Hyperspectral images captured by hyperspectral sensors, or imaging spectrometers, deployed on various spatial platforms are characterized by unified image and spectrum information and rich spectral information. Unlike conventional and multispectral images, hyperspectral images possess hundreds of spectral features. The high dimensionality, high inter-band correlation, and spectral mixing of hyperspectral images pose significant challenges to their classification.

[0005] Most existing hyperspectral image classification methods are based on convolutional neural networks, which typically require a sufficient number of labeled samples for training, making hyperspectral image classification very difficult. Furthermore, most convolutional neural network architectures feed images into the network in batches, which prevents the network from capturing relationships between pixels over long distances. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a hyperspectral image classification method and system based on a dynamic hypergraph convolutional network, thereby overcoming one or more problems caused by the limitations and defects of the relevant technology at least to a certain extent.

[0007] In a first aspect, the present application provides a hyperspectral image classification method based on a dynamic hypergraph convolutional network, comprising:

[0008] Acquiring a hyperspectral image and performing image preprocessing on the hyperspectral image;

[0009] Performing superpixel segmentation on the hyperspectral image after image preprocessing to obtain a plurality of superpixel points, each superpixel point corresponding to a plurality of original pixel points;

[0010] constructing a supergraph according to the first spectral characteristics of the superpixel points to obtain a first supergraph;

[0011] Performing a hypergraph convolution process on the first hypergraph to obtain a second spectral feature of the superpixel point;

[0012] reconstructing a hypergraph according to the second spectral characteristics of the superpixel points to obtain a second hypergraph;

[0013] Performing dynamic hypergraph convolution processing on the second hypergraph to obtain a third spectral feature of each superpixel point;

[0014] Converting the spectral characteristics of the superpixel into spectral characteristics of the corresponding original pixel according to the correspondence between the superpixel and the original pixel;

[0015] The original pixels are classified based on the spectral characteristics of the original pixels.

[0016] Optionally, the step of performing superpixel segmentation on the hyperspectral image after image preprocessing to obtain a plurality of superpixel points includes:

[0017] Simple linear iterative clustering is used to cluster and group the original pixels in the hyperspectral image after image preprocessing to generate a number of superpixel points.

[0018] Optionally, the step of constructing a hypergraph according to the first spectral feature of the superpixel point to obtain a first hypergraph specifically includes:

[0019] Calculate and obtain a first spectral feature of each superpixel point based on the spectral features of the original pixel points of the hyperspectral image after image preprocessing, wherein the first spectral feature of each superpixel point is the average value of the spectral features of all original pixel points contained in the superpixel point;

[0020] A first hyperedge is constructed based on the similarity of the first spectral features of the superpixel points to obtain a hypergraph association matrix, thereby completing the construction of the first hypergraph.

[0021] Optionally, in the step of constructing a first hyperedge based on the similarity of the first spectral features of the superpixel points, the first hyperedge e i The definition formula is as follows:

[0022]

[0023] Among them, v i is a superpixel node, is the node v with superpixel i A neighborhood consisting of several other superpixel nodes whose similarity meets the preset threshold.

[0024] Optionally, in the step of performing a hypergraph convolution process on the first hypergraph to obtain the second spectral feature of the superpixel point, the formula of the hypergraph convolution is:

[0025]

[0026] Among them, X represents the spectral characteristics of the input superpixel point, H is the hypergraph association matrix, and D v and D e are the node degree matrix and the hyperedge degree matrix respectively, θ is the trainable parameter matrix, and Y is the spectral features of the superpixel points output by the hypergraph convolution.

[0027] Optionally, in the step of reconstructing the hypergraph according to the second spectral characteristics of the superpixel points to obtain the second hypergraph, a KNN method and a k-means clustering method are used to construct a second hyperedge according to the second spectral characteristics of the superpixel points.

[0028] Optionally, the step of performing dynamic hypergraph convolution processing on the second hypergraph to obtain a third spectral feature of each superpixel point specifically includes:

[0029] Performing vertex convolution on the second hypergraph to aggregate the second spectral feature of each superpixel point to the second hyperedge to obtain a hyperedge feature;

[0030] Performing hyperedge convolution on the hyperedge feature to aggregate adjacent hyperedge features to the superpixel point to obtain a third spectral feature of the superpixel point.

[0031] Optionally, the step of performing vertex convolution on the second hypergraph to aggregate the second spectral feature of each superpixel point to the second hyperedge to obtain a hyperedge feature specifically includes:

[0032] Performing a global average pooling operation on the second spectral feature of the superpixel point to obtain a fourth spectral feature of the superpixel point;

[0033] Performing a one-dimensional convolution operation on the fourth spectral feature of the superpixel in the spectral dimension, and obtaining a weight matrix for each spectral dimension through a Sigmoid activation function;

[0034] Multiplying the weight matrix by the second spectral feature of the superpixel to obtain a fifth spectral feature of the superpixel; and using a multilayer perceptron to learn and obtain a transformation matrix based on the fifth spectral feature of the superpixel.

[0035] The fifth spectral feature of the superpixel point is transformed using the transformation matrix, and the transformed feature is compressed using one-dimensional convolution to obtain the superedge feature.

[0036] Optionally, image preprocessing is performed by principal component analysis or convolutional neural network.

[0037] On the other hand, the present application also provides a hyperspectral image classification device based on a dynamic hypergraph convolutional network, comprising:

[0038] An image dimension reduction module, used for acquiring a hyperspectral image and performing image preprocessing on the hyperspectral image;

[0039] A pixel segmentation module is used to perform super-pixel segmentation on the hyperspectral image after image preprocessing to obtain a plurality of super-pixel points, each of which corresponds to a plurality of original pixel points;

[0040] A first hypergraph construction module, configured to construct a hypergraph according to the first spectral characteristics of the superpixel points to obtain a first hypergraph;

[0041] a hypergraph convolution module, configured to perform hypergraph convolution processing on the first hypergraph to obtain a second spectral feature of the superpixel;

[0042] A second hypergraph construction module, configured to reconstruct a hypergraph according to the second spectral characteristics of the superpixel points to obtain a second hypergraph;

[0043] a dynamic graph convolution module, configured to perform dynamic hypergraph convolution processing on the second hypergraph to obtain a third spectral feature of each superpixel point;

[0044] A feature conversion module, configured to convert the spectral features of the superpixels into spectral features of the corresponding original pixels based on the correspondence between the superpixels and the original pixels;

[0045] A pixel classification module is used to classify the original pixel points based on the spectral characteristics of the original pixel points.

[0046] The hyperspectral image classification method and system based on a dynamic hypergraph convolutional network obtains a hyperspectral image and performs image preprocessing on the hyperspectral image; performs superpixel segmentation on the preprocessed hyperspectral image to obtain a plurality of superpixels, each superpixel corresponding to a plurality of original pixels; constructs a hypergraph based on the first spectral characteristics of the superpixels to obtain a first hypergraph; performs hypergraph convolution on the first hypergraph to obtain a second spectral characteristic of the superpixels; reconstructs a hypergraph based on the second spectral characteristics of the superpixels to obtain a second hypergraph; performs dynamic hypergraph convolution on the second hypergraph to obtain a third spectral characteristic of each superpixel; converts the spectral characteristics of the superpixels into spectral characteristics of the corresponding original pixels based on the correspondence between the superpixels and the original pixels; and classifies the original pixels based on the spectral characteristics of the original pixels. The two hypergraph constructions enable dynamic extraction of the spectral characteristics of the superpixels / original pixels. Furthermore, since the entire hyperspectral image is used as input, rather than being divided into blocks like most CNNs, the network can better extract global features. At the same time, the use of superpixel segmentation technology to divide the original pixels into superpixels greatly reduces the amount of calculation, and the labels of some unknown pixels can be determined through the superpixel labels, which is equivalent to implicitly increasing the number of samples. This allows the model proposed in this application to achieve higher classification accuracy with a small number of training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of the process of the hyperspectral imaging method based on dynamic hypergraph convolutional network of the present application.

[0048] Figure 2 A schematic diagram of the network architecture of this application is shown.

[0049] Figure 3 A schematic diagram showing the dimensionality reduction and dimensionality reshaping of the present application is shown.

[0050] Figure 4 A schematic diagram showing the positional relationship between target pixels and similar pixels in a hyperspectral image is shown.

[0051] Figure 5 A schematic diagram of vertex convolution is shown.

[0052] Figure 6 A schematic diagram of hyperedge convolution is shown.

[0053] Figures 7a-7e The pseudo-color image, ground truth image, training sample image, test sample image and classification structure image are shown respectively.

[0054] Figure 8A functional module diagram of the hyperspectral imaging device based on the dynamic hypergraph convolutional network of the present application is shown. DETAILED DESCRIPTION

[0055] The following describes the embodiments of the present application through specific examples. Those skilled in the art will readily understand the other advantages and benefits of the present application from the disclosure herein. The present application may also be implemented or applied through various other specific embodiments, and the details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present application.

[0056] See also Figure 1-8 It should be noted that the diagrams provided in this embodiment are only for schematically illustrating the basic concept of the present application. Therefore, the diagrams only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complex.

[0057] Hyperspectral images are continuous, hundreds-channel images generated by hyperspectral remote sensing systems by capturing and recording solar radiation signals that carry ground feature information. Hyperspectral images contain rich spectral and spatial information. Because different components absorb light differently, defects are more prominently reflected in the image at specific wavelengths. This allows hyperspectral images to not only reveal external quality characteristics of a sample, such as size, shape, and defects, but also fully capture differences in the sample's internal physical structure and chemical composition. These characteristics give hyperspectral imaging technology its unique advantages in precision agriculture.

[0058] Precision agriculture is a new type of agriculture that comprehensively integrates information technology with agricultural production. Powered by information technology, precision agriculture utilizes advanced technologies such as 3S (GPS, GIS, and RS) combined with modern agricultural techniques. Based on spatial variation, it implements a comprehensive system of modern agricultural operations and management, using positioning, timing, and quantitative methods. Precision agriculture maximizes agricultural productivity and is an effective path to achieving sustainable agricultural development characterized by high quality, high yield, low energy consumption, and environmental friendliness.

[0059] Hyperspectral image classification presents significant challenges due to its high dimensionality, high inter-band correlation, and spectral mixing. Existing hyperspectral image classification methods are mostly based on convolutional neural networks, which typically require a sufficient number of labeled samples for training, making hyperspectral image classification difficult. Furthermore, most convolutional neural network architectures feed images into the network in batches, which can result in the network failing to capture relationships between pixels over long distances.

[0060] Based on this, the present application discloses a hyperspectral image classification method based on a dynamic hypergraph convolutional network, which can be applied to hyperspectral image classification in precision agriculture. Figure 1 FIG1 shows a hyperspectral image classification method based on a dynamic hypergraph convolutional network according to an embodiment of the present application. Figure 1 As shown, the hyperspectral image classification method based on the dynamic hypergraph convolutional network includes:

[0061] S10: Acquire a hyperspectral image and perform image preprocessing on the hyperspectral image;

[0062] S20: performing super-pixel segmentation on the hyperspectral image after image preprocessing to obtain a plurality of super-pixel points, each super-pixel point corresponding to a plurality of original pixel points;

[0063] S30: constructing a supergraph according to the first spectral characteristics of the superpixel points to obtain a first supergraph;

[0064] S40: performing a hypergraph convolution process on the first hypergraph to obtain a second spectral feature of the superpixel;

[0065] S50: reconstructing a hypergraph according to the second spectral characteristics of the superpixel to obtain a second hypergraph;

[0066] S60: performing dynamic hypergraph convolution processing on the second hypergraph to obtain a third spectral feature of each superpixel point;

[0067] S70: Converting the spectral features of the superpixel into spectral features of the corresponding original pixel according to the correspondence between the superpixel and the original pixel;

[0068] S80: Classifying the original pixel points based on the spectral characteristics of the original pixel points.

[0069] It should be noted that the hyperspectral image classification method based on the dynamic hypergraph convolutional network of this embodiment can be applied not only to the field of precision agriculture, but also to the fields of aerospace, food safety, medical diagnosis, industrial classification quality inspection of assembly lines, military field, geological survey and so on.

[0070] The hyperspectral image classification method based on dynamic hypergraph convolutional network in this embodiment is Figure 2 The network architecture shown is implemented, and the technical solution of this embodiment will be explained in detail below in combination with the network architecture.

[0071] First, step S10 is executed: a hyperspectral image is acquired, and image preprocessing is performed on the hyperspectral image.

[0072] Since hyperspectral images are different from ordinary images and hyperspectral images, the spectral bands of hyperspectral images have hundreds of dimensions. Applying principal component analysis (PCA) to hyperspectral images can effectively remove spectral redundant bands and facilitate better extraction of spectral features. Figure 3 As shown in Figure 1, through the PCA dimensionality reduction operation, that is, the image preprocessing operation, the number of spectral bands of the original hyperspectral image is reduced from c dimension to d dimension.

[0073] It should be noted that in other embodiments, the hyperspectral image may be preprocessed using a one-dimensional convolutional neural network (1D CNN) to remove redundant bands in the hyperspectral image, thereby facilitating better extraction of spectral features. For example, the hyperspectral image may be preprocessed using BN and two-dimensional convolution (1×1 convolution kernel).

[0074] Next, step S20 is executed: super-pixel segmentation is performed on the hyperspectral image after image preprocessing to obtain a plurality of super-pixel points, each of which corresponds to a plurality of original pixel points.

[0075] Since there are thousands of original pixels in hyperspectral images, although we can regard the original pixels in the image as nodes in a hypergraph, the construction of the hypergraph requires an adjacency matrix, which is very large and will greatly increase the computational cost.

[0076] To solve this problem, Figure 2 and 3 As shown, in the network architecture of the application, simple linear iterative clustering (SLIC) is used to group the raw pixels of the hyperspectral image after image preprocessing. Roughly n raw pixels are clustered using k-means to generate a superpixel, and the m superpixels generated by SLIC are used as nodes in the hypergraph.

[0077]

[0078] H and W represent the height and width of the hyperspectral image, respectively. Superpixel segmentation can segment similar pixels into uniform regions based on the weighted Euclidean distance of the spectrum and space. The superpixels S obtained after segmentation are {s1, s2, ...s m It should be noted that in actual processing, the number of original pixels contained in each superpixel is not the same.

[0079] Through superpixel segmentation, each original pixel corresponds to a superpixel, and the final classification of the hyperspectral image is performed for each original pixel. The spectral characteristics of the obtained superpixel need to be assigned to each original pixel.

[0080] Since different superpixels contain different numbers of original pixels, it is difficult to calculate directly through the network. Therefore, np.reshape can be used to reshape the dimension of the hyperspectral image after superpixel segmentation, and the length and width of the original three-dimensional hyperspectral image are multiplied to convert it into a two-dimensional matrix, such as Figure 3 As shown. Figure 3 It can be seen that the dimensions of the two-dimensional matrix after dimension reshaping are H×W and d, d is the number of spectral bands of the hyperspectral image after dimensionality reduction, and the value of H×W represents the number of all original pixels in the hyperspectral image.

[0081] A mapping matrix O∈R is defined in the encoder and decoder HW×m Convert superpixels to original pixels and vice versa. HW×d It represents the 2D matrix reshaped from the 3D hyperspectral image along the spectral space dimension. The mapping matrix O is represented as

[0082]

[0083] Feature transformation is achieved through the following formula:

[0084]

[0085]

[0086] in is the mapping matrix O normalized by column, V represents the node composed of superpixels, is the node formed by the superpixel pad after feature extraction, Denotes the original pixel obtained after the decoder operation. After the SLIC operation, dimension reshaping, and encoder (Encoder), V is obtained. The segmented features can be viewed as an undirected graph G = (V, E), where V and E represent nodes and edges, respectively.

[0087] Next, step S30 is executed: constructing a hypergraph according to the first spectral characteristics of the superpixel points to obtain a first hypergraph.

[0088] A hypergraph is a generalized graph model. Compared to a simple graph where all edges have degree 2, a hypergraph is characterized by the ability of a single hyperedge to connect multiple nodes. Hypergraph descriptions are more flexible and easier to extend in multimodal and heterogeneous situations. Hypergraphs can be combined with adjacency matrices to model high-order relationships between objects. This demonstrates powerful data representation capabilities in many computer vision tasks, such as semantic segmentation, graph data segmentation, 3D object classification, and multimodal data analysis. Hyperedges in a hypergraph can connect multiple vertices, which helps the network capture more features of distantly similar nodes, which is particularly valuable for hyperspectral image classification.

[0089] In an exemplary embodiment, the step of constructing a hypergraph according to the first spectral characteristics of the superpixel points to obtain a first hypergraph specifically includes:

[0090] Calculate and obtain a first spectral feature of each superpixel point based on the spectral features of the original pixel points of the hyperspectral image after image preprocessing, wherein the first spectral feature of each superpixel point is the average value of the spectral features of all original pixel points contained in the superpixel point;

[0091] A first hyperedge is constructed based on the similarity of the first spectral features of the superpixel points to obtain a hypergraph association matrix, thereby completing the construction of the first hypergraph.

[0092] This application constructs each superpixel as a node of a supergraph. The node feature of the supergraph (i.e., the first spectral feature of the superpixel) is the average value of all pixel features contained in the superpixel. Assuming that the i-th superpixel is composed of j original pixels, the spectral vector d of each original pixel can be expressed as d α ∈R 1×d (α=1,2,…,j) Node feature f β (That is, the first spectral feature corresponding to the super pixel point) is calculated as follows:

[0093]

[0094] The classification of a raw pixel in a hyperspectral image may benefit not only from neighboring pixels but also from distant pixels. Unlike convolutional CNNs, which have a smaller receptive field and cannot capture effective information from distant pixels, graph structures are not restricted by distance. They take the entire hyperspectral image as input, rather than segmenting it into blocks as in most CNN-based architectures. This allows the network to better extract global features. Figure 4 The positional relationship between target pixels and similar pixels in hyperspectral images is illustrated, which further reflects the advantages of the graph-based network framework.

[0095] This application extracts features by constructing a hypergraph model. Figure 1 Unlike edges that connect only two nodes, each hyperedge in a hypergraph can connect multiple nodes. This multi-node connection allows for better feature extraction. Each pixel can be connected to multiple similar pixels, allowing for more comprehensive feature extraction.

[0096] In order to better obtain the global features of the hyperspectral image, the KNN method is used to construct hyperedges based on the similarity of spectral features between nodes. i , select and vi The most similar n nodes v j , which is close to the target node, that is, n other nodes v whose similarity to the target node meets the preset threshold j Composed of node v i Neighborhood Super Edge e i The definition formula is as follows

[0097]

[0098] Therefore, each node corresponds to a hyperedge, that is, the number of hyperedges is the same as the number of nodes, both of which are m.

[0099] The incidence matrix H∈R of the hypergraph m×m Can be defined as

[0100]

[0101] When constructing a hypergraph, the weight of each hyperedge is set to 1, so the weight matrix of the hyperedge W∈R m×m That is the identity matrix. Node degree matrix D v ∈R m×m Expressed as

[0102]

[0103] Hyperedge degree matrix D e ∈R m×m Expressed as

[0104]

[0105] Next, step S40 is executed: performing a hypergraph convolution process on the first hypergraph to obtain a second spectral feature of the superpixel point.

[0106] Hypergraph convolution is a convolution method based on the hypergraph model. It can compute complex high-order data correlations through the hypergraph structure and update node features through information propagation. Hypergraph convolution effectively extracts high-order correlations in the hypergraph through node-edge-node transformations.

[0107] The architecture of this application uses superpixels as nodes and uses the KNN method to obtain hyperedges, resulting in a hypergraph structure. To further extract features, we perform two (or three or more) hypergraph convolution operations on the constructed hypergraph. The output value of the last layer of hypergraph convolution is used as the second spectral feature of the superpixel.

[0108] For a hypergraph G = (V, E, W), where V and E represent the vertex set and hyperedge set respectively, and the diagonal matrix W represents the hyperedge weight matrix. The number of vertices is the number of superpixels divided, m, and the formula for hypergraph convolution is

[0109]

[0110] in Represents the features of m nodes and c1 dimension, that is, the spectral features of the input superpixel points, H∈R m×m is the hypergraph incidence matrix, D v and D e are the node degree matrix and the hyperedge degree matrix, Is a trainable parameter matrix that applies theta filters to the nodes of the hypergraph to extract features. After the convolution operation, we get That is, Y is the spectral feature of the superpixel point output by the hypergraph convolution. The feature of the node after the hypergraph convolution operation is given by Convert to l represents the number of layers (corresponding to the number of hypergraph convolutions, in Figure 2 The value is 2), and the hypergraph convolution formula of each layer is

[0111]

[0112] Next, step S50 is performed: a hypergraph is reconstructed based on the second spectral characteristics of the superpixel to obtain a second hypergraph. During the construction of the second hypergraph, a KNN method and a k-means clustering method are used to form second hyperedges based on the second spectral characteristics of the superpixel.

[0113] After two hypergraph convolution operations, the vertex feature obtained is X = [x1; x2; ...; x m ], where x i (i=1,2,…,m represents the spectral characteristics of the i-th superpixel vertex. Figure 2 The dynamic hypergraph module DHG in

[15] uses the features of these vertices to reconstruct a hypergraph. In the DHG module, we use the KNN method and the k-means clustering method to update the hypergraph structure based on local features and global features respectively. The KNN method is used to generate basic hyperedges, and the set of adjacent hyperedges is expanded through the clustering algorithm, namely k-means clustering. For the KNN method, we calculate each vertex u and its k-1 (k value is configurable, for example, it can be 12, 18, etc.) nearest vertices. These neighboring vertices together with the vertex u constitute a hyperedge. For the k-means method, we divide all vertices into S clusters according to the Euclidean distance. For each vertex, the nearest S-1 clusters will be assigned as the adjacent hyperedges of the vertex. Local and global features can be extracted through the dynamic hypergraph construction method.

[0114] Next, step S60 is executed: performing dynamic hypergraph convolution processing on the second hypergraph to obtain a third spectral feature of each superpixel point.

[0115] In an exemplary embodiment, the step of performing dynamic hypergraph convolution processing on the second hypergraph to obtain the third spectral feature of each superpixel point includes:

[0116] Vertex convolution is performed on the second hypergraph to aggregate the second spectral feature of each superpixel point to the second hyperedge to obtain a hyperedge feature.

[0117] Performing hyperedge convolution on the hyperedge feature to aggregate adjacent hyperedge features to the superpixel point to obtain a third spectral feature of the superpixel point.

[0118] In an exemplary embodiment, when performing vertex convolution on the second hypergraph to aggregate the second spectral features of each superpixel point to the second hyperedge to obtain the hyperedge feature, first, a global average pooling operation is performed on the second spectral features of the superpixel point to obtain the fourth spectral feature of the superpixel point; then, a one-dimensional convolution operation is performed on the fourth spectral feature of the superpixel point in the spectral dimension, and a weight matrix of each spectral dimension is obtained through a Sigmoid activation function; the weight matrix is ​​multiplied by the second spectral feature of the superpixel point to obtain the fifth spectral feature of the superpixel point; based on the fifth spectral feature of the superpixel point, a multilayer perceptron is used to learn to obtain a transformation matrix; the fifth spectral feature of the superpixel point is transformed using the transformation matrix, and the transformed feature is compressed using one-dimensional convolution to obtain the hyperedge feature.

[0119] The following will use specific examples to illustrate vertex convolution (Vertex Conv) and hyperedge convolution (HyperedgeConv).

[0120] Vertex Conv

[0121] like Figure 5 As shown in Figure 1, assuming that there are n vertices on each hyperedge in the second hypergraph, and the spectral dimension feature of each vertex is d-dimensional, the input original vertex feature is first subjected to a global average pooling (GAP) operation, and then a one-dimensional convolution operation is performed on the spectral dimension with a convolution kernel size of k. The weight matrix W of each spectral dimension is obtained by the Sigmoid activation function. The formula is:

[0122] W=σ(Conv1 k (y))

[0123] Where Conv1 represents a one-dimensional convolution, k is the size of the convolution kernel, and y is the feature after the global average pooling operation (that is, the fourth spectral feature of the superpixel point). The obtained weight matrix W is multiplied by the corresponding elements of the original vertex feature to obtain a new vertex feature (That is, the fifth spectral feature of the superpixel point.) We use a multi-layer perceptron (MLP) to learn the transformation matrix T from the new vertex feature map, perform feature arrangement and weighting, and use one-dimensional convolution to compress the transformed features. The formula is as follows

[0124]

[0125]

[0126] Through the above operations, the vertex features are aggregated to the hyperedge to obtain the hyperedge feature X e , and further extract the spectral features.

[0127] Hyperedge Convolution

[0128] The hyperedge features obtained by vertex convolution are aggregated into centroid vertex features through hyperedge convolution. Figure 6 As shown in Figure 2, for hyperedge convolution, we use an attention mechanism and a multi-layer perceptron (MLP) to generate the weight of each hyperedge. The output centroid vertex feature is calculated as the weighted sum of the input hyperedge features. The process formula is

[0129] w=softmax(X e W+b)

[0130]

[0131] Among them, X e The hyperedge x where the centroid vertex is located is obtained by vertex convolution e is composed of all adjacent hyperedge features. n represents the number of hyperedges in the adjacent hyperedge set, X represents the centroid vertex feature obtained after hyperedge convolution (that is, the third spectral feature of the superpixel point), and W and b are learnable parameters.

[0132] Next, S70: according to the correspondence between the super pixel and the original pixel, the spectral feature of the super pixel is converted into the spectral feature of the corresponding original pixel.

[0133] After the super-pixel feature extraction, the spectral features of the super-pixel points are converted into the spectral features of the original pixels through the decoder.

[0134] Finally, S80: classify the original pixel points based on the spectral characteristics of the original pixel points.

[0135] Specifically, the spectral features of the original pixels are input into the fully connected layer to achieve the classification of the original pixels. The expression of the fully connected layer is:

[0136]

[0137] in, represents the original pixel obtained after the decoder operation, FC() is the fully connected layer, and Y is the final pixel-level classification result.

[0138] After classification, the loss function can be calculated to evaluate the accuracy of the classification.

[0139] The loss function used is the cross entropy loss function, the formula is

[0140]

[0141] Among them, C is the total number of categories of original pixels in the hyperspectral image, T is the total number of original pixels for training, and if the tth original pixel belongs to the cth class, the true value is is the predicted output for the t-th pixel.

[0142] The dataset classification diagram is as follows Figures 7a-7e As shown, Figures 7a-7e The pseudo-color image, ground truth image, training sample image, test sample image and classification structure diagram are shown respectively. As can be seen from the figure, the training sample image only uses a small number of pixels as training samples ( Figure 7e The test sample set uses pixels other than the training samples. The reason for using a small number of training samples is that the feature labels of some unknown pixels can be determined using the feature labels of superpixels, which implicitly increases the number of samples. This allows the proposed model to achieve high classification accuracy with a small number of training samples.

[0143] It should be noted that the step division of the various methods above is only for the purpose of clear description. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0144] Figure 8 This is a functional module diagram of a hyperspectral imaging device based on a dynamic hypergraph convolutional network provided in an embodiment of the present application. Figure 5As shown, the hyperspectral imaging device 11 based on the dynamic hypergraph convolutional network includes an image acquisition and dimensionality reduction module 111, a pixel segmentation module 112, a first hypergraph construction module 113, a hypergraph convolution module 114, a second hypergraph construction module 115, a dynamic graph convolution module 116, a feature conversion module 117 and a pixel classification module 118.

[0145] Among them, the image acquisition and dimensionality reduction module 111 is used to acquire a hyperspectral image and perform image preprocessing on the hyperspectral image; the pixel segmentation module 112 is used to perform superpixel segmentation on the hyperspectral image after image preprocessing to obtain a plurality of superpixel points, each superpixel point corresponds to a plurality of original pixel points; the first supergraph construction module 113 is used to construct a supergraph according to the first spectral characteristics of the superpixel points to obtain a first hypergraph; the supergraph convolution module 114 is used to perform supergraph convolution processing on the first supergraph to obtain the second spectral characteristics of the superpixel points; the second supergraph construction module 115 is used to re-construct the supergraph according to the second spectral characteristics of the superpixel points to obtain a second hypergraph; the dynamic graph convolution module 116 is used to perform dynamic supergraph convolution processing on the second supergraph to obtain the third spectral characteristics of each superpixel point; the feature conversion module 117 is used to convert the spectral characteristics of the superpixel into the spectral characteristics of the corresponding original pixel points according to the correspondence between the superpixel points and the original pixel points; the pixel point classification module 118 is used to classify the original pixel points based on the spectral characteristics of the original pixel points.

[0146] It should be noted that the hyperspectral imaging device 11 based on a dynamic hypergraph convolutional network of this embodiment corresponds to the hyperspectral imaging method based on a dynamic hypergraph convolutional network described above. The functional modules in the hyperspectral imaging device 11 based on a dynamic hypergraph convolutional network correspond to corresponding steps in the hyperspectral imaging method based on a dynamic hypergraph convolutional network. The hyperspectral imaging device 11 based on a dynamic hypergraph convolutional network of this embodiment can be implemented in conjunction with the hyperspectral imaging method based on a dynamic hypergraph convolutional network. Accordingly, the relevant technical details mentioned in the hyperspectral imaging device 11 based on a dynamic hypergraph convolutional network of this embodiment can also be applied to the hyperspectral imaging method based on a dynamic hypergraph convolutional network described above.

[0147] It should be noted that, when actually implemented, the aforementioned functional modules can be fully or partially integrated into a single physical entity, or physically separated. Furthermore, these modules can all be implemented in the form of software called by a processing element; or all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, while others are implemented in the form of hardware. Furthermore, these modules can all or partly be integrated together, or implemented independently. The processing element described herein can be an integrated circuit with signal processing capabilities. During implementation, some or all of the steps of the aforementioned method, or the aforementioned functional modules, can be completed by hardware integrated logic circuits in the processor element or by software instructions.

[0148] In summary, the hyperspectral image classification method and system based on the dynamic hypergraph convolutional network of the present application obtains a hyperspectral image and performs image preprocessing on the hyperspectral image; performs superpixel segmentation on the hyperspectral image after image preprocessing to obtain a number of superpixels, each superpixel corresponding to a number of original pixels; constructs a supergraph according to the first spectral characteristics of the superpixels to obtain a first hypergraph; performs supergraph convolution on the first hypergraph to obtain the second spectral characteristics of the superpixels; reconstructs the supergraph according to the second spectral characteristics of the superpixels to obtain a second hypergraph; performs dynamic supergraph convolution on the second hypergraph to obtain the third spectral characteristics of each superpixel; converts the spectral characteristics of the superpixels into the spectral characteristics of the corresponding original pixels according to the correspondence between the superpixels and the original pixels; and classifies the original pixels based on the spectral characteristics of the original pixels. Through the two supergraph constructions, the spectral characteristics of the superpixels / the spectral characteristics of the original pixels can be dynamically extracted, and since the entire hyperspectral image is used as input instead of being divided into blocks like most CNNs, this allows the network to better extract global features. At the same time, the use of superpixel segmentation technology to divide the original pixels into superpixels greatly reduces the amount of calculation, and the labels of some unknown pixels can be determined through the superpixel labels, which is equivalent to implicitly increasing the number of samples. This allows the model proposed in this application to achieve higher classification accuracy with a small number of training samples.

[0149] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application.

[0150] The above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A hyperspectral image classification method based on a dynamic hypergraph convolutional network, characterized in that: include: Acquiring a hyperspectral image and performing image preprocessing on the hyperspectral image; Performing superpixel segmentation on the hyperspectral image after image preprocessing to obtain a plurality of superpixel points, each superpixel point corresponding to a plurality of original pixel points; A hypergraph is constructed based on the first spectral features of the superpixel points to obtain a first hypergraph, and the steps include: calculating and obtaining the first spectral features of each superpixel point based on the spectral features of the original pixel points of the hyperspectral image after image preprocessing, wherein the first spectral feature of each superpixel point is the average value of the spectral features of all original pixel points contained in the corresponding superpixel point; constructing a first hyperedge based on the similarity of the first spectral features of the superpixel points to obtain a hypergraph association matrix, thereby completing the construction of the first hypergraph; Performing a hypergraph convolution process on the first hypergraph to obtain a second spectral feature of the superpixel point; reconstructing a hypergraph according to the second spectral characteristics of the superpixel points to obtain a second hypergraph; The second hypergraph is subjected to dynamic hypergraph convolution processing to obtain the third spectral feature of each superpixel point, and the steps include: performing vertex convolution on the second hypergraph to aggregate the second spectral feature of each superpixel point to a second hyperedge to obtain a hyperedge feature; wherein, a global average pooling operation is performed on the second spectral feature of the superpixel point to obtain a fourth spectral feature of the superpixel point; a one-dimensional convolution operation is performed on the fourth spectral feature of the superpixel point in the spectral dimension, and a weight matrix of each spectral dimension is obtained through a Sigmoid activation function; the weight matrix is ​​multiplied by the second spectral feature of the superpixel point to obtain a fifth spectral feature of the superpixel point; according to the fifth spectral feature of the superpixel point, a multilayer perceptron is used to learn to obtain a transformation matrix; the fifth spectral feature of the superpixel point is transformed using the transformation matrix, and the transformed feature is compressed using one-dimensional convolution to obtain the hyperedge feature; the hyperedge feature is subjected to hyperedge convolution to aggregate adjacent hyperedge features to the superpixel point to obtain the third spectral feature of the superpixel point; Converting the spectral characteristics of the superpixel into spectral characteristics of the corresponding original pixel according to the correspondence between the superpixel and the original pixel; The original pixels are classified based on the spectral characteristics of the original pixels.

2. The hyperspectral image classification method based on dynamic hypergraph convolutional network according to claim 1, characterized in that: The step of performing superpixel segmentation on the hyperspectral image after image preprocessing to obtain a plurality of superpixel points includes: Simple linear iterative clustering is used to cluster and group the original pixels in the hyperspectral image after image preprocessing to generate a number of superpixel points.

3. The hyperspectral image classification method based on dynamic hypergraph convolutional network according to claim 1, characterized in that: In the step of constructing a first hyperedge based on the similarity of the first spectral features of the superpixel points, the first hyperedge e i The definition formula is as follows: Among them, v i is a superpixel node, is the node v with superpixel i A neighborhood consisting of several other superpixel nodes whose similarity meets the preset threshold.

4. The hyperspectral image classification method based on dynamic hypergraph convolutional network according to claim 2, characterized in that: In the step of performing a hypergraph convolution process on the first hypergraph to obtain the second spectral feature of the superpixel point, the formula of the hypergraph convolution is: Among them, X is the spectral feature of the input superpixel point, H is the hypergraph correlation matrix, and D v and D e are the node degree matrix and the hyperedge degree matrix respectively, θ is the trainable parameter matrix, and Y is the spectral feature of the superpixel output by the hypergraph convolution.

5. The hyperspectral image classification method based on dynamic hypergraph convolutional network according to claim 1, characterized in that: In the step of reconstructing the hypergraph according to the second spectral characteristics of the superpixel points to obtain the second hypergraph, the second hyperedge is constructed using the KNN method and the k-means clustering method according to the second spectral characteristics of the superpixel points.

6. The hyperspectral image classification method based on a dynamic hypergraph convolutional network according to any one of claims 1 to 5, characterized in that: Image preprocessing is performed through principal component analysis or convolutional neural networks.

7. A hyperspectral image classification device based on a dynamic hypergraph convolutional network, characterized in that: include: An image acquisition and dimensionality reduction module is used to acquire a hyperspectral image and perform image preprocessing on the hyperspectral image; A pixel segmentation module is used to perform super-pixel segmentation on the hyperspectral image after image preprocessing to obtain a plurality of super-pixel points, each super-pixel point corresponding to a plurality of original pixel points; A first hypergraph construction module is configured to construct a hypergraph based on the first spectral features of the superpixels to obtain a first hypergraph; wherein, based on the spectral features of the original pixels of the hyperspectral image after image preprocessing, the first spectral features of each superpixel are calculated and obtained, wherein the first spectral feature of each superpixel is the average value of the spectral features of all original pixels contained in the superpixel; and a first hyperedge is constructed based on the similarity of the first spectral features of the superpixels to obtain a hypergraph association matrix, thereby completing the construction of the first hypergraph; a hypergraph convolution module, configured to perform hypergraph convolution processing on the first hypergraph to obtain a second spectral feature of the superpixel; A second hypergraph construction module, configured to reconstruct a hypergraph according to the second spectral characteristics of the superpixel points to obtain a second hypergraph; A dynamic graph convolution module is used to perform dynamic supergraph convolution processing on the second supergraph to obtain the third spectral feature of each superpixel point, including: performing vertex convolution on the second supergraph to aggregate the second spectral feature of each superpixel point to a second hyperedge to obtain a hyperedge feature; wherein, a global average pooling operation is performed on the second spectral feature of the superpixel point to obtain a fourth spectral feature of the superpixel point; a one-dimensional convolution operation is performed on the fourth spectral feature of the superpixel point in the spectral dimension, and a weight matrix of each spectral dimension is obtained through a Sigmoid activation function; the weight matrix is ​​multiplied by the second spectral feature of the superpixel point to obtain a fifth spectral feature of the superpixel point; according to the fifth spectral feature of the superpixel point, a multilayer perceptron is used to learn to obtain a transformation matrix; the fifth spectral feature of the superpixel point is transformed using the transformation matrix, and the transformed feature is compressed using one-dimensional convolution to obtain the hyperedge feature; the hyperedge convolution is performed on the hyperedge feature to aggregate adjacent hyperedge features to the superpixel point to obtain the third spectral feature of the superpixel point; A feature conversion module, configured to convert the spectral features of the superpixels into spectral features of the corresponding original pixels based on the correspondence between the superpixels and the original pixels; A pixel classification module is used to classify the original pixel points based on the spectral characteristics of the original pixel points.

Images