Hyperspectral image classification method based on residual enhancement hyperspectral fusion hypergraph neural network

Abstract

A hyperspectral image classification method based on a residual-enhanced spatial-spectral fusion hypergraph neural network includes the following steps: S1: Extracting spectral-spatial features of the hyperspectral image using different feature extraction strategies, i.e., flattening the original image into a matrix to calculate spectral features, and generating spatial features from the horizontal and vertical coordinates of pixels; S2: Generating hypergraphs from features of different modalities and fusing them to generate a multimodal hypergraph association matrix; and calculating hyperedges and vertices based on the hypergraph structure; S3: Inputting the hypergraph and the original hyperspectral features into a residual-enhanced hypergraph neural network for training, dynamically updating the hypergraph through backpropagation to obtain the predicted classification label for each pixel. This invention can fully represent the rich and complex information in hyperspectral images and solves the problem of significant performance degradation of hyperspectral images with increasing depth in traditional deep learning, thus achieving the best training effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of computer vision, remote sensing image data processing, and specifically relates to a hyperspectral image classification method based on residual enhancement space-spectrum fusion hypergraph neural network. TECHNICAL BACKGROUND

[0002] Compared with traditional panchromatic images or multispectral images, hyperspectral remote sensing images have a large amount of spectral information. Therefore, they have satisfactory performance and greater research value in classic land cover classification tasks, but the large amount of spectral information also brings the problem of information redundancy. How to accurately and effectively extract hyperspectral data information and explore potential features has become the key to the hyperspectral image classification task.

[0003] Traditional machine learning classification methods that rely on spectral features (such as support vector machines, neural networks, kernel methods, multinomial logistic regression, etc. and their variants) have the Hughes phenomenon and are sensitive to noise, resulting in low classification accuracy. Since deep learning has achieved good results in computer vision, we want to replace the traditional manually designed feature extraction method with a deep model distributed feature expression to solve the various problems caused by the diversification of hyperspectral data structure and the influence of complex noise. The space-spectrum fusion strategy can fully exploit and utilize the local pixel dependence within the hyperspectral image, and improve the supervised classification performance of the hyperspectral image by utilizing the neighborhood aggregation attribute of the ground object. The hypergraph structure can represent more complex information, but the traditional hypergraph neural network has the problem of over-smoothing, that is, its performance significantly decreases when the layers are stacked. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the present application provides a hyperspectral image classification method based on residual enhancement space-spectrum fusion hypergraph neural network, which can accurately and effectively extract hyperspectral image information and mine the potential features of hyperspectral data, to solve the problem of significant performance degradation of traditional hypergraph neural networks with increasing depth, and has good practicality and robustness.

[0005] To solve the above technical problems, the present application provides the following technical solutions:

[0006] A hyperspectral image classification method based on residual enhancement space-spectrum fusion hypergraph neural network, comprising the following steps:

[0007] Step S1: Extract the spectral-spatial features of the hyperspectral image by different feature extraction strategies, that is, calculate the spectral features by flattening the original image into a matrix, and generate the spatial features from the horizontal and vertical coordinates of the pixels;

[0008] Step S2: generate hypergraphs for different modal features and fuse them to generate a multi-modal hypergraph correlation matrix; and calculate the hyperedges and vertices according to the hypergraph structure;

[0009] Step S3: input the hypergraph and hyperspectral original features into the residual-enhanced hypergraph neural network for training, dynamically update the hypergraph through back propagation, and obtain the predicted classification label of each pixel.

[0010] Further, the step S1 comprises the following steps:

[0011] Step S11: hyperspectral image dataset reading: read the preprocessed training set and test set with labeled information, and select 5-15 test samples for each classification category;

[0012] Step S12: spectral feature extraction: flatten the original image to obtain the spectral feature X spe , and the specific details are to read the image N represents the number of image pixels, and d represents the number of spectral channels. All channels before the last two channels are selected, and the distance matrix is calculated to obtain the spectral feature X spe ;

[0013] Step S13: spatial feature extraction: obtain the spatial feature by pixel coordinates. Refer to formula (1):

[0014] X spa [i]=[x(i),y(j)] (1)

[0015] Wherein, x(i) and y(i) represent the horizontal and vertical coordinates of pixel i, respectively.

[0016] Further, the step S2 comprises the following steps:

[0017] Step S21: generate a hypergraph association matrix: construct the spectral feature X spe and the spatial feature X spa obtained in step S1 into hypergraphs H spe and H spa respectively through formula (2):

[0018]

[0019] Wherein, X i ∈Ν k (X j ) represents the vertex X i is the k nearest neighbors of the vertex X j , mean represents the average Euclidean distance between all k nearest neighbors, σ is an adjustable hyperparameter, and e is a hyperedge.

[0020] Step S22: calculate the hyperedge and vertex from the hypergraph, refer to formula (3):

[0021]

[0022] where d(v) denotes the diagonal matrix of the top degree, w(e) is the hyperedge weight parameter, and δ(e) denotes the diagonal matrix of the edge degree.

[0023] Further, in the step S3, constructing the residual-enhanced hypergraph neural network comprises the following steps:

[0024] Step S31: Hypergraph Laplacian matrix generation: The Laplacian matrix is a kind of matrix representation of a graph, and the Laplacian matrix of a traditional graph is defined as:

[0025] L = D - A (4)

[0026] where D is the degree matrix of the graph, A is the adjacency matrix of the graph, and the adjacency matrix of the hypergraph can be defined as:

[0027]

[0028] The hypergraph Laplacian matrix has the following transformation:

[0029]

[0030] Therefore, after derivation, the convolution operation on the hypergraph for a given image feature X can be defined as:

[0031]

[0032] where D v is the hyperedge degree matrix, D e is the vertex degree matrix, H is the incidence matrix of the original hypergraph, W represents the weight matrix of the hyperedge, the elements on the diagonal line represent the weight of each hyperedge, and Θ represents the hyperparameter.

[0033] Step S32: Construction of residual-enhanced hypergraph neural network: A linear layer is added at the beginning and end of the entire network, respectively, to realize linear combination of the processed data through a linear layer, adopt a nonlinear activation function, and use a normalization operation before and after the activation function; The operation is to input the hypergraph and hyperspectral image into the network at the same time, first normalize and Dropout the original hyperspectral features to prevent overfitting; Subsequently, it is transmitted into the residual-enhanced hypergraph convolution layer, and the propagation process at the lth layer is defined as:

[0034] X (l+1) = σ(((1-α l )YX l +α l X (0) )((1-β l )I+βW (l))) (8)

[0035] where, alpha l , beta l are hyperparameters, which are set to 0.5 and 0.1 respectively, and I is the identity matrix.

[0036] In the present application, the introduction of the residual can effectively avoid the over-smoothing problem caused by the traditional hypergraph neural network, and the hypergraph neural network based on residual enhancement can effectively fuse multiple hypergraph information. Therefore, the present application has important significance in the field of computer vision and remote sensing.

[0037] The present application has the advantages that it can accurately and effectively extract hyperspectral image information and mine the potential features of hyperspectral data, solve the problem of significant performance degradation caused by traditional hypergraph neural networks with increasing depth, and has good practicality and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a flowchart of a hyperspectral image classification method based on residual enhanced space spectrum fusion hypergraph neural network; DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with specific embodiments and with reference to the drawings.

[0040] Referring to Figure 1 A hyperspectral image classification method based on residual enhanced space spectrum fusion hypergraph neural network, which comprises three modules, namely hyperspectral space spectrum feature extraction, hypergraph generation, hypergraph dynamic update and feature image classification. The method comprises the following steps:

[0041] Step S1: Extract the spectral-spatial features of the hyperspectral image by different feature extraction strategies, i.e. flatten the original image to calculate the spectral features, and generate the spatial features from the horizontal and vertical coordinates of the pixels; the process is as follows:

[0042] Step S11: hyperspectral image dataset reading: read the preprocessed training set and the test set with labeled information, select 5-15 test samples for each classification category, and divide the image into four parts for classification for the labeled sample with a large number. The original image read is where n, m, d are the length and width in the spatial dimension and the number of spectral channels, respectively;

[0043] Step S12: spectral feature extraction: flatten the original image to calculate the spectral feature X spe , the specific details are to read the image N represents the number of image pixels, d represents the number of spectral channels, all channels before the last two channels are selected, the distance matrix is calculated to obtain the spectral feature X spe ;

[0044] Step S13: spatial feature extraction: spatial features are obtained through pixel coordinates Referring to formula (1):

[0045] X spa [i] = [x(i), y(j)] (1)

[0046] Wherein, x(i) and y(i) represent the horizontal and vertical coordinates of pixel i respectively.

[0047] Figure 1 After the hollow spectral feature extraction is completed, the hollow spectral features are fused to generate a hypergraph.

[0048] Step S2: features of different modalities are respectively generated into a hypergraph and fused to generate a multi-modal hypergraph correlation matrix; and a hyperedge and a vertex are calculated according to the hypergraph structure, the process being as follows:

[0049] Step S21: generating a hypergraph correlation matrix: the hypergraph is defined as G(V, E, W), W is a hyperedge weight matrix, in the specific implementation, the hypergraph is represented by a correlation matrix H, the size of H is |V|×|E|, V and E represent vertices and hyperedges of the hypergraph respectively. The spectral feature X spe and the spatial feature X spa are obtained in step S1. spe and H spa are respectively constructed through formula (2):

[0050]

[0051] Wherein, X i ∈Ν k (X j ) represents the vertex X i is the k nearest neighbors of the vertex X j , mean represents the average Euclidean distance between all k nearest neighbors, sigma is an adjustable hyperparameter, and e is a hyperedge; preferably, the nearest neighbors k of the spectral and spatial hypergraphs are both set to 16.

[0052] Step S22: calculating a hyperedge and a vertex from the hypergraph, referring to formula (3):

[0053]

[0054] Wherein, d(v) represents a diagonal matrix of vertex degree, w(e) is a hyperedge weight parameter, and delta(e) represents a diagonal matrix of edge degree.

[0055] Step S3: input the hypergraph and hyperspectral original features into the residual enhanced hypergraph neural network for training, dynamically update the hypergraph through back propagation, and obtain the predicted classification label of each pixel, the process is as follows:

[0056] Step S31: hypergraph Laplacian matrix generation: the multi-modal features X [X1, X2, X3, …, X m generated by step (1) are used to calculate the hypergraph association matrix H i of each X i using formula (2), and are concatenated to generate H [H1, H2, H3, …, H m ]. The Laplacian matrix of the traditional graph is defined as:

[0057] L=D-A (4)

[0058] Wherein, D is the degree matrix of the graph, A is the adjacency matrix of the graph, and the adjacency matrix of the hypergraph can be defined as:

[0059]

[0060] The hypergraph Laplacian matrix is transformed as follows:

[0061]

[0062] Therefore, the convolution operation on the hypergraph can be defined as:

[0063]

[0064] Wherein, D v represents the hyperedge degree matrix, D e represents the vertex degree matrix, H is the association matrix of the original hypergraph, W represents the weight matrix of the hyperedge, the elements on the diagonal line represent the weight of each hyperedge, and Θ represents the hyperparameter;

[0065] Step S32: construction of residual enhanced hypergraph neural network: a linear layer is added at the beginning and end of the whole network respectively, the processed data is first passed through a linear layer to realize linear combination of data, a nonlinear activation function is used, and a normalization operation is used before and after the activation function; The operation is to input the hypergraph and hyperspectral image into the network at the same time, first, the hyperspectral original features are normalized and Dropout processed to prevent overfitting; Then, it is transmitted into the residual enhanced hypergraph convolution layer, although the residual connection in ResNet solves the problem of network depth training, but it can only alleviate the problem of over-smoothing, the residual convolution here uses two operations of initial residual and identity mapping, not only solves the problem of over-smoothing, but also can continuously improve the classification effect with the increase of network depth, the propagation process in the lth layer is defined as:

[0066] X (l+1) = σ(((1 - a l )YX l + a l X (0) ((1 - β l )I + βW (l) )) (8)

[0067] where a l , β l are hyperparameters, set to 0.5 and 0.1 respectively, and I is the identity matrix.

[0068] Initial residual: In order to simulate the skip connection in ResNet, the smooth representation YX (l) is combined with the initial connection of the first layer X (0) . The initial residual connection can make the output of each layer of the network at least retain a small part of the initial input features;

[0069] Identity mapping: In the l-th layer, the identity matrix I is added to the weight matrix W (l) . β l can ensure the attenuation of the weight matrix to be adaptively increased with the increase of the number of layers. Specifically, β l = log(λ / (l+1)+1), λ represents a hyperparameter, and is set to 0.1;

[0070] In order to increase the nonlinear relationship between each layer of the neural network, a Relu operation is added to each layer. A linear layer is added to the last layer of the network for data pixel label output, so as to realize the lower layer classification task. The whole network is gradient descent, and the learning rate can be dynamically adjusted. The initial value of the learning rate is set to 0.01, and the learning rate is dynamically halved every 30 training batches.

[0071] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and is only for the purpose of description. The protection scope of the present application should not be regarded as being limited to the specific forms described in the present embodiments, and the protection scope of the present application also extends to the equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A hyperspectral image classification method based on residual enhancement hyperspectral fusion supergraph neural network, characterized in that, The method comprises the following steps: Step S1: Extracting spectral-spatial features of the hyperspectral image by different feature extraction strategies, i.e., calculating spectral features by flattening the original image into a matrix, and generating spatial features from the horizontal and vertical coordinates of pixels; Step S2: Generating a hypergraph for each feature of different modalities and performing fusion to generate a multi-modal hypergraph correlation matrix; and calculating hyperedges and vertices according to the hypergraph structure; Step S3: Inputting the hypergraph and the original hyperspectral features into a residual-enhanced hypergraph neural network for training, dynamically updating the hypergraph through back propagation, and obtaining the predicted classification label of each pixel; The step S2 comprises the following steps: Step S21: generating hypergraph incidence matrix: constructing hypergraph incidence matrix H by formula (2) based on the spectral features and spatial features obtained in step S1 and spatial features by formula (2) respectively and : （2）； wherein, denotes a vertex is a vertex of a nearest neighbor, denotes all nearest neighbor vertices, is a tunable hyperparameter, is a hyperedge; Step S22: Calculating hyperedges and vertices from the hypergraph, referring to formula (3): （3）； wherein, is a diagonal matrix representing the top degree, is a hyperedge weight parameter, is a diagonal matrix representing the edge degree; The step S3 comprises the following steps: Step S31: Hypergraph Laplacian matrix generation: The Laplacian matrix is a matrix representation of a graph, and the Laplacian matrix of a traditional graph is defined as: （4）； wherein, is the degree matrix of the graph, is the adjacency matrix of the graph, the adjacency matrix of a hypergraph can be defined as: （5）； The hypergraph Laplacian matrix has the following transformation: （6）； Thus, by derivation, for a given image feature The convolution operation on a hypergraph can be defined as: （7）； wherein, denotes the superadjacency matrix, denotes the vertex degree matrix, is the incidence matrix of the original hypergraph, denotes the weight matrix of the hyperedges, the elements on the diagonal representing the weight of each hyperedge, denotes the hyperparameter; Step S32: residual enhanced hypergraph neural network construction: a linear layer is added at the beginning and end of the whole network respectively, the processed data is first passed through a linear layer to realize linear combination of data, a nonlinear activation function is used, and a normalization operation is used before and after the activation function; operation is to input hypergraph and hyperspectral image into the network at the same time, first, the original features of hyperspectral are normalized and Dropout processed to prevent overfitting; then into the residual enhanced hypergraph convolution layer, the propagation process of the first layer is defined as: Layer = Layer + 1 Layer = Layer + 1 （8）； wherein, , are hyperparameters, set to 0.5 and 0.1, respectively, is the identity matrix.

2. The residual enhancement hyperspectral image classification method based on spectral fusion supergraph neural network according to claim 1, characterized in that, The step S1 comprises the following steps: Step S11: Hyperspectral image dataset reading: reading the preprocessed training set and the test set with labeled information, and selecting 5-15 test samples for each classification category; Step S12: spectrum feature extraction: the original image is flattened to obtain the spectrum feature , the specific details are reading the image , , which represents the number of image pixels, , which represents the number of spectrum channels, all channels before the last two channels are selected, and the distance matrix is calculated to obtain the spectrum feature ; Step S13: Spatial feature extraction: Obtain spatial features by pixel coordinates Referring to Equation (1): （1）； wherein, and denote the horizontal and vertical coordinates of the pixel respectively.

Images