A hyperspectral image classification method, system and storage medium
By constructing a hyperspectral image classification model that combines Transformer and graph convolutional neural networks, and employing a follow-pixel block mechanism and a multi-layer Graphormer encoder, the problem of insufficient global and local feature extraction in existing technologies is solved, thereby improving the accuracy and efficiency of hyperspectral image classification.
Patent Information
- Application Number
- CN202310868403.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-14
- Publication Date
- 2025-12-26
- Estimated Expiration
- 2043-07-14
AI Technical Summary
Existing technologies struggle to effectively combine the advantages of Transformer and graph convolutional neural networks to extract global and local spatial-spectral features from hyperspectral images for image classification, resulting in insufficient classification accuracy.
A hyperspectral image classification model is constructed, which adopts a pixel block following mechanism, a pixel block spectral embedding module, and a multi-layer Graphormer encoder. It combines Transformer and graph convolutional neural network. The hyperspectral image is converted into regions by the pixel block following mechanism, local spectral information is extracted by the pixel block spectral embedding module, and the hyperspectral image is classified by the multi-layer Graphormer encoder.
It improves the accuracy of hyperspectral image classification, reduces computational complexity, preserves local spatial features, enhances the interactive model of spatial spectral features, and improves classification accuracy.
Smart Images
Figure CN117132885B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of image processing, in particular to a hyperspectral image classification method, system and storage medium. BACKGROUND
[0002] Recently, both the Transformer and the graph convolutional neural network have made gratifying progress in hyperspectral image classification. The method based on the Transformer has the ability to extract and model the non-local features between the spectral and spatial information of the image, while the graph convolutional neural network has a unique aggregation mechanism, which makes it good at extracting the feature relationship of the interaction between the neighborhood vertices. How to take advantage of the strengths of the two and comprehensively extract the global and local spatial-spectral features of the hyperspectral image for image classification is a problem worth studying. SUMMARY
[0003] The purpose of the present application is to provide a hyperspectral image classification method, system and storage medium, which can improve the accuracy of hyperspectral image classification.
[0004] To achieve the above purpose, the present application provides the following scheme:
[0005] A hyperspectral image classification method, comprising:
[0006] constructing a hyperspectral image classification model; the hyperspectral image classification model comprises a following pixel block mechanism, a pixel block spectral embedding module and a multi-layer Graphormer encoder; the following pixel block mechanism is used to convert the hyperspectral image from pixels to regions to generate a following pixel block; the pixel block spectral embedding module is used to mine the local spectral information of each frequency band in the following pixel block to construct a neighborhood convolution mechanism and extract HSI spectral features; the multi-layer Graphormer encoder is used to classify the hyperspectral image according to the HSI spectral features; the multi-layer Graphormer encoder comprises a multi-layer Graphormer encoder; each Graphormer encoder is a Transformer combined with a self-attention mechanism and a graph convolution;
[0007] training the hyperspectral image classification model;
[0008] using the trained hyperspectral image classification model to classify the hyperspectral image.
[0009] Optionally, the following pixel block mechanism comprises the following formula:
[0010] The top-left corner coordinates (x p ,y p ) of the following pixel block are represented as:
[0011]
[0012]
[0013] The coordinates (x q ,y q ) of the following pixel block upper right corner are represented as:
[0014]
[0015]
[0016] wherein W and H are the width and height of the hyperspectral image respectively, m is the width and height of the following pixel block and is an odd number, and the coordinates of the classified pixel x are (x center ,y center ).
[0017] Optionally, the pixel block spectrum embedding module specifically comprises:
[0018] performing feature fusion on the current spectral band and the adjacent spectral band in the following pixel block;
[0019] combining the fused features and the original spectral information by using a residual mechanism to obtain output features;
[0020] performing cascade operation on the output features to obtain an output feature map;
[0021] performing regularization on the output feature map by using BN and ReLU to obtain HSI spectral features.
[0022] Optionally, each Graphormer encoder comprises a multi-layer perceptron (MLP), layer regularization, a graph residual block, layer random deactivation, a multi-head self-attention module and second layer regularization.
[0023] Optionally, the hyperspectral image classification model training adopts a cross-entropy loss function.
[0024] A hyperspectral image classification system comprises:
[0025] The hyperspectral image classification model construction unit is configured to construct a hyperspectral image classification model, which comprises a following pixel block mechanism, a pixel block spectrum embedding module, and a multi-layer Graphormer encoder.
[0026] The hyperspectral image classification model training unit is configured to train the hyperspectral image classification model.
[0027] The hyperspectral image classification unit is configured to classify the hyperspectral image by using the trained hyperspectral image classification model.
[0028] A storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the method.
[0029] According to the specific embodiments of the present application, the following technical effects are achieved:
[0030] The hyperspectral image classification method and system provided by the present application comprehensively utilize the advantages of the Transformer and the graph convolutional neural network, combine the two structures into a unified Graphormer encoder, establish a global and local feature interaction model for hyperspectral image classification, and propose a spatial-spectral feature enhanced hyperspectral image classification model (S 2 GFormer). A following pixel block mechanism is proposed to convert pixels in the hyperspectral image into pixel blocks while retaining local spatial features and reducing the calculation cost. Secondly, a pixel block spectrum embedding module is used to extract the spectral features of the pixel blocks, and a neighborhood convolution is used to extract the comprehensive spectral information in the hyperspectral image. Finally, a multi-layer Graphormer encoder is used to extract representative spatial-spectral features from the pixel blocks for hyperspectral image classification. The three modules are jointly integrated into a unified end-to-end network, and each module interacts with each other to improve the classification accuracy of the hyperspectral image. BRIEF DESCRIPTION OF DRAWINGS
[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings.
[0032] Figure 1 A high-spectral image classification method provided by the present application is shown in the flowchart.
[0033] Figure 2 A principle diagram of a high-spectral image classification method provided by the present application is shown in the flowchart.
[0034] Figure 3 A principle diagram of a Graphormer encoder is shown in the flowchart. DETAILED DESCRIPTION
[0035] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.
[0036] The purpose of the present application is to provide a high-spectral image classification method, system and storage medium, which can improve the accuracy of high-spectral image classification.
[0037] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail with reference to the drawings and specific embodiments.
[0038] As shown in Figure 1 and Figure 2 A high-spectral image classification method provided by the present application includes:
[0039] S101, a hyperspectral image classification model is constructed; the hyperspectral image classification model comprises: a following pixel block mechanism, a pixel block spectrum embedding module, and a multi-layer Graphormer encoder; the following pixel block mechanism is used to convert the hyperspectral image from pixels to regions, generating a following pixel block; the pixel block spectrum embedding module is used to mine local spectral information of each frequency band in the following pixel block to construct a neighborhood convolution mechanism and extract HSI spectral features; the multi-layer Graphormer encoder is used for hyperspectral image classification according to the HSI spectral features; the multi-layer Graphormer encoder comprises: a multi-layer Graphormer encoder; each Graphormer encoder is a Transformer combined with a self-attention mechanism and a graph convolution.
[0040] The hyperspectral image contains a large number of pixels. If each pixel is taken as S 2 The input of GFormer will result in a huge amount of calculation. In order to solve this problem, there are mainly two methods at present, namely the superpixel segmentation method and the batch input method. The superpixel segmentation method can divide the entire image into some superpixels, thereby reducing the number of input nodes. The small sample method aims to divide the graph into multiple subgraphs and calculate the node features of each sample respectively. However, the superpixel segmentation method inputs all samples into the network at one time, which brings a great burden to the memory. The batch input method is inefficient and tedious. In addition, the widely used pixel block method cannot completely preserve the local features of the image, which reduces the integrity of the feature input. In order to overcome the above shortcomings, a following pixel block mechanism is proposed, which can reduce the S 2 The input calculation complexity of GFormer while preserving the local features of the hyperspectral image.
[0041] The following pixel block mechanism is adopted to convert the hyperspectral image from pixels to regions, while preserving the local spatial spectral structure and reducing the calculation complexity.
[0042] Given a classification pixel x(x center ,y center ), take the pixel block with a size of m x m around the classification pixel x as S 2 The input unit of GFormer. Since the pixel block is a square region, the position of the pixel block in the image can be determined by calibrating the coordinates of the left upper corner and the right lower corner of the pixel block. The left upper corner coordinate (x p ,y p ) of the following pixel block can be expressed as:
[0043]
[0044]
[0045] where W and H are the width and height of the image, respectively. m is an odd number. Similarly, the bottom-right corner coordinates (x q ,y q ) are formed as:
[0046]
[0047]
[0048] Using ((x p ,y p ), (x q ,y q )), the algorithm can accurately define the pixel block. Through the above analysis, it can be observed that the following pixel block mechanism expands the spatial information of the image edge points, retaining more abundant spatial features, which is essentially different from the selection of convolution regions in the form of Padding by CNN. In addition, compared with the pixel block division in ViT, the following pixel mechanism avoids the problem of edge classification of pixels. Compared with constructing global node information, the following pixel block mechanism removes redundant spatial information in the form of aggregating local spatial information, making classification and calculation more efficient.
[0049] The following pixel block generated by the following pixel block mechanism is taken as input, and a pixel block spectral embedding module is used to construct a neighborhood convolution mechanism by mining the local spectral information of each band in the hyperspectral image, and to adaptively extract the representative spectral features of the pixel block.
[0050] The pixel block is used to retain the local features of the hyperspectral image. However, the algorithm still does not solve the problem of how to extract the representative spectral features of the pixel block. The most mainstream method at present is to average the spectral values of all pixels in the pixel block, however, this method cannot solve the problem of spectral feature information redundancy, and is easily disturbed by spectral noise. At present, spectral feature extraction pays too much attention to global spectral features, while ignoring local spectral features. The Transformer can establish non-local dependence of spectral sequences, but ignores local information. In order to better integrate local information while using Transformer to establish global attention.
[0051] First, the features of the current spectral band and the adjacent spectral bands are fused, and the i-th adjacent spectral band can be expressed as:
[0052]
[0053] where k is the number of adjacent spectra, is the i-th adjacent spectrum, N is the total number of spectra, and Concat(·) is the fusion operation. m is expressed as:
[0054]
[0055] Subsequently, the feature spectrum is combined with the original spectral information using a residual mechanism, and the residual cascade mechanism is represented as:
[0056]
[0057] X″ i = X + X′ i (8)
[0058] F = Concat (X″1, X″2, …, X″ N ) (9)
[0059] where W i and b i are the weights and biases of the i-th neighborhood convolution to obtain the neighborhood feature X′, the output feature X″ i is obtained after the residual cascade, and the output feature map F is calculated by performing a concatenation operation on X″ i .
[0060] To prevent overfitting, the output is regularized using BN and ReLU to improve classification performance, i.e.:
[0061]
[0062] where F′ is the output. μ and σ are the expected value and variance of the output feature, respectively. γ and β are trainable parameters of the network.
[0063] The following pixel block is regarded as input. A multi-layer Graphormer encoder is adopted to model global and local space-spectral interactions by combining self-attention mechanisms and graph convolution into a newly designed Transformer, and spatial-spectral features are extracted for hyperspectral image classification.
[0064] In the S 2 GFormer framework, a Graphormer is proposed to extract space-spectral features representing pixel blocks by combining the advantages of graph convolutional neural networks and Transformers. The architecture of the Graphormer encoder proposed in this section is shown in Figure 3 The multi-layer graph encoder consists of four identical blocks. Each Graphormer encoder contains six sub-modules, namely multi-layer perceptron (MLP), layer regularization, graph residual block, layer random dropout, multi-head self-attention module, and second layer regularization. In the Graphormer encoder, a graph convolution is embedded in the traditional Transformer encoder to model fine-grained local interaction information.
[0065] In the Graphormer encoder, the multi-head self-attention module utilizes multiple self-attention functions to learn the context feature representation. Given an input pixel block sequence P = {F'1, F'2,..., F'h}, where d is the spectral dimension of the pixel block. By learning the trainable parameters {W n ,W Q ,W K ,W V}, the input pixel block sequence is first projected to Q, K, and V, i.e.:
[0066]
[0067] The algorithm can split the three feature representations into h different subspaces, i.e. Q = {Q 1 ,Q 2 ,..., Q h}, Using these subspaces, self-attention can be performed on each subspace respectively. Then the subspace output can be calculated as:
[0068]
[0069] where Att(·) is the attention function, and the semantic-related and K h are quantified by scaling softmax and dot product. Then, the outputs are concatenated to form the final output Y.
[0070] Multi-head self-attention helps to extract long-distance dependencies, but local information in complex pixel block sequences is easily ignored. Therefore, the algorithm proposes a graph residual block to refine the local spatial-spectral features contained in the pixel block sequence by extracting the relationship between different pixel blocks.
[0071] The comprehensive features Y generated by the multi-head self-attention module can be improved by using graph convolution to extract local information, and its output Y' is:
[0072]
[0073] where W G and σ(·) are the graph adjacency matrix, trainable parameters, and activation function, respectively. In the proposed module, Gaussian error linear unit (GeLU) is used to perform the non-linearization of the network.
[0074] Subsequently, the output Y" of the Graphormer encoder can be represented as:
[0075]
[0076] The graph residual block improves the spatial spectral locality of the features by encoding the graph structure in the network.
[0077] S102, training the hyperspectral image classification model;
[0078] S 2 The feature extraction algorithm of the GFormer adopts a cross-entropy loss function to punish the difference between the label examples and the output of the proposed multi-layer graph encoder, trains the network parameters, and finally predicts the label of each pixel.
[0079] Formulas (3) to (6) illustrate the hyperspectral image preprocessing process, and the output thereof serves as the input of the subsequent network; subsequently, the pixel block spectral embedding module is introduced to extract the spectral features of the pixel block; finally, a multi-layer graph encoder is designed to extract the spatial-spectral features of the pixel block sequence, S 2 The output of the GFormer is The S2GFormer algorithm adopts a cross-entropy loss function to punish the difference between the label examples and the output of the proposed multi-layer graph encoder, which can be expressed as:
[0080]
[0081] The training process of the algorithm:
[0082]
[0083]
[0084] S103, classifying the hyperspectral image by using the trained hyperspectral image classification model.
[0085] Corresponding to the method described in the above embodiment, the present application also provides a hyperspectral image classification system, comprising:
[0086] A hyperspectral image classification model construction unit is configured to construct a hyperspectral image classification model; the hyperspectral image classification model comprises a following pixel block mechanism, a pixel block spectral embedding module, and a multi-layer Graphormer encoder; the following pixel block mechanism is configured to convert a hyperspectral image from a pixel to a region to generate a following pixel block; the pixel block spectral embedding module is configured to mine the local spectral information of each frequency band in the following pixel block to construct a neighborhood convolution mechanism and extract HSI spectral features; the multi-layer Graphormer encoder is configured to classify the hyperspectral image according to the HSI spectral features; the multi-layer Graphormer encoder comprises a multi-layer Graphormer encoder; each Graphormer encoder is a Transformer combined with a self-attention mechanism and a graph convolution;
[0087] The hyperspectral image classification model training unit is configured to train the hyperspectral image classification model.
[0088] The hyperspectral image classification unit is configured to classify the hyperspectral image by using the trained hyperspectral image classification model.
[0089] In order to perform the method corresponding to the above-mentioned embodiments, to achieve the corresponding functions and technical effects, a storage medium has computer program instructions stored thereon, which, when executed by a processor, implement the method.
[0090] Based on the above description, the technical solutions of the present application, in essence or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the various embodiments of the method of the present application. The aforementioned computer storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes.
[0091] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.
[0092] The principles and implementation modes of the present application are described by using specific examples in this paper. The above-mentioned examples are only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.
Claims
1. A hyperspectral image classification method, characterized in that, include: A hyperspectral image classification model is constructed, comprising: a following pixel block mechanism, a pixel block spectral embedding module, and a multi-layer Graphormer encoder. The following pixel block mechanism converts hyperspectral images from pixels into regions, generating following pixel blocks. The pixel block spectral embedding module mines local spectral information of each frequency band within the following pixel blocks to construct a neighborhood convolution mechanism and extract HSI spectral features. The multi-layer Graphormer encoder performs hyperspectral image classification based on HSI spectral features. The multi-layer Graphormer encoder comprises: multiple layers of Graphormer encoders; each Graphormer encoder is a Transformer combining self-attention and graph convolution. Training a hyperspectral image classification model; The trained hyperspectral image classification model is used to classify hyperspectral images; The pixel block spectrum embedding module specifically includes: Feature fusion will be performed by following the current spectral band and adjacent spectral bands in the pixel block; The fused features are combined with the original spectral information using a residual mechanism to obtain the output features; The output features are concatenated to obtain the output feature map; The output feature map is regularized using BN and ReLU to obtain the HSI spectral features.
2. The hyperspectral image classification method of claim 1, wherein, The following pixel block mechanism includes the following formula: Follow the pixel block upper left corner coordinates is represented as: ； ； Follow the pixel block right upper corner coordinates is represented as: ； ； wherein, and are width and height of the hyperspectral image, respectively, are width and height of the following pixel block, respectively, and are odd numbers, and the coordinates of the classified pixels are .
3. The hyperspectral image classification method of claim 1, wherein, Each Graphormer encoder includes: a multilayer perceptron (MLP), layer regularization, graph residual block, layer random deactivation, multi-head self-attention module, and second layer regularization.
4. The hyperspectral image classification method of claim 1, wherein, The hyperspectral image classification model is trained using the cross-entropy loss function.
5. A hyperspectral image classification system for implementing a hyperspectral image classification method according to any one of claims 1-4, characterized in that, include: A hyperspectral image classification model construction unit is used to construct a hyperspectral image classification model. The hyperspectral image classification model includes: a following pixel block mechanism, a pixel block spectral embedding module, and a multi-layer Graphormer encoder. The following pixel block mechanism converts hyperspectral images from pixels into regions, generating following pixel blocks. The pixel block spectral embedding module mines the local spectral information of each frequency band in the following pixel blocks to construct a neighborhood convolution mechanism and extract HSI spectral features. The multi-layer Graphormer encoder is used to classify hyperspectral images based on HSI spectral features. The multi-layer Graphormer encoder includes: multiple layers of Graphormer encoders; each Graphormer encoder is a Transformer combining a self-attention mechanism and graph convolution. The hyperspectral image classification model training unit is used to train the hyperspectral image classification model. The hyperspectral image classification unit is used to classify hyperspectral images using a trained hyperspectral image classification model.
6. A storage medium having stored thereon computer program instructions, characterized in that, When the computer program instructions are executed by the processor, a hyperspectral image classification method as described in any one of claims 1-4 is implemented.