Hyperspectral image classification system and method based on self-supervised learning

Abstract

The invention is suitable for the technical field of hyperspectral image classification, and provides a hyperspectral image classification system based on self-supervised learning. The invention also provides a hyperspectral image classification method based on self-supervised learning. The method comprises the following steps: carrying out data processing and feature reconstruction on input hyperspectral image data; performing shallow feature extraction and long-distance dependence modeling on the preprocessed hyperspectral image data, and using parameters of a reconstruction path model to guide a classification process; and performing classification result output on the features extracted by the classification path model. According to the method, efficient feature extraction and classification of the hyperspectral image data are realized through self-supervised learning and the combined classification encoder, and the classification precision and generalization ability are improved, so that efficient and robust hyperspectral image classification is realized, and the method has very high practicability and generalizability.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral image classification, and in particular relates to a hyperspectral image classification system and method based on self-supervised learning. Background Art

[0002] In the field of hyperspectral image classification, hyperspectral imaging technology, as an advanced remote sensing method that can capture rich spectral information of ground objects, has been widely used in many fields such as environmental monitoring, agriculture, and mineral exploration. Each pixel of a hyperspectral image contains rich spectral bands that can reflect the physical and chemical properties of an object, so it has great potential in fine classification tasks. However, due to the high dimensionality and complex spectral structure of hyperspectral images, traditional classification methods are difficult to fully utilize this information, resulting in limited classification performance.

[0003] At present, the commonly used hyperspectral image classification methods include classical machine learning methods based on support vector machines (SVM), k-nearest neighbors (KNN), random forests, etc., as well as deep learning methods based on convolutional neural networks (CNN) and recurrent neural networks (RNN). Although these methods can achieve classification tasks to a certain extent, they have their own shortcomings in feature extraction. Taking CNN as an example, it mainly extracts spatial features through local convolution operations, and it is difficult to capture global spectral information, which makes it limited in performance when facing long-distance dependencies and complex spectral relationships; RNN has limited performance when processing spectral sequences and is difficult to model long-distance spectral dependencies; in addition, many traditional classification methods often only focus on the spectral information of hyperspectral images, while ignoring the spatial correlation between pixels, which makes the model insufficient in complex scenes.

[0004] In recent years, the Transformer network based on the self-attention mechanism has been widely used in image and natural language processing tasks due to its excellent global modeling ability, especially the Vision Transformer (ViT), which can capture the global semantic information in the image through the self-attention mechanism, without relying on the local perception of CNN. However, due to the essential difference between hyperspectral images and traditional RGB images, these Transformer-based models cannot fully capture the subtle spectral information and complex spatial information relationships in hyperspectral images; in addition, hyperspectral data is often affected by sensor noise, light scattering and atmospheric interference, which makes denoising an important task in hyperspectral classification. Although some models such as autoencoders (such as MAE and MAEST) have made progress in denoising, their denoising modules are mainly concentrated on the reconstruction path and have not been fully integrated into the classification process. The ability to capture spatial information is still insufficient, which limits the improvement of classification performance by denoising. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a hyperspectral image classification system based on self-supervised learning, aiming to solve the problems raised in the above background technology.

[0006] The embodiment of the present invention is implemented as follows: a hyperspectral image classification system based on self-supervised learning, comprising:

[0007] Reconstruction path model for data processing and feature reconstruction;

[0008] The classification path model is used for shallow feature extraction and long-distance dependency modeling, and the parameters of the reconstructed path model are used to guide the classification process;

[0009] The classification and output model is used to output the classification results of the features extracted by the classification path model.

[0010] Preferably, the reconstructed path model includes:

[0011] A mask unit is used to perform a mask operation on the input hyperspectral image data to cover up part of the data;

[0012] The ViT encoder unit is used to encode visible data and effectively extract global spatial and spectral features in the data;

[0013] The lightweight decoder unit is used to decode the features encoded by the ViT encoder unit and reconstruct the complete data.

[0014] Preferably, the classification path model includes:

[0015] CNN feature extractor unit, used to extract shallow semantic features of hyperspectral image data and capture local features of image data;

[0016] The spatial-spectral joint self-attention unit is used to use the parameters of the reconstruction path model to guide the classification process of the features. It uses the spatial self-attention mechanism and the spectral self-attention mechanism to model in the spatial and spectral dimensions respectively, captures the long-range dependencies in the data, and enhances the model's ability to capture complex spectral patterns and spatial features.

[0017] Preferably, the classification and output model comprises: a classifier unit for classifying the features extracted by the classification path model by calculating the probability value of each category, and the category with the highest probability value is output as the classification result as the final classification label of the input hyperspectral image.

[0018] Preferably, the classifier unit is a linear layer.

[0019] Another object of the embodiment of the present invention is to provide a hyperspectral image classification method based on self-supervised learning, comprising the following steps:

[0020] Perform data processing and feature reconstruction on the input hyperspectral image data;

[0021] The pre-processed hyperspectral image data is subjected to shallow feature extraction and long-distance dependency modeling, and the parameters of the reconstructed path model are used to guide the classification process;

[0022] The features extracted by the classification path model are classified and output as results.

[0023] Preferably, the step of performing data processing and feature reconstruction on the input hyperspectral image data is specifically:

[0024] Preprocess the input hyperspectral image data;

[0025] Perform masking operation on the preprocessed data to cover up part of the data;

[0026] ViT encoder is used to encode visible data and effectively extract global spatial and spectral features in the data;

[0027] The encoded features are decoded using a lightweight decoder to reconstruct the complete data.

[0028] Preferably, the step of performing shallow feature extraction and long-distance dependency modeling on the pre-processed hyperspectral image data and using the parameters of the reconstructed path model to guide the classification process is specifically:

[0029] The shallow semantic features of the hyperspectral image data are extracted through the CNN feature extractor to capture the local features of the image;

[0030] The features are passed to an encoder equipped with a spatial-spectral joint self-attention mechanism, and the pre-trained model parameters in the reconstruction path are used to guide the classification process. The spatial self-attention mechanism and the spectral self-attention mechanism are used to model in the spatial and spectral dimensions respectively to capture long-range dependencies in the data and enhance the model's ability to capture complex spectral patterns and spatial features.

[0031] Preferably, the step of outputting classification results of the features extracted by the classification path model is specifically as follows:

[0032] Pass the extracted features into a linear layer;

[0033] The linear layer performs classification by calculating the probability value of each category;

[0034] The class with the highest probability is selected as the classification result output, and the result is used as the final classification label of the input hyperspectral image.

[0035] A hyperspectral image classification system based on self-supervised learning provided by an embodiment of the present invention comprises a reconstruction path module and a classification path module. Efficient feature extraction and classification of hyperspectral image data are achieved through self-supervised learning and a joint classification encoder. Specifically, better spatial and spectral information fusion is achieved through a spatial-spectral joint self-attention mechanism and local feature extraction. Not only can the global spectral dependency of the hyperspectral image data be captured, but also the influence of noise on classification can be better handled. By adding a CNN feature extractor unit and a spatial-spectral joint self-attention unit to the classification path module, the classification accuracy and generalization ability of the system can be improved without relying on large-scale labeled data, thereby achieving efficient and robust hyperspectral image classification, which has strong practicality and generalizability. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flow chart of a classification method of a hyperspectral image classification system based on self-supervised learning provided in Example 1 of the present invention;

[0037] Figure 2 A schematic diagram of a CNN feature extractor unit provided in Embodiment 2 of the present invention;

[0038] Figure 3 A schematic diagram of a JCE module provided in Embodiment 2 of the present invention;

[0039] Figure 4 Classification diagram results of multiple classification systems provided in Example 3 of the present invention on the Indian Pines dataset. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0042] Example 1: A hyperspectral image classification system based on self-supervised learning, wherein the classification method is as follows: Figure 1 As shown, the specific steps include:

[0043] Data processing and feature reconstruction in the reconstruction path module: First, the input hyperspectral image data is preprocessed, and then a mask operation is performed through the mask unit to cover up part of the data; then, the visible data is encoded using the standard VisionTransformer (ViT) encoder unit. The ViT encoder unit is based on a multi-head self-attention mechanism and can effectively extract global spatial and spectral features from the data; then, the lightweight decoder unit decodes the encoded features and reconstructs the complete data. Through this self-supervised learning reconstruction task, the module learns the latent spatial-spectral structural characteristics of the data without relying on labels and improves robustness to noise;

[0044] Shallow feature extraction and long-distance dependency modeling in the classification path module: First, the shallow semantic features of the hyperspectral image data are extracted through the convolutional neural network (CNN) feature extractor unit. The extractor unit mainly captures local features of the image, such as edges, textures, and other information. Since the characteristics of hyperspectral images are high dimensions and rich spectral information, the CNN feature extractor unit can effectively reduce the feature complexity and extract shallow features of the image while maintaining spatial information; then, these features are passed to an encoder unit equipped with a spatial-spectral joint self-attention mechanism. The encoder unit uses the pre-trained parameters in the reconstruction path module to guide the classification process, and uses the spatial self-attention mechanism and the spectral self-attention mechanism to model in the spatial and spectral dimensions respectively, thereby capturing long-distance dependencies in the data and enhancing the module's ability to capture complex spectral patterns and spatial features;

[0045] Classification and output: After the encoder of the classification path module completes the modeling of spatial-spectral features, the extracted features will be passed to a linear layer, which acts as a classifier unit to achieve classification by calculating the probability value of each category; finally, the category with the highest probability is selected as the classification result output, and the result is used as the system's final classification label for the input hyperspectral image.

[0046] Example 2: Given a hyperspectral image, the size of the image is H×W×B, where H represents the height of the image, W represents the width of the image, and B represents the number of bands of the image. The classification method specifically includes the following steps:

[0047] (1) Preprocessing the hyperspectral image: Instead of directly using the entire image as input, a group spectral embedding (GSE) method is used to preprocess the hyperspectral image. The feature embedding in the standard ViT can be expressed as:

[0048] n=vm (1)

[0049] in, represents the trainable spectral weights, is the output feature after linear transformation. The GSE method is improved on this basis, not only considering a single band, but also including the feature embedding of adjacent bands. Therefore, the feature embedding can be expressed as:

[0050] N=VM (2)

[0051] in, q represents the number of adjacent bands, and V is the trainable weight, which is continuously adjusted and optimized through model learning. It can be expressed as:

[0052] M=[m1,…,m i …,m b ] (3)

[0053] in, This means that the current band m i It forms a group with its q adjacent bands (including itself) and converts the m at the corresponding position i Replaced with a one-dimensional vector, in order to make full use of spatial information, the image block is used as input, and the size of each image block p is h×w. After expansion, m i The size becomes therefore

[0054] (2) In the reconstruction path module, in order to perform mask operation on part of the data, an improved method MGSE is adopted, which is expressed as:

[0055]

[0056] Among them, the function Represents a mask function that randomly masks d (0≤d≤1) groups with a ratio of , and the encoder only processes the remaining visible groups. 1-d represents the ratio of unmasked groups (e.g. 25%). The masked groups will be excluded, which significantly reduces the computational and memory costs. Similar to the standard ViT, the encoder embeds these groups by adding linear projections of position embeddings and processes the data through the reconstruction path module;

[0057] (3) Input the data preprocessed by MGSE into the reconstruction path:

[0058] The preprocessed images are input into the standard ViT encoder and decoder for pre-training, and the trained model parameters are applied to the classification path module to guide the classification task;

[0059] (4) Input the data preprocessed by GSE into the classification path module:

[0060] The hyperspectral image data is preprocessed using the method in (1). In order to effectively capture the local semantic information in the hyperspectral image, a simple but efficient CNN-based feature extractor is designed. The extractor is integrated at the beginning of the classification path module to perform preprocessing before the feature map is input into the encoder. Considering the limited number of training samples in the hyperspectral image classification task, a minimalist design is adopted, using only a small number of convolutional blocks while still achieving good results.

[0061] like Figure 2 As shown in the figure, each convolution layer is followed by a batch normalization (BN) layer and a rectified linear unit (ReLU). The convolution kernel size of the first three layers is set to 1 to ensure that the size of the feature map remains unchanged, thereby maximizing the extraction of spatial information. Before the fourth convolution layer, the feature maps of the first three layers are concatenated, and then a convolution layer with a convolution kernel size of 3 is used for feature fusion. This step is crucial for capturing spectral domain information while maintaining the original size of the feature map, so that the feature map is restored to its original size before entering the CNN feature extractor.

[0062] (5) To address the limitations in capturing spatial information and enhance the interaction between spatial and spectral dimensions, the JCE module is introduced, e.g. Figure 3 As shown, JCE (Joint Classification Encoder) is one of the units in the classification path module and is also a core unit designed for hyperspectral image classification tasks. It is based on the model weight parameter output of the reconstruction path and requires complete and unmasked training data as input. These data need to be grouped and embedded through the GSE module before they can be processed by JCE:

[0063] This module uses two self-attention mechanisms: spectral self-attention mechanism and spatial self-attention mechanism. Given a sequence, JCE can be expressed as:

[0064] S HOE =SPE(SPA(S) T ) T (5)

[0065] Among them, SPA represents the spatial self-attention mechanism, SPE represents the spectral self-attention mechanism, and T represents the transposition operation. In the Transformer architecture, three weight matrices need to be obtained through linear transformation The spatial self-attention mechanism can be expressed as:

[0066]

[0067] in, represents matrix multiplication, d a Used to stabilize the gradient, Softmax(·) represents the softmax function;

[0068] After being processed by the spatial attention module, After that, the data is passed to the spectral attention unit, which needs to be transposed, i.e. The spectral self-attention mechanism is similar to the spatial self-attention mechanism and can be expressed as:

[0069]

[0070] Similarly, the sizes of the three weight matrices are After applying these two attention mechanisms, we need to re-attention S″ HCE Perform a transposition operation with a size of (x, y). After these operations, the module can effectively capture the global dependencies between different tags, thereby improving classification accuracy;

[0071] In addition, recognizing the importance of cross-layer feature fusion, the JCE module introduces a residual connection similar to ResNet. Given the limited number of hyperspectral image training samples and the small number of network layers, a residual-like connection is applied after every two encoder layers to enhance feature fusion. This method improves feature extraction capabilities and accelerates the convergence of the model.

[0072] (6) The data processed by the classification path is input into the classifier, and the linear layer is used as the classifier to achieve classification by calculating the probability value of each category. Finally, the category with the highest probability is selected as the classification result output, and the result is used as the final classification label of the system for the input hyperspectral image.

[0073] Example 3: Comparison of classification diagrams of multiple classification systems including Example 1 on the IndianPines dataset, the results are as follows: Figure 4 As shown, according to Figure 4 It can be seen that due to the lack of a local information extraction module, ViT generates more noise, and the results of its classification model present scattered and incoherent classification points. In contrast, a direct comparison of the classification graphs of MAEST and Example 1 systems shows that Example 1 processes the boundary area more smoothly, thanks to the introduction of the CNN feature extractor. In addition, compared with advanced Transformer models such as SpectralFormer and MorphFormer, Example 1 shows superior results in modeling long-distance pixel blocks, highlighting the effectiveness of the JCE module.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A hyperspectral image classification system based on self-supervised learning, characterized in that: include: Reconstruction path model for data processing and feature reconstruction; The classification path model is used for shallow feature extraction and long-distance dependency modeling, and the parameters of the reconstructed path model are used to guide the classification process; The classification and output model is used to output the classification results of the features extracted by the classification path model.

2. The hyperspectral image classification system based on self-supervised learning according to claim 1, characterized in that: The reconstruction path model includes: A mask unit is used to perform a mask operation on the input hyperspectral image data to cover up part of the data; The ViT encoder unit is used to encode visible data and effectively extract global spatial and spectral features in the data; The lightweight decoder unit is used to decode the features encoded by the ViT encoder unit and reconstruct the complete data.

3. The hyperspectral image classification system based on self-supervised learning according to claim 1, characterized in that: The classification path model includes: CNN feature extractor unit, used to extract shallow semantic features of hyperspectral image data and capture local features of image data; The spatial-spectral joint self-attention unit is used to use the parameters of the reconstruction path model to guide the classification process of the features. It uses the spatial self-attention mechanism and the spectral self-attention mechanism to model in the spatial and spectral dimensions respectively, captures the long-range dependencies in the data, and enhances the model's ability to capture complex spectral patterns and spatial features.

4. The hyperspectral image classification system based on self-supervised learning according to claim 1, characterized in that: The classification and output model includes: a classifier unit, which is used to classify the features extracted by the classification path model by calculating the probability value of each category, and the category with the highest probability value is output as the classification result as the final classification label of the input hyperspectral image.

5. The hyperspectral image classification system based on self-supervised learning according to claim 4, characterized in that: The classifier unit is a linear layer.

6. A hyperspectral image classification method based on self-supervised learning, characterized in that: The following steps are involved: Perform data processing and feature reconstruction on the input hyperspectral image data; The pre-processed hyperspectral image data is subjected to shallow feature extraction and long-distance dependency modeling, and the parameters of the reconstructed path model are used to guide the classification process; The features extracted by the classification path model are classified and output as results.

7. The hyperspectral image classification method based on self-supervised learning according to claim 6 is characterized in that: The step of performing data processing and feature reconstruction on the input hyperspectral image data is specifically as follows: Preprocess the input hyperspectral image data; Perform masking operation on the preprocessed data to cover up part of the data; ViT encoder is used to encode visible data and effectively extract global spatial and spectral features in the data; The encoded features are decoded using a lightweight decoder to reconstruct the complete data.

8. The hyperspectral image classification method based on self-supervised learning according to claim 7 is characterized in that: The steps of performing shallow feature extraction and long-distance dependency modeling on the pre-processed hyperspectral image data and using the parameters of the reconstructed path model to guide the classification process are specifically as follows: The shallow semantic features of the hyperspectral image data are extracted through the CNN feature extractor to capture the local features of the image; The features are passed to an encoder equipped with a spatial-spectral joint self-attention mechanism, and the pre-trained model parameters in the reconstruction path are used to guide the classification process. The spatial self-attention mechanism and the spectral self-attention mechanism are used to model in the spatial and spectral dimensions respectively to capture long-range dependencies in the data and enhance the model's ability to capture complex spectral patterns and spatial features.

9. The hyperspectral image classification method based on self-supervised learning according to claim 8, characterized in that: The step of outputting the classification result of the features extracted by the classification path model is specifically as follows: Pass the extracted features into the linear layer; The linear layer performs classification by calculating the probability value of each category; The class with the highest probability is selected as the classification result output, and the result is used as the final classification label of the input hyperspectral image.

Images