Method for realizing remote sensing hyperspectral image classification based on deep learning
Through the residual depth of deep learning, the convolution model can be separated and the improved Transformer architecture, the multi-scale features of high-spectral images are extracted, which solves the problem of insufficient classification accuracy in the prior art and achieves higher classification accuracy.
Patent Information
- Application Number
- CN202411026359.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-08
AI Technical Summary
The existing hyperspectral image classification methods have deteriorated performance when processing complex data and failed to make full use of multi-scale information, resulting in insufficient classification accuracy.
Using a deep learning-based residual depth separable convolution model combined with an improved Transformer architecture, the spectral and spatial features of high-spectral images are extracted through 3D depth separable convolution modules, extrusion-excitation modules and residual connections, and through multi-heads Self-attention mechanism and fully connected layer enable more accurate classification.
The multi-scale features of hyperspectral images are effectively extracted, improving the accuracy of pixel-level classification, and achieving state-of-the-art classification performance on multiple public data sets.
Smart Images

Figure CN118918377A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing, and in particular relates to a method for realizing remote sensing hyperspectral image classification based on deep learning. Background Art
[0002] Hyperspectral imagery (HSI) has become an important tool in remote sensing and earth observation, with high spectral resolution and fine spatial details. Hyperspectral imaging technology can capture hundreds of continuous narrow spectral bands across the entire electromagnetic spectrum, allowing accurate identification and detection of materials even if their spectral characteristics are slightly different. The fusion of spectral and spatial information in HSI data has generated a large number of applications, including land cover classification, precision agriculture, target identification, and biodiversity conservation. With the continuous development of spectral imaging technology, it has become increasingly urgent to fully tap the potential of HSI, and one of the important application areas is land cover classification.
[0003] Initially, some statistical methods such as principal component analysis (PCA), factor analysis (FA) and independent component analysis (ICA) were used to reduce the dimension and extract key spectral information, or signal processing methods such as discrete cosine transform (DCT), discrete wavelet transform (DWT), linear discriminant analysis (LDA) and nonparametric discriminant analysis (NDA) were used to extract spectrum information to achieve HSI classification. However, these methods only study the statistical properties and mathematical optimization of the data, which may cause the HSI data to lose its original physical and chemical characteristics during the classification process, which may lead to the loss of information in key frequency bands. In addition, these methods also ignore the spatial information in HSI, resulting in limited classification accuracy.
[0004] With the rise of machine learning, methods such as support vector machines (SVM) with nonlinear kernels, random forests (RF), and K-NearestNeighbors (KNN) are widely used for HSI classification. However, these methods usually achieve good results when dealing with smaller and less complex data. As the size and complexity of training data increase, their performance tends to plateau or decrease. This is because traditional machine learning methods have inherent limitations in data fitting and representation capabilities.
[0005] In 2012, Andrew Ng's team used deep learning to achieve great success in image processing, and scholars at home and abroad began to use deep learning to conduct extensive research on HSI classification. Y. Chen, Hamida, Charmisha and others used convolutional neural network CNN to learn the spectral and spatial information in HSI and realize remote sensing land cover classification. Subsequently, Z. Meng, Z. Xue and Paoletti and others improved CNN and introduced residual network, which alleviated the information loss and gradient descent problems of traditional CNN and improved the network's ability to classify HSI. In addition to CNN-based methods, some other networks with good performance, such as recurrent neural network (RNN), long short-term memory network (LSTM) and generative adversarial network (GAN), have also been used for HSI classification tasks.
[0006] However, CNN introduces too many layers, which increases the model complexity and computational cost. The main problem of GAN is the inconsistent label assignment on spectra with different spatial variations and the limited generalization ability of new spectral features. In addition, RNN is not suitable for training models at the same time because HSI usually contains many samples, which limits the performance of the classifier.
[0007] The Transformer architecture has shown strong performance in extracting long-term dependencies, limited generalization of spectral features, spectral sequence properties, and high-level semantic features. L.Sun et al. designed a spectral spatial feature tokenization transformer (SSFTT), which integrates the hierarchical CNN module for low-level feature extraction with the transformer encoder module to effectively capture and represent high-level semantic features. Roy used the spectral-spatial morphological attention transformer (MorphFormer) to combine spectral and spatial morphological convolution with self-attention to enhance the interaction between the structure and shape information of the image tag. However, the above methods do not fully consider the use of multi-scale information, resulting in insufficient accuracy of image classification. Summary of the invention
[0008] The purpose of the present invention is to provide a method for remote sensing hyperspectral image classification based on deep learning.
[0009] The technical solution of the present invention is as follows:
[0010] A method for remote sensing hyperspectral image classification based on deep learning, comprising:
[0011] Acquire hyperspectral image data;
[0012] Inputting the hyperspectral data into a residual deep separable convolution model to extract data and obtain an output result;
[0013] The output results are classified to obtain the classification results of the ground cover of the hyperspectral image data.
[0014] Furthermore, the residual depthwise separable convolution model consists of a 3D depthwise separable convolution module, a squeeze-excitation module, a residual connection and an improved transformer.
[0015] Furthermore, the improved transformer consists of a multi-head self-attention block, a multi-layer perceptron layer and two normalization layers.
[0016] Furthermore, the hyperspectral data is input into the residual deep separable convolution model for data extraction, and the output result is specifically:
[0017] Using hyperspectral image data as input, extract three-dimensional patches;
[0018] The patch is passed through a 3D deep separable convolution module to extract the spectral and spatial features of the hyperspectral image;
[0019] The squeeze-excitation module recalibrates the extracted spatial features to assign different weights according to the importance of each spectral channel;
[0020] Then, the channel dependency is fully captured through sigmoid and ReLU activation functions to obtain the weight of each spectral channel;
[0021] Finally, dot product is performed on the obtained spectral channel weights and the input data, and the result is output;
[0022] The output results are reduced in dimension through the improved transformer, and then the reduced-dimensional results are marked;
[0023] Finally, the embedded semantic tag sequence is imported into the improved transformer module to obtain the output result.
[0024] Furthermore, the output results of the transformer module improved by the fully connected layer are classified to obtain the classification results of the ground cover of the hyperspectral image data.
[0025] Furthermore, the step of extracting the spectral and spatial features of the hyperspectral image by passing the patch through a 3D depth-separable convolution module is as follows:
[0026] The depthwise separable aggregation module imports p patches each time, and the depthwise convolution uses a convolution kernel to convolve the patches of these p channels. The calculated value at the jth special spatial position in the i-th layer is:
[0027]
[0028] Among them, the convolution kernel at channel c is a p×K α ×K β ×K γ Tensor; K α , K β and K γ are the width, height and depth of the convolution kernel, p is the number of input channels, b is the bias term, and then all features are merged to form the output result;
[0029] The point convolution layer operates on the output results and fuses the information of each channel.
[0030] Furthermore, the results after dimensionality reduction are marked as follows:
[0031] First, flatten the data to get T flat , which is a one-dimensional vector with a length of D = d × s′ 2 ;
[0032] Then a learnable linear transformation is performed on the one-dimensional vector of each patch to map it to a higher-dimensional embedding space, thereby helping the model capture more complex features T i =T flat ·W proj +b proj ,in, represents the weighting matrix, b proj is the bias vector, h is the embedding dimension, is the embedding of the i-th patch;
[0033] To add the position information of each marker in the sequence, the CLS marker is connected to the HSIpatch marker, resulting in T in =T cls ⊙T flat′ ; Among them, T in The dimension is N is the number of patches, T cls is a randomly initialized learnable tensor, T in is a sequence of embedded semantic tags.
[0034] Compared with the prior art, the present invention has the following advantages:
[0035] First, the residual deep separable convolutional model proposed in this paper combines three-dimensional residual calculation, SE block and attention mechanism. This model can effectively extract rich spectral and spatial features from hyperspectral image data and can be used for pixel-level classification of multispectral images.
[0036] Secondly, the present invention also improves the calculation method of the attention score in the transformer module to better capture the dependencies between pixels and achieve better hyperspectral image classification results.
[0037] Finally, experiments are conducted on three public datasets and the proposed framework achieves state-of-the-art results in pixel-level classification of multispectral images. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The accompanying drawings generally illustrate various embodiments by way of example and not limitation, and together with the description and claims, serve to illustrate the embodiments of the invention. Where appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present apparatus or method.
[0039] Figure 1 Flowchart of RDSSEFormer designed for the present invention.
[0040] Figure 2 This is a structural diagram of the 3D DSC used in the present invention.
[0041] Figure 3 This is a structural diagram of the SE module used in the present invention.
[0042] Figure 4 The residual connection established by the present invention.
[0043] Figure 5 This is the improved ViT structure of the present invention.
[0044] Figure 6 Flowchart for improved Q, K and V solutions
[0045] Figure 7 Classification map of the Houston 2013 dataset. The OA results are in brackets: (a) pseudo-color image; (b) ground truth; (c) SVM (75.25%); (d) RF (78.76%); (e) 3DCNN (94.65%); (f) HybridSN (94.05%); (g) DHS (98.80%); (h) HResNetAM (96.22%); (i) Morphspectral (98.51%); (j) SpetralFormer (97.61%); (k) SSFTT (98.92%); (l) Ours (99.23%).
[0046] Figure 8Classification map of the University of Pavia dataset. OA results in brackets: (a) pseudo-color image; (b) ground truth; (c) SVM (78.89%); (d) RF (82.10%); (e) 3DCNN (97.55%); (f) HybridSN (97.25); (g) DHS (98.16%); (h) HResNetAM (99.14%); (i) Morphspectral (98.19%); (g) SpetralFormer (98.64%); (k) SSFTT (99.21%); (l) Ours (99.90%).
[0047] Fig. 9 Classification map of Indian pine dataset. OA results in brackets: (a) pseudo-color image; (b) ground truth; (c) SVM (72.36%); (d) RF (74.25%); (e) 3DCNN (94.33%); (f) HybridSN (92.32); (g) DHS (96.62%); (h) HResNetAM (97.23%); (i) Morphspectral (98.21%); (j) SpetralFormer (95.25%); (k) SSFTT (98.07%); (l) Ours (99.90%).
[0048] Fig.10 2D graphical visualization of features extracted by RDSSEFormer via t-SNE; (a) Pava University; (b) Houston 2013; (c) Indianapolis. DETAILED DESCRIPTION
[0049] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0050] The purpose of this invention is to propose an integrated network architecture based on 3D depthwise separable convolution (DSC), squeeze-excitation (SE) module, residual connection and improved transformer. This architecture is residual depthwise separable convolution-SEtransformer (RDSSEFormer), such as Figure 1As shown in Figure 1, it is used to achieve remote sensing HSI land cover classification. The hyperspectral data first passes through the DSC-SE module, which is a learnable module that integrates 3D deep separable convolution with SE blocks, that is, SE is embedded after the convolution module to utilize the spatial and spectral information of the HSI data while reducing the interference of non-critical redundant spatial and spectral features. In addition, a residual connection is added between the two DSC-SE modules to improve the classification accuracy. This overall structure is called residual DSC-SE (RDS-SE). Since the amount of hyperspectral data is relatively small compared to deep learning, based on this feature, we improve the Vison Transformer (ViT) connected after the RDS-SE module. It splits the features extracted by RDS-SE into multiple patches, each of which is then converted into a feature vector, and captures the relationship between patches through a self-attention mechanism. The improved ViT can improve the clarity of the attention score distribution, thereby better adapting to small datasets. Finally, a fully connected layer is used to identify the extracted features and assign a land cover category to each pixel in the hyperspectral.
[0051] The technical solution for implementing the present invention is as follows:
[0052] RDSSEFormer design method, the specific steps are:
[0053] Step 101: Use As the original HSI data cube, m×n represents the HSI spatial size and l represents the number of spectral bands. Subsequently, three-dimensional patch extraction is performed on the HSI data. Created by H, s×s represents the window size.
[0054] Step 102: Figure 1 As shown, the patch first passes through DSC. The DSC structure is detailed in Figure 2 The patch first undergoes depth convolution and then point convolution to obtain the spectral and spatial features of HSI.
[0055] The calculation method of DSC is as follows:
[0056] Step 201, the deep convolution network imports p patches each time, that is, the input data size is p×s×s×l. The deep convolution uses a convolution kernel to convolve the patches of these p channels, and the calculated value at the j-th special spatial position (c, α, β, γ) in the i-th layer is: Convolution kernel at channel c is a p×K α ×K β ×K γ Tensor. K α , K βand K γ are the width, height and depth of the convolution kernel, p is the number of input channels, b is the bias term. Then all features are combined to form the output result
[0057] Step 202: The point convolution layer operates on the result of the depth convolution to fuse the information of each channel. It uses a 1×1×1 convolution kernel to convolve the channels after the depth convolution, that is, it directly acts on all channels of each pixel, and the result is: Convolution Kernel The size of is p×p′, which converts the input channel number p into the output channel number p′. The size of the output result is p′×s′×s′×l′, which belongs to the c′th channel.
[0058] Step 103: Figure 3 As shown in Figure 2, the SE module recalibrates the features extracted by DSC and assigns different weights to each spectral channel according to its importance. The SE module first “squeezes” F sq () operation, retaining spectral features through global average pooling The size is l′×1×1×1, which compresses the information of each sample in dimensions other than the spectral dimension into a single value while keeping the spectral dimension l′ unchanged.
[0059] Step 104, the SE module then "stimulates" F ex () operation, using sigmoid and ReLU activation functions to fully capture channel dependencies and obtain the weight Y of each spectral channel ex =F ex (Y sq , W) = σ(W2δ(W1Y sq )), where δ refers to the ReLU function, σ is the sigmoid function, W1 is a dimensionality increase layer, and W2 is a dimensionality reduction layer.
[0060] Step 105: The SE module finally combines the obtained spectral channel weights with the input data Y′ c′,α,β,γ Dot product: Y Ds-SE =Y DSC Y ex Among them, the output result Y DS-SE It has the same size as the input data, and each spectral channel is scaled according to its corresponding weight.
[0061] Step 106: Figure 4 As shown in Figure 2, we establish a residual connection between two DS-SE layers. The resulting module is called RDS-SE, and the output is Y RDS-SE=σ(Y+F(Y)), where F() refers to the calculation function of DS-SE. The output size is p′×s′×s′×l′.
[0062] Step 107: Figure 4 As shown in Figure 2, we propose an improved ViT to process small-scale datasets such as the HSI dataset. We first transform the data Y output by the RDS-SE module into RDs-SE Perform two-dimensional convolutional layer downsampling to obtain Y cube , the size is d×s′×s′, where d is the number of downsampled output channels and s′ is the length and width of the patch.
[0063] Step 108: tokenize the result after dimensionality reduction.
[0064] The calculation method of Tokenize is as follows:
[0065] Step 301: The first step is to flatten the data to obtain T flat , which is a one-dimensional vector with a length of D = d × s′ 2 .
[0066] Step 302: Then, a learnable linear transformation is performed on the one-dimensional vector of each patch to map it to a higher-dimensional embedding space, thereby helping the model capture more complex features T i =T flat ·W proj +b proj ,in, represents the weighting matrix, b proj is the bias vector, h is the embedding dimension, is the embedding of the ith patch.
[0067] Step 303: To add the position information of each marker in the sequence, Figure 5 As shown, connect the CLS tag to the HSIpatch tag to obtain T in =T cls ⊙T flat ′. Among them, T in The dimension is N is the number of patches, T cls is a randomly initialized learnable tensor. T in is an embedded sequence of semantic tags that contains information about the content of the patch and its location in the image, providing the Transformer model with the ability to handle both sequential and spatial structures.
[0068] Step 109: Import the embedded semantic tag sequence into the improved transformer module, which consists of Figure 5As shown in Figure 1, there is a multi-head self-attention (MSA) block, a multi-layer perceptron (MLP) layer, and two normalization layers (LN). Figure 6 The present invention adopts diagonal masking to eliminate self-labeling relationships and learnable softmax parameters to change the calculation method of the attention mechanism, so that ViT is more suitable for small datasets like HSI. Diagonal masking is to set the values on the diagonal of the matrix multiplied by the query value Q and the key value K to zero. The learnable softmax parameter, the attention score obtained is As a learnable parameter, τ replaces the high temperature parameter in the traditional attention score calculation formula.
[0069] Step 110: Finally, a fully connected layer is used to classify the output result of ViT to obtain the classification result of the ground coverage of the HSI data.
[0070] Embodiment 1:
[0071] Figure 7 , Figure 8 and Fig. 9 The method of the present invention is presented in the classification maps obtained on three hyperspectral remote sensing datasets, and compared with the existing state-of-the-art methods. Compared with the state-of-the-art methods, the classification results of adjacent pixels in the hyperspectral remote sensing images by the method of the present invention are closer to the ground truth. The network of the present invention effectively utilizes the unique features and multi-scale information of hyperspectral data, and can better present texture and edge details.
[0072] Fig.10 The features extracted by the model are analyzed using the t-SNE method. In all three datasets, samples from similar classes are clustered together, with minimal intra-class variance.
[0073] Although the specific embodiments of the present invention are described in conjunction with the accompanying drawings, it is apparent to those skilled in the art that several modifications, substitutions and improvements may be made without departing from the principles of the present invention, and these should also be considered to fall within the scope of protection of the present invention.
[0074] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A method for remote sensing hyperspectral image classification based on deep learning, It is characterized in that include: Acquire hyperspectral image data; Inputting the hyperspectral data into a residual deep separable convolution model to extract data and obtain an output result; The output results are classified to obtain the classification results of the ground cover of the hyperspectral image data.
2. The method for realizing remote sensing hyperspectral image classification based on deep learning according to claim 1, It is characterized in that The residual depth-wise separable convolution model consists of a 3D depth-wise separable convolution module, a squeeze-excitation module, a residual connection and an improved transformer.
3. The method for realizing remote sensing hyperspectral image classification based on deep learning according to claim 1, It is characterized in that The improved transformer consists of a multi-head self-attention block, a multi-layer perceptron layer and two normalization layers.
4. The method for realizing remote sensing hyperspectral image classification based on deep learning according to claim 1, It is characterized in that The hyperspectral data is input into the residual deep separable convolution model for data extraction, and the output result is specifically: Using hyperspectral image data as input, extract three-dimensional patches; The patch is passed through a 3D deep separable convolution module to extract the spectral and spatial features of the hyperspectral image; The squeeze-excitation module recalibrates the extracted spatial features to assign different weights according to the importance of each spectral channel; Then, the channel dependency is fully captured through sigmoid and ReLU activation functions to obtain the weight of each spectral channel; Finally, dot product is performed on the obtained spectral channel weights and the input data, and the result is output; The output results are reduced in dimension through the improved transformer, and then the reduced-dimensional results are marked; Finally, the embedded semantic tag sequence is imported into the improved transformer module to obtain the output result.
5. The method for realizing remote sensing hyperspectral image classification based on deep learning according to claim 1, It is characterized in that The output results of the transformer module improved by the fully connected layer are classified to obtain the classification results of the ground cover of the hyperspectral image data.
6. The method for remote sensing hyperspectral image classification based on deep learning according to claim 4, It is characterized in that The specific method of extracting the spectral and spatial features of the hyperspectral image by passing the patch through the 3D deep separable convolution module is as follows: The depthwise separable aggregation module imports p patches each time, and the depthwise convolution uses a convolution kernel to convolve the patches of these p channels. The calculated value at the jth special spatial position in the i-th layer is: Among them, the convolution kernel at channel c is a p×K α ×K β ×K γ Tensor; K α , K β and K γ are the width, height and depth of the convolution kernel, p is the number of input channels, b is the bias term, and then all features are merged to form the output result; The point convolution layer operates on the output results and fuses the information of each channel.
7. The method for realizing remote sensing hyperspectral image classification based on deep learning according to claim 4, It is characterized in that The results after dimensionality reduction are marked as follows: First, flatten the data to get T flat , which is a one-dimensional vector with a length of D = d × s′ 2 ; Then a learnable linear transformation is performed on the one-dimensional vector of each patch to map it to a higher-dimensional embedding space, thereby helping the model capture more complex features T i =T flat ·W proj +b proj ,in, represents the weighting matrix, b proj is the bias vector, h is the embedding dimension, is the embedding of the i-th patch; To add the position information of each marker in the sequence, the CLS marker is connected to the HSIpatch marker, resulting in T in =T cls ⊙T flat ′; where T in The dimension is N is the number of patches, T cls is a randomly initialized learnable tensor, T in is a sequence of embedded semantic tags.
Citation Information
Patent Citations
Depth separable convolution hyperspectral image classification method based on residual connection
CN112115972A
Hyperspectral image classification method based on depth spectral space inverse residual network
CN113935433A
Flue-cured tobacco leaf image grading method combining CNN and Transform
CN115439690A
KPIs missing value interpolation method based on improved Transform and masking training
CN115658672A
Picture classification method based on probability graph and ViT model
CN115953617A
