Hyperspectral image classification method based on improved swin-transformer network
By using an improved Swin-Transformer network, combined with a spatial spectral recombination module and a cross-layer fusion module, the problem of insufficient information utilization in hyperspectral image classification is solved, achieving higher classification accuracy and better generalization ability.
Patent Information
- Application Number
- CN202211641963.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-20
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2042-12-20
AI Technical Summary
Existing hyperspectral image classification methods struggle to effectively utilize the rich spectral information in hyperspectral images, resulting in insufficient classification accuracy. This is particularly evident when there are numerous categories with similar spectral features, leading to significant performance bottlenecks.
An improved Swin-Transformer network, combined with a spatial spectral reconstruction module and a cross-layer fusion module, is used to achieve pixel-by-pixel classification of hyperspectral images through data preprocessing and network improvement. The spatial spectral reconstruction module reduces high-dimensional data to two-dimensional data and makes non-adjacent pixels closer together, while the cross-layer fusion module avoids information loss.
It improved the accuracy of hyperspectral image classification, enhanced the network's efficiency in mining spectral information, reduced information loss, and achieved higher classification accuracy.
Smart Images
Figure CN116051886B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of remote sensing hyperspectral image processing, and particularly relates to a hyperspectral image classification method based on an improved Swin-Transformer network. BACKGROUND
[0002] Hyperspectral remote sensing technology provides an innovative way for studying spatial phenomena and processes in the earth's surface and atmosphere based on the data fused with spectral and spatial information obtained by imaging hyperspectral sensor systems. If the hyperspectral image data has sufficient quality, the target outside a certain distance can be observed and analyzed at a level close to laboratory spectral analysis in theory. At present, imaging hyperspectral remote sensing technology has been successfully applied to more and more disciplines and fields, including geology, ecology, forestry, snow and ice, soil, environment, hydrology, disaster management, urban monitoring, atmospheric research, agriculture, fishery, ocean and even national security. It can be seen that the analysis and research of hyperspectral images have far-reaching practical significance, and the accurate classification of the ground objects represented by the pixels in the hyperspectral images has important research value.
[0003] Generally speaking, each ground object will emit, reflect and absorb electromagnetic waves, so different ground objects will produce different reflection values in the same wavelength range, and multiple consecutive wave bands constitute the spectral curve of the ground object. Therefore, each pixel in the hyperspectral image can be considered as a high-dimensional data cube composed of many spectra, and the goal of hyperspectral image classification is to assign a corresponding class label to each pixel using the rich spectral information in the data cube. At present, many classification algorithms based on spectral information have been proposed. These pixel-by-pixel classification algorithms have the advantages of easy extension and calculation. However, these algorithms ignore the spatial continuity of ground object distribution, i.e. the spatial correlation between adjacent pixels, and only rely on the spectral information of hyperspectral data, so that the constructed classifier cannot well fit such high-dimensional data, which is called Hughes phenomenon. In addition, the same object different spectrum phenomenon also exists in hyperspectral data, i.e. the same ground object may produce different spectral curves due to different light intensities, shadows and other factors. In summary, these problems make the effect of pixel-by-pixel classification method unsatisfactory. Therefore, how to effectively solve the above problems becomes the key to improving the performance of hyperspectral image classification.
[0004] Hyperspectral image classification techniques mainly fall into two categories: traditional machine learning methods and deep learning methods. Traditional optimization methods such as principal component analysis and linear discriminant analysis have the characteristics of simplicity and easy implementation, but can only extract low-dimensional spectral and spatial feature information. However, traditional machine learning involves manual design to extract and learn features, and incorrect dimension reduction process can lose a lot of spectral information, resulting in only shallow feature information. Therefore, it is difficult for traditional methods to achieve high classification accuracy. In recent years, deep learning technology has shown its powerful feature extraction ability in various tasks, and compared with traditional methods, deep learning needs less artificial constraints to deeply mine data features, and the end-to-end method reduces the problem of information loss caused by data dimension reduction in preprocessing. Deep learning methods can fully mine the deep features of hyperspectral data, thereby achieving higher classification accuracy than traditional methods.
[0005] Although many deep learning methods have been able to obtain good classification results, they still have deficiencies in representing spectral information, mainly including the following two aspects: first, as the mainstream backbone architecture, CNN shows strong ability in extracting spatial structure information and local context information from HS images. However, CNN is difficult to capture sequence properties, especially long-term dependencies. This inevitably encounters a performance bottleneck in the hyperspectral image classification task, especially when the number of categories to be classified is large and the spectral features are very similar. Second, RNN is designed for sequence data, which learns spectral features from hyperspectral images in an ordered and cumulative manner. This mode is extremely dependent on the order of spectral bands and is prone to gradient disappearance, making it difficult to learn long-term dependencies. In summary, appropriate data preprocessing needs to be performed according to the characteristics of hyperspectral data, and the existing network needs to be improved to effectively utilize the rich spectral information in hyperspectral images to improve the classification accuracy. SUMMARY
[0006] In order to solve the problem that the existing hyperspectral image classification method cannot effectively utilize the rich spectral information in the hyperspectral image, the present application proposes to use the Swin-Transformer network to classify the hyperspectral image pixel by pixel. Since the hyperspectral image has a high dimension, the data is reduced, and a spatial and spectral recombination module is proposed. At the same time, the non-adjacent spectrum becomes adjacent in this process, and the pixels around the center pixel are also considered, thereby effectively reducing the difficulty of the network in mining spectral information.
[0007] In addition, in order to avoid information loss in the interlayer feedforward process, a cross-layer fusion module is proposed. By fusing the output of the current layer in the network with the output of the previous layer, the information is transmitted from the shallow layer to the deep layer, thereby avoiding the loss of effective information in the feedforward process.
[0008] To achieve the above object of the application, the technical scheme adopted by the application is as follows:
[0009] A hyperspectral image classification method based on an improved Swin-Transformer network, the specific steps are as follows:
[0010] S1, according to the characteristics of hyperspectral data, a spatial-spectral recombination module is proposed to preprocess the data.
[0011] The spatial-spectral recombination module is established, which changes the data of one pixel in the hyperspectral data from one-dimensional data to two-dimensional data, and moves the spectral bands of the transformed data, so that the originally non-adjacent spectral bands become adjacent and are considered together in the network learning process. In addition, other pixels adjacent to the pixel are also considered, and the additional information provided is used as auxiliary information to improve the classification accuracy of the central pixel. Finally, the processed data is output to the hyperspectral image classification network;
[0012] S2, improve the Swin-Transformer network, add a cross-layer fusion module in the network to avoid information loss in the interlayer feedforward process. By fusing the output of the current layer with the output of the previous layer in the network, the information is transmitted from the shallow layer to the deep layer, thereby avoiding the loss of effective information in the feedforward process;
[0013] The cross-layer fusion module is composed of a downsampling operation and a fully connected layer. The downsampling operation changes the size of the feature map, and the fully connected layer changes the channels of the feature map. Finally, the output of the residual connection is fused with the output of the basic block.
[0014] S3, insert the spatial-spectral recombination module into the improved Swin-Transformer network, and use the public hyperspectral image data as training data, the training process is as follows:
[0015] First, visualize the data, use different colors to represent different categories to be classified, select part of the pixel points as the training data of the Swin-Transformer model, and the remaining all labeled parts as test data (here, public data sets are used, on these data sets, the ground cover type to which each pixel in the hyperspectral image belongs has been labeled, for example, the ground cover type corresponding to the first pixel is forest). Then the training data is processed by the spatial-spectral recombination module and input into the improved Swin-Transformer model for multiple rounds of training. After the training is completed, the model is evaluated on the test set, and the hyperparameters of the model are fine-tuned and retrained. Finally, compare the experimental results of multiple groups, and get a model with strong robustness, good generalization ability and high precision.
[0016] Further, the processing flow of the spatial spectrum recombination module in S1 is as follows:
[0017] S11, the width and height of the recombined data are calculated according to the number of channels of the hyperspectral data. The formula is as follows:
[0018]
[0019] where C is the number of channels of the hyperspectral data.
[0020] S12, on the basis of the original data, the adjacent channels are moved, all channels are moved one unit to the right, and the last channel is moved to the first position. All channels are moved two units to the right, and the last two channels are moved to the first and second positions. Repeat the above operation, move the channels n times.
[0021] S13, the hyperspectral data after moving the channels is unfolded along the channel dimension to obtain two-dimensional data (the shape of the data is WxHx1), and finally a plurality of two-dimensional data are stacked in the channel dimension (the shape of the data is WxHxn).
[0022] S14, after introducing adjacent pixels and performing the above transformation, all data are stacked in the channel direction. Assuming that 4 adjacent pixels are introduced, the shape of the stacked data is WxHx4n.
[0023] Further, the working flow of the cross-layer fusion module in S2 is as follows:
[0024] Let the feature maps output by the network L-1 layer and L layer be and The output after processing by the cross-layer fusion module is where represents the fusion of the L-1 layer and the L layer, represents the network parameters that can be learned by the cross-layer fusion module.
[0025] Further, the improved Swin-Transformer model in S3 is trained in multiple rounds, and the flow is as follows:
[0026] S31, prepare the data set to be used for training.
[0027] S32, load the data set using dataloader.
[0028] S33, take batch size data from dataloader and input the improved Swin-Transformer model.
[0029] S34, input the prediction result of the model and the label into the loss function to calculate the loss.
[0030] S35, according to the calculated loss value, the weight of the network is updated by back propagation.
[0031] S36, the steps of S33 to S35 are iterated for epoch times to obtain a finally trained model.
[0032] Further, the hyperparameters include: batch size, epoch, initial learning rate, learning rate change ratio, learning rate adjustment strategy
[0033] Further, the hyperspectral image classification method is realized through the existing deep learning framework Pytorch and the corresponding programming library.
[0034] Further, the spatial spectrum recombination module can realize plug and play.
[0035] Compared with the prior art, the advantages of the present application are:
[0036] 1. The present application realizes a more accurate pixel-by-pixel classification of hyperspectral images by using an improved Swin-Transformer network, and achieves higher accuracy than previous methods.
[0037] 2. A plug-and-play module is proposed, which can effectively reduce the dimensionality of hyperspectral images and improve the efficiency of network mining spectral information.
[0038] 3. A cross-layer fusion module is proposed, which can effectively reduce the loss of effective information in the network during the feedforward process. BRIEF DESCRIPTION OF DRAWINGS
[0039] Figure 1 The framework diagram of the hyperspectral image classification method of the embodiment of the present application is shown in the figure.
[0040] Figure 2 The flowchart of the hyperspectral image classification method of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical scheme and advantages of the present application clearer, the following embodiments are described based on the drawings.
[0042] As shown in Figure 1 and 2 The hyperspectral image classification method based on the improved Swin-Transformer network, the specific steps are:
[0043] S1, according to the characteristics of hyperspectral data, a spatial spectrum recombination module is proposed to preprocess the data.
[0044] Through observation and analysis of hyperspectral image data, it can be seen that the hyperspectral data can be regarded as a high-dimensional data cube. For such data, the neural network model will be difficult to effectively utilize the rich spectral information in the hyperspectral data. In view of this problem, a spatial-spectral recombination method is proposed, which converts one-dimensional spectral vectors into two-dimensional feature maps, thereby realizing dimension reduction of the data. At the same time in this process, not only the originally non-adjacent pixels can become adjacent, but also the pixels adjacent to the center pixel are considered. Finally, the processed data is output to the hyperspectral image classification network;
[0045] S2, improve the Swin-Transformer network, add a cross-layer fusion module in the network to avoid information loss in the interlayer feedforward process. By fusing the output of the current layer in the network with the output of the previous layer, the information is transmitted from the shallow layer to the deep layer, thereby avoiding the loss of effective information in the feedforward process;
[0046] S3, insert the spatial-spectral recombination module into the improved Swin-Transformer network, and use the public hyperspectral image data as the training data. The training process is divided into three steps:
[0047] First, visualize the data, with different colors representing different classes to be classified. Manually select some pixels as training data for the model, and the remaining all labeled parts as test data. Then the training data is processed by the spatial-spectral recombination module and input into the improved Swin-Transformer model for multiple rounds of training. After training, the model is evaluated on the test set, and the hyperparameters of the model are fine-tuned and retrained. Finally, compare the experimental results of multiple groups to obtain a model with strong robustness, good generalization ability and high precision;
[0048] Finally, the present application uses overall accuracy (OA), average accuracy (AA) and Kappa coefficient as evaluation indexes to evaluate the classification performance of the model, and compares it with the traditional machine learning method and deep learning method that have been proposed. The methods participating in the comparison are: KNN, RF, SVM, 1-D CNN, 2-D CNN, RNN, miniGCN, ViT, SpectralFormer, SRF (ours).
[0049] Table 1: Performance of methods participating in comparison on test set
[0050]
[0051] The experimental results are shown in Table 1. It can be observed from the table that the improved Swin-Transformer network has the best performance in hyperspectral image classification, not only has the highest prediction accuracy for most classes, but also has the highest scores on OA, AA and Kappa three evaluation indexes.
[0052] Those skilled in the art will understand that the embodiments described herein are for the purpose of helping the reader understand the implementation method of the present application, and should be understood as the protection scope of the present application is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the protection scope of the present application.
Claims
1. A hyperspectral image classification method based on an improved Swin-Transformer network, comprising the following specific steps: S1. According to the characteristics of hyperspectral data, a spatial-spectral recombination module is proposed for data preprocessing; The spatial-spectral recombination module changes the data of one pixel in the hyperspectral data from one-dimensional data to two-dimensional data, and moves the spectral bands of the transformed data, so that the originally non-adjacent spectral bands become adjacent and are considered together in the network learning process; In addition, other pixels adjacent to the pixel are also considered, and the additional information provided is used as auxiliary information to improve the classification accuracy of the central pixel; Finally, the processed data is output to the hyperspectral image classification network; S2. Improve the Swin-Transformer network, add a cross-layer fusion module to the network to avoid information loss in the interlayer feedforward process; By fusing the output of the current layer with the output of the previous layer in the network, the information is transmitted from the shallow layer to the deep layer, thereby avoiding the loss of effective information in the feedforward process; The cross-layer fusion module is composed of a downsampling operation and a fully connected layer; The downsampling operation changes the size of the feature map, and the fully connected layer changes the channels of the feature map; Finally, the output of the residual connection is fused with the output of the basic block; S3. Insert the spatial-spectral recombination module into the improved Swin-Transformer network, and use the public hyperspectral image data as training data, the training process is as follows: First, visualize the data, use different colors to represent different classes to be classified, select part of the pixel points as the training data of the Swin-Transformer model, and the remaining all labeled parts as test data; Then, after the training data is processed by the spatial-spectral recombination module, it is input into the improved Swin-Transformer model for multiple rounds of training, and after the training is completed, the model is evaluated on the test set, and the hyperparameters of the model are fine-tuned and retrained; Finally, compare the experimental results of multiple groups, and get a model with strong robustness, good generalization ability and high precision.
2. The hyperspectral image classification method based on the improved Swin-Transformer network according to claim 1, characterized in that: The processing flow of the spatial-spectral recombination module in S1 is as follows: S11. Calculate the width and height of the recombined data according to the channel number of the hyperspectral data; The formula is as follows: Where C is the channel number of the hyperspectral data; S12. On the basis of the original data, move the adjacent channels, move all channels to the right by one unit, and move the last channel to the first position; Move all channels to the right by two units, and move the last two channels to the first and second positions; Repeat the above operation to move the channels n times; S13. Expand the hyperspectral data after channel movement along the channel dimension to obtain two-dimensional data with a shape of WxHx1, and finally stack multiple two-dimensional data in the channel dimension, and the data shape is WxHxn; S14. By introducing adjacent pixels and performing the above transformation, stack all data in the channel direction; Assuming 4 adjacent pixels are introduced, the shape of the stacked data is WxHx4n.
3. The hyperspectral image classification method based on the improved Swin-Transformer network according to claim 1, characterized in that: The workflow of the cross-layer fusion module in S2 is as follows: Let the feature maps outputted by the network of the (L-1)th layer and the Lth layer be and The output after processing by the cross-layer fusion module is wherein represents the fusion representation of the (L-1)th layer and the Lth layer, represents the network parameters that can be learned by the cross-layer fusion module.
4. The hyperspectral image classification method based on the improved Swin-Transformer network according to claim 1, characterized in that: The improved Swin-Transformer model in S3 is trained in multiple rounds, and the process is as follows: S31, prepare the data set to be used for training; S32, use dataloader to load the data set; S33, take batch size data from dataloader and input the improved Swin-Transformer model; S34, input the prediction result of the model and the label into the loss function to calculate the loss; S35, according to the calculated loss value, update the weight of the network by back propagation; S36, loop S33 to S35 steps for epoch times to get the final trained model.
5. The hyperspectral image classification method based on the improved Swin-Transformer network according to claim 1, characterized in that: Hyperparameters include: batch size, epoch, initial learning rate, learning rate change ratio and learning rate adjustment strategy.
6. The hyperspectral image classification method based on the improved Swin-Transformer network according to any one of claims 1-5, characterized in that: The hyperspectral image classification method is realized through the existing deep learning framework Pytorch and the corresponding programming library.
7. The hyperspectral image classification method based on the improved Swin-Transformer network according to any one of claims 1-5, characterized in that: The spatial spectral recombination module can realize plug and play.