A transformer-based generative adversarial network method

Abstract

This invention discloses a Transformer-based Generative Adversarial Network (GAN) method for the field of Hyperspectral Image Classification (HIC). This method introduces the Transformer into GANs and proposes a Transformer-based GAN with residual upscale (TRUG) for HIC. TRUG includes a generator G and a discriminator D. In G, we propose a residual upscale (RU) module, which can improve the resolution of the generated image. In D, we employ Transformer blocks with progressively decreasing scales and use a grid self-attention mechanism in the first layer to better extract image features. Furthermore, GANs are prone to training instability; to address this issue, we improve the normalization algorithm and add relative position encoding. TRUG is the first Transformer-based GAN applied to HIC.

Description

Technical Field

[0001] This invention relates to hyperspectral image classification methods, and in particular to a Transformer-based Generative Adversarial Network (GAN) method, belonging to the field of remote sensing information processing technology. Background Technology

[0002] With the development of technology, hyperspectral image classification (HIC) has been widely used in many fields. In recent years, deep learning (DL) models have been applied to the field of HIC.

[0003] With the development of deep learning and the increase in model parameters, overfitting has become a significant challenge. To alleviate this problem, Zhang et al. focused on developing a simpler network. They proposed a 1D capsule network that is easy to implement and lighter than ordinary 3D convolutions. However, Mou et al. argued that one-dimensional convolutions might cause loss of pixel information when representing hyperspectral pixels, so they proposed a novel recurrent neural network (RNN) structure. However, RNNs suffer from inefficiency when processing sequential information. When processing sequential data, the Transformer, with its attention mechanism, is better able to address the inefficiency problem compared to RNNs. Currently, combining Transformers with CNNs to learn image features is a common approach. However, Transformers have a large number of parameters, making them prone to overfitting when training with small samples like HSI. An important method to alleviate overfitting is to increase the amount of training data. Many researchers have addressed this by increasing the amount of data. This specifically includes data flipping, cropping, translation, and generative models. Generative models alleviate this problem by generating high-quality samples. Generative Adversarial Networks (GANs) are typical generative models, mainly composed of a generator G and a discriminator D. GANs can fundamentally solve the problem of insufficient data samples, thereby addressing overfitting. Therefore, many researchers design GANs to alleviate the problem of insufficient samples. Zhu et al. used 1D GANs as spectral classifiers and 3D GANs as spatial classifiers. Furthermore, many researchers combine GANs with other techniques. However, GANs always suffer from imbalanced training data and mode collapse. To address the imbalanced training data problem, Wang et al. adapted D to a single classifier and proposed an adaptive DropBlock regularization method to solve the mode collapse problem.

[0004] GANs suffer from instability, and most researchers have focused on addressing this issue, leading to the introduction of various regularization methods, but few have altered the network structure. For CNNs, the convolution operator has a local receptive field, making them unable to handle long-range dependencies. However, HSI (Hyperspective Injection) provides more spectral sequence information. Therefore, this method uses the Transformer as its basic framework, which is better suited for handling global information and also excels at processing sequence information. Currently, no one in the HSI field has incorporated the Transformer into GANs. Therefore, this method combines the ideas of Transformer and GANs, proposing a Generative Adversarial Network with Residual Upscale (TRUG) module. Summary of the Invention

[0005] This invention introduces the Transformer into GANs and proposes a Transformer-based Generative Adversarial Network (TRUG) with a residual upscale module for HIC. TRUG consists of a generator G and a discriminator D. In G, we propose a residual upscale (RU) module, which improves the resolution of the generated image. In D, we employ Transformer blocks with progressively decreasing sizes and use a grid self-attention mechanism in the first layer to better extract image features. Furthermore, GANs are prone to training instability; to address this, we improve the normalization algorithm and add relative position encoding.

[0006] The specific steps of this invention are as follows:

[0007] S1: The original data is reduced in dimensionality using PCA to obtain Xpca, and Xpca is input into the discriminator D to learn the features of its real samples;

[0008] S2: Divide Xpca into several patches in the discriminator D and embed them;

[0009] S3: Input the data after embedding into the Transformer's Block, learn its features, and then downsample the obtained features to reduce their size. Repeat this step three times to obtain the final discriminative features.

[0010] S4: Input one-dimensional random noise Z∈R into generator G. B*LGiven class label C, the noise Z is reconstructed into a feature map X∈R with resolution (H×W) using a multi-layer perceptron (MLP). B*H*W*C Furthermore, the obtained feature map X is input into the Transformer Block for further feature extraction;

[0011] S5: The features obtained in S4 are used to improve the resolution of the feature map through a residual upscale (RU) module. The specific steps of the residual upscale module are as follows: a Kronecker product is performed between the feature map X before the module and the feature map Xnew after the module to generate a high-resolution Xup. The specific formula is as follows:

[0012] X ⊗Xnew =Xup

[0013] S6: Input the feature map Xup obtained in S5 into the Swing Transformer (ST) to further extract the features Xst between different windows, and further improve the resolution of the obtained feature map Xst through the RU module to obtain the feature Xstnew;

[0014] S7: Compress the channel dimensions of Xstnew to match those of Xpca to obtain fake samples. data ∈R B *M*N*C ;

[0015] S8: Fake the generated fake samples data The samples Xpca and real samples are input into the discriminator D. The discriminative features obtained from S3 are input into the softmax function for classification and distinguishing between true and false samples, resulting in the final classification result. At the same time, the loss of the true / false samples and the classification result is fed back to the generator so that it can continuously learn and generate higher quality samples.

[0016] Compared with the prior art, the technical solution of the present invention has the following technical effects:

[0017] (1) GAN can generate fake images that are similar to real data, which can alleviate the problem of insufficient training samples;

[0018] (2) RU in TRUG can improve image resolution, enabling G to generate high-quality samples;

[0019] (3) Networks based on the Swing Transformer basic module can obtain different feature information through window exchange;

[0020] (4) Compared with traditional GAN, TRUG combines the loss function for identifying real and fake samples and classification, which can alleviate the training mode collapse problem caused by traditional GAN;

[0021] (5) TRUG is the first Transformer-based GAN applied to HIC. Compared with ordinary GANs, it can achieve higher HSI classification accuracy;

[0022] (6) Expressive information is enhanced through attention mechanisms. Attached Figure Description

[0023] Figure 1 This is a framework diagram of the TRUG of the present invention.

[0024] Figure 2 Visualization of samples of different sizes generated for datasets IP and PU.

[0025] Figure 3 OA for different samples generated for different datasets.

[0026] Figure 4 OA for datasets IP and PU, indicating whether samples generated using the RU module were used.

[0027] Figure 5 Visual comparison of classification images obtained from the IP dataset using different methods; (a) False colorimage; (b) Ground truth; (c) SVM; (d) CNN; (e) 3D CNN; (f) HybridSN; (g) DPRN; (h) Transformer; (i) ViT; (j) TRUG.

[0028] Figure 6 Visual comparison of classification images obtained from the UP dataset using different methods; (a) False colorimage; Ground truth; (c) SVM; (d) CNN; (e) 3D CNN; (f) HybridSN; (g) DPRN; (h) Transformer; (i) ViT; (j) TRUG. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] Figure 1 This is a framework diagram of the TRUG of the present invention.

[0031] We selected two publicly available HSI datasets, Indian Pines (IP) and University of Pavia (UP), to verify the effectiveness of the proposed method.

[0032] All datasets were divided into two parts: a training set and a test set. Since GANs are highly sensitive to small sample sizes, we classified each class and selected 10% of the samples from each class for training. Experimental results were evaluated using three metrics: overall accuracy (OA), average accuracy (AA), and Kappa coefficient (Kappa). Furthermore, to avoid biased estimations, 10 independent tests were performed using PyTorch on a computer equipped with an Intel Core i5 processor and an RTX 3090 GPU.

[0033] The specific steps for each test are as follows:

[0034] S1: Dimensionality reduction of the original data using PCA is obtained. X pca and will X pca The input is fed into the discriminator D to learn the features of its real samples;

[0035] S2: In discriminator D, X pca Divide into several patches and embed them;

[0036] S3: Input the data after embedding into the Transformer's Block, learn its features, and then downsample the obtained features to reduce their size. Repeat this step three times to obtain the final discriminative features.

[0037] S4: Input one-dimensional random noise into generator G Z∈R B*L And class label C, noise is separated by a multi-layer perceptron (MLP). ZReconstructed into a feature map with a resolution of (H×W) X∈R B*H*W*C And the resulting feature map X The input is fed into the Transformer Block for further feature extraction;

[0038] S5: The features obtained in S4 are used to improve the resolution of the feature map through a residual upscale (RU) module. The specific steps of the residual upscale module are as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] X Feature maps after modules X new Perform a Kronecker product between them to generate a high-resolution image. X up The specific formula is as follows:

[0039] X⊗X new =X up

[0040] S6: The feature map obtained in S5 X up The input is further processed into a Swin Transformer (ST) to extract features between its different windows. Xst and the obtained feature map Xst The resolution is further improved by using the RU module to obtain features. X stnew ;

[0041] S7: Will X stnew The channel dimension is compressed to the same level as X pca The channel dimensions are consistent to obtain fake samples Fake data ∈R B *M*N*C ;

[0042] S8: Generate fake samples Fake data Compared with real samples X pca The samples are input together into the discriminator D. The discriminative features obtained from S3 are input into the softmax for classification and distinguishing between true and false samples, and the final classification result is obtained. At the same time, the loss of the true and false samples and the classification result are fed back to the generator so that it can continuously learn and generate higher quality samples.

[0043] To verify the effectiveness of the present invention, ablation tests and comparative experiments were conducted.

[0044] A. Ablation experiment

[0045] (1) The sample size and the size of the images generated for visual analysis are important parameters. In the experiment, we generated images of different sizes using the dataset, specifically 16 and 32 pixels. The visualization of the features of the generated images is shown below. Figure 2 As shown. Feature analysis has always been a challenge, especially its intuitive analysis. For evaluating the quality of generated samples, the most intuitive approach is to compare them with real images for visual analysis. Figure 2 (a) Shows a comparison of real and fake images generated by dataset PU from early to late training stages, from top to bottom. Figure 2 (b) is a visualization of 32×32 fake images generated from the IN dataset, displayed from top to bottom according to the length of the training time. As can be seen from the figure, the real and fake images in the early stages of training are significantly different, while similar parts appear in the middle and later stages. The image learning process can still be observed during training.

[0046] Regarding the size of the generated samples, we conducted parameter experiments on two datasets respectively, such as... Figure 3 As shown, a larger sample size leads to better classification performance, but due to hardware limitations, we could not further increase the sample size for the experiment. The figure also shows that the size of the generated samples significantly impacts the experimental results for different datasets. For the IP dataset, a sample size of 64 yielded an accuracy (OA) of 94.56%, which is 14.49% higher than the OA achieved with a sample size of 16. The highest classification accuracy of 96.76% was obtained with a sample size of 16. This indicates that different datasets exhibit different classification accuracies at different sizes. Therefore, in subsequent experiments, we used a sample size of 64 for IP and 16 for PU.

[0047] (2) Analysis of RU: We chose TransGAN as a comparative experiment, which uses the traditional method of improving image resolution, namely UpScaling. From Figure 4 It can be seen that the classification accuracy of GANs using the RU module is significantly higher than that using traditional methods. Experimental results show that the RU module does indeed have a certain effect.

[0048] B. Comparative Experiment

[0049] We present classification accuracies achieved on the IP and UP datasets using different methods. Comparison methods include SVM, CNN, 3D CNN, HybridSN, Deep Pyramidal Residual Networks (DPRN), and more recently, Transformer and ViT. In our experiments, we used 10% of the training set.

[0050] As shown in Table 1, TRUG outperforms all other methods. For the IP dataset, the OA of our proposed method is significantly higher than that of SVM, CNN, 3D CNN, and ViT, and about 4.43% higher than that of Transformer, HybridSN, and DPRN. TRUG's OA, AA, and Kappa are 97.85%, 97.67%, and 97.55%, respectively. This indicates that TRUG performs well in classifying HSI. Furthermore, we can observe that for the UP dataset, the proposed method achieves the best performance (i.e., OA=99.83%, AA=99.67%, Kappa=99.77%), which is about 1% better than other deep learning-based methods and 4% to 5% better than traditional methods. Classification graphs for different datasets are shown below. Figure 5 and Figure 6 As shown, the classification map obtained by the method proposed in this invention is clearer than that obtained by other methods.

[0051] Table 1

[0052]

[0053] The above are merely specific embodiments of this application and are not intended to limit this application in any way. Any simple modifications, equivalent changes, or alterations made to the above embodiments based on the technical essence of this application shall still fall within the protection scope of the technical solution of this application.

Claims

1. A Transformer-based generative adversarial network method, characterized in that, Includes the following steps: S1: The original hyperspectral image data is reduced in dimensionality using PCA to obtain Xpca, and Xpca is input into the discriminator D to learn the features of its real samples. S2: Divide Xpca into several patches in the discriminator D and embed them; S3: Input the data after embedding into the Transformer's Block, learn its features, and then downsample the obtained features to reduce their size. Repeat step S3 three times to obtain the final discriminative features. S4: Input one-dimensional random noise Z∈R into generator G. B*L Given class label C, a multilayer perceptron is used to reconstruct the noise Z into a feature map X∈R with resolution (H×W). B*H*W*C The obtained feature map X is then input into the Transformer Block for further feature extraction. S5: The features obtained in S4 are used to improve the resolution of the feature map through the residual upgrade module. The specific steps of the residual upgrade module are: to perform a Kronecker product between the feature map X before the module and the feature map Xnew after the module to generate a high-resolution Xup. S6: Input the feature map Xup obtained in S5 into SwingTransformer to further extract the features Xst between its different windows, and further improve the resolution of the obtained feature map Xst through the RU module to obtain the feature Xstnew; S7: Compress the channel dimensions of Xstnew to match the channel dimensions of Xpca to obtain fake samples. data ∈R B*M*N*C ; S8: Fake the generated fake samples data The samples Xpca and real samples are input into the discriminator D. The discriminative features obtained from S3 are input into the softmax for classification and to distinguish between true and false samples, thus obtaining the final classification result. At the same time, the loss of the true / false samples and the classification result is fed back to the generator so that it can continuously learn and generate higher quality samples.

Images