Hyperspectral image classification method based on SACVAEGAN
By using self-attention mechanism and spectral regularization CVAEGAN in hyperspectral image classification, the problem of small label data and latent vector correspondence difficulties is solved, and the hyperspectral image is improved. High classification accuracy.
Patent Information
- Application Number
- CN202011569729.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-12-26
AI Technical Summary
Due to the small amount of label data in hyperspectral image classification, it is difficult for the existing technology to accurately distinguish different categories. It is difficult for GAN networks to correspond with potential vectors to categories when generating virtual samples, resulting in limited classification accuracy.
A conditional variational autoencoder (CVAEGAN) based on the self-attention mechanism is used to generate virtual samples, and a potential vector classifier module is added to improve feature extraction and model stability through self-attention and spectral regularization, combined with residual The difference network extracts features from spatial and spectral perspectives.
It significantly improves the accuracy of hyperspectral image classification, solves the problem of potential vectors and categories in the GAN network, improves the stability and training speed of the model, and achieves higher classification accuracy.
Smart Images

Figure CN112633386B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of hyperspectral image classification, and in particular to a method for classifying hyperspectral images. Background Art
[0002] With the continuous development of remote sensing technology, hyperspectral images (HSI) have made significant breakthroughs in the field of earth observation. Unlike traditional three-channel color images, HSI can collect images in hundreds of spectral bands at the same time, with very rich spectral information. Therefore, hyperspectral images are widely used in satellite remote sensing, crop observation, mineral exploration and other fields.
[0003] In the hyperspectral data processing technology, the classification problem has always been one of the most active subjects. There are usually two classification methods for the classification of hyperspectral data: spectral classifier and spectral-spatial classifier. Traditional hyperspectral image classification algorithms usually include support vector machine (SVM), K nearest neighbor (KNN), maximum likelihood, neural network and logistic regression. However, since the same material may have spectral differences and different materials may have similar spectral features, it is difficult to accurately distinguish different categories only by spectral information. In order to solve the above problems, some scholars have proposed a method to combine spectral information with spatial information to improve classification performance.
[0004] Since traditional classification methods require a lot of experience and parameter settings, deep learning methods have been widely used in hyperspectral image classification in recent years. In particular, convolutional neural networks (CNNs) have received great attention. For example, W.Hu proposed a five-layer deep CNN model to extract the spectral features of HSI, which achieved good classification performance. W.Li proposed a pixel pair method as a classifier for deep spectra. When training data is scarce, this method has achieved good results, but this method mainly performs convolution operations in the spectral domain and ignores spatial details. Ying Li proposed a method of extracting spectral-spatial features using 3D convolution, which fully considers the various characteristics of hyperspectral data and obtains good classification results.
[0005] Although deep learning-based methods have made great progress in HSI classification, they still face some problems, namely, the amount of labeled data is too small. Based on this, some scholars have proposed using the GAN network model to alleviate the phenomenon of limited hyperspectral data. The GAN network model usually includes a generation model G and a discriminant model D. Models G and D are trained in an adversarial manner. G tries to generate real pseudo samples as much as possible through a random vector Z, while D tries to identify real samples and pseudo samples generated by G. The two continue to confront each other until D cannot successfully identify pseudo samples. By correctly using samples generated by GAN that can be used as virtual samples, the classification accuracy can be improved and the situation of limited hyperspectral training data can be reduced. Summary of the invention
[0006] The present invention aims to solve the above problems and proposes a SACVAEGAN (Self-Attention based Conditional Variational Auto-Encoder Generative Adversarial Networks) hyperspectral image classification method. The present invention adopts CVAEGAN (Conditional Variational Auto-Encoder) as the basic structure to solve the problem of small training data of hyperspectral images. The decoder in CVAEGAN generates virtual samples of hyperspectral images, expands the training data, and thus improves the accuracy of classification. At the same time, the present invention adds a latent vector classifier module on the basis of CVAEGAN to classify the latent vectors corresponding to the hyperspectral data, so that it cooperates with the decoder and the sample classifier for training, solves the problem that the randomly generated latent vectors in the GAN network are difficult to correspond to the category, and thus improves the accuracy. The self-attention mechanism and the spectral regularization method are applied in the decoder, the encoder and the discriminator. The self-attention mechanism can enable the network model to better extract the features of the hyperspectral data, and the spectral regularization method can improve the stability of the model. In the sample classifier, features are extracted from both spatial and spectral perspectives, and the structure of the residual network is added to improve the effect of model classification.
[0007] In order to achieve the above object, the technical solution and experimental steps adopted by the present invention are as follows:
[0008] (1) First, the hyperspectral image data is preprocessed.
[0009] (1a) First, the edges of the original hyperspectral data are padded with zeros so that a window size of patchsize×patchsize can be taken with each point as the center. The patchsize of the IndianPines dataset and the Salinas dataset is 28, and the patchsize of the PaviaU dataset is 24.
[0010] (1b) Randomly select K points as training labels, where the IndianPines and PaviaU datasets use 500 points as training labels, the Salinas dataset uses 200 labels as training labels, and the rest are used as test labels.
[0011] (1c) Obtain a sample set. Obtain a sample set of hyperspectral images, with the randomly obtained K training labels as the center, and divide the training data into a window size of patchsize×patchsize, and the rest into test data of patchsize×patchsize.
[0012] (2) Building a network model
[0013] After data preprocessing, the next step is to build a network model. The training network model consists of five parts: conditional variational autoencoder, discriminator, sample classifier and latent vector classifier.
[0014] (2a) The conditional variational autoencoder is divided into a decoder (i.e., generator) and an encoder. The encoder is mainly used to generate the latent vector corresponding to the real hyperspectral data. The encoder combines the self-attention mechanism and the spectral normalization method. The decoder is mainly used to generate the corresponding virtual hyperspectral data based on the latent vector. The decoder combines the self-attention mechanism and the spectral normalization method.
[0015] (2b) The main function of the discriminator is to determine whether the input hyperspectral data is true or false. The discriminator combines the self-attention mechanism and spectral normalization.
[0016] (2c) The main function of the sample classifier is to classify the input hyperspectral data. It consists of two branches that obtain the spatial features and spectral features of the hyperspectral image, combined with the residual network to extract features.
[0017] (2d) The main function of the latent vector classifier is to classify the randomly generated latent vectors, and then assign a category to the virtual hyperspectral data generated by the generator according to the latent vector for the next operation.
[0018] (3) Training the network
[0019] After the data and model are processed, training begins. The training process is mainly divided into four parts, namely training the conditional variational autoencoder, discriminator, sample classifier and latent vector classifier.
[0020] (3a) First, the discriminator is trained. The training of the discriminator is divided into three steps, which are to input the real hyperspectral data, the latent vector generated by the encoder in the conditional variational autoencoder, the virtual hyperspectral data generated by the decoder, and the virtual hyperspectral data generated by the randomly generated latent vector into the discriminator for training, calculate the loss function and optimize the parameters of the discriminator.
[0021] (3b) Train the conditional variational autoencoder. The training is divided into five steps, which are: putting the real hyperspectral data into the encoder to generate the corresponding latent vector and calculate the corresponding loss function. Input the virtual hyperspectral data and the real hyperspectral data corresponding to the latent vector generated by the encoder into the discriminator and calculate the corresponding loss function. Put the virtual hyperspectral data and the real hyperspectral data corresponding to the latent vector generated by the encoder into the sample classifier and calculate the corresponding loss function. Input the randomly generated latent vector and its corresponding virtual hyperspectral data into the latent vector classifier and the sample classifier respectively, and calculate the corresponding loss function. Input the latent vector generated by the encoder into the latent vector classifier and calculate the corresponding loss function.
[0022] (3c) Classify the latent vector classifier. The training of the latent vector classifier mainly involves inputting the latent vector corresponding to the real hyperspectral data into the latent vector classifier for classification and calculating the loss function.
[0023] (3d) Training the sample classifier. The training of the sample classifier is mainly divided into three steps. The real hyperspectral data is input into the sample classifier to calculate the classification loss function. The real hyperspectral data and the virtual hyperspectral data are respectively input into the sample classifier to calculate the corresponding loss function. The randomly generated latent vector is input into the latent vector classifier for classification, and the virtual hyperspectral data generated according to the randomly generated latent vector is input into the classifier for classification, and the corresponding loss function is calculated according to the classification results.
[0024] (4) Hyperspectral image classification
[0025] After the model training is completed, it is tested. The test results are compared with the true values to obtain the classification results and calculate the accuracy. The present invention adds a latent vector classifier to solve the correspondence problem between the randomly generated latent vectors and the categories in the GAN model. The experimental results show that the accuracy of the model in classifying hyperspectral data is significantly improved after adding the latent vector classifier. The self-attention mechanism and spectral regularization method are added to the network model to improve the stability of the model and the training speed. In the sample classifier, features are extracted from the two perspectives of space and spectrum, and the residual network structure is added to improve the accuracy of the sample classifier in classifying hyperspectral data. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a flow chart of the present invention.
[0027] Figure 2 It is the overall structural diagram of the present invention.
[0028] Figure 3 It is the structural diagram of the sample classifier.
[0029] Figure 4 This is the structural diagram of the conditional variational autoencoder, with the encoder on the upper side and the decoder on the lower side.
[0030] Figure 5 It is the structural diagram of the discriminator.
[0031] Figure 6 It is the structural diagram of the latent vector classifier.
[0032] Figure 7 (a) is the Indian Pines hyperspectral image used in the present invention, (b) is the Pavia U hyperspectral image used in the present invention, and (c) is the Salinas hyperspectral image used in the present invention.
[0033] Figure 8 (a) is the result of SVM classification of Indian Pines hyperspectral image. (b) is the result of 2dCNN classification of Indian Pines hyperspectral image. (c) is the result of 3dCNN classification of Indian Pines hyperspectral image. (d) is the result of DCGAN classification of Indian Pines hyperspectral image. (e) is the result of the invention classification of Indian Pines hyperspectral image.
[0034] Fig. 9(a) is the result of SVM classification of PaviaU hyperspectral image. (b) is the result of 2dCNN classification of PaviaU hyperspectral image. (c) is the result of 3dCNN classification of PaviaU hyperspectral image. (d) is the result of DCGAN classification of PaviaU hyperspectral image. (e) is the result of the invention classification of PaviaU hyperspectral image.
[0035] Fig.10 (a) is the result of SVM classification of Salinas hyperspectral image. (b) is the result of 2dCNN classification of Salinas hyperspectral image. (c) is the result of 3dCNN classification of Salinas hyperspectral image. (d) is the result of DCGAN classification of Salinas hyperspectral image. (e) is the result of invention classification of Salinas hyperspectral image. DETAILED DESCRIPTION
[0036] The present invention will be further described below in conjunction with the accompanying drawings.
[0037] See attached Figure 1 , the experimental steps of the present invention are as follows:
[0038] Step 1: Data preprocessing
[0039] (1) Data edge filling and normalization
[0040] The data is padded with zeros at the edges so that each available data point can be used as the center and the window size is
[0041] The data is of patchsize×patchsize. The data is normalized.
[0042] (2) Obtaining a sample set
[0043] Randomly select K data points from the original data, take the point as the center, and divide the window size into
[0044] The patchsize×patchsize data block is used as the training set. The remaining data points are used as the center and the patchsize×patchsize data block is used as the test set.
[0045] Step 2: Build the model
[0046] The model mainly consists of four parts: conditional variational autoencoder, discriminator, sample classifier and latent vector classifier.
[0047] The conditional variational autoencoder consists of an encoder and a decoder. In order to better utilize the encoder to extract features and generate corresponding latent vectors. The role of the encoder is to generate the corresponding latent vector according to the input sample, and the role of the decoder is to generate sample data of the corresponding category according to its input latent vector. Two branches are used in the encoder. One branch uses 2-D convolution to process the data to obtain the spatial features of the data, and the other branch uses 1-D convolution to process the central pixel data of the data block to obtain the spectral features of the data. After extracting the spatial features and spectral features, the two are spliced to continue the next operation. Both branches are composed of convolutional layers, fully connected layers, batch normalization, spectral normalization, Dropout and self-supervision mechanisms. Except for the last convolutional layer, the remaining convolutional layers are normalized by spectral normalization to satisfy the constraint of Lipschitz=1. The activation function of the convolutional layer is 'ReLU'. The structure of the decoder is similar to that of the encoder except that it is a single branch. It is composed of inverse convolutional layers, fully connected layers, batch normalization, spectral normalization, Dropout and self-supervision mechanisms. Except for the last convolutional layer, the remaining convolutional layers are normalized by spectral normalization to satisfy the constraint of Lipschitz=1. Using batch normalization can enable higher learning rates to accelerate convergence by normalizing the data of each training mini-batch. Using Dropout can prevent overfitting when the training sample is small. Using spectral normalization can reduce the possibility of mode collapse in the GAN network and improve its stability. Using a self-supervision mechanism can better extract feature information. After the convolution and self-supervision mechanism, the encoder flattens it and inputs it into the fully connected layer, and uses the 'sigmoid' function for activation. After the deconvolution layer and the self-supervision layer, the decoder flattens it and inputs it into the fully connected layer, and finally uses the 'Tanh' function for activation.
[0048] The main function of the discriminator is to determine whether the input sample data is real sample data or virtual sample data. The model structure consists of convolutional layer, fully connected layer, batch normalization, spectral normalization, Dropout and self-supervision mechanism. Except for the last convolutional layer, the remaining convolutional layers are normalized by spectral normalization to satisfy the constraint of Lipschitz=1. The activation function of the convolutional layer is 'LeakyReLU'. Using batch normalization can make higher learning rates accelerate convergence by normalizing the data of each training mini-batch. Using Dropout can prevent overfitting when the training sample is small. Using spectral normalization can reduce the possibility of mode collapse in the GAN network and improve its stability. Using the self-supervision mechanism, it can better extract feature information. After passing through the convolutional layer and the self-supervision layer, the discriminator flattens it and inputs it into the fully connected layer, and finally uses the 'Sigmoid' function for activation.
[0049] The main function of the sample classifier is to classify the input sample data. The model structure is mainly divided into two branches. One branch uses 2-D convolution to process the data and obtain the spatial features of the data. The other branch uses 1-D convolution to process the central pixel data of the data block and obtain the spectral features of the data. After extracting the spatial features and spectral features, the two are spliced to continue the next operation. Both branches are composed of convolutional layers, fully connected layers, batch normalization, Dropout and residual network structures. The activation function of the convolutional layer is 'LeakyReLU'. Using batch normalization can enable a higher learning rate to accelerate convergence by normalizing the data of each training small batch. Using Dropout can prevent overfitting when the training sample is small. Using the residual network structure deepens the depth of the network and improves the accuracy of the network. After passing through the convolutional layer and the residual network layer, the sample classifier flattens it and inputs it into the fully connected layer, and finally activates it through the 'LogSoftmax' function.
[0050] The main function of the latent vector classifier is to classify the latent vector. The model structure consists of a convolutional layer, a fully connected layer, and Dropout. The activation function of the convolutional layer is 'ReLU'. Using batch normalization can enable a higher learning rate to accelerate convergence by normalizing the data of each training mini-batch. Using Dropout can prevent overfitting when the training sample is small. The latent vector classifier is first adjusted to a fixed size through a fully connected layer, then its shape is changed and input to the convolutional layer, then it is flattened and input to the fully connected layer, and finally activated by the 'LogSoftmax' function.
[0051] Batch Normalization (BN), which normalizes the activations of the previous layer in each batch. In other words, it transforms the mean activation of the previous layer to 0 and the standard deviation of the activation to 1. Assume that the size of the batch is n, and is the activation value obtained from the previous layer, The calculation formula for batch normalization is:
[0052]
[0053] in, Represents the output of samples in a batch after batch normalization. and express Correspondingly, γ and β represent the hyperparameters of learning.
[0054] The main function of spectral normalization is to make the parameters in the convolution operation satisfy the constraint of Lipschitz = 1, so as to make the network structure more stable and reduce the occurrence of mode collapse. The implementation method is to divide the network parameters of each layer by the spectral norm of the parameter matrix of that layer to satisfy the constraint of Lipschitz = 1.
[0055] The main function of the self-attention mechanism is to make the image generation higher quality, thereby improving the classification performance of the classifier. Most GAN-based image generation models are built using convolutional layers. Convolutional layers process information in local neighborhoods, so using convolutional layers alone to model long-distance correlations in images is computationally inefficient. Therefore, the self-attention mechanism is introduced so that the generator and discriminator can effectively model the relationship between widely separated spatial regions.
[0056] Assume that the image feature x∈R in the previous hidden layer C×N Convert to two feature spaces f and g to calculate attention. Where f(x) = W f x,g(x)=W g x.
[0057]
[0058] β ij It indicates the involvement of region i when the model synthesizes the image content of region j, that is, the relevance. The output of the attention layer is o = (o1, o2, ..., o j ,...,o N )∈R C×N , C is the number of channels, and N is the number of feature positions where the previous hidden layer features are located.
[0059]
[0060] Finally, the output of the attention layer is further multiplied by a scale parameter and added to the input feature map. Therefore, the final output is:
[0061] y i =γo i +x i (4)
[0062] γ is a learnable scalar, which is initially set to 0. The purpose is to limit the network to focus on neighborhood information and then slowly distribute the weights to other long-distance features.
[0063] The residual network consists of a series of residual blocks. A residual block can be expressed as:
[0064] x l+1 =x l +F(x l ,W l ) (5)
[0065] where x l It is a direct mapping, that is, directly passing the data of the previous layer. l ,W l ) is the residual part, which is generally composed of 2 to 3 convolutions and represents the part generated by convolution. The residual network is easy to optimize and can increase the depth to improve the accuracy.
[0066] Step 3: Train the network
[0067] The training of the conditional variational autoencoder is divided into four parts, namely, training the encoder part of the conditional variational autoencoder, training the decoder using the discriminator, and training the decoder using the sample classifier and the latent vector classifier. The loss function when training the conditional variational autoencoder is:
[0068] L G =L E +L GD +L GC +L aux_real (6)
[0069] in
[0070]
[0071] In the above formula, υ and ξ are the mean and variance of the potential vector output by the encoder network, x is the real data, and x' is the generated virtual data.
[0072]
[0073]
[0074] In the above formula, P r , P z are the distributions of the real data and the latent variables respectively. D , f C They refer to the covariance of the output of the intermediate layer of the discriminator and the output of the intermediate layer of the classifier respectively.
[0075]
[0076] Where x represents real data or virtual data generated by the latent vector corresponding to the real data, O C represents the final output of the sample classifier, O aux represents the output of the latent vector classifier.
[0077] The training of the discriminator is mainly divided into three parts. The real data, the virtual samples generated according to the latent vector corresponding to the real data, and the virtual samples generated according to the randomly generated latent vector are input into the discriminator to calculate the corresponding loss function. The loss function of training the discriminator is:
[0078] L D =L D_real +L D_fake +L D_fake_random (11)
[0079] in
[0080]
[0081]
[0082]
[0083] Where x represents the real data, P r represents the real data distribution. z represents the potential vector corresponding to the real data, P z represents the distribution of z. z_random represents a randomly generated latent vector. P z_random Represents the distribution of z_random.
[0084] The training of the sample classifier is mainly divided into three parts. The real data, the virtual samples generated according to the latent vector corresponding to the real data, and the virtual samples generated according to the randomly generated latent vector are input into the sample classifier to calculate the corresponding loss function. The loss function of the training sample classifier is:
[0085] L C =L C_real +L C_fake +L C_fake_random (14)
[0086] in
[0087] L C_real = -E[logP(c|x r )](15)
[0088] L C_fake =||f(x r )-f(x g )|| (16)
[0089] L C_fake_random = -E[logP(c|x g_random )] (17)
[0090] where x r represents the real data, x g represents the virtual data generated by the decoder according to the latent vector corresponding to the real data, x g_random represents virtual data generated by the decoder based on randomly generated latent vectors. c represents the category. f represents the intermediate layer output of the sample classifier.
[0091] There is only one part in training a latent vector classifier, which is to input the latent vector corresponding to the real data into the latent vector classifier for classification and calculate the corresponding loss function. The loss function for training a latent vector classifier is
[0092] L aux = -E[logP(c|z)] (18)
[0093] Where z is the potential vector corresponding to the real data.
[0094] In addition, the classifier, discriminator and generator all use the Adam optimization algorithm with a batch size of 32, the weight decay is set to 0.0005, the learning rate of the generator is 0.0001, the learning rate of the discriminator is 0.0002, and the learning rate of both classifiers is 0.0001. When iterating to 20, 40, 80, and 100, the learning rate becomes 0.7 times the current learning rate. The number of iterations is 500. The IndianPines dataset has a data block size of 28*28 and a training set size of 500. The PaviaU dataset has a data block size of 24*24 and a training set size of 500. The Salinas dataset has a data block size of 28*28 and a training set size of 200. The latent variable z size is 100.
[0095] Step 4: Hyperspectral image classification
[0096] Compare the test value and the true value output by the classifier to obtain the classification result and calculate the accuracy.
[0097] Step 5: Output the classified image results.
[0098] Experiment and analysis
[0099] 1. Experimental conditions
[0100] The hardware test platform of the present invention is: processor Intel (R) Core (TM) i5-9300H CPU, main frequency is 2.40GHz, memory is 16GB, graphics card is GTX1660Ti; software platform is Windows 10 operating system and PyCharm2019. The programming language is Python, and the network structure is implemented using the PyTorch deep learning framework.
[0101] 2. Experimental Data
[0102] The performance evaluation of the present invention mainly uses three data sets: Indian Pines data set, Pavia University data set in Italy, and Salinas data set in California, USA.
[0103] The Indian Pines dataset was collected by the Airborne Visible Infrared Imaging Spectrometer (AVIRIS) in 1992 at the Indian Pines Test Site in northwestern Indiana. The image originally had 220 bands, and after removing useless bands, 200 bands remained. The image has a total of 16 types of samples. Table 1 shows the distribution of the number of samples of each type in the Indian Pines image, as well as the number of each type in the training set and test set of the present invention on the dataset.
[0104] Table 1
[0105]
[0106]
[0107] The Pavia University dataset of Pavia City, Italy is a part of the hyperspectral data of the image of Pavia City, Italy, taken in 2003 using a German airborne reflective optical spectroscopic imager. The spectral imager continuously images 115 bands within the wavelength range of 0.43-0.86μm, and the spatial resolution of the image is 1.3m. Among them, 12 bands are eliminated due to the influence of noise, so the images formed by the remaining 103 spectral bands are generally used. The size of the data is 610×340, and it contains 9 categories of samples. Table 2 shows the distribution of the number of samples of each category in the Pavia University image, as well as the number of each category in the training set and test set of the present invention on this dataset.
[0108] Table 2
[0109]
[0110] The Salinas dataset is an image of the Salinas Valley in California, USA, taken by the AVIRIS imaging spectrometer. Its spatial resolution reaches 3.7m. The image originally has 224 bands, and after removing several invalid bands, 204 bands remain. The image size is 512×217, with a total of 16 types of samples. Table 3 shows the distribution of the number of samples of each category in the Salinas image, as well as the number of each category in the training set and test set of the present invention on this dataset.
[0111] Table 3
[0112]
[0113] 3. Performance comparison
[0114] The four prior art comparison and classification methods used in the present invention are as follows:
[0115] (1) Hyperspectral images are classified using SVM (Support Vector Machine) based on RBF (Radial Basis Function) kernel.
[0116] (2) The hyperspectral image classification method proposed by Luo Jianhua et al. in the hyperspectral remote sensing image classification based on deep convolutional neural network. It is referred to as the convolutional neural network classification method. First, the hyperspectral data is reduced in dimension by PCA (Principal Component Analysis), and then the hyperspectral image is classified by 2D convolution.
[0117] (3) The hyperspectral image classification method proposed by Amina Ben Hamida et al. in 3-D Deep Learning Approach for Remote Sensing Image Classification. It is referred to as the 3D convolutional network classification method. 3D convolution is used to classify hyperspectral images.
[0118] (4) The hyperspectral image classification method proposed by Lin Zhu et al. in Generative Adversarial Networks for Hyperspectral Image Classification. It is referred to as the generative adversarial network classification method. DCGAN (Deep Convolutional Generative Adversarial Networks) is used to classify hyperspectral images.
[0119] In the experiment, the following three indicators were used to evaluate the performance of the present invention:
[0120] The first evaluation indicator is the overall accuracy (OA), which indicates the proportion of correctly classified samples to all samples. The larger the value, the better the classification effect.
[0121] The second evaluation indicator is the average accuracy (AA), which represents the average classification accuracy of each category. The larger the value, the better the classification effect.
[0122] The third evaluation indicator is the chi-square coefficient (Kappa), which represents the different weights in the confusion matrix. The larger the value, the better the classification effect.
[0123] Table 4 shows the accuracy of the classification of the hyperspectral image of Indian Pines in the present invention and the comparison
[0124] Table 5 shows the accuracy and comparison of the hyperspectral image classification of Pavia University in Pavia, Italy by the present invention.
[0125] Table 6 shows the accuracy and comparison of the hyperspectral image classification of the Salinas Valley in California, USA by the present invention.
[0126] Table 4
[0127]
[0128] Table 5
[0129]
[0130]
[0131] Table 6
[0132]
[0133]
[0134] It can be seen from Tables 4, 5 and 6 that for the same hyperspectral data set, the hyperspectral classification method proposed in the present invention has better classification effect than other methods. For the University of Pavia data set, the Indian pine data set and the Salinas Valley data set in California, the classification performance of the network is about 3%, 9% and 2% better than the current optimal method classification performance, respectively.
[0135] in addition Figure 8 , Fig. 9 and Fig.10The classification diagram is shown, and its visualized classification effect is consistent with the results listed in Table 4, Table 5 and Table 6. From the visualization of the results, the classification diagram achieved by the present invention has a better effect.
[0136] Table 7
[0137]
[0138] Table 7 is a comparison of the training and testing time of the present invention, SVM, 2dCNN, 3dCNN and DCGAN.
[0139] As can be seen from Table 7, compared with other methods, the training time of the present invention is much longer due to the complex model structure. The test time of the present invention is similar to that of DCGAN, but a little longer than that of 2dCNN and 3dCNN.
[0140] In summary, the present invention proposes a hyperspectral image classification method with a SACVAEGAN network structure. A self-supervision mechanism is added on the basis of CVAEGAN, which can enable the network to better learn the characteristics of hyperspectral images. We have also modified other structures of the network, added a residual network to the sample classifier, making the network structure deeper and more accurate; at the same time, two branches for extracting spectral features and spatial features are added to the sample classifier to better extract the characteristics of hyperspectral images; and a latent vector classifier module is added to the network, so that the latent vector generated by SACVAEGAN is more accurate, and a more accurate label can also be given to the randomly generated latent vector. Experimental results show that the present invention has higher classification accuracy than the prior art.
Claims
1. A hyperspectral image classification method based on SACVAEGAN, characterized by: The following steps are included (1) First, preprocess the hyperspectral image data; (2) Constructing a network model; After data preprocessing, the next step is to build a network model; the training network model consists of four parts: conditional variational autoencoder, discriminator, sample classifier and latent vector classifier; (3) Train the network; After the data and model are processed, training begins; the training process is mainly divided into four parts, namely, training the conditional variational autoencoder, discriminator, sample classifier, and latent vector classifier; (4) Hyperspectral image classification; After the model training is completed, it is tested; the test results are compared with the true values to obtain the classification results and calculate the accuracy; In step (3), (3a) first trains the discriminator; the training of the discriminator is divided into three steps, namely, inputting the real hyperspectral data, the latent vector generated by the encoder in the conditional variational autoencoder, the virtual hyperspectral data generated by the decoder, and the virtual hyperspectral data generated by the randomly generated latent vector into the discriminator for training, calculating the loss function and optimizing the parameters of the discriminator; (3b) Training the conditional variational autoencoder; The training is divided into five steps, which are: putting the real hyperspectral data into the encoder to generate the corresponding latent vector to calculate the corresponding loss function; inputting the virtual hyperspectral data and the real hyperspectral data corresponding to the latent vector generated by the encoder into the discriminator to calculate the corresponding loss function; putting the virtual hyperspectral data and the real hyperspectral data corresponding to the latent vector generated by the encoder into the sample classifier to calculate the corresponding loss function; inputting the randomly generated latent vector and its corresponding virtual hyperspectral data into the latent vector classifier and the sample classifier respectively to calculate the corresponding loss function; inputting the latent vector generated by the encoder into the latent vector classifier to calculate the corresponding loss function; (3c) classifying the latent vector classifier; the training of the latent vector classifier is to input the latent vector corresponding to the real hyperspectral data into the latent vector classifier for classification and calculate the loss function; (3d) Training the sample classifier; the training of the sample classifier is divided into three steps: inputting the real hyperspectral data into the sample classifier to calculate the classification loss function; inputting the real hyperspectral data and the virtual hyperspectral data into the sample classifier to calculate the corresponding loss function; inputting the randomly generated latent vector into the latent vector classifier for classification, and inputting the virtual hyperspectral data generated according to the randomly generated latent vector into the classifier for classification, and calculating the corresponding loss function according to the classification results.
2. The hyperspectral image classification method based on SACVAEGAN according to claim 1, characterized in that: In step (1), (1a) first fill the edges of the original hyperspectral data with zeros so that data with a window size of patchsize×patchsize can be taken with each point as the center, where the patchsize of the IndianPines dataset and the Salinas dataset is 28, and the patchsize of the PaviaU dataset is 24; (1b) Randomly select K points as training labels, of which the IndianPines and PaviaU datasets use 500 points as training labels, the Salinas dataset uses 200 labels as training labels, and the rest are used as test labels; (1c) Obtaining a sample set: Obtain a sample set of hyperspectral images, centering on the randomly acquired K training labels, dividing the training data into a window size of patchsize×patchsize, and dividing the rest into test data of patchsize×patchsize.
3. The hyperspectral image classification method based on SACVAEGAN according to claim 1, characterized in that: In step (2), (2a) the conditional variational autoencoder is divided into a decoder and an encoder; the encoder is used to generate a latent vector corresponding to the real hyperspectral data; the encoder combines the self-attention mechanism and the spectral normalization method; The decoder is used to generate corresponding virtual hyperspectral data based on the latent vector. The decoder combines the self-attention mechanism and the spectral normalization method. (2b) The discriminator is used to determine whether the input hyperspectral data is real or fake. The discriminator combines the self-attention mechanism and spectral normalization. (2c) The sample classifier is used to classify the input hyperspectral data. It consists of two branches that obtain the spatial features and spectral features of the hyperspectral image, combined with the residual network to extract features; (2d) The role of the latent vector classifier is to classify the randomly generated latent vectors, and then assign a category to the virtual hyperspectral data generated by the generator based on the latent vector.
