Incremental hyperspectral image classification method based on virtual sample expansion

Abstract

The invention provides an incremental hyperspectral image classification method based on virtual sample expansion, and belongs to the technical field of remote sensing image processing and application. Firstly, data preprocessing is carried out on original data of a hyperspectral image; secondly, constructing a virtual sample expansion network and training the virtual sample expansion network in an initial stage; thirdly, building and training an incremental hyperspectral image classification model in an initial stage; and finally, training an incremental hyperspectral image classification model in an incremental stage by using a virtual sample expansion network and momentum knowledge distillation, performing classification and outputting a result. According to the invention, incremental learning can be carried out on the hyperspectral images collected from different hyperspectral imagers, so that the classification capability of not forgetting old classes is realized while new classes are learned; according to the method, a feasible solution is provided for realizing hyperspectral image classification in an incremental manner, so that the hyperspectral image classification theory is further improved from large-scale practical application.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing and application, and relates to an incremental hyperspectral image classification method, and in particular to an incremental hyperspectral image classification method based on virtual sample expansion. Background Art

[0002] Hyperspectral image classification is a pixel-level image classification task for ground objects and is a key step in many remote sensing applications. Incremental hyperspectral image classification is a task-scalable classification model that can continuously identify new ground object categories. When faced with new ground object categories, it incrementally expands the recognition ability of the model without retraining the model from scratch, thereby alleviating computing and memory pressure. Incremental hyperspectral image classification can significantly improve the efficiency of large-width and long-time series earth observation tasks.

[0003] Researchers have conducted extensive research on remote sensing image classification methods and models and achieved excellent results:

[0004] Qiqihar University has proposed a new remote sensing scene image classification method in its patented technology "Remote sensing scene image classification method based on multi-level feature dense fusion" (patent application number: 202111037856.2, authorization announcement number: CN113705718B). This method achieves the classification of remote sensing scene images by establishing a lightweight convolutional neural network based on dual-branch multi-level feature dense fusion. This remote sensing scene image classification method can classify wide geographical areas, but cannot finely classify fine-grained scenes.

[0005] Yunnan University proposed a hyperspectral image classification method in its patented technology "Hyperspectral Image Classification Method, Device and Electronic Equipment" (Patent Application No.: 202310869709.4, Authorization Announcement No.: CN116595208B). This method uses a dual-channel twin network based on convolutional residual blocks and spatial attention mechanism for training, and uses transfer learning to classify hyperspectral images in small sample scenarios. This method can achieve fine classification, but cannot achieve new image categories.

[0006] Xidian University proposed an image classification method based on linear programming incremental learning in its patented technology "Image classification method based on linear programming incremental learning" (patent application number: CN201911348984.1, authorization announcement number: CN111222545B). This method calculates the feature mean vector of the new category and uses linear programming to update the classifier weights when there is very little new category data, thereby achieving rapid learning and maintaining the classification ability of the original category. This method can perform incremental image classification when there is little new category data, but can only perform incremental classification on one new category.

[0007] Chunyan Yu et al. proposed an incremental learning method for distillation-constrained prototype representation networks in "Distillation-Constrained Prototype Representation Network for Hyperspectral Image Incremental Classification" published in Volume 62 of IEEE Trans. Geoscience and Remote Sensing in 2024. This method uses initial category samples for training to establish an initial model in the initial stage, and uses a dual knowledge distillation module to transfer knowledge in the incremental learning stage to maintain the recognition ability of the original category. This method can learn new categories without directly using historical data, does not forget old categories, and has good generalization ability.

[0008] In the existing incremental hyperspectral image classification methods, either the old category data is used directly, which requires a lot of storage resources, or the old category data is used indirectly. As the incremental learning stage increases, the old category features generated by the model will have feature shift. These shortcomings are the key issues that restrict the incremental hyperspectral image classification from theory to large-scale earth observation applications. Summary of the invention

[0009] The purpose of the present invention is to overcome the defects of the prior art and provide an incremental hyperspectral image classification method based on virtual sample expansion, which is used to solve the problem of irrational utilization of old class samples in the current incremental hyperspectral image classification technology, reduce storage resource usage and prevent feature shift.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] An incremental hyperspectral image classification method based on virtual sample expansion is disclosed. The incremental hyperspectral image classification method firstly performs data preprocessing on the original data of the hyperspectral image. Secondly, a virtual sample expansion network is constructed and the virtual sample expansion network of the initial stage is trained. Thirdly, an incremental hyperspectral image classification model of the initial stage is constructed and trained. Finally, the incremental hyperspectral image classification model of the incremental stage is trained by using the virtual sample expansion network and momentum knowledge distillation, and the classification is performed and the results are output.

[0012] The following steps are involved:

[0013] Step 1: Use a hyperspectral imager to collect raw data of hyperspectral images, perform data preprocessing and divide the data set. Specifically:

[0014] Step 1.1: Use a hyperspectral imager to collect raw data of hyperspectral images, perform data correction and band selection, and obtain hyperspectral images.

[0015] Step 1.2, PCA technology is used to reduce the dimension of the hyperspectral image obtained in step 1.1, and a sliding window is used to divide the hyperspectral image into small blocks of shape N×N×C, where C is the image dimension after dimensionality reduction and N is the size of the sliding window.

[0016] Step 1.3, in order to divide the data set, the training process of the incremental hyperspectral image classification model needs to be divided into several stages (0, 1, 2, ..., t..., T), where T represents the number of incremental stages; for each stage t, the incremental hyperspectral image classification model is assigned a new set of categories C that need to be learned t ,in, Denotes the mth category of stage t. Define C i and C j Represent the category sets assigned to stage i and stage j respectively, where where i≠j, is an empty set, indicating that the newly learned hyperspectral image categories are different in each stage.

[0017] Step 1.4: divide the data set obtained after processing in step 1.3 into a training data set and a test data set in proportion. Usually, the proportion of the training data set is set to 10% or 5%.

[0018] Step 2: construct a virtual sample expansion network, which consists of a variational autoencoder and an adversarial autoencoder. Its main function is to generate virtual samples to enhance the current training samples and generate pseudo samples of old categories. Specifically:

[0019] Step 2.1, the basic structure of the variational autoencoder includes an encoder and a decoder. The structure of the encoder is: convolution layer → 3D convolution layer → 3D convolution layer → 2D convolution layer → pooling layer → linear layer → latent space; the structure of the decoder is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer. The loss function L used when optimizing the variational autoencoder VAE It consists of two parts, namely KL divergence L KL and the reconstruction loss L re The loss function can be expressed by formula (1) to formula (4).

[0020] z=μ+ε×σ (1)

[0021]

[0022]

[0023] L VAE =L KL +L re (4)

[0024] Where ε is the random noise drawn from the normal distribution; μ represents the mean of the latent space variable in the encoder; σ represents the standard deviation of the latent space variable in the encoder; N represents the number of samples; μ (i) represents the mean of the i-th sample; σ (i) Represents the standard deviation of the i-th sample; I represents the image input to the variational autoencoder, with a shape of N×N×C; Represents the image output by the variational autodecoder; x represents the width index; y represents the height index; and z represents the channel index.

[0025] Step 2.2, the basic structure of the adversarial autoencoder includes an encoder, a decoder and a discriminator, wherein the decoder is also a generator. The structure of the encoder is: convolution layer → 3D convolution layer → 3D convolution layer → 2D convolution layer → pooling layer → linear layer → latent space; the structure of the decoder is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer; the structure of the discriminator is: latent space → linear layer → linear layer. The reconstruction loss of the adversarial autoencoder is the same as that of the variational autoencoder. The losses of the discriminator and the generator are shown in formula (5) and formula (6), where L D represents the discriminator loss, L G represents the generator loss.

[0026]

[0027] in, represents the calculation of expectation; Represents the expected probability that the data x generated by the generator is identified by the discriminator as real data; represents the expected probability that the real data x is identified as the real data by the discriminator; D(·) represents the predicted value of the variable by the discriminator; P g represents the distribution of generated samples (latent space); P r represents the distribution of real samples (samples from normal distribution); x represents the distribution of samples from P g and P r A random sample of .

[0028] Step 3: According to the stages divided in step 1.3, the virtual sample expansion network is trained in the initial stage (t=0), specifically:

[0029] Step 3.1, initialize the weights and bias parameters of the variational autoencoder and adversarial autoencoder in the virtual sample expansion network respectively.

[0030] Step 3.2, according to the category set assigned in step 1.3, input the training set and test set of each category in the category set in the initial stage into the virtual sample expansion network constructed in step 2, and train according to the set training rounds. Each round of training includes: forward propagation, calculation of loss function, back propagation, optimization of model parameters, and cyclic iterative training. The loss function consists of two parts, see steps 2.1 and 2.2 respectively.

[0031] Step 3.3, for each category, save the variational autoencoder and the decoder model in the adversarial autoencoder trained in step 3.2 to form a virtual sample expansion network. In the virtual sample expansion network, the decoder model of the variational autoencoder and the decoder model of the adversarial autoencoder are respectively used to generate two different types of virtual samples.

[0032] Step 4: Build an incremental hyperspectral image classification model: Build a 10-layer feature extraction module, followed by a Softmax classifier. Specifically:

[0033] In step 4.1, the structure of the feature extraction module is as follows: convolution layer → first residual module → second residual module → first pooling layer → third residual module → fourth residual module → second pooling layer → Flatten layer → normalization layer → linear layer.

[0034] In step 4.2, a Softmax classifier is connected after the linear layer of the feature extraction module as the initial form of the incremental hyperspectral image classification model.

[0035] Step 5: According to the stages divided in step 1.3, a training data set for the initial stage (t=0) is constructed to perform hyperspectral image classification in the initial stage. Specifically:

[0036] Step 5.1, generate random Gaussian noise and input it into the decoder model saved in step 3.3 to generate data enhancement samples of the initial training samples. The training samples and the data enhancement samples are combined to construct the initial training set D0.

[0037] Step 5.2: Input the initial training set obtained in step 5.1 into the feature extraction module and Softmax classifier constructed in step 4, perform iterative training through forward propagation, loss calculation, back propagation and optimization parameter update, obtain the classification results of the initial stage, and save the parameters of the feature extraction module and Softmax classifier to obtain the incremental hyperspectral image classification model. The loss function used in the loss calculation is the cross entropy loss L ce , as shown in formula (7).

[0038]

[0039] Among them, C 0 is the category set in the initial stage; p c is the prediction sample x i The probability of belonging to category c; when y = c, δ = 1, otherwise δ = 0; c represents the category index; y i Represents sample x i The real label.

[0040] Step 6: According to the stages and category sets divided in step 1.3, the training data corresponding to the category set are input into the subsequent incremental learning stage (t>0) in sequence, and the momentum knowledge distillation and the virtual sample expansion network of step 3 are used to further update the parameters of the incremental hyperspectral image classification model obtained in the previous stage, so that the incremental hyperspectral image classification model gradually has the classification ability for all stage learning categories.

[0041] Specific:

[0042] Step 6.1, in the incremental learning stage (t>0), the training samples corresponding to the current training stage (ie, new class samples) are input into the virtual sample expansion network of step 2, and the corresponding decoder is trained and saved.

[0043] Step 6.2, generate random Gaussian noise and input it into the decoder saved in step 6.1, use the decoder to generate data enhancement samples of the current training samples and old class virtual samples, combine the new class samples, new class enhancement samples and old class virtual samples to construct the incremental training set D t , where t>0.

[0044] Step 6.3, momentum knowledge distillation: Use exponential moving average to update the parameters of the incremental hyperspectral image classification model, as shown in formula (8), and use the knowledge distillation loss L kd Keep the classification ability of the old category, as shown in formula (9):

[0045] F t =α*F t +(1-α)*F t-1 (8)

[0046]

[0047] Among them, F t is the incremental hyperspectral image classification model parameter at stage t, F t-1 is the incremental hyperspectral image classification model parameter of stage t-1, α is the set hyperparameter, x i represents a training sample, D t represents the incremental training set, F t (x i ) and F t-1 (x i ) represent the incremental hyperspectral image classification models that input training samples into stage t and stage t-1, respectively.

[0048] Step 6.4: The incremental training set D obtained in step 6.2 t Input into the feature extraction module and Softmax classifier in step 4, update the incremental hyperspectral image classification model to obtain the normalized probability. Use the label and the normalized probability to calculate the cross entropy loss, and calculate the knowledge distillation loss. The total loss function is the sum of the cross entropy loss and the knowledge distillation loss. Perform training according to a fixed round, forward propagation, calculation of loss function, back propagation, and optimization of model parameters. Save the incremental hyperspectral image classification model after training. The incremental hyperspectral image classification model can keep the memory of the old categories when learning the new category classification in the current stage.

[0049] Step 7: Use the incremental hyperspectral image classification model trained in step 6 to perform classification and output the results. Specifically:

[0050] Step 7.1: Input the image to be classified into the incremental hyperspectral image classification model to obtain the category prediction result of each pixel.

[0051] Step 7.2, save the classification results as image files or other formats, and output them to a specified directory or storage medium.

[0052] Step 7.3, visualize the classification results to visually observe the classification results of the incremental hyperspectral image classification model, including mapping the predicted category labels to different colors and superimposing them on the original image.

[0053] An incremental hyperspectral image classification system based on virtual sample expansion specifically includes the following modules:

[0054] Acquisition module: used to obtain raw data of hyperspectral images;

[0055] Storage medium: used to store acquired hyperspectral image data and computer programs; the computer programs include a data set construction program, a virtual sample expansion network program, an initial stage program, an incremental learning stage program, and a result saving program;

[0056] Processor: used to load and execute data and computer programs in storage media. When the computer program is executed, the processor is used to:

[0057] Read the hyperspectral raw data collected and stored by the hyperspectral imager; use the data set construction program to perform data preprocessing and organize and divide the data set; use the virtual sample extension network program to train the model; use the initial stage program to classify the initial training samples; use the incremental learning stage program to update and adjust the model to maintain the classification effect of the old categories; use the result saving program to save the classification results to the storage medium.

[0058] A terminal device of an incremental hyperspectral image classification system, the terminal device includes an acquisition module, a storage medium and a processor, the storage medium is a memory system in the terminal device, used to store programs and data, and the processor is used to execute program instructions stored in the storage medium. The acquisition module can be a hyperspectral imager carried on a satellite, an unmanned aerial vehicle, etc. The storage medium can be a high-speed RAM memory or a non-volatile memory. The processor can be a central processing unit (CPU), or a field-programmable gate array (FPGA) or other programmable logic device, which is used to load and execute program instructions to realize the corresponding method flow or corresponding function.

[0059] Beneficial effects of the present invention:

[0060] (1) The present invention provides an incremental hyperspectral image classification method based on virtual sample expansion, which incrementally expands the recognition capability of the model when facing new ground object categories, without the need to retrain the model from scratch;

[0061] (2) Compared with other incremental hyperspectral image classification methods, the virtual sample expansion network provided by the present invention does not need to use old category data and can effectively suppress feature shift phenomenon;

[0062] (3) The momentum knowledge distillation algorithm adopted by the present invention can effectively maintain the classification ability of old categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is the overall framework diagram of the present invention.

[0064] Figure 2 It is the ground truth map and label map of the hyperspectral dataset.

[0065] Figure 3 This is the result graph and label graph of the incremental hyperspectral image classification model implemented in the present invention. Specific implementation methods

[0067] The incremental hyperspectral image classification using the PaviaU dataset is taken as an example to further illustrate the implementation of the present invention.

[0068] An incremental hyperspectral image classification method based on virtual sample expansion specifically comprises the following steps:

[0069] Step 1: Collect the raw data of hyperspectral images, perform data preprocessing and divide the data set. Specifically:

[0070] Step 1.1, this example uses the PaviaU dataset, which is a hyperspectral image obtained by the ROSIS sensor. The dataset has 115 spectral bands, each with a width of 0.43μm to 0.86μm. After removing 12 spectra of specific water absorption and noise, the remaining 103 spectra are used as experimental data for this study. The size of the PaviaU dataset is 610×340×103, and the number of ground object categories is 9.

[0071] Step 1.2: Use PCA technology to reduce the data dimension of the hyperspectral image obtained in step 1.1 from 103 to 15, and then use a sliding window of 11×11 to segment the hyperspectral image into small blocks of 11×11×15. In this example, there are 42776 small blocks in total.

[0072] Step 1.3, in order to divide the data set, the training process of the incremental hyperspectral image classification model needs to be divided into three stages; three categories are assigned to each stage, and the categories assigned to each stage are different.

[0073] Step 1.4, the data set obtained after processing in step 1.3 is divided into a training data set and a test data set according to a ratio. In this example, the ratio of the training data set is set to 5%.

[0074] Step 2: construct a virtual sample expansion network, which consists of a variational autoencoder and an adversarial autoencoder. Its main function is to generate virtual samples to enhance the current training samples and generate pseudo samples of old categories. Specifically:

[0075] Step 2.1, the basic structure of the variational autoencoder includes an encoder and a decoder. The structure of the encoder is: convolution layer → 3D convolution layer → 3D convolution layer → 2D convolution layer → pooling layer → linear layer → latent space; the structure of the decoder is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer. The loss function L used when optimizing the variational autoencoder VAE It consists of two parts, namely KL divergence L KL and the reconstruction loss L re The loss function can be expressed by formulas (1) to (4).

[0076] z=μ+ε×σ (1)

[0077]

[0078]

[0079] L VAE =L KL +L re (4)

[0080] Where ε is the random noise drawn from the normal distribution; μ represents the mean of the latent space variable in the encoder; σ represents the standard deviation of the latent space variable in the encoder; N represents the number of samples; μ (i) represents the mean of the i-th sample; σ (i) Represents the standard deviation of the i-th sample; I represents the image input to the variational autoencoder, with a shape of N×N×C; Represents the image output by the variational autodecoder; x represents the width index; y represents the height index; and z represents the channel index.

[0081] Step 2.2, the basic structure of the adversarial autoencoder includes an encoder, a decoder and a discriminator, wherein the decoder is also a generator. The structure of the encoder is: convolution layer → 3D convolution layer → 3D convolution layer → 2D convolution layer → pooling layer → linear layer → latent space; the structure of the decoder is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer; the structure of the discriminator is: latent space → linear layer → linear layer. The reconstruction loss of the adversarial autoencoder is the same as that of the variational autoencoder. The losses of the discriminator and the generator are shown in formula (5) and formula (6), where L Drepresents the discriminator loss, L G represents the generator loss.

[0082]

[0083] in, represents the calculation of expectation; Represents the expected probability that the data x generated by the generator is identified by the discriminator as real data; represents the expected probability that the real data x is identified as the real data by the discriminator; D(·) represents the predicted value of the variable by the discriminator; P g represents the distribution of generated samples (latent space); P r represents the distribution of real samples (samples from normal distribution); x represents the distribution of samples from P g and P r A random sample of .

[0084] Step 3: According to the stages divided in step 1.3, the virtual sample expansion network is trained in the initial stage (t=0), specifically:

[0085] Step 3.1, initialize the weights and bias parameters of the variational autoencoder and adversarial autoencoder in the virtual sample expansion network respectively.

[0086] Step 3.2, according to the category set assigned in step 1.3, the training set and test set of the three categories in the category set of the initial stage are input into the virtual sample expansion network constructed in step 2, and 200 rounds of training are performed, each round of training includes: forward propagation, calculation of loss function, back propagation, optimization of model parameters, and cyclic iterative training. The loss function consists of two parts, see steps 2.1 and 2.2 respectively.

[0087] Step 3.3, for each category, save the variational autoencoder and the decoder model in the adversarial autoencoder trained in step 3.2 to form a virtual sample expansion network. In the virtual sample expansion network, the decoder part of the variational autoencoder and the decoder model of the adversarial autoencoder are respectively used to generate two different types of virtual samples.

[0088] Step 4: Build an incremental hyperspectral image classification model: Build a 10-layer feature extraction module, followed by a Softmax classifier. Specifically:

[0089] In step 4.1, the structure of the feature extraction module is as follows: convolution layer → first residual module → second residual module → first pooling layer → third residual module → fourth residual module → second pooling layer → Flatten layer → normalization layer → linear layer.

[0090] In step 4.2, a Softmax classifier is connected after the linear layer of the feature extraction module as the initial form of the incremental hyperspectral image classification model.

[0091] Step 5: According to the stages divided in step 1.3, a training data set for the initial stage (t=0) is constructed to perform hyperspectral image classification in the initial stage. Specifically:

[0092] Step 5.1, generate random Gaussian noise and input it into the decoder model saved in step 3.3 to generate data enhancement samples of the initial training samples. The training samples and the data enhancement samples are combined to construct the initial training set D0.

[0093] Step 5.2: Input the initial training set obtained in step 5.1 into the feature extraction module and Softmax classifier constructed in step 4. Perform 200 rounds of iterative training through forward propagation, loss calculation, back propagation and optimization parameter update to obtain the classification results of the initial stage. At the same time, save the parameters of the feature extraction module and Softmax classifier to obtain the incremental hyperspectral image classification model. The loss function used in the loss calculation is the cross entropy loss L ce , as shown in formula (7).

[0094]

[0095] Among them, C 0 is the category set in the initial stage; p c is the prediction sample x i The probability of belonging to category c; when y = c, δ = 1, otherwise δ = 0; c represents the category index; y i Represents sample x i The real label.

[0096] Step 6: According to the stages and category sets divided in step 1.3, the training data corresponding to the category sets are input into the subsequent incremental stages 1 and 2 in sequence, and the momentum knowledge distillation and the virtual sample expansion network in step 3 are used to further update the parameters of the incremental hyperspectral image classification model obtained in the previous stage, so that the incremental hyperspectral image classification model gradually has the classification ability for all stage learning categories. Specifically:

[0097] Step 6.1, in the incremental learning stage (t>0), the training samples corresponding to the current training stage (i.e., new class samples) are input into the virtual sample expansion network of step 2, trained for 20 rounds and the corresponding decoder is saved.

[0098] Step 6.2, generate random Gaussian noise and input it into the decoder model saved in step 6.1, use the decoder to generate data enhancement samples of the current training samples and old class virtual samples, combine the new class samples, new class enhancement samples and old class virtual samples to construct the incremental training set D t , where t>0.

[0099] Step 6.3, momentum knowledge distillation: Use exponential moving average to update the parameters of the incremental hyperspectral image classification model, as shown in formula (8), and use the knowledge distillation loss L kd Keep the classification ability of the old category, as shown in formula (9):

[0100] F t =α*F t +(1-α)*F t-1 (8)

[0101]

[0102] Among them, F t is the incremental hyperspectral image classification model parameter at stage t, F t-1 is the incremental hyperspectral image classification model parameter at stage t-1, α is set to 0.01, x i represents a training sample, D t represents the incremental training set, F t (x i ) and F t-1 (x i ) represent the incremental hyperspectral image classification models that input training samples into stage t and stage t-1, respectively.

[0103] Step 6.4: The incremental training set D obtained in step 6.2 t Input into the feature extraction module and Softmax classifier in step 4, update the incremental hyperspectral image classification model to obtain the normalized probability. Use the label and the normalized probability to calculate the cross entropy loss, and calculate the knowledge distillation loss. The total loss function is the sum of the cross entropy loss and the knowledge distillation loss. Train for 200 rounds, forward propagation, calculation of loss function, back propagation, and optimization of model parameters. Save the incremental hyperspectral image classification model after training. The incremental hyperspectral image classification model can keep the memory of the old categories when learning the new category classification in the current stage.

[0104] Step 7: Use the incremental hyperspectral image classification model trained in step 6 to perform classification and output the results. Specifically:

[0105] Step 7.1: Input the image to be classified into the incremental hyperspectral image classification model to obtain the category prediction result of each pixel.

[0106] Step 7.2, save the classification results in .mat format and output them to the specified directory.

[0107] Step 7.3, visualize the classification results to visually observe the classification results of the incremental hyperspectral image classification model, including mapping the predicted nine category labels into nine different colors and superimposing them on the original image.

[0108] Furthermore, when training the incremental hyperspectral image classification model, the Adam algorithm was used for optimization. In the initial stage, 100 iterations were performed and the initial learning rate was 10 -2 ; The learning rate decays 10 times at the 45th and 90th iterations. 50 iterations are performed in the incremental phase, with an initial learning rate of 10 -2 The learning decays by a factor of 10 at the 30th iteration, and the batch size is set to 128. When training the virtual sample expansion network, the Adam algorithm is used for 20 iterations, and the learning rate is set to 10 -3 , the batch size is set to 128.

[0109] The above model is used to calculate the three commonly used evaluation indicators for hyperspectral image classification: overall accuracy (OA), average accuracy (AA), and Kappa coefficient. The larger the values ​​of these three indicators, the better the classification results.

[0110] In order to simulate the real hyperspectral image classification, the experiment adopted a multi-stage incremental learning scenario setting. For the nine categories of the PaviaU dataset, three categories were selected for training in the initial stage, and three categories were incrementally learned in two incremental learning stages. Table 1 lists the classification accuracy at each training stage, including the accuracy of each category and the overall classification index.

[0111] Table 1 Classification results of the PaviaU dataset in this example

[0112]

[0113] According to the classification result data in Table 1, in the multi-stage incremental learning scenario, the method of the present invention can maintain the classification ability of the old category without using the old category data, and the classification accuracy decreases slowly, which proves the effectiveness of the method of the present invention in incremental classification. Figure 2 and Figure 3 It can be seen that the classification result map of the method of the present invention is close to the real ground truth map, with only a small number of misclassified points. In short, the method of the present invention can effectively realize incremental hyperspectral image classification.

[0114] The above-mentioned examples only express the implementation modes of the present invention, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. An incremental hyperspectral image classification method based on virtual sample expansion, characterized in that: The incremental hyperspectral image classification method first preprocesses the raw data of the hyperspectral image; secondly, constructs a virtual sample extension network and trains the virtual sample extension network in the initial stage; thirdly, builds and trains the incremental hyperspectral image classification model in the initial stage; finally, uses the virtual sample extension network and momentum knowledge distillation to train the incremental hyperspectral image classification model in the incremental stage, performs classification and outputs the results.

2. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 1 is characterized in that: The following steps are involved: Step 1: Use a hyperspectral imager to collect raw data of hyperspectral images, perform data preprocessing and divide the data set. Specifically: Step 1.1, using a hyperspectral imager to collect raw data of a hyperspectral image, perform data correction and band selection, and obtain a hyperspectral image; Step 1.2, PCA technology is used to reduce the dimension of the hyperspectral image obtained in step 1.1, and a sliding window is used to divide the hyperspectral image into small blocks of shape N×N×C, where C is the dimension of the image after dimensionality reduction processing, and N is the size of the sliding window; Step 1.3, in order to divide the data set, the training process of the incremental hyperspectral image classification model is divided into several stages (0, 1, 2, ..., t..., T), where T represents the number of incremental stages; for each stage t, the incremental hyperspectral image classification model is assigned a new set of categories C that need to be learned t ,in, represents the mth category of stage t; define C i and C j Represent the category sets assigned to stage i and stage j respectively, where where i≠j, is an empty set, indicating that the newly learned hyperspectral image categories are different in each stage; Step 1.4, divide the data set obtained after processing in step 1.3 into a training data set and a test data set in proportion; Step 2: construct a virtual sample expansion network, which consists of a variational autoencoder and an adversarial autoencoder. Its main function is to generate virtual samples to enhance the current training samples and generate pseudo samples of old categories. Specifically: Step 2.1, the basic structure of the variational autoencoder includes an encoder and a decoder; the loss function L used when optimizing the variational autoencoder VAE It consists of two parts, namely KL divergence L KL and the reconstruction loss L re ; Step 2.2, the basic structure of the adversarial autoencoder includes an encoder, a decoder and a discriminator, wherein the decoder is also a generator; Step 3: According to the stages divided in step 1.3, the virtual sample expansion network is trained in the initial stage (t=0), specifically: Step 3.1, respectively initialize the weights and bias parameters of the variational autoencoder and adversarial autoencoder in the virtual sample expansion network; Step 3.2, according to the category set assigned in step 1.3, the training set and test set of each category in the category set in the initial stage are input into the virtual sample expansion network constructed in step 2, and the training is performed iteratively according to the set training rounds; Step 3.3, saving the variational autoencoder and the decoder model in the adversarial autoencoder trained in step 3.2 for each category to form a virtual sample expansion network; in the virtual sample expansion network, the decoder model of the variational autoencoder and the decoder model of the adversarial autoencoder are respectively used to generate two different types of virtual samples; Step 4: Build an incremental hyperspectral image classification model: Build a feature extraction module, and the last linear layer of the feature extraction module is followed by a Softmax classifier as the initial form of the incremental hyperspectral image classification model; Step 5: According to the stages divided in step 1.3, construct the initial training data set and perform the initial hyperspectral image classification. Specifically: Step 5.1, generate random Gaussian noise and input it into the decoder model saved in step 3.3 to generate data augmentation samples of the initial training samples. The training samples and data augmentation samples are combined to construct the initial training set D0; Step 5.2, input the initial training set obtained in step 5.1 into the feature extraction module and Softmax classifier constructed in step 4, perform iterative training through forward propagation, loss calculation, back propagation and optimization parameter update, obtain the classification results of the initial stage, and save the parameters of the feature extraction module and Softmax classifier to obtain an incremental hyperspectral image classification model; Step 6: According to the stages and category sets divided in step 1.3, the training data corresponding to the category set are sequentially input into the subsequent incremental learning stage of t>0, and the momentum knowledge distillation and the virtual sample expansion network of step 3 are used to further update the parameters of the incremental hyperspectral image classification model obtained in the previous stage, so that the incremental hyperspectral image classification model gradually has the classification ability for all stage learning categories; specifically: Step 6.1, in the incremental learning phase when t>0, input the training sample corresponding to the current training phase into the virtual sample expansion network of step 2, and train and save the corresponding decoder; Step 6.2, generate random Gaussian noise and input it into the decoder saved in step 6.1, use the decoder to generate data enhancement samples of the current training samples and old class virtual samples, combine the new class samples, new class enhancement samples and old class virtual samples to construct the incremental training set D t , where t>0; Step 6.3, Momentum Knowledge Distillation: Update the parameters of the incremental hyperspectral image classification model using exponential moving average and use the knowledge distillation loss L kd Maintaining the classification capabilities of old categories; Step 6.4: The incremental training set D obtained in step 6.2 t Input into the feature extraction module and Softmax classifier in step 4, update the incremental hyperspectral image classification model to obtain the normalized probability; use the label and the normalized probability to calculate the cross entropy loss, and calculate the knowledge distillation loss. The total loss function is the sum of the cross entropy loss and the knowledge distillation loss; train according to a fixed round, forward propagation, calculation of loss function, back propagation, and optimization of model parameters; save the incremental hyperspectral image classification model after training. The incremental hyperspectral image classification model can keep the memory of the old categories when learning the new category classification in the current stage; Step 7: Use the incremental hyperspectral image classification model trained in step 6 to perform classification and output the results. Specifically: Step 7.1, input the image to be classified into the incremental hyperspectral image classification model to obtain the category prediction result of each pixel; Step 7.2, save the classification results as image files or other formats, and output them to a specified directory or storage medium; Step 7.3, visualize the classification results to intuitively observe the classification results of the incremental hyperspectral image classification model; including mapping the predicted category labels to different colors and superimposing them on the original image.

3. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2 is characterized in that: In step 2.1: The structure of the encoder is: convolution layer → 3D convolution layer → 3D convolution layer → 2D convolution layer → pooling layer → linear layer → latent space; The structure of the decoder is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer; The loss function is expressed by formula (1) to formula (4); z=μ+ε×σ (1) L VAE =L KL +L re (4) Where ε is the random noise drawn from the normal distribution; μ represents the mean of the latent space variable in the encoder; σ represents the standard deviation of the latent space variable in the encoder; N represents the number of samples; μ (i) represents the mean of the i-th sample; σ (i) Represents the standard deviation of the i-th sample; I represents the image input to the variational autoencoder, with a shape of N×N×C; Represents the image output by the variational autodecoder; x represents the width index; y represents the height index; and z represents the channel index.

4. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2 is characterized in that: In step 2.2: The structure of the encoder is: convolution layer → 3D convolution layer → 3D convolution layer → 2D convolution layer → pooling layer → linear layer → latent space; the reconstruction loss of the adversarial autoencoder is the same as the reconstruction loss of the variational autoencoder; The structure of the decoder is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer; the structure of the discriminator is: latent space → linear layer → linear layer; The losses of the discriminator and generator are shown in formula (5) and formula (6), where L D represents the discriminator loss, L G represents the generator loss; in, represents the calculation of expectation; Represents the expected probability that the data x generated by the generator is identified by the discriminator as real data; represents the expected probability that the real data x is identified as the real data by the discriminator; D(·) represents the predicted value of the variable by the discriminator; P g represents the distribution of generated samples; P r represents the distribution of real samples; x represents the distribution of samples from P g and P r A random sample of .

5. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2 is characterized in that: In the step 3.2, each round of training in the iterative training process includes: forward propagation, calculation of loss function, back propagation, and optimization of model parameters.

6. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2 is characterized in that: In the step 4, the feature extraction module has a 10-layer structure, which is: convolution layer → first residual module → second residual module → first pooling layer → third residual module → fourth residual module → second pooling layer → Flatten layer → normalization layer → linear layer.

7. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2 is characterized in that: In step 5.2, the loss function used in the loss calculation is the cross entropy loss L ce , as shown in formula (7); Among them, C 0 is the category set in the initial stage; p c is the prediction sample x i The probability of belonging to category c; when y = c, δ = 1, otherwise δ = 0; c represents the category index; y i Represents sample x i The real label.

8. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2 is characterized in that: In step 6.3, formula (8) is used to update the parameters of the incremental hyperspectral image classification model, and formula (9) is used to maintain the classification ability of the old category: F t =α*F t +(1-a)*F t-1 (8) Among them, F t is the incremental hyperspectral image classification model parameter at stage t, F t-1 is the incremental hyperspectral image classification model parameter of stage t-1, α is the set hyperparameter, x i represents a training sample, D t represents the incremental training set, F t (x i ) and F t-1 (x i ) represent the incremental hyperspectral image classification models that input training samples into stage t and stage t-1, respectively.

9. An incremental hyperspectral image classification system for implementing the incremental hyperspectral image classification method according to any one of claims 1 to 8, characterized in that: The incremental hyperspectral image classification system includes the following modules: an acquisition module: used to obtain raw data of hyperspectral images; Storage medium: used to store acquired hyperspectral image data and computer programs; the computer programs include a data set construction program, a virtual sample expansion network program, an initial stage program, an incremental learning stage program, and a result saving program; Processor: used to load and execute data and computer programs in storage media. When the computer program is executed, the processor is used to: Read the hyperspectral raw data collected and stored by the hyperspectral imager; use the data set construction program to perform data preprocessing and organize and divide the data set; use the virtual sample extension network program to train the model; use the initial stage program to classify the initial training samples; use the incremental learning stage program to update and adjust the model to maintain the classification effect of the old categories; use the result saving program to save the classification results to the storage medium.

10. A terminal device of the incremental hyperspectral image classification system according to claim 9, characterized in that: The terminal device includes an acquisition module, a storage medium and a processor. The storage medium is a memory system in the terminal device for storing programs and data. The processor is used to execute program instructions stored in the storage medium. The acquisition module is a hyperspectral imager.

Images