An incremental hyperspectral image classification method based on virtual sample expansion

Abstract

The present invention provides an incremental hyperspectral image classification method based on virtual sample expansion, which belongs to the field of remote sensing image processing and application technology. First, data preprocessing is performed on the original data of the hyperspectral image; second, a virtual sample expansion network is constructed and the virtual sample expansion network of the initial stage is trained; third, an incremental hyperspectral image classification model of the initial stage is constructed and trained; finally, the incremental hyperspectral image classification model of the incremental stage is trained using the virtual sample expansion network and momentum knowledge distillation, and classification is performed and the results are output. The present invention can perform incremental learning on hyperspectral images collected from different hyperspectral imagers, thereby achieving the classification ability of learning new categories while not forgetting old categories; the present invention provides a feasible solution for incremental hyperspectral image classification, bringing the hyperspectral image classification theory one step closer to large-scale practical application.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing and application, and relates to an incremental hyperspectral image classification method, and in particular to an incremental hyperspectral image classification method based on virtual sample expansion. Background Art

[0002] Hyperspectral image classification, a pixel-level classification task for ground objects, is a key step in many remote sensing applications. Incremental hyperspectral image classification is a scalable classification model that can continuously identify new object categories. It incrementally expands the model's recognition capabilities for new object categories, eliminating the need to retrain the model from scratch and alleviating computational and memory pressures. Incremental hyperspectral image classification can significantly improve the efficiency of wide-bandwidth, long-time-series Earth observation missions.

[0003] Researchers have conducted extensive research on remote sensing image classification methods and models and achieved excellent results:

[0004] Qiqihar University has proposed a new remote sensing scene image classification method in its patented technology, "Remote Sensing Scene Image Classification Method Based on Multi-level Feature Dense Fusion" (Patent Application No.: 202111037856.2, Authorization Announcement No.: CN113705718B). This method achieves remote sensing scene image classification by establishing a lightweight convolutional neural network based on a dual-branch multi-level feature dense fusion. This remote sensing scene image classification method can classify broad geographic areas, but it cannot accurately classify fine-grained scenes.

[0005] Yunnan University has proposed a hyperspectral image classification method in its patented technology, "Hyperspectral Image Classification Method, Device, and Electronic Device" (Patent Application No.: 202310869709.4, Authorization Announcement No.: CN116595208B). This method uses a dual-channel Siamese network trained with convolutional residual blocks and a spatial attention mechanism, and employs transfer learning to classify hyperspectral images in small sample scenarios. This method achieves fine classification but cannot classify new image categories.

[0006] Xidian University has proposed an image classification method based on incremental learning using linear programming in its patented technology, "Image Classification Method Based on Incremental Learning Using Linear Programming" (Patent Application Number: CN201911348984.1, Authorization Announcement Number: CN111222545B). This method, when new class data is scarce, calculates the mean vector of the new class's features and uses linear programming to update the classifier weights, thereby achieving rapid learning while maintaining the ability to classify the original class. This method is capable of incremental image classification when new class data is scarce, but it can only incrementally classify one new class.

[0007] In their paper "Distillation-Constrained Prototype Representation Network for Hyperspectral Image Incremental Classification," published in IEEE Trans. Geoscience and Remote Sensing, Volume 62, in 2024, Chunyan Yu et al. proposed an incremental learning method for a distillation-constrained prototype representation network. This method uses samples from the initial category for training to establish an initial model. During the incremental learning phase, a dual knowledge distillation module performs knowledge transfer to maintain the ability to recognize the original category. This method can learn new categories without directly using historical data, retains old categories, and exhibits good generalization capabilities.

[0008] Existing incremental hyperspectral image classification methods either directly use old class data, requiring significant storage resources, or indirectly use old class data. As the incremental learning phase increases, the old class features generated by the model experience feature shifts. These shortcomings are key issues hindering the advancement of incremental hyperspectral image classification from theoretical perspectives to large-scale Earth observation applications. Summary of the Invention

[0009] The purpose of the present invention is to overcome the defects of the prior art and provide an incremental hyperspectral image classification method based on virtual sample expansion, which is used to solve the problem of irrational utilization of old class samples in the current incremental hyperspectral image classification technology, reduce storage resource usage and prevent feature shift.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] A novel incremental hyperspectral image classification method based on virtual sample expansion is proposed. First, the raw hyperspectral image data is preprocessed. Second, a virtual sample expansion network is constructed and trained in the initial phase. Third, an incremental hyperspectral image classification model is constructed and trained in the initial phase. Finally, the incremental hyperspectral image classification model is trained in the incremental phase using the virtual sample expansion network and momentum knowledge distillation, and classification is performed and output.

[0012] The following steps are involved:

[0013] Step 1: Use a hyperspectral imager to collect raw hyperspectral image data, perform data preprocessing and data set division, specifically:

[0014] Step 1.1: Use a hyperspectral imager to collect raw data of hyperspectral images, perform data correction and band selection, and obtain hyperspectral images.

[0015] In step 1.2, PCA technology is used to reduce the dimensionality of the hyperspectral image obtained in step 1.1. A sliding window is used to split the hyperspectral image into small blocks of shape N×N×C, where C is the image dimension after dimensionality reduction and N is the size of the sliding window.

[0016] Step 1.3: In order to divide the data set, the training process of the incremental hyperspectral image classification model needs to be divided into several stages (0, 1, 2, ..., t..., T), where T represents the number of incremental stages; for each stage t, the incremental hyperspectral image classification model is assigned a new category set C that needs to be learned. t ,in, Denotes the mth category of stage t. Define C i and C j denote the category sets assigned to stage i and stage j respectively, where Where i≠j, It is an empty set, indicating that the categories of newly learned hyperspectral images in each stage are different.

[0017] Step 1.4: Divide the dataset obtained after step 1.3 into a training dataset and a test dataset in proportion. The proportion of the training dataset is usually set to 10% or 5%.

[0018] Step 2: Construct a virtual sample expansion network. The virtual sample expansion network consists of a variational autoencoder and an adversarial autoencoder. Its main function is to generate virtual samples to enhance the current training samples and generate pseudo samples of old categories. Specifically:

[0019] Step 2.1: The basic structure of the variational autoencoder includes an encoder and a decoder. The encoder structure is: convolution layer → 3D convolution layer → 3D convolution layer → 2D convolution layer → pooling layer → linear layer → latent space; the decoder structure is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer. The loss function L used in optimizing the variational autoencoder is: VAE It consists of two parts, namely KL divergence L KL and reconstruction loss L re The loss function can be expressed by formula (1) to formula (4).

[0020] z=μ+ε×σ (1)

[0021]

[0022]

[0023] L VAE =L KL +L re (4)

[0024] Where ε is the random noise drawn from the normal distribution; μ represents the mean of the latent space variable in the encoder; σ represents the standard deviation of the latent space variable in the encoder; N represents the number of samples; μ (i) represents the mean of the i-th sample; σ (i) Represents the standard deviation of the i-th sample; I represents the image input to the variational autoencoder, with a shape of N×N×C; Represents the image output by the variational autodecoder; x represents the width index; y represents the height index; and z represents the channel index.

[0025] Step 2.2, the basic structure of the adversarial autoencoder includes an encoder, a decoder, and a discriminator, wherein the decoder is also a generator. The structure of the encoder is: convolution layer → 3D convolution layer → 3D convolution layer → 2D convolution layer → pooling layer → linear layer → latent space; the structure of the decoder is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer; the structure of the discriminator is: latent space → linear layer → linear layer. The reconstruction loss of the adversarial autoencoder is the same as that of the variational autoencoder. The losses of the discriminator and the generator are shown in formulas (5) and (6), where L D represents the discriminator loss, L G represents the generator loss.

[0026]

[0027] in, represents the calculation of expectation; Represents the expected probability that the data x generated by the generator is identified by the discriminator as real data; represents the expected probability that the real data x is identified as real data by the discriminator; D(·) represents the predicted value of the variable by the discriminator; P g represents the distribution of generated samples (latent space); P r represents the distribution of real samples (samples from normal distribution); x represents the distribution of samples from P g and P r A random sample of .

[0028] Step 3: According to the stages divided in step 1.3, the virtual sample expansion network is trained in the initial stage (t=0). Specifically:

[0029] In step 3.1, the weights and bias parameters of the variational autoencoder and adversarial autoencoder in the virtual sample extended network are initialized respectively.

[0030] In step 3.2, based on the category set assigned in step 1.3, the training and test sets for each category in the initial category set are input into the virtual sample expansion network constructed in step 2. Training is performed according to the set training rounds. Each training round includes forward propagation, loss function calculation, backpropagation, and model parameter optimization, and the training is repeated iteratively. The loss function consists of two parts, as described in steps 2.1 and 2.2, respectively.

[0031] In step 3.3, the decoder models of the variational autoencoder and the adversarial autoencoder trained in step 3.2 are saved for each category to form a virtual sample expansion network. In the virtual sample expansion network, the decoder model of the variational autoencoder and the decoder model of the adversarial autoencoder are used to generate two different types of virtual samples.

[0032] Step 4: Build an incremental hyperspectral image classification model: Build a 10-layer feature extraction module, followed by a Softmax classifier. Specifically:

[0033] In step 4.1, the structure of the feature extraction module is as follows: convolutional layer → first residual module → second residual module → first pooling layer → third residual module → fourth residual module → second pooling layer → Flatten layer → normalization layer → linear layer.

[0034] In step 4.2, a Softmax classifier is connected after the linear layer of the feature extraction module as the initial form of the incremental hyperspectral image classification model.

[0035] Step 5: Based on the stages divided in step 1.3, construct a training data set for the initial stage (t=0) and perform hyperspectral image classification in the initial stage. Specifically:

[0036] In step 5.1, random Gaussian noise is generated and input into the decoder model saved in step 3.3 to generate data augmentation samples for the initial training samples. The initial training set D0 is constructed by combining the training samples and the data augmentation samples.

[0037] In step 5.2, the initial training set obtained in step 5.1 is input into the feature extraction module and Softmax classifier constructed in step 4. Through forward propagation, loss calculation, back propagation and optimization parameter update, iterative training is performed to obtain the classification results of the initial stage. At the same time, the parameters of the feature extraction module and Softmax classifier are saved to obtain the incremental hyperspectral image classification model. The loss function used in the loss calculation is the cross entropy loss L ce , as shown in formula (7).

[0038]

[0039] Among them, C 0 is the category set in the initial stage; p c is the predicted sample x i The probability of belonging to category c; when y = c, δ = 1, otherwise δ = 0; c represents the category index; y i Represents sample x i The true label.

[0040] In step 6, according to the stages and category sets divided in step 1.3, the training data corresponding to the category set are input into the subsequent incremental learning stage (t>0) in sequence, and the momentum knowledge distillation and the virtual sample expansion network in step 3 are used to further update the parameters of the incremental hyperspectral image classification model obtained in the previous stage, so that the incremental hyperspectral image classification model gradually has the classification ability for all stages of learning categories.

[0041] Specifically:

[0042] Step 6.1: In the incremental learning phase (t>0), the training samples corresponding to the current training phase (i.e., new class samples) are input into the virtual sample expansion network of step 2, and the corresponding decoder is trained and saved.

[0043] Step 6.2: Generate random Gaussian noise and input it into the decoder saved in step 6.1. Use the decoder to generate data enhancement samples of the current training samples and virtual samples of the old class. Combine the new class samples, new class enhancement samples and old class virtual samples to construct the incremental training set D. t , where t>0.

[0044] Step 6.3, Momentum Knowledge Distillation: Update the parameters of the incremental hyperspectral image classification model using exponential moving average, as shown in formula (8), and use the knowledge distillation loss L kd Maintain the classification ability of the old category, as shown in formula (9):

[0045] F t =α*F t +(1-α)*F t-1 (8)

[0046]

[0047] Among them, F t is the incremental hyperspectral image classification model parameter at stage t, F t-1 is the incremental hyperspectral image classification model parameter of stage t-1, α is the set hyperparameter, x i represents a training sample, D t represents the incremental training set, F t (x i ) and F t-1 (x i ) represent the incremental hyperspectral image classification models that input training samples into stage t and stage t-1, respectively.

[0048] Step 6.4: The incremental training set D obtained in step 6.2 is t This input is fed into the feature extraction module and Softmax classifier from step 4, and the incremental hyperspectral image classification model is updated to obtain normalized probabilities. The labels and normalized probabilities are used to calculate the cross-entropy loss and the knowledge distillation loss. The total loss function is the sum of the cross-entropy loss and the knowledge distillation loss. Training is performed in a fixed number of rounds, including forward propagation, loss function calculation, backpropagation, and model parameter optimization. The trained incremental hyperspectral image classification model is saved. This model retains its memory of previous categories while learning new ones.

[0049] Step 7: Use the incremental hyperspectral image classification model trained in step 6 to perform classification and output the results. Specifically:

[0050] In step 7.1, the image to be classified is input into the incremental hyperspectral image classification model to obtain the category prediction result of each pixel.

[0051] Step 7.2: Save the classification results as image files or other formats and output them to a specified directory or storage medium.

[0052] Step 7.3: Visualize the classification results to visually observe the classification results of the incremental hyperspectral image classification model. This includes mapping the predicted category labels to different colors and overlaying them on the original image.

[0053] An incremental hyperspectral image classification system based on virtual sample expansion includes the following modules:

[0054] Acquisition module: used to obtain raw data of hyperspectral images;

[0055] Storage medium: used to store acquired hyperspectral image data and computer programs; the computer programs include a data set construction program, a virtual sample expansion network program, an initial stage program, an incremental learning stage program, and a result saving program;

[0056] Processor: used to load and execute data and computer programs from storage media. When the computer program is executed, the processor is used to:

[0057] Read the hyperspectral raw data collected and stored by the hyperspectral imager; use the data set construction program to perform data preprocessing and organize and divide the data set; use the virtual sample extension network program to train the model; use the initial stage program to classify the initial training samples; use the incremental learning stage program to update and adjust the model to maintain the classification effect of the old categories; use the result saving program to save the classification results to the storage medium.

[0058] A terminal device for an incremental hyperspectral image classification system includes an acquisition module, a storage medium, and a processor. The storage medium is a memory system within the terminal device for storing programs and data, and the processor is configured to execute program instructions stored in the storage medium. The acquisition module can be a hyperspectral imager mounted on a satellite, unmanned aerial vehicle, or the like. The storage medium can be high-speed RAM or non-volatile memory. The processor can be a central processing unit (CPU), a field-programmable gate array (FPGA), or other programmable logic device, configured to load and execute program instructions to implement a corresponding method flow or function.

[0059] Beneficial effects of the present invention:

[0060] (1) The present invention provides an incremental hyperspectral image classification method based on virtual sample expansion, which incrementally expands the recognition capability of the model when facing new ground object categories without retraining the model from scratch;

[0061] (2) Compared with other incremental hyperspectral image classification methods, the virtual sample expansion network proposed in this paper does not require the use of old category data and can effectively suppress feature offset phenomena;

[0062] (3) The momentum knowledge distillation algorithm adopted in this invention can effectively maintain the classification ability of old categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is the overall framework diagram of the present invention.

[0064] Figure 2 is the ground truth map and label map of the hyperspectral dataset.

[0065] Figure 3 This is the result graph and label graph of the incremental hyperspectral image classification model implemented in this invention. Specific implementation methods

[0066] The incremental hyperspectral image classification using the PaviaU dataset is taken as an example to further illustrate the embodiments of the present invention.

[0067] An incremental hyperspectral image classification method based on virtual sample expansion specifically includes the following steps:

[0068] Step 1: Collect the raw data of hyperspectral images, perform data preprocessing and data set division, specifically:

[0069] In Step 1.1, this example uses the PaviaU dataset, a hyperspectral image acquired by the ROSIS sensor. This dataset has 115 spectral bands, each ranging in width from 0.43 μm to 0.86 μm. After removing 12 spectra with specific water absorption and noise, the remaining 103 spectra were used as experimental data for this study. The PaviaU dataset has a size of 610 × 340 × 103 and contains nine object classes.

[0070] In step 1.2, PCA is used to reduce the data dimension of the hyperspectral image obtained in step 1.1 from 103 to 15. Then, an 11×11 sliding window is used to segment the hyperspectral image into 11×11×15 patches. In this example, there are 42,776 patches in total.

[0071] In step 1.3, in order to divide the data set, the training process of the incremental hyperspectral image classification model needs to be divided into three stages; three categories are assigned to each stage, and the categories assigned to each stage are different.

[0072] Step 1.4: Divide the dataset obtained after processing in step 1.3 into a training dataset and a test dataset in proportion. In this example, the proportion of the training dataset is set to 5%.

[0073] Step 2: Construct a virtual sample expansion network. The virtual sample expansion network consists of a variational autoencoder and an adversarial autoencoder. Its main function is to generate virtual samples to enhance the current training samples and generate pseudo samples of old categories. Specifically:

[0074] Step 2.1: The basic structure of the variational autoencoder includes an encoder and a decoder. The encoder structure is: convolution layer → 3D convolution layer → 3D convolution layer → 2D convolution layer → pooling layer → linear layer → latent space; the decoder structure is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer. The loss function L used in optimizing the variational autoencoder is: VAE It consists of two parts, namely KL divergence L KL and reconstruction loss L re The loss function can be expressed by formulas (1) to (4).

[0075] z=μ+ε×σ (1)

[0076]

[0077]

[0078] L VAE =L KL +L re (4)

[0079] Where ε is the random noise drawn from the normal distribution; μ represents the mean of the latent space variable in the encoder; σ represents the standard deviation of the latent space variable in the encoder; N represents the number of samples; μ (i) represents the mean of the i-th sample; σ (i) Represents the standard deviation of the i-th sample; I represents the image input to the variational autoencoder, with a shape of N×N×C; Represents the image output by the variational autodecoder; x represents the width index; y represents the height index; and z represents the channel index.

[0080] Step 2.2, the basic structure of the adversarial autoencoder includes an encoder, a decoder, and a discriminator, wherein the decoder is also a generator. The structure of the encoder is: convolution layer → 3D convolution layer → 3D convolution layer → 2D convolution layer → pooling layer → linear layer → latent space; the structure of the decoder is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer; the structure of the discriminator is: latent space → linear layer → linear layer. The reconstruction loss of the adversarial autoencoder is the same as that of the variational autoencoder. The losses of the discriminator and the generator are shown in formulas (5) and (6), where L Drepresents the discriminator loss, L G represents the generator loss.

[0081]

[0082] in, represents the calculation of expectation; Represents the expected probability that the data x generated by the generator is identified by the discriminator as real data; represents the expected probability that the real data x is identified as real data by the discriminator; D(·) represents the predicted value of the variable by the discriminator; P g represents the distribution of generated samples (latent space); P r represents the distribution of real samples (samples from normal distribution); x represents the distribution of samples from P g and P r A random sample of .

[0083] Step 3: According to the stages divided in step 1.3, the virtual sample expansion network is trained in the initial stage (t=0). Specifically:

[0084] In step 3.1, the weights and bias parameters of the variational autoencoder and adversarial autoencoder in the virtual sample extended network are initialized respectively.

[0085] In step 3.2, based on the category set assigned in step 1.3, the training and test sets for the three categories in the initial category set are fed into the virtual sample expansion network constructed in step 2. Training is performed for 200 rounds. Each round of training consists of the following steps: forward propagation, loss function calculation, backpropagation, and model parameter optimization, with training being repeated iteratively. The loss function consists of two parts, as described in steps 2.1 and 2.2, respectively.

[0086] In step 3.3, for each category, the decoder model of the variational autoencoder and the adversarial autoencoder trained in step 3.2 are saved to form a virtual sample expansion network. In the virtual sample expansion network, the decoder part of the variational autoencoder and the decoder model of the adversarial autoencoder are used to generate two different types of virtual samples.

[0087] Step 4: Build an incremental hyperspectral image classification model: Build a 10-layer feature extraction module, followed by a Softmax classifier. Specifically:

[0088] In step 4.1, the structure of the feature extraction module is as follows: convolutional layer → first residual module → second residual module → first pooling layer → third residual module → fourth residual module → second pooling layer → Flatten layer → normalization layer → linear layer.

[0089] In step 4.2, a Softmax classifier is connected after the linear layer of the feature extraction module as the initial form of the incremental hyperspectral image classification model.

[0090] Step 5: Based on the stages divided in step 1.3, construct a training data set for the initial stage (t=0) and perform hyperspectral image classification in the initial stage. Specifically:

[0091] In step 5.1, random Gaussian noise is generated and input into the decoder model saved in step 3.3 to generate data augmentation samples for the initial training samples. The training samples and data augmentation samples are combined to construct the initial training set D0.

[0092] In step 5.2, the initial training set obtained in step 5.1 is input into the feature extraction module and Softmax classifier constructed in step 4. Through forward propagation, loss calculation, back propagation and optimization parameter update, 200 rounds of iterative training are performed to obtain the classification results of the initial stage. At the same time, the parameters of the feature extraction module and Softmax classifier are saved to obtain the incremental hyperspectral image classification model. The loss function used in the loss calculation is the cross entropy loss L ce , as shown in formula (7).

[0093]

[0094] Among them, C 0 is the category set in the initial stage; p c is the predicted sample x i The probability of belonging to category c; when y = c, δ = 1, otherwise δ = 0; c represents the category index; y i Represents sample x i The true label.

[0095] Step 6: Based on the stages and category sets divided in step 1.3, the training data corresponding to the category sets are input into the subsequent incremental stages 1 and 2 in sequence. Momentum knowledge distillation and the virtual sample expansion network in step 3 are used to further update the parameters of the incremental hyperspectral image classification model obtained in the previous stage, so that the incremental hyperspectral image classification model gradually has the classification ability for all stages of learning categories. Specifically:

[0096] Step 6.1: In the incremental learning stage (t>0), the training samples corresponding to the current training stage (i.e., new class samples) are input into the virtual sample expansion network of step 2, trained for 20 rounds, and the corresponding decoder is saved.

[0097] Step 6.2: Generate random Gaussian noise and input it into the decoder model saved in step 6.1. Use the decoder to generate data enhancement samples of the current training samples and virtual samples of the old class. Combine the new class samples, new class enhancement samples and old class virtual samples to construct the incremental training set D. t , where t>0.

[0098] Step 6.3, Momentum Knowledge Distillation: Update the parameters of the incremental hyperspectral image classification model using exponential moving average, as shown in formula (8), and use the knowledge distillation loss L kd Maintain the classification ability of the old category, as shown in formula (9):

[0099] F t =α*F t +(1-α)*F t-1 (8)

[0100]

[0101] Among them, F t is the incremental hyperspectral image classification model parameter at stage t, F t-1 is the incremental hyperspectral image classification model parameter at stage t-1, α is set to 0.01, x i represents a training sample, D t represents the incremental training set, F t (x i ) and F t-1 (x i ) represent the incremental hyperspectral image classification models that input training samples into stage t and stage t-1, respectively.

[0102] Step 6.4: The incremental training set D obtained in step 6.2 is t This is input into the feature extraction module and Softmax classifier from step 4, and the incremental hyperspectral image classification model is updated to obtain normalized probabilities. The labels and normalized probabilities are used to calculate the cross-entropy loss and the knowledge distillation loss. The total loss function is the sum of the cross-entropy loss and the knowledge distillation loss. Training is performed for 200 epochs, including forward propagation, loss function calculation, backpropagation, and model parameter optimization. The trained incremental hyperspectral image classification model is saved. This model retains the memory of previous categories while learning new ones.

[0103] Step 7: Use the incremental hyperspectral image classification model trained in step 6 to perform classification and output the results. Specifically:

[0104] In step 7.1, the image to be classified is input into the incremental hyperspectral image classification model to obtain the category prediction result of each pixel.

[0105] Step 7.2: Save the classification results in .mat format and output them to the specified directory.

[0106] Step 7.3: Visualize the classification results to visually observe the classification results of the incremental hyperspectral image classification model. This involves mapping the nine predicted category labels into nine different colors and overlaying them onto the original image.

[0107] Furthermore, when training the incremental hyperspectral image classification model, the Adam algorithm was used for optimization. In the initial stage, 100 iterations were performed and the initial learning rate was 10. -2 The learning rate decays by a factor of 10 at the 45th and 90th iterations respectively. The incremental phase is performed for 50 iterations with an initial learning rate of 10 -2 The learning rate is set to 10 and the batch size is set to 128. When training the virtual sample expansion network, the Adam algorithm is used for 20 iterations and the learning rate is set to 10. -3 , the batch size is set to 128.

[0108] The above model is used to calculate three commonly used evaluation indicators for hyperspectral image classification: overall accuracy (OA), average accuracy (AA), and Kappa coefficient. The larger the values of these three indicators, the better the classification results.

[0109] To simulate real-world hyperspectral image classification, the experiment employed a multi-stage incremental learning scenario. For the nine categories in the PaviaU dataset, three categories were selected for training in the initial phase, and three categories were incrementally learned in each of the two incremental learning phases. Table 1 lists the classification accuracy at each training phase, including the accuracy for each category and the overall classification metric.

[0110] Table 1 Classification results of the PaviaU dataset in this example

[0111]

[0112] According to the classification results in Table 1, in the multi-stage incremental learning scenario, the method of the present invention can maintain the classification ability of the old category without using the old category data, and the classification accuracy decreases slowly, which proves the effectiveness of the method of the present invention in incremental classification. Figure 2 and Figure 3 It can be seen that the classification result map of the method of the present invention is close to the real ground truth map, with only a small number of misclassified points. In short, the method of the present invention can effectively realize incremental hyperspectral image classification.

[0113] The above examples merely express the embodiments of the present invention, but they should not be understood as limiting the scope of the patent of the present invention. It should be pointed out that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. An incremental hyperspectral image classification method based on virtual sample expansion, characterized in that: The following steps are involved: Step 1: Use a hyperspectral imager to collect raw hyperspectral image data, perform data preprocessing and data set division. In order to divide the data set, the training process of the incremental hyperspectral image classification model is divided into several stages (0, 1, 2, ..., t..., T), where T represents the number of incremental stages; for each stage t, the incremental hyperspectral image classification model is assigned a new category set C that needs to be learned. t ,in, represents the mth category at stage t; Step 2: construct a virtual sample expansion network, which consists of a variational autoencoder and an adversarial autoencoder to generate virtual samples to enhance the current training samples and generate pseudo samples of old categories; Step 3: According to the stages divided in step 1, the virtual sample expansion network is trained in the initial stage (t=0); Step 4: Build an incremental hyperspectral image classification model: Build a feature extraction module, and the last linear layer of the feature extraction module is followed by a Softmax classifier as the initial form of the incremental hyperspectral image classification model; Step 5: According to the stages divided in step 1, construct the initial stage training data set and perform the initial stage hyperspectral image classification; Step 6: Based on the stages and category sets divided in step 1, the training data corresponding to the category sets are sequentially input into the subsequent incremental learning stages (t>0). Momentum knowledge distillation and the virtual sample expansion network in step 3 are used to further update the parameters of the incremental hyperspectral image classification model obtained in the previous stage, so that the incremental hyperspectral image classification model gradually has the classification ability for all stages of learning categories. Specifically: Step 6.1: In the incremental learning phase with t>0, the training sample corresponding to the current training phase is input into the virtual sample expansion network of step 2, and the corresponding decoder is trained and saved; Step 6.2: Generate random Gaussian noise and input it into the decoder saved in step 6.

1. Use the decoder to generate data enhancement samples of the current training samples and virtual samples of the old class. Combine the new class samples, new class enhancement samples and old class virtual samples to construct the incremental training set D. t , where t>0; Step 6.3, Momentum Knowledge Distillation: Update the parameters of the incremental hyperspectral image classification model using exponential moving average and use the knowledge distillation loss L kd Maintaining the categorization power of old categories; Step 6.4: The incremental training set D obtained in step 6.2 is t Input the labels into the feature extraction module and Softmax classifier in step 4, update the incremental hyperspectral image classification model to obtain normalized probabilities; use the labels and normalized probabilities to calculate the cross entropy loss and the knowledge distillation loss. The total loss function is the sum of the cross entropy loss and the knowledge distillation loss; train for a fixed number of rounds; and save the incremental hyperspectral image classification model after training. Step 7: Use the incremental hyperspectral image classification model trained in step 6 to perform classification and output the results.

2. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 1, characterized in that: Step 1 specifically includes: Step 1.1: Use a hyperspectral imager to collect raw data of hyperspectral images, perform data correction and band selection, and obtain hyperspectral images; Step 1.2: Use PCA technology to reduce the dimensionality of the hyperspectral image obtained in step 1.

1. Use a sliding window to split the hyperspectral image into small blocks of shape N×N×C, where C is the image dimension after dimensionality reduction and N is the size of the sliding window. Step 1.3, divide the dataset and assign the incremental hyperspectral image classification model to each stage t to a new set of categories C that need to be learned t ;Define C i and C j denote the category sets assigned to stage i and stage j respectively, where Where i≠j, is an empty set, indicating that the newly learned hyperspectral image categories are different in each stage; Step 1.4: Divide the dataset obtained after step 1.3 into a training dataset and a test dataset in proportion; Step 2 specifically includes: Step 2.1, the basic structure of the variational autoencoder includes an encoder and a decoder; the loss function L used in optimizing the variational autoencoder VAE It consists of two parts, namely KL divergence L KL and reconstruction loss L re ; Step 2.2: The basic structure of the adversarial autoencoder includes an encoder, a decoder, and a discriminator, where the decoder is also a generator. Step 3 specifically includes: Step 3.1, respectively initialize the weights and bias parameters of the variational autoencoder and adversarial autoencoder in the virtual sample expansion network; Step 3.2: Based on the category set assigned in step 1.3, the training set and test set of each category in the initial category set are input into the virtual sample expansion network constructed in step 2, and the training is iterated according to the set training rounds. Step 3.3: Save the variational autoencoder and the decoder model in the adversarial autoencoder trained in step 3.2 for each category to form a virtual sample expansion network; in the virtual sample expansion network, the decoder model of the variational autoencoder and the decoder model of the adversarial autoencoder are used to generate two different types of virtual samples respectively; Step 5 specifically includes: Step 5.1: Generate random Gaussian noise and input it into the decoder model saved in step 3.3 to generate data augmentation samples for the initial training samples. The training samples and data augmentation samples are combined to construct the initial training set D0. In step 5.2, the initial training set obtained in step 5.1 is input into the feature extraction module and Softmax classifier constructed in step 4. Through forward propagation, loss calculation, backpropagation and optimization parameter update, iterative training is performed to obtain the classification results of the initial stage. At the same time, the parameters of the feature extraction module and Softmax classifier are saved to obtain the incremental hyperspectral image classification model. Step 7 specifically includes: Step 7.1: Input the image to be classified into the incremental hyperspectral image classification model to obtain the category prediction result of each pixel; Step 7.2: Save the classification results as an image file or other format and output them to a designated directory or storage medium; Step 7.3: Visualize the classification results to visually observe the classification results of the incremental hyperspectral image classification model; this includes mapping the predicted category labels to different colors and superimposing them on the original image.

3. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2, characterized in that: In step 2.1: The structure of the encoder is: convolution layer → 3D convolution layer → 3D convolution layer → 2D convolution layer → pooling layer → linear layer → latent space; The structure of the decoder is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer; The loss function is expressed by formula (1) to formula (4); z=μ+ε×σ (1) L VAE =L KL +L re (4) Where ε is the random noise drawn from the normal distribution; μ represents the mean of the latent space variable in the encoder; σ represents the standard deviation of the latent space variable in the encoder; N represents the number of samples; μ (i) represents the mean of the i-th sample; σ (i) Represents the standard deviation of the i-th sample; I represents the image input to the variational autoencoder, with a shape of N×N×C; Represents the image output by the variational autodecoder; x represents the width index; y represents the height index; and z represents the channel index.

4. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2, characterized in that: In step 2.2: The structure of the encoder is: convolutional layer → 3D convolutional layer → 3D convolutional layer → 2D convolutional layer → pooling layer → linear layer → latent space; the reconstruction loss of the adversarial autoencoder is the same as the reconstruction loss of the variational autoencoder; The structure of the decoder is: latent space → linear layer → 2D deconvolution layer → 3D deconvolution layer → 3D deconvolution layer → linear layer; the structure of the discriminator is: latent space → linear layer → linear layer; The losses of the discriminator and generator are shown in formula (5) and formula (6), where L D represents the discriminator loss, L G represents the generator loss; in, represents the calculation of expectation; Represents the expected probability that the data x generated by the generator is identified by the discriminator as real data; represents the expected probability that the real data x is identified as real data by the discriminator; D(·) represents the predicted value of the variable by the discriminator; P g represents the distribution of generated samples; P r Represents the distribution of real samples; x represents the distribution from P g and P r A random sample of .

5. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2, characterized in that: In step 3.2, each round of training in the iterative training process includes: forward propagation, calculation of loss function, back propagation, and optimization of model parameters.

6. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2, characterized in that: In step 4, the feature extraction module has a 10-layer structure, which is: convolution layer → first residual module → second residual module → first pooling layer → third residual module → fourth residual module → second pooling layer → Flatten layer → normalization layer → linear layer.

7. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2, characterized in that: In step 5.2, the loss function used in the loss calculation is the cross entropy loss L ce , as shown in formula (7); Among them, C 0 is the category set in the initial stage; p c is the predicted sample x i The probability of belonging to category c; when y = c, δ = 1, otherwise δ = 0; c represents the category index; y i Represents sample x i The true label.

8. The incremental hyperspectral image classification method based on virtual sample expansion according to claim 2, characterized in that: In step 6.3, the parameters of the incremental hyperspectral image classification model are updated using formula (8), and the classification ability of the old category is maintained using formula (9): F t =α*F t +(1-a)*F t-1 (8) Among them, F t is the incremental hyperspectral image classification model parameter at stage t, F t-1 is the incremental hyperspectral image classification model parameter of stage t-1, α is the set hyperparameter, x i represents a training sample, D t represents the incremental training set, F t (x i ) and F t-1 (x i ) represent the incremental hyperspectral image classification models that input training samples into stage t and stage t-1, respectively.

9. An incremental hyperspectral image classification system implementing the incremental hyperspectral image classification method according to any one of claims 1 to 8, characterized in that: The incremental hyperspectral image classification system includes the following modules: an acquisition module: used to obtain raw data of hyperspectral images; Storage medium: used to store acquired hyperspectral image data and computer programs; the computer programs include a data set construction program, a virtual sample expansion network program, an initial stage program, an incremental learning stage program, and a result saving program; Processor: used to load and execute data and computer programs from storage media. When the computer program is executed, the processor is used to: Read the hyperspectral raw data collected and stored by the hyperspectral imager; use the data set construction program to perform data preprocessing and organize and divide the data set; use the virtual sample extension network program to train the model; use the initial stage program to classify the initial training samples; use the incremental learning stage program to update and adjust the model to maintain the classification effect of the old categories; use the result saving program to save the classification results to the storage medium.

10. A terminal device of the incremental hyperspectral image classification system according to claim 9, characterized in that: The terminal device includes an acquisition module, a storage medium and a processor. The storage medium is a memory system in the terminal device for storing programs and data. The processor is used to execute the programs stored in the storage medium. The acquisition module is a hyperspectral imager.

Images