Sparse Adversarial Attack Method for Hyperspectral Image Classification Model Based on Deep Learning

Abstract

The present invention discloses a sparse adversarial attack method for a hyperspectral image classification model based on deep learning. A band selection algorithm based on an attention mechanism selects a specific subset of bands to generate a specific spectral band image. The generator of the sparse band perturbation generation module generates an adversarial perturbation according to the specific spectral band image, and the discriminator judges whether the input sample is a clean sample or an adversarial sample. Through the confrontation between the generator and the discriminator, the level of the perturbation generated by the generator is improved. The sparse band perturbation generation module is used to generate adversarial band data. The clean band data and the adversarial band data are combined to be restored to the original data size to obtain sparse band adversarial samples, which are input into the target hyperspectral image classification model to obtain incorrect classification results, thus completing the adversarial attack. Starting from the intrinsic attributes of hyperspectral, the present invention fully considers the influence of spectral bands on classification, and while achieving the purpose of adversarial attack by adding perturbations only on a few specific bands, reduces the amount of perturbation.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a sparse adversarial attack method for a hyperspectral image classification model based on deep learning. Background Art

[0002] In recent years, satellite remote sensing has made significant breakthroughs. By processing, imaging, and analyzing the electromagnetic wave signals received by sensors, it is possible to observe and identify surface targets. Hyperspectral imaging is a spectral image in which the spectral resolution is subdivided into the range of 10-2λ order of magnitude. Hyperspectral remote sensing images can identify ground object targets that cannot be distinguished in panchromatic or multispectral remote sensing images, and are of great significance in the fields of meteorology, environment, agriculture, and military.

[0003] With the development of deep learning, researchers have continuously introduced deep learning methods into the field of hyperspectral image classification and proposed hyperspectral image classification models based on deep neural networks. However, due to the lack of understanding and analysis of the internal working mechanism, the deep neural network model, as a "black box", itself has vulnerabilities, which will greatly limit the practical application of such hyperspectral image classification technologies. Conducting adversarial attack research on existing hyperspectral image classification models and methods based on deep neural networks is the first step to detect, verify, and repair such vulnerabilities, thus effectively promoting the development of hyperspectral image classification systems towards interpretability, credibility, and reliability.

[0004] The existing research on adversarial attacks for hyperspectral image classification is limited to migrating adversarial attack algorithms for ordinary images to the hyperspectral field, ignoring the influence of the unique spectral information of hyperspectral images on the attack performance. And because ordinary RGB images have only three channels, ordinary adversarial attack algorithms mainly focus on the high-dimensional information of pixels in the spatial domain. When migrating them to the hyperspectral classification field, the adversarial attack algorithm will generate perturbations in each spectral band of the hyperspectral image, which will make the entire generated adversarial perturbation have a higher perturbation level compared to ordinary images, and will greatly damage the invisibility of hyperspectral adversarial samples. This method conducts sparse adversarial attacks on hyperspectral image classification according to the influence of different spectral bands on the classification of hyperspectral images, so as to maintain a high attack success rate while having a small perturbation amount. The relevant technical solutions are as follows:

[0005] The invention patent CN112633386A discloses a hyperspectral image classification method based on SACVAEGAN, which uses CVAENGAN (conditional variational autoencoder) to generate false hyperspectral images and uses a latent vector classifier to give a classification result to the image to supplement the problem of insufficient training samples of hyperspectral images. It generates complete hyperspectral images and classification results given by the latent vector classifier, rather than weak adversarial samples that cannot be recognized. And in the existing technical solution, only useless bands (with a value of 0) in a broad sense are removed during training. However, in fact, the information correlation between adjacent hyperspectral bands is relatively strong and there is a lot of information redundancy, and the entire band information can be reconstructed based on individual bands.

[0006] The invention patent CN112699899A discloses a hyperspectral image feature extraction method based on a generative adversarial network. The features obtained by using the discriminator of the GAN to extract features from the hyperspectral image are high-dimensional deep information, which is relatively abstract and does not require prior information, so it is unsupervised.

[0007] The invention patent CN112836666A discloses a hyperspectral image classification and recognition method, which extracts three different features of the hyperspectral image: spectral information, spatial information, and spectral-spatial information by introducing an attention mechanism. Band selection is not possible, resulting in a very high redundancy of training data.

[0008] CN114418027A discloses a hyperspectral image feature band selection method based on a band attention mechanism. However, when performing band selection, an MLP network is used, and prediction and analysis are carried out through 3D ResNet. Less effective information is concerned, the result error is large, and the purpose of sparse attack cannot be achieved. Summary of the Invention

[0009] Object of the Invention: The object of the present invention is to solve the deficiencies existing in the prior art and provide a sparse adversarial attack method for a deep learning hyperspectral image classification model.

[0010] Technical Solution: A sparse adversarial attack method for a deep learning hyperspectral image classification model according to the present invention includes the following steps:

[0011] Step S10, data preprocessing

[0012] First, perform feature normalization on the hyperspectral image data (to accelerate training and reduce the risk of overfitting), then adopt the method of padding 0 at the edges of the original image to avoid the inability to extract edge pixels. Then, for each pixel, take the surrounding pixel points to form a pixel block of size s×s, extract the spectral information of each pixel point in the formed pixel block to form the third dimension, and finally remove the meaningless background information to form the final training data set and test data set;

[0013] Step S20: Since there is a high degree of correlation between adjacent spectral bands of hyperspectral images, high-dimensional hyperspectral data not only increases the time complexity and space complexity but also leads to the so-called Hughes phenomenon or curse of dimensionality. Moreover, the existence of redundant information will also cause the adversarial attack algorithm to generate redundant adversarial perturbations and prevent the most targeted attacks. To achieve a high attack effect with the least amount of perturbation, it is necessary to consider the impact of the connection relationship between different bands on the classification task. The weight relationship between different bands is learned through a band selection algorithm based on the attention mechanism. Here, the DARecNet-BS network is used to perform band selection on the hyperspectral images in the training dataset to obtain a specific subset of bands, and finally, a specific spectral band image x is obtained.

[0014] The DARecNet-BS network regards the band selection problem as a spectral reconstruction problem and applies a band attention mechanism and a position attention mechanism to capture long-range non-linear context information in the spectral and spatial directions, so as to more effectively select an information-rich subset of bands.

[0015] Step S30: The structure of the generative adversarial network GAN is adopted to implement a sparse band perturbation generation module, and the sparse band perturbation generation module is trained to generate sparse adversarial attack samples.

[0016] The sparse band perturbation generation module includes a generator G, a discriminator D, and a target hyperspectral image classification model f. The generator G generates a specific perturbation G(x) according to the specific spectral band image x. The discriminator D judges whether the input sample is a clean sample or an adversarial sample. The target hyperspectral image classification model f takes the sparse band adversarial sample as the input and outputs its loss L. adv ; In the continuous confrontation between the generator G and the discriminator D, the performance of the generated perturbation is continuously improved. Finally, the generated perturbation G(x) is added to the specific spectral band image x to obtain the adversarial band x + G(x).

[0017] Step S40: The adversarial sample is input into the target hyperspectral image classification network to complete the attack.

[0018] The clean band data and the adversarial band data are combined to restore the original data size to obtain a sparse band adversarial sample, which is input into the target hyperspectral image classification model f to obtain the final classification result, and it is expected to be misclassified as other random labels, finally completing the adversarial attack.

[0019] Furthermore, in step S10, the top 15% of the bands with the highest weight values are selected to form a specific subset of bands.

[0020] Furthermore, to optimize the performance of the generator and improve the effectiveness and efficiency of the generator in generating adversarial samples, in step 30, the generator G of the sparse band perturbation generation module is trained. The generator loss function during the training process includes three key components: adversarial loss, perceptual loss, and fooling loss;

[0021] The total loss of the generator G is expressed as:

[0022]

[0023] where α and β are the weight coefficients assigned to the perceptual loss term and the fooling loss term, respectively;

[0024] Adversarial loss is expressed as follows:

[0025]

[0026] x is the image of a specific spectral band, and G(x) is the perturbation; the adversarial loss can prompt the discriminator D to recognize the sample as a real sample and improve the overall quality of the generated adversarial samples;

[0027] To limit the amplitude of the perturbation, the perceptual loss L p is shown as follows:

[0028]

[0029] L p is the perceptual loss of the generator, ε represents the hyperparameter that controls the minimum allowable perturbation intensity, and ‖G(x)‖ p represents the L p norm of the perturbation;

[0030] The main objective of the above perceptual loss is to force the generated adversarial samples to be very similar to the real samples. The combined application of the perceptual loss and the adversarial loss has a synergistic inhibitory effect on the generation of perturbations, thereby reducing the perceptual difference between the clean samples and the adversarial samples;

[0031] To maximize the difference between the prediction made by the target network and the actual true label, so that the target network misclassifies the adversarial samples, the fooling loss L adv is shown as follows:

[0032]

[0033] where l f is the cross-entropy loss function, f is the target model, and y true is the true label.

[0034] Furthermore, to accurately distinguish adversarial samples from clean samples, the discriminator D is trained using the loss function L D which is expressed as:

[0035]

[0036] Through the loss function L D it is possible to maximize D(x) and minimize D(x + G(x)), thereby identifying clean samples and adversarial samples and improving the performance of the discriminator.

[0037] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0038] (1) The present invention uses band selection to select an important subset of wavebands that contains most of the information in the original band set. It retains the main physical properties contained in the hyperspectral image, can effectively reduce the dimensionality of the original data, thereby removing redundant information, and is more conducive to the training of the network.

[0039] (2) The present invention obtains a specific spectral band image through band selection. Its adversarial attack only needs to generate perturbations for a very small number of wavebands. Compared with other adversarial attack algorithms, it fully considers the intrinsic properties of hyperspectral images, greatly reduces the amount of perturbation, and thus achieves the purpose of sparse attack.

[0040] (3) The present invention uses a GAN network to generate sparse adversarial samples, that is, uses the generator G to generate adversarial perturbations and combines them with the original image to make the target model misclassify, and uses its discriminator D to promote the generator G to generate more perfect perturbations.

[0041] (4) The present invention provides a unique generator loss function to minimize the amplitude of the generated adversarial perturbations. This loss function consists of three parts. The adversarial loss is mainly responsible for making the perturbations generated by the generator undetectable, the perceptual loss is mainly responsible for minimizing the amplitude of the perturbations generated by the generator, and the fooling loss is mainly responsible for making the target model misclassify; the three cooperate with each other to improve the effectiveness and efficiency of the generator in generating adversarial samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of the overall framework of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0043] The technical solution of the present invention will be described in detail below, but the protection scope of the present invention is not limited to the described embodiments.

[0044] Due to the redundant information in the numerous bands of hyperspectral images, current existing adversarial attack algorithms will generate unnecessary perturbations, and these redundant perturbations violate the original intention of generating adversarial samples, that is, making the model misclassify with as few perturbations as possible. To solve this problem, the present invention uses a band selection algorithm based on the attention mechanism to learn the weight relationship between different bands, so as to select a specific subset of wavebands. In addition, a new sparse spectral perturbation generation module is designed based on the framework of the generative adversarial network, so as to generate adversarial perturbations on a specific subset of wavebands. To make the generated adversarial samples closer to real samples, the present invention newly designs a generator loss function and a discriminator loss function to optimize the overall quality of the adversarial samples. As Figure 1 shown, the sparse adversarial attack method for a deep learning hyperspectral image classification model of the present invention includes the following steps:

[0045] Step S10, data preprocessing

[0046] First, perform feature normalization on the hyperspectral image, then adopt the method of padding 0 at the edges of the original image, then take the surrounding pixel points of each pixel to form a pixel block of size s×s, extract the spectral information of each pixel point formed to form the third dimension, and finally remove the meaningless background information to form the final training data set and test data set;

[0047] Step S20, use the DARecNet-BS network to perform band selection on the hyperspectral images in the training data set, learn the weight relationship between different bands, and obtain the subset of wavebands with the highest weight value. Obtain the specific spectral band image x, so as to realize the selection of the target band to be attacked;

[0048] Step S30, train the sparse band perturbation generation module to generate sparse adversarial attack samples

[0049] The sparse band perturbation generation module includes a generator G, a discriminator D, and a target model f. The generator G generates a special perturbation G(x) according to the specific spectral band image x. The discriminator D judges whether the input sample is a clean sample or an adversarial sample. The target model f takes the sparse band adversarial sample as the input and outputs its loss L adv ; in the continuous confrontation between the generator G and the discriminator D, the performance of the generated perturbation is continuously improved; finally, the generated perturbation G(x) is added to the specific spectral band image x to obtain the adversarial band x+G(x);

[0050] Among them, the generator G includes three convolutional layers and three deconvolutional layers, and each neuron adopts the nonlinear activation function ReLu(). The discriminator D includes four convolutional layers. In the first three layers, each neuron adopts the LeakyReLU() activation function, and the last output layer adopts the Sigmoid activation function.

[0051] Step S40: Input the adversarial sample into the target hyperspectral image classification network to complete the attack; Combine the clean band data and the adversarial band data to restore it to the original data size to obtain a sparse band adversarial sample, and input it into the target hyperspectral image classification model f to obtain the final classification result, which is expected to be misclassified as other random labels, and finally complete the adversarial attack.

[0052] It can be seen that different from the full-band perturbation addition method of the existing adversarial attack algorithms, the present invention starts from the inherent attributes of hyperspectral, considers the influence of spectral bands on classification, and reduces the perturbation amount while achieving the purpose of adversarial attack by adding perturbations only on a few specific bands.

[0053] Embodiment

[0054] Adopt the technical solution of the present invention to generate specific adversarial perturbations for the spectra that have a greater impact on the results, and the perturbation amount is small. Use multiple publicly available hyperspectral image classification evaluation datasets: Pavia University dataset, Salinas dataset, and Indian Pines dataset.

[0055] In this embodiment, hyperspectral image classification models HybridSN, SSRN, and SACNet are used to verify the technical effects of the present invention. The existing attack algorithms and adversarial attack algorithms for comparison use FGSM, PGD, and C&W.

[0056] The metrics adopted in this embodiment are the attack success rate (ASR), L0 norm, and L2 norm. Among them, the larger the ASR, the more successful attack times, and the smaller the L0 and L2 norms, the smaller the adversarial perturbation.

[0057] The specific results are shown in Tables 1 to 3:

[0058] Table 1 Comparison experiment results on the PaviaU dataset

[0059]

[0060] Table 2 Comparison experiment results on the Salinas dataset

[0061]

[0062] Table 3 Comparison experiment results on the Indian Pines dataset

[0063]

[0064] As can be seen from Tables 1 to 3, for the Pavia University dataset, Salinas dataset, and IndianPines dataset, the attack success rates of existing ordinary attack algorithms on hyperspectral images are not as high as those on ordinary RGB images. This is because hyperspectral images have more spectral information, which will affect the generation of perturbations in adversarial attack algorithms. And hyperspectral images contain a large amount of spectral redundant information, which will also cause redundant adversarial perturbations to be generated when the adversarial attack algorithm generates perturbations, greatly increasing the perturbation amount of adversarial samples. However, the present invention selects important bands through band selection, greatly reducing the magnitude of the perturbation amount.

[0065] Taking the comparison of the hyperspectral image classification model HybridSN on the PaviaU dataset as an example, the L0 norms of the three advanced attack algorithms are 16880, 15222, and 17250 respectively. The L0 norm of the algorithm in this study is only 2529. The L2 norms of the three advanced attack algorithms are 32.02, 20.36, and 53.78 respectively. The algorithm in this study is only 20.32. In addition, we not only ensure the reduction of the perturbation amount, but also have a certain improvement in the attack success rate. It has been improved from the highest 84.73% to 94.2%.

[0066] In summary, on all datasets, the attack success rate of the present invention shows that the ASR index is almost greater than 90%. The value of 89.41% on the Salinas dataset for SACNet is also higher than the international advanced level; moreover, while ensuring the attack success rate, the level of the perturbation amount is greatly reduced.

Claims

1. A sparse adversarial attack method for a deep learning-based hyperspectral image classification model, characterized in that, It includes the following steps: Step S10, data preprocessing First, perform feature normalization on the hyperspectral image. Then, adopt the method of padding 0 at the edges for the data after feature normalization. Next, for each pixel, take the surrounding pixel points to form a pixel block of size s×s, extract the spectral information of each pixel in the formed pixel block to form the third dimension, and finally remove the meaningless background information to form the final training dataset and test dataset; Step S20, use the DARecNet-BS network to perform band selection on the hyperspectral images in the training dataset, learn the weight relationship between different bands, obtain the subset of bands with the highest weight value, and get the specific spectral band image x, so as to realize the selection of the target band to be attacked; Step S30, adopt the GAN structure of the generative adversarial network to implement the sparse band perturbation generation module, and then train the sparse band perturbation generation module to generate sparse adversarial attack samples; The sparse band perturbation generation module includes a generator G, a discriminator D, and a target hyperspectral image classification model f. The generator G generates a specific perturbation G(x) based on the specific spectral band image x. The discriminator D determines whether the input sample is a clean sample or an adversarial sample. The target model f takes the sparse band adversarial sample as input and outputs its loss L adv ; during the continuous confrontation between the generator G and the discriminator D, the performance of the generated perturbation is continuously improved; finally, the generated perturbation G(x) is added to the specific spectral band image x to obtain the adversarial band x + G(x); Step S40, input the adversarial samples into the target hyperspectral image classification network to complete the attack; Combine the clean band data and the adversarial band data to restore them to the original data size to obtain the sparse band adversarial samples, input them into the target hyperspectral image classification model f to get the final classification result, which is expected to be misclassified as other random labels, and finally complete the adversarial attack.

2. The sparse adversarial attack method for a deep learning hyperspectral image classification model according to claim 1, wherein, In the step S10, select the top 15% of the bands with the highest weight values to form a specific band subset.

3. The sparse adversarial attack method for a deep learning hyperspectral image classification model according to claim 1, wherein In the step S30, the generator G of the sparse band perturbation generation module is trained, and the generator loss function in the training process includes three key components: adversarial loss, perceptual loss, and fooling loss; The total loss of the generator G is expressed as: where α and β are the weight coefficients assigned to the perceptual loss term and the fooling loss term respectively; Adversarial loss is expressed as follows: x is the specific spectral band image, and G(x) is the perturbation; Perceptual loss L p is expressed as follows: L p is the perceptual loss of the generator, ε represents the hyperparameter that controls the minimum allowable perturbation strength, and ‖G(x)‖ p represents the L p norm of the perturbation; Fooling loss L adv is expressed as follows: l f is the cross-entropy loss function, f refers to the hyperspectral image classification model, and y true is the true label.

4. The sparse adversarial attack method for a deep learning hyperspectral image classification model according to claim 1, characterized in that In step S30, the discriminator D of the sparse band perturbation generation module is trained, and the loss function L of the discriminator D D is expressed as: Through the loss function L D Maximize D(x) and minimize D(x + G(x)) to determine whether the generated samples are clean samples or adversarial samples.

Images