Hyperspectral image adversarial sample generation method and hyperspectral target detection model test method

Abstract

The application discloses a hyperspectral image adversarial sample generation method and a hyperspectral target detection model test method, wherein the adversarial sample generation method comprises the following steps: acquiring an original hyperspectral image, and selecting a proxy model; processing the original hyperspectral image by using the proxy model to obtain a prediction result; based on the proxy model, a mixed image is made in combination with spectral features and spatial features of a target category; the original hyperspectral image and the mixed image are respectively input into the proxy model to correspondingly obtain two shallow layer features; a multi-path mixed loss function is constructed based on the two shallow layer features and the prediction result, and the proxy model is iteratively trained by using the multi-path mixed loss function for back propagation to obtain an adversarial sample. The adversarial sample generated by the method comprises rich spectral information, and the effective test of the hyperspectral target detection model can be realized.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security technology, specifically relating to a method for generating adversarial examples from hyperspectral images and a method for testing hyperspectral target detection models. Background Technology

[0002] Hyperspectral images are captured by hyperspectral cameras and possess advantages such as spatial recognizability, a wide range of spectral bands, high spectral resolution, and integrated image and spectrum representation. Unlike ordinary images, hyperspectral images not only contain the location and shape information of targets but also rich spectral information. In a hyperspectral image, each pixel can be viewed as a high-dimensional vector, with the pixel value corresponding to the spectral reflectance at a specific wavelength. Because different parts of the same object absorb different spectral bands to varying degrees, hyperspectral images are more effective than ordinary images in reflecting the internal differences between targets. Therefore, hyperspectral technology is widely used in high-precision fields such as remote sensing, image scene classification, medical imaging, and military target recognition.

[0003] With the development of artificial intelligence technology, deep neural networks are increasingly being incorporated into hyperspectral target detection technology. However, recent studies have shown that deep neural networks are susceptible to interference from attack algorithms, posing a security risk to artificial intelligence. Common attack algorithms primarily generate adversarial examples by adding adversarial perturbations to the original samples, misleading the trained network into making incorrect predictions. Therefore, it is necessary to conduct performance tests to verify the anti-attack capabilities of hyperspectral target detection networks (hyperspectral target detection models). Generally, adversarial examples are used to attack the hyperspectral target detection model under test, and the network's ability to resist attacks is analyzed based on the model's prediction results. The quality of the adversarial examples has a significant impact on the test results.

[0004] Most existing adversarial example generation methods are designed for the RGB image domain. In the RGB image domain, the generated adversarial examples mislead the model's classification primarily due to the spatial features of the image, with only a small amount of color features. However, hyperspectral images contain rich spectral information. If adversarial example generation methods for RGB images are directly applied to hyperspectral images, the unique spectral information of hyperspectral images will not be effectively utilized. This results in limited information in the generated adversarial examples, restricting their role in misleading the model's correct classification and thus affecting the model's testing performance.

[0005] Therefore, it is necessary to develop adversarial example generation methods specifically for hyperspectral images to test hyperspectral target detection models, thereby inspiring and motivating adversarial defense research in the hyperspectral field and improving the security of deep neural networks in the hyperspectral field. Summary of the Invention

[0006] To address the shortcomings of the prior art, this invention proposes a method for generating adversarial examples from hyperspectral images and a method for testing hyperspectral target detection models. Experiments demonstrate that the adversarial examples generated using the method of this invention can significantly reduce the classification accuracy of hyperspectral target detection models, enabling effective testing of hyperspectral target detection models.

[0007] The technical problem to be solved by this invention is achieved through the following technical solution:

[0008] In a first aspect, the present invention proposes a method for generating adversarial examples from hyperspectral images, comprising:

[0009] Acquire the original hyperspectral image and select a surrogate model; the surrogate model is a hyperspectral classification model that includes an encoding table and an attention mechanism;

[0010] The original hyperspectral image is processed using a surrogate model to obtain the prediction results;

[0011] Based on the surrogate model, a hybrid image is created by combining the spectral and spatial features of the target category;

[0012] The original hyperspectral image and the mixed image are input into the surrogate model respectively, resulting in two shallow features. The shallow features are the features output by the last convolutional layer in the several consecutive convolutional layers used for dimensionality reduction and feature extraction in the first part of the surrogate model.

[0013] Based on two shallow features and prediction results, a multi-path mixed loss function is constructed, and the network is backpropagated and iteratively trained using the multi-path mixed loss function to obtain adversarial examples.

[0014] Secondly, this invention proposes a testing method for a hyperspectral target detection model, comprising:

[0015] Adversarial examples are generated using the hyperspectral image adversarial example generation method provided in the first aspect of the present invention;

[0016] Adversarial examples are input into the hyperspectral target detection model to be tested, so as to obtain the test results of the hyperspectral target detection model in terms of resistance to attacks.

[0017] The beneficial effects of this invention are:

[0018] 1. The hyperspectral image adversarial example generation method provided by this invention first selects a hyperspectral classification model including an encoding table and an attention mechanism as a surrogate model. Then, based on the selected surrogate model, a hybrid image including the spectral and spatial features of the target category is created. Next, the shallow features of the original hyperspectral image and the hybrid image are obtained using the surrogate model, as well as the prediction results output by the surrogate model. Based on this, a multi-path mixing loss function is constructed. Finally, the surrogate model is backpropagated and iteratively trained using the multi-path mixing loss function to obtain adversarial examples. This method fully utilizes the shallow and deep output features of the surrogate model, as well as the spatial and spectral features of the hyperspectral data, making the generated adversarial examples more information-rich. This enhances the role of adversarial examples in misleading the model to correctly classify the data, thereby improving the effectiveness of adversarial examples in model testing.

[0019] 2. The hyperspectral image adversarial example generation method provided by this invention improves the Saliency Maps algorithm proposed in the paper "Deep Inside Convolutional Networks: Visualizing Image Classification Models and SaliencyMaps" to hyperspectral image data. The improved SaliencyMaps algorithm selectively selects spectral channels for each target category and generates a hybrid image accordingly. Compared with the hybrid image generated by randomly assigning channels to each target category, the feature mixing is more thorough, making fuller use of spectral information in the generated adversarial examples.

[0020] 3. The hyperspectral image adversarial example generation method provided in this invention designs a dual-path hybrid loss function. One path is KL divergence, which can reduce the distribution distance between the shallow features of the hybrid image and the shallow features of the original image. The other path is cross-entropy loss, which is used to assist the attack, considering that the prediction accuracy of the network has a crucial impact on the network. These two loss paths can jointly extract similar parameter gradient distributions between different models, further improving the effectiveness of the generated adversarial examples.

[0021] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a hyperspectral image adversarial sample generation method provided in an embodiment of the present invention;

[0023] Figure 2 This is an overall framework diagram of the hyperspectral image adversarial sample generation method provided in the embodiments of the present invention;

[0024] Figure 3These are synthetic true-color images of the PaviaU dataset used in the embodiments of the present invention;

[0025] Figure 4 This is a line graph showing the channel sensitivity of each target category on the PaviaU dataset, obtained using the improved Saliency Maps algorithm provided in this embodiment of the invention.

[0026] Figure 5 This is a schematic diagram of the process of synthesizing a hybrid vector provided in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the process of mixing spatial information provided in an embodiment of the present invention;

[0028] Figure 7 This is a comparison of synthetic true-color images of adversarial samples generated using the HGMF algorithm of this invention and other attack algorithms. Detailed Implementation

[0029] Since hyperspectral target detection models often employ deep learning-based neural networks, they are susceptible to interference from attack algorithms. Therefore, researching adversarial attacks on existing deep neural network-based hyperspectral target detection models to inspire and motivate adversarial defense research in the hyperspectral field is crucial for improving the security of deep network security. Based on the rich spectral information inherent in hyperspectral images, this invention specifically proposes an adversarial sample generation method and a testing method for hyperspectral target detection models.

[0030] Since the ultimate goal of the attack is to reduce the accuracy of a trained network in identifying adversarial examples, the proposed method of testing a hyperspectral target detection model using adversarial examples can also be understood as attacking the target model using adversarial examples. This invention primarily utilizes hyperspectral gradient mixture features to generate adversarial examples; therefore, the hyperspectral image adversarial example generation method of this invention can also be called the Hyperspectral Gradient Mix Feature Attack (HGMF) method, or simply the HGMF method.

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example 1

[0033] Please refer to the above. Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating a hyperspectral image adversarial sample generation method provided in an embodiment of the present invention; Figure 2 This is an overall framework diagram of the hyperspectral image adversarial example generation method provided in this embodiment of the invention. The hyperspectral image adversarial example generation method provided in this embodiment specifically includes the following steps:

[0034] Step 1: Acquire the original hyperspectral image and select a surrogate model; the surrogate model is a hyperspectral classification model that includes an encoding table and an attention mechanism.

[0035] Optionally, as an implementation, this embodiment uses the publicly available PaviaU dataset as the original hyperspectral image. This dataset includes continuous imaging of 115 bands in the 0.43-0.86 μm wavelength range. Twelve bands were removed due to noise, so the image is composed of the remaining 103 spectral bands. The data contains a total of 2,207,400 pixels, but it includes a large number of background pixels. Only 42,776 pixels contain ground features, which include nine types of ground features, such as asphalt roads and bricks.

[0036] It should be noted that the PaviaU dataset consists of two files: the original data file and the corresponding ground truth file. A true-color illustration of the PaviaU dataset can be found in the reference file. Figure 3 As shown.

[0037] Furthermore, for the surrogate model, this embodiment can select a pre-trained hyperspectral image classification DNN model f as the surrogate model.

[0038] In one optional implementation, this embodiment uses SAC-Net, proposed by Yonghao Xu et al. in their paper "Universal Adversarial Examples in Remote Sensing: Methodology and Benchmark," as the proxy model. SAC-Net, by introducing a self-attention model and encoding table, effectively reduces the attack capability of adversarial methods for RGB images when extended to the hyperspectral domain. Choosing a network with certain defensive capabilities as the proxy model is crucial for obtaining parameter information regarding these defensive features, which is essential against black-box attacks.

[0039] Step 2: Use a surrogate model to process the original hyperspectral image and obtain the prediction results.

[0040] Specifically, the hyperspectral image I is input into the surrogate model f to obtain the model's prediction result f(I).

[0041] Step 3: Based on the surrogate model, create a hybrid image by combining the spectral and spatial features of the target category.

[0042] The ultimate goal of adversarial attacks is to reduce the accuracy of a trained network in identifying adversarial examples. Therefore, by fusing the spatial and spectral information of all ground features, it can be ensured that the category is different from any input image. The more thorough the fusion, the stronger the perturbation effect. Thus, image mixing requires mixing the spectral information of all categories in the original dataset and shuffling the spatial information of the entire image according to certain rules.

[0043] Specifically, step 3 includes the following sub-steps:

[0044] 31) Based on the surrogate model, obtain the channel sensitivity line chart for each target category.

[0045] First, based on the surrogate model, the SaliencyMaps algorithm proposed in the paper "Deep Inside Convolutional Networks: Visualizing Image Classification Models and Saliency Maps" is improved, and the improved SaliencyMaps algorithm is used to obtain the weight of each spectral channel of each pixel in the hyperspectral image on the classification influence.

[0046] Specifically, the calculation process of the improved Saliency Maps algorithm is as follows:

[0047] For a complex DNN model f, although it is not a linear model, it can be transformed into a near-linear model by performing a first-order Taylor expansion of the formula describing the surrogate model f around the given input data I, thereby obtaining the weight of each spectral channel of each pixel on the classification effect; that is:

[0048] f(I)≈w T (I)+b;

[0049] In the formula, w represents the weight, describing the influence of each spectral channel of each pixel in the hyperspectral image on the model's correct classification. That is, the influence of each spectral channel of a pixel i in the image on the model's correct classification. Specifically, the position corresponding to i can be expressed as (i... x i y i c ).

[0050] Weight w at position (i) x i y i c The larger the value at (), the greater the influence of the c-th spectral channel of pixel i on the final correct recognition by the network. w is calculated as follows:

[0051]

[0052] The symbol T represents transpose, b represents a constant term, and I0 represents an instance of a hyperspectral image.

[0053] Using PyTorch's automatic differentiation function, w can be obtained. w describes the influence of each spectral channel of each pixel in the hyperspectral image on the classification, so its shape is consistent with the shape of the selected PaviaU dataset.

[0054] Then, based on the location information of each target category in the ground truth corresponding to the original hyperspectral image, all data in the weights are extracted according to location and category, and the average value is taken to obtain the channel sensitivity spectral vector of each target category.

[0055] Finally, using the channel as the x-axis and the channel sensitivity spectral vector as the y-axis, a line graph of the channel sensitivity for each target category is obtained.

[0056] When the PaviaU dataset is selected as the original hyperspectral image, channel sensitivity spectral vectors for nine target categories can be obtained. See also... Figure 4 , Figure 4 This is a line graph of channel sensitivity for each category obtained on the PaviaU dataset using the improved Saliency Maps algorithm provided in this embodiment of the invention.

[0057] 32) Combine the line graph to analyze the light-sensitive regions of each target category and generate a hybrid vector.

[0058] To perform spectral information mixing on an image, the first step is to obtain a mixing vector.

[0059] Alternatively, as one implementation method, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the process of synthesizing a hybrid vector according to an embodiment of the present invention. For a hyperspectral image containing C channels and N target categories, each category is assigned... One channel.

[0060] It should be noted that when there are surplus channels, the extra channels can be allocated to target categories that require special attention, based on actual needs.

[0061] Specifically, since the sensitive spectral bands of different target categories are likely to be the same, it is necessary to create hybrid images based on specific attack requirements. For example, if the goal is to significantly reduce the classification accuracy of a particular key category, and the sensitive spectral band of this category is also the sensitive spectral band of one or more other categories, then the conflicting sensitive spectral bands can be assigned to the category that needs to be focused on.

[0062] For example, refer to Figure 4 The line graphs showing the channel sensitivity for different target categories are shown. For the categories of "asphalt road" and "shade", the sensitive spectral bands are both between the 80th and 100th spectral channels, which can be allocated as needed.

[0063] After the channels are allocated, the sensitive channels for each target category are combined to generate a blend vector, denoted as β. mix , mixture vector β mix The length of is equal to C.

[0064] 33) Based on the mixing vector, the original hyperspectral image is mixed with spectral features to obtain a primary mixed image that only mixes spectral features.

[0065] Optionally, as one implementation method, this embodiment can add the mixing vector to the original hyperspectral image data according to a certain ratio to obtain a primary mixed image that only mixes spectral features, as expressed by the formula:

[0066] I spe_mix =w spe ×β mix +I;

[0067] In the formula, I spe_mix This represents a primary blending image that only mixes spectral features, w spe ∈(0,1) represents the mixing ratio.

[0068] 34) Perform spatial feature blending on the primary blended image to obtain the final blended image.

[0069] Optionally, as one implementation method, this embodiment can divide the primary mixed image into several data blocks in space, and randomly arrange the several data blocks to obtain the final mixed image.

[0070] For example, see Figure 6 , Figure 6This is a schematic diagram of the process of mixing spatial information provided in an embodiment of the present invention. In this embodiment, the primary mixed image, which only mixes spectral features, is divided into n equal parts vertically and horizontally using a block-by-block transformation method. The adjacent rows of these n data blocks are swapped vertically, and the adjacent columns are swapped horizontally to obtain a new hyperspectral image, which is the final mixed image required.

[0071] Experiments show that a larger value of n is more favorable to the final experimental results. In this embodiment, the value of n is set to 9.

[0072] This invention provides a method for generating adversarial examples for hyperspectral images. It improves the Saliency Maps algorithm proposed in the paper "Deep Inside Convolutional Networks: Visualizing Image Classification Models and Saliency Maps" and applies it to hyperspectral image data. The improved Saliency Maps algorithm selectively selects spectral channels for each target category and generates a hybrid image accordingly. Compared to hybrid images generated by randomly assigning channels to each target category, the feature mixing is more thorough, allowing the generated adversarial examples to make fuller use of spectral information.

[0073] Step 4: Input the original hyperspectral image and the mixed image into the surrogate model respectively to obtain two shallow features; the shallow features are the features output by the last convolutional layer in the several consecutive convolutional layers used for dimensionality reduction and feature extraction in the front end of the surrogate model.

[0074] Specifically, the original hyperspectral image I and the mixed image I mix Input the proxy model f respectively, and obtain the shallow features f of both. s (I) and f s (I mix );

[0075] Step 5: Construct a multi-path mixed loss function based on two shallow features and prediction results, and use the multi-path mixed loss function to perform backpropagation iterative training on the proxy model to obtain adversarial examples.

[0076] Optionally, as an implementation method, the multi-path mixing loss function designed in this embodiment is a dual-path mixing loss function, which is generated by two mixing loss functions according to a certain weight ratio; wherein, the first loss function is the KL divergence between the shallow features of the original hyperspectral image and the shallow features of the mixed image, denoted as L. KL Its expression is:

[0077]

[0078] In the formula, θ represents the parameters of the surrogate model f, I represents the original hyperspectral image, and I mix f represents a blended image s (I) represents the shallow features of the original hyperspectral image, f s (I mix ) represents the shallow features of the blended image, where w, h, and k represent the length, width, and dimension of the feature, respectively; n w n h n k These represent shallow features f respectively. s (I) Total length, total width and total dimensions.

[0079] Since a smaller KL divergence indicates a higher similarity between two shallow features, resulting in better performance, the KL divergence loss function contains a negative sign. Minimizing the KL divergence is equivalent to maximizing this expression.

[0080] Furthermore, the second loss function is the cross-entropy loss between the network prediction and the ground truth, denoted as L. ce Its expression is:

[0081]

[0082] In the formula, f(I) represents the prediction result obtained by the surrogate model after processing the original hyperspectral image I, w, h, and c represent the length, width, and number of spectral bands of the feature, respectively, and I w I h I c represents the length, width, and total number of spectral bands of the original hyperspectral image I, respectively, and y represents the ground truth.

[0083] The final dual-path mixing loss L double for:

[0084] L double (θ,I)=L KL (θ,I)+λL ce (θ,I);

[0085] Where λ (λ∈(0,1)) represents the cross-entropy loss L ce The weighting parameter. In this embodiment, λ can be 0.2.

[0086] Then, the network is backpropagated and iteratively trained using a multi-path hybrid loss function to obtain adversarial examples, specifically including:

[0087] Using the dual-path hybrid loss function L doubleThe gradient is calculated by backpropagation on the surrogate model, and then added as an adversarial perturbation to the original hyperspectral image. The specific calculation method is as follows:

[0088]

[0089] In the formula, L double This represents the dual-path hybrid loss function. and Let represent the adversarial examples obtained in the (t+1)th and tth iterations, respectively; α represents the step size, used to limit the perturbation applied in a single iteration from being too large. In this embodiment, α can be 0.00001. θ represents the parameters of the surrogate model f, and y represents the ground truth. Denotes the gradient operator, |||| ∞ Representing the infinity norm, the function clip() is used to convert values ​​outside the range of image pixel representation to the correct (0,1) interval. Its expression is:

[0090]

[0091] Understandably, in backpropagation iterative training, each iteration yields an adversarial example, and the final adversarial example selected is generated in the last iteration.

[0092] This invention proposes a dual-path hybrid loss function. One path is KL divergence, which measures the degree of difference between two distributions. The smaller the difference between the two distributions, the smaller the KL divergence. The larger the difference between the two distributions, the larger the KL divergence. When the two distributions are identical, the KL divergence is 0. Because KL divergence can measure the difference between two distributions, this invention uses KL divergence as one loss path to reduce the distribution distance between the shallow features of the blended image and the shallow features of the original image. The other path is cross-entropy loss. Considering that the prediction accuracy of the network has a crucial impact on the network, cross-entropy loss is used to assist in attacks. These two loss paths can jointly extract similar parameter gradient distributions between different DNN models, further improving the effectiveness of the generated adversarial examples.

[0093] Another embodiment of the present invention proposes a testing method for a hyperspectral target detection model, comprising:

[0094] Adversarial examples are generated using the hyperspectral image adversarial example generation method described above;

[0095] The generated adversarial samples are input into the hyperspectral target detection model to be tested, so as to obtain the test results of the hyperspectral target detection model in terms of resisting attacks.

[0096] This invention first selects a hyperspectral classification model, including an encoding table and an attention mechanism, as a surrogate model. Then, based on the selected surrogate model, a hybrid image including the spectral and spatial features of the target category is created. Next, the shallow features of the original hyperspectral image and the hybrid image, as well as the prediction results output by the surrogate model, are obtained using the surrogate model. A multipath mixing loss function is then constructed based on this. Finally, the surrogate model is iteratively trained using backpropagation using the multipath mixing loss function to obtain adversarial examples. This method fully utilizes the shallow and deep output features of the surrogate model, as well as the spatial and spectral features of the hyperspectral data, resulting in generated adversarial examples with richer information. This enhances the role of adversarial examples in misleading the model's correct classification, thereby improving the effectiveness of adversarial examples in model testing.

[0097] The beneficial effects of the method proposed in this invention will be verified and explained through simulation experiments below.

[0098] I. Experimental conditions:

[0099] The hardware platform for the simulation experiment of this invention is as follows: the CPU is a 12th Gen Intel(R) Core(TM) i9-12900K with a frequency of 3.20GHz; the GPU is an NVIDIA GeForce RTX3090; the software for the simulation experiment of this invention is PyTorch and Anaconda.

[0100] The simulation comparison test of this invention uses attack methods including the present invention and four traditional image adversarial attack methods, namely: FGSM, iFGSM, PGD, and C&W algorithms; the datasets used include four publicly available hyperspectral datasets: PaviaU, IndianPines, Salinas, and Houston; the proxy model for creating adversarial examples is SAC-Net; the target models for black-box attacks are four hyperspectral image classification networks: Hybrid Spectral Net, 3D-CNN, Deep Feature Fusion Network (DFFN), and SSFN.

[0101] The FGSM (Fast Gradient Sign Method) algorithm used in the simulation experiments of this invention was proposed by Goodfellow in his paper "Explaining and Harnessing Adversarial Examples." It is a gradient-based algorithm for generating adversarial examples. Its goal is to maximize the loss function to obtain adversarial examples. Typically, training a deep neural network is a process of minimizing the loss function, with the parameters used in training being the network's parameters. The FGSM algorithm can be understood as a gradient ascent algorithm, moving along the gradient direction and using the plus sign to find the maximum value of the loss function. The parameters used in training are the original image, and the gradient is calculated only once. By maximizing the loss function, the gradient of the original image is obtained, and this gradient is added to the original image to obtain the adversarial example.

[0102] The i-FGSM algorithm used in the simulation experiments of this invention is an iterative version based on the FGSM algorithm. To overcome the limitation of the FGSM algorithm, which only calculates the gradient once, i-FGSM first sets a step size for the gradient in each iteration process, and then iterates the gradient ascent process several times. The adversarial examples obtained after this iteration are more powerful than those generated by the FGSM algorithm.

[0103] The PGD (Proximal Gradient Descent) algorithm used in the simulation experiments of this invention has more iterations than the I-FGSM algorithm. Furthermore, during the iteration process, it applies random adversarial perturbations to the adversarial examples obtained in the previous iteration, thereby avoiding saddle points that may be encountered during optimization. It is currently recognized as the most powerful white-box attack method and is one of the benchmark algorithms used to evaluate model robustness.

[0104] The C&W (Carlini & Wagner) algorithm used in the simulation experiments of this invention is a set of optimized adversarial attack methods proposed by Carlini and Wagner. They can generate L0, L2, and L... ∞ Adversarial examples CW0, CW2, and CW under norm constraints ∞ In addition to achieving a 100% attack success rate on normally trained DNN models in MNIST, CIFAR10, and ImageNet, the C&W algorithm can also break defensive distillation models that many attack algorithms cannot penetrate.

[0105] The PaviaU dataset used in the simulation experiments of this invention has been described in detail above and will not be repeated here.

[0106] The Indian Pines dataset used in the simulation experiments of this invention was collected from farmland in Indiana, USA. A 145×145 pixel section of this dataset was cropped and labeled for hyperspectral image classification testing. Since the cropped area consists entirely of crops, there are a total of 16 classes. Therefore, different land cover types have relatively similar spectral curves, and the distribution of samples within these 16 classes is extremely uneven.

[0107] The Salinas dataset used in the simulation experiments of this invention is an image of the Salinas Valley in California, USA. The image contains 204 spectral bands and has a size of 512×217. Of these, 54,129 pixels are suitable for classification, and these pixels are divided into 16 classes.

[0108] The Houston dataset used in the simulation experiment of this invention has an image size of 349×1905 and a total of 144 spectral bands. Among them, 15029 pixels can be used for classification, and these pixels are divided into 15 categories.

[0109] The surrogate model SAC-Net used in the simulation experiment of this invention has been described in detail above and will not be repeated here.

[0110] The Hybrid Spectral Net (HSNet) black-box attack target model used in the simulation experiments of this invention is a deep learning model for hyperspectral image classification. HSNet combines traditional spectral analysis methods with deep learning techniques, aiming to improve the classification performance and feature extraction capabilities of hyperspectral data. A key feature of HSNet is the fusion of spectral and spatial features to obtain more representative feature representations.

[0111] The black-box attack target model 3D-CNN network used in the simulation experiment of this invention is a convolutional neural network for processing three-dimensional data. Based on traditional convolutional neural networks (CNNs), it introduces a temporal dimension (or another third dimension) to process three-dimensional data, enabling the extraction of richer and more accurate feature representations from the three-dimensional data, thus helping to solve problems with temporal or stereoscopic perception attributes. Because hyperspectral images have a large number of spectral bands, which is also a type of three-dimensional data that 3D-CNN networks excel at processing, this model also performs well in the field of hyperspectral classification.

[0112] The Deep Feature Fusion Network (DFFN) used in the simulation experiments of this invention is a deep learning model for image processing and computer vision tasks. DFFN aims to extract richer and more accurate image features by fusing feature maps from multiple convolutional neural networks. The main idea of ​​DFFN is to fuse feature maps from different levels to obtain a more global and local feature representation. By introducing multiple CNN branches, each extracting features at a different level, DFFN can capture semantic information at different levels. Then, the fusion module fuses these feature maps layer by layer to produce a more representative feature representation.

[0113] The black-box attack target model Spectral-Spatial-FeatureNetworks (SSFN) used in the simulation experiments of this invention introduces spectral attention and spatial attention mechanisms into a deep convolutional neural network to fully utilize the correlation between spectral and spatial features. Specifically, SSFN processes different spectral bands of hyperspectral data through multiple parallel convolutional branches and uses an attention mechanism to adjust the weights of these spectral branches. Simultaneously, SSFN also includes a convolutional branch for learning spatial features to capture the spatial relationships between pixels.

[0114] II. Experimental Content and Result Analysis

[0115] Content 1: Using four publicly available datasets—PaviaU, Indian Pines, Salinas, and Houston—the proxy model SAC-Net is trained to achieve a classification accuracy that fits the target dataset. Then, using these four datasets, general adversarial examples are generated on the trained proxy model SAC-Net using FGSM, iFGSM, PGD, C&W algorithms, and the HGMF algorithm proposed in this invention.

[0116] Content 2: Use these four datasets to train three network models: Hybrid Spectral Net, 3D-CNN, and Deep Feature Fusion Network (DFFN), respectively, so that their classification accuracy reaches a fitted state on each dataset.

[0117] Content 3: The adversarial examples created using five different algorithms in Content 1 are input into the three network models trained in Content 2 according to the dataset, and the experimental results are compared.

[0118] The results of the above simulation experiments are summarized in Table 1. In Table 1, the first column represents the target model that was attempted to be attacked, the second column represents the dataset, the third column represents the classification accuracy of the target model when using the original image, and the fourth to eighth columns represent the classification accuracy of the target model when using adversarial examples created using FGSM, iFGSM, PGD, C&W, and HGMF algorithms as data, respectively.

[0119] Table 1. Comparison of attack performance on four target networks using four datasets and five adversarial attack algorithms.

[0120]

[0121]

[0122] As shown in Table 1, the adversarial samples generated by the HGMF adversarial attack method proposed in this invention achieved the lowest accuracy under most conditions when attacking four target networks using the four datasets mentioned above. This indicates that the HGMF attack method has high attack performance and good attack effect, surpassing existing adversarial attack methods. In other words, the adversarial samples generated using the method provided in this invention are more effective in testing the performance of defending target models against attacks.

[0123] For further details, please see Figure 7 , Figure 7 This is a comparison of synthetic true-color images of adversarial samples generated using the HGMF algorithm of this invention and other attack algorithms. It can be seen that the synthetic true-color images of adversarial samples generated by the HGMF algorithm of this invention are indistinguishable to the naked eye compared to those generated by other comparison algorithms, ensuring the adversarial samples are visually indistinguishable.

[0124] This verifies the effectiveness of the present invention.

[0125] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for generating adversarial examples from hyperspectral images, characterized in that, include: Acquire the original hyperspectral image and select a surrogate model; wherein the surrogate model is a hyperspectral classification model that includes an encoding table and an attention mechanism; The original hyperspectral image is processed using the surrogate model to obtain the prediction result; Based on the aforementioned proxy model, a hybrid image is created by combining the spectral and spatial features of the target category; The original hyperspectral image and the mixed image are respectively input into the surrogate model to obtain two shallow features; wherein, the shallow features are the features output by the last convolutional layer in the several consecutive convolutional layers used for dimensionality reduction and feature extraction in the front end of the surrogate model; Based on the two shallow features and the prediction results, a multi-path hybrid loss function is constructed, and the proxy model is backpropagated and iteratively trained using the multi-path hybrid loss function to obtain adversarial examples; Specifically, based on the surrogate model, a hybrid image is created by combining the spectral and spatial features of the target category, including: Based on the aforementioned proxy model, the improved Saliency Maps algorithm is used to obtain the weights of each spectral channel of each pixel in the hyperspectral image on the classification effect, including: in the input data The surrounding area will describe the proxy model. The formula is expanded using a first-order Taylor series to transform the model into a linear-like model, thereby obtaining the weights of each spectral channel of each pixel on the classification effect; the surrogate model The first-order Taylor formula is expressed as: ； In the formula, The weights represent the degree of influence of each spectral channel of each pixel in the hyperspectral image on the model's correct classification. They are calculated as follows: ； symbol T Indicates transpose. Represents a constant term. Examples of hyperspectral images; Based on the location information of each target category in the ground truth corresponding to the original hyperspectral image, all data in the weights are extracted according to location and category, and the average value is taken to obtain the channel sensitivity spectral vector of each target category. Using the channel as the x-axis and the channel sensitivity spectral vector as the y-axis, a channel sensitivity line graph for each target category is obtained; Based on the aforementioned line graph, the light-sensitive regions of each target category are analyzed, and a blending vector is generated; Based on the mixing vector, the original hyperspectral image is subjected to spectral feature mixing to obtain a primary mixed image that only mixes spectral features; Spatial feature blending is performed on the primary blended image to obtain the final blended image.

2. The hyperspectral image adversarial example generation method according to claim 1, characterized in that, Based on the aforementioned line graph, the light-sensitive regions of each target category are analyzed, and a blending vector is generated, specifically including: For a sheet containing One channel, Hyperspectral images of pixels of each target category, assigning each category... There are 10 channels; when there are any surplus channels, the excess channels are allocated to target categories that require special attention, based on actual needs. Once the channels are allocated, the sensitive channels for each target category are combined to generate a hybrid vector.

3. The hyperspectral image adversarial example generation method according to claim 1, characterized in that, Based on the mixing vector, the original hyperspectral image is subjected to spectral feature mixing to obtain a primary mixed image that only mixes spectral features, specifically including: The mixing vector is added to the original hyperspectral image data at a certain ratio to obtain a primary mixed image that only mixes spectral features.

4. The hyperspectral image adversarial example generation method according to claim 1, characterized in that, Spatial feature blending is performed on the primary blended image to obtain the final blended image, specifically including: The primary blended image is spatially divided into several data blocks, and the data blocks are randomly arranged to add spatial features to the primary blended image, resulting in the final blended image.

5. The hyperspectral image adversarial example generation method according to claim 1, characterized in that, The multi-path hybrid loss function is a dual-path hybrid loss function; the dual-path hybrid loss function is generated by two hybrid loss functions according to a certain weight ratio; Wherein, the first-path loss function is the KL divergence between the shallow features of the original hyperspectral image and the shallow features of the mixed image, denoted as . Its expression is: ； In the formula, Representative proxy model The parameters, Represents the original hyperspectral image. Represents a mixed image. This represents the shallow features of the original hyperspectral image. Represents shallow features of the blended image. These represent the length, width, and dimension of the feature, respectively. , , These represent shallow features. Total length, total width, and total dimensions; The second loss function is the cross-entropy loss between the network prediction and the ground truth, denoted as... Its expression is: ； In the formula, The proxy model represents the original hyperspectral image. Process the obtained prediction results, These represent the length, width, and number of spectral bands of the feature, respectively. , , These represent the original hyperspectral images. The length, width, and total number of spectral bands, It represents the ground truth.

6. The hyperspectral image adversarial example generation method according to claim 5, characterized in that, The network is backpropagated and iteratively trained using the aforementioned multipath hybrid loss function to obtain adversarial examples, specifically including: Use dual-path hybrid loss function The gradient is calculated by backpropagation on the surrogate model, and the gradient is added as an adversarial perturbation to the original hyperspectral image. The specific calculation method is as follows: ； In the formula, This represents the dual-path hybrid loss function. and They represent the first t +1 and the t Adversarial examples obtained in the next iteration; This indicates the step size, used to limit the perturbation applied in a single iteration from being too large. Representative proxy model The parameters, Represents ground truth. Represents the gradient operator, Representing the infinity norm, a function Used to convert values ​​that are outside the range of image pixel representation to the correct value. An interval, its expression is: ； The adversarial sample generated in the last iteration is selected as the final adversarial sample.

7. A testing method for a hyperspectral target detection model, characterized in that, include: Adversarial examples are generated using the hyperspectral image adversarial example generation method according to any one of claims 1-6; The adversarial sample is input into the hyperspectral target detection model to be tested to obtain the test results of the hyperspectral target detection model in terms of resisting attacks.

Images