An adversarial attack method for a multi-modal remote sensing image classification network

Abstract

The application discloses a kind of multi-modal remote sensing image classification network-oriented attack method, to the optical remote sensing image TOP with three wave bands and the digital elevation model image DSM of one wave band two remote sensing image data sources, a kind of multi-modal remote sensing image classification network-oriented attack technology is proposed, attack effect is more remarkable, attack time efficiency is higher, to be used for evaluating and improving the robustness of multi-modal remote sensing image classification network.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an adversarial attack method for a multi-modal remote sensing image classification network. BACKGROUND

[0002] Remote sensing image classification is still a challenging task in the field of remote sensing. In recent years, with the rapid development of aerospace technology, a large number of remote sensing images of different sensors have been generated, and combining multiple sensor images to effectively classify ground objects has become a research hotspot. Multi-modal remote sensing images have different observations of ground objects, for example, a topographic (TOP) image expresses high spatial resolution information of a remote sensing scene, and a digital surface model (DSM) image provides height information of a ground object. The multi-modal remote sensing image classification method based on the advantages of multi-modal remote sensing data can effectively reduce the uncertainty in the classification task. In recent years, deep neural networks have been widely introduced into the multi-modal remote sensing image classification task. Compared with single-modal remote sensing image classification methods, multi-modal remote sensing image classification methods can effectively utilize the correlation between the two modalities to achieve higher classification accuracy.

[0003] In recent years, the vulnerability of deep neural networks has also attracted widespread attention from scholars. Adding a small, human-undetectable perturbation to the original clean sample can cause the deep neural network to produce an erroneous prediction with a high confidence. Such a sample added with a perturbation is called an adversarial sample, and the process of generating an adversarial sample is called adversarial attack. Therefore, to make the network more robust, it is necessary to understand all the risks faced by the network in advance and find adversarial samples with better attack performance. Various attack methods have been proposed to generate adversarial samples, such as the Fast Gradient Sign Method (FGSM), the Itersative FGSM (I-FGSM), the C&W algorithm based on the optimization objective function, and the DeepFool algorithm. However, existing adversarial attack techniques only consider the attack effect under a single modality, and lack of attack implementation for multi-modal classification networks. When attacking multi-modal remote sensing image classification networks, not only the success rate of the attack, the concealment of the perturbation, and the timeliness of the attack need to be considered, but also the collaborative attack ability between different modalities needs to be considered. Therefore, for multi-modal remote sensing image classification networks, high-quality multi-modal adversarial samples need to be generated to further evaluate and improve the robustness of multi-modal remote sensing image classification networks. SUMMARY

[0004] The purpose of the application is to provide an adversarial attack method for a multi-modal remote sensing image classification network, which has more significant attack effect and higher attack time efficiency.

[0005] The application adopts the following technical scheme: a multi-modal remote sensing image classification network facing an attack method, comprising the following steps:

[0006] Step one, constructing a multi-modal training sample set T and a test sample set S;

[0007] Step two, inputting the training sample set T and the test sample set S in step one into a target model f to be attacked;

[0008] Step three, constructing a target attack class t according to the target model f to be attacked in step two;

[0009] Step four, building a multi-modal adversarial attack network; the multi-modal adversarial attack network is composed of a perturbation generation network and a discriminator network of each mode;

[0010] Step five, multi-modal adversarial sample generation;

[0011] Step six, inputting the multi-modal adversarial samples in step five into the corresponding perturbation generation network, discriminator network and target model f respectively, and constructing a multi-modal generation loss function and a multi-modal discrimination loss function;

[0012] Step seven, alternately training the multi-modal generation loss function and the multi-modal discrimination loss function in step six, updating the perturbation generation network and the discriminator network, and completing the training of the multi-modal adversarial attack network;

[0013] Step eight, inputting the test sample into the multi-modal adversarial attack network in step seven to generate a corresponding test adversarial sample.

[0014] Further, after step eight, step nine is further included, as follows: inputting the training sample into the multi-modal adversarial attack network to obtain an adversarial sample of the training sample, adding the adversarial sample of the training sample to the training sample set, and retraining the target model f in step two; repeating steps three to eight.

[0015] Further, the multi-modal is an optical remote sensing image and a digital elevation model image.

[0016] Further, the specific process of constructing the multi-modal generation loss function and the multi-modal discrimination loss function is as follows:

[0017] Step 6.1, constructing a multi-modal generation loss function, as shown in formula (C):

[0018]

[0019] Among them: are the optical remote sensing image adversarial loss and the digital elevation model image adversarial loss respectively;

[0020] The super parameters α, β, γ are weight coefficients of the perception loss, the deception loss and the collaborative attack loss.

[0021] L f For the deception loss, the deception loss L f is defined as formula (H) shown.

[0022] L c For the collaborative loss, the definition is shown as formula (I).

[0023] The definitions are shown as formula (D) and (E):

[0024]

[0025]

[0026] Wherein: D T (x′ T ) represents the generated optical remote sensing image adversarial sample x′ T input into the optical remote sensing image discriminator network D T The output probability obtained is D D (x′ D ) represents the generated digital elevation model image adversarial sample x′ D input into the digital elevation model image discriminator network D D The output probability obtained is D respectively, the optical remote sensing image perception loss and the digital elevation model image perception loss, the definitions are shown as formula (F) and (G):

[0027]

[0028]

[0029] Wherein: ε is a super parameter controlling the minimum allowable perturbation intensity, ||| P represents the l P norm of the perturbation;

[0030] Formula (H) is:

[0031] f=l f (f(x′ T , x′ D ), t) (H);

[0032] Wherein: f is the target model to be attacked, l f is the cross-entropy loss, and the target label t of the target attack is obtained by step three;

[0033] Formula (I) is:

[0034] L C = l C (T(G T (x T )),G D (x D )) (I);

[0035] wherein: T is a band variation function; l C is a cosine similarity measure function;

[0036] Step 6.2, constructing a multi-modal discriminative loss function, as shown in equation (J):

[0037]

[0038] wherein: are the discriminative loss functions for multi-modal optical remote sensing images and digital elevation model images, respectively, defined as equations (K) and (L) respectively:

[0039]

[0040]

[0041] wherein: D T (x T ) represents the output probability obtained by inputting the original multi-modal optical remote sensing image training sample x T to the multi-modal optical remote sensing image discriminative network D T ; D D (x D ) represents the output probability obtained by inputting the original digital elevation model image training sample x D to the digital elevation model image discriminative network D D ; D T (x′ T ) represents the output probability obtained by inputting the generated optical remote sensing image adversarial sample x′ T to the optical remote sensing image discriminative network D T ; D D (x′ D ) represents the output probability obtained by inputting the generated digital elevation model image adversarial sample x′ D to the digital elevation model image discriminative network D D .

[0042] Further, in step two, the target model f is a trained multi-source remote sensing image classification network.

[0043] Further, in step three, the predicted probability score P = [p1, p2,..., p n ] of the sample is obtained by forward calculation through the target model f.T ;

[0044] According to the original label of the sample and the number of categories, One-Hot encoding is performed to obtain an encoding vector, for example, if the original sample label is 1, the encoding vector is h = [1, 0, …, 0] T , and the vector length is the number of categories of the sample;

[0045] According to the encoding vector, an inverse mask is obtained The inverse mask is multiplied by the prediction probability score P to obtain the prediction probability value p' = [0, p2, …, p n ] of other categories except the original label category probability of the sample; T Difference value processing is performed on p', a larger value is subtracted from the position of the original label, that is, the difference value s = p' - h x 1e10, and the position number of the maximum value of the difference value s is the target category;

[0046] Where: p n n represents the probability score of the nth output category.

[0047] Further, in step four, the multi-modal adversarial attack network is composed of an optical remote sensing image perturbation generation network G T , an optical remote sensing image discriminator network D T , a digital elevation model image perturbation generation network G D , and a digital elevation model image discriminator network D D .

[0048] Further, in step five, the multi-modal adversarial sample generation process is as follows:

[0049] The optical remote sensing image training sample x T is input into the optical remote sensing image perturbation generation network G T , the optical remote sensing image perturbation is generated, and the generated optical remote sensing image perturbation is added to the input optical remote sensing image training sample to obtain the TOP adversarial sample x' T , as shown in formula (A):

[0050] x' T = x T + G T (x T ) (A);

[0051] Where: G T (x T ) represents the TOP perturbation generated by the optical remote sensing image perturbation generation network.

[0052] The digital elevation model image training sample x D is input into the digital elevation model image perturbation generation network D TIn the specific implementation, the digital elevation model image perturbation is generated, and the generated perturbation is added to the input digital elevation model image training sample to obtain a digital elevation model image adversarial sample x' D As shown in formula (B):

[0053] x′ D = x D + G D (x D ) (B);

[0054] Wherein: G D (x D ) represents the digital elevation model image perturbation generated by the digital elevation model image perturbation generation network.

[0055] Further, in step eight, the specific process of attacking the multi-modal remote sensing image classification network with the test sample is as follows:

[0056] The TOP test sample is input into the trained TOP perturbation generation network, and the TOP perturbation is output, and the TOP perturbation is added to the input TOP test sample to obtain the adversarial sample of the TOP test sample At the same time, the DSM test sample is input into the trained DSM perturbation generation network, and the DSM perturbation is output, and the DSM perturbation is added to the input DSM test sample to obtain the adversarial sample of the DSM test sample The adversarial attack on the multi-modal remote sensing image classification network is completed; the specific representation is as follows:

[0057]

[0058]

[0059] Further, the specific process of step nine is as follows:

[0060] The optical remote sensing image training sample x T is input into the trained optical remote sensing image perturbation generation network to output the optical remote sensing image perturbation, and the optical remote sensing image perturbation is added to the input optical remote sensing image training sample to obtain the adversarial sample x′ T of the optical remote sensing image training sample.

[0061] The digital elevation model image training sample x D is input into the trained digital elevation model image perturbation generation network to output the digital elevation model image perturbation, and the digital elevation model image perturbation is added to the input digital elevation model image training sample to obtain the adversarial sample x′ D; x T and x D are added to the training sample set to obtain a new training sample set T', that is, {x T , x D , x T , x D} is in T', and the target model f of step two is retrained.

[0062] The beneficial effects of the present application are: 1. The method considers joint multiple modalities to generate multi-source disturbances, and makes the generated adversarial samples more realistic. 2. A multi-modal generation loss function and a multi-modal discrimination loss function are designed to establish the relationship between different modal disturbances, which can effectively reduce the interference strength added to each data source while maintaining a high attack success rate. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 is a research flow block diagram of an adversarial attack method for a multi-source remote sensing image classification network of the present application;

[0064] Figure 2 is a data set used in the experiment of the present application;

[0065] 2a is the Potsdam data set; 2b is the Vaihingen data set;

[0066] Figure 3 is a structural diagram of the target model, the disturbance generation network and the discriminator network of the present application;

[0067] Figure 4 is a classification graph before and after adversarial training of different methods on the Potsdam data set;

[0068] Figure 5 is a classification graph before and after adversarial training of different classification methods on the Vaihingen data set. DETAILED DESCRIPTION

[0069] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0070] An adversarial attack method for a multi-modal remote sensing image classification network of the present application, as shown in FIG. Figure 1 , includes the following steps:

[0071] Step one, construct a multi-modal training sample set T and a test sample set S;

[0072] The above multi-modal is three-band optical remote sensing image (TOP) and one-band digital elevation model image (DSM).

[0073] Input two multi-modal data TOP and DSM, and the true class maps corresponding to the two modalities, as shown in Figure 2 A 27x27 pixel size space window is drawn around each pixel on the two multi-modal data TOP and DSM to extract samples, and sample pairs are extracted from TOP and DSM, respectively, to form a sample set. Part of the sample pairs are selected to form a training sample set, and the remaining sample pairs are selected to form a test sample set. T , x D}∈T, x T and are the TOP training samples and the TOP test samples, respectively, and x D and are the DSM training samples and the DSM test samples, respectively.

[0074] Step two, input the training sample set T and the test sample set S in step one into the target model f to be attacked.

[0075] In step two, the target model f is a trained multi-source remote sensing image classification network. As shown in Figure 3 (a), the construction is as follows: two data sources are input, the data input size of the TOP data source is 28x28x3, and the data input size of the DSM data source is 28x28x1. Each data source has 5 layers of network to extract the basic features of each data source, and then the outputs of the two data sources are connected, and 4 layers of network are used to extract and classify their common features. The network parameters of the target model f are shown in Table 1. The target model f is trained in advance, and the parameters of the target model remain unchanged during the attack process.

[0076] Table 1 Network parameters of target model f

[0077]

[0078] Step three, construct a target attack class t according to the target model f to be attacked in step two.

[0079] In step three, each pair of samples x T , x D in the training sample set T is forward calculated by the pre-trained target model f to obtain the output probability score of each sample on each class, and the target attack class is selected. Assuming that the training sample pair is x T , x D , the output probability score is P=[p1, p2,..., p n ] T , p nThis represents the probability score of the nth class in the output, and the target class is constructed according to the following principles:

[0080] The predicted probability score P = [p1, p2, ..., p] of the sample is obtained by forward computation of the target model f. n ] T ;

[0081] One-Hot encoding is performed on the original label and number of categories of the sample to obtain the encoding vector. For example, if the original sample label is 1, then the encoding vector is h = [1, 0, ..., 0]. T The length of the vector is the number of classes in the sample;

[0082] The inverse mask is obtained from the encoded vector. Multiplying the inverse mask by the predicted probability score P yields the predicted probability values ​​p′ = [0, p2, ..., p1] for each class other than the original label class probability of the sample. n ] T To avoid duplicate labels between the target category and the original category, a difference is applied to p′, subtracting a large value from the position of the original label, i.e., the difference s = p′ - h × 1e10. The position number of the maximum value of the difference s is the target category.

[0083] Where: p n This represents the probability score of the nth class in the output.

[0084] Step 4: Construct a multimodal adversarial attack network; the multimodal adversarial attack network consists of perturbation generation networks and discriminator networks for each mode;

[0085] In step four, the multimodal adversarial attack network is generated by the optical remote sensing image perturbation generation network G. T Optical remote sensing image discriminator network D T Digital Elevation Model Image Perturbation Generator Network G D and the digital elevation model image discriminator network D D composition.

[0086] like Figure 3 As shown in (b)(c). The perturbation generation network is an encoder-decoder structure, where G is the encoder. T The input image size for the first convolutional layer is 28×28×3, G D The input size of the first convolutional layer is 28×28×1. The number of filters in the third layer of the decoding structure depends on the number of channels in the image. T For 3, G D The value is 1. Apart from this parameter setting, the perturbation generation network G... T and G D The other parameter settings are the same, as shown in Table 2; TOP discriminator network DT The input picture size of the first layer convolutional layer is 28x28x3, and the DSM discriminator network D D The input picture size of the first layer convolutional layer is 28x28x1, and the discriminator D T and D D The parameter settings are the same as shown in Table 3.

[0087] Table 2 Network parameters of the disturbance generation network

[0088]

[0089] Table 3 Discriminator network parameters

[0090]

[0091] Step five, multi-modal adversarial sample generation;

[0092] In the step five, the multi-modal adversarial sample generation process is as follows:

[0093] The optical remote sensing image training sample x T is input into the optical remote sensing image disturbance generation network G T , and the optical remote sensing image disturbance is generated, and the generated optical remote sensing image disturbance is added to the input optical remote sensing image training sample to obtain the TOP adversarial sample x' T , as shown in formula (A):

[0094] x' T = x T + G T (x T ) (A);

[0095] Wherein: G T (x T ) represents the TOP disturbance generated by the optical remote sensing image disturbance generation network.

[0096] The digital elevation model image training sample x D is input into the digital elevation model image disturbance generation network D T , and the digital elevation model image disturbance is generated, and the generated disturbance is added to the input digital elevation model image training sample to obtain the digital elevation model image adversarial sample x' D , as shown in formula (B):

[0097] x' D = x D + G D (x D ) (B);

[0098] Wherein: G D (x D) represents a digital elevation model image perturbation generated by the digital elevation model image perturbation generation network.

[0099] Step six, input the multi-modal adversarial samples in step five into their corresponding perturbation generation network, discriminator network and target model f respectively, and construct a multi-modal generation loss function and a multi-modal discrimination loss function;

[0100] In step six, the specific process of constructing the multi-modal generation loss function and the multi-modal discrimination loss function is as follows:

[0101] The TOP adversarial samples are input into the TOP discriminator and the target model f respectively, and the DSM adversarial samples are input into the DSM discriminator and the target function f respectively, and according to the TOP perturbation, the TOP adversarial samples, the output of the TOP discriminator and the output of the TOP in the target model, and the DSM perturbation, the DSM adversarial samples, the output of the DSM discriminator and the output of the DSM in the target model, a multi-modal generation loss function and a multi-modal discrimination loss function are constructed; The specific construction is as follows:

[0102] Step 6.1, construct a multi-modal generation loss function, as shown in formula (C):

[0103]

[0104] Wherein: respectively, the multi-modal optical remote sensing image adversarial loss and the digital elevation model image adversarial loss; to constrain the generated TOP adversarial samples to be closer to the input TOP training samples, and to constrain the generated DSM adversarial samples to be closer to the input DSM training samples.

[0105] The hyperparameters α, β, γ are the weight coefficients of the perception loss, the deception loss and the collaborative attack loss;

[0106] L f is the deception loss, which is wrongly classified as the t-th class, and the deception loss L f is defined as formula (H);

[0107] L c is the collaborative loss to achieve multi-modal collaborative attack, and its definition is shown in formula (I);

[0108] is defined as formula (D) and (E):

[0109]

[0110]

[0111] Wherein: D T (x′ T) represents the generated optical remote sensing image adversarial sample x' T Input to the optical remote sensing image discriminator network D T The output probability obtained; D D (x′ D ) represents the generated digital elevation model image adversarial sample x′ D Input to the digital elevation model image discriminator network D D The output probability obtained; respectively, the optical remote sensing image perception loss and the digital elevation model image perception loss, to constrain the perturbation intensity of the TOP perturbation and the DSM perturbation respectively, are defined as shown in equations (F) and (G):

[0112]

[0113]

[0114] Where: ε is a hyperparameter that controls the minimum allowed perturbation intensity, ||| P represents the l P norm of the perturbation; in the embodiment, the l ∞ norm is used to constrain the perturbation, forcing the generated adversarial sample to be closer to the real sample.

[0115] Equation (H) is:

[0116] L f = l f (f(x′ T , x′ D ), t) (H):

[0117] Where: f is the target model to be attacked, l f is the cross-entropy loss, and the target label t of the target attack is obtained from step three;

[0118] Equation (I) is:

[0119] L C = l C (T(G T (x T )), G D (x D )) (I);

[0120] Where: T is the frequency band variation function; because the TOP data has three frequency bands, while the perturbation of the DSM has one frequency band, it is necessary to repeat the expansion of the perturbation of the DSM to three frequency bands for similarity calculation. l C is the cosine similarity measure function, used to measure the similarity of the perturbation under different modalities.

[0121] Step 6.2, constructing a multi-modal discriminator loss function, as shown in formula (J):

[0122]

[0123] Wherein: are the discriminator loss functions of the multi-modal optical remote sensing image and digital elevation model image respectively, defined as formula (K) and formula (L) respectively:

[0124]

[0125]

[0126] Wherein: D T (x T ) represents the output probability obtained by inputting the original multi-modal optical remote sensing image training sample x T to the multi-modal optical remote sensing image discriminator network D T ; D D (x D ) represents the output probability obtained by inputting the original digital elevation model image training sample x D to the digital elevation model image discriminator network D D ; D T (x′ T ) represents the output probability obtained by inputting the generated optical remote sensing image adversarial sample x′ T to the optical remote sensing image discriminator network D T ; D D (x′ D ) represents the output probability obtained by inputting the generated digital elevation model image adversarial sample x′ D to the digital elevation model image discriminator network D D .

[0127] Step seven, alternately training the multi-modal generation loss function and the multi-modal discriminator loss function in the step six, updating each perturbation generation network and discriminator network, and completing the training of the multi-modal adversarial attack network;

[0128] The multi-modal generator loss and the multi-modal discriminator loss are used to optimize the model alternately trained, and the TOP perturbation generation network, the TOP discriminator network, the DSM perturbation generation network, and the DSM discriminator network are trained respectively according to the following steps:

[0129] Step 7.1, updating the TOP discriminator D T ; keeping the DSM discriminator D D , the DSM perturbation generator G D , and the TOP perturbation generator G Twith the gradient descent method, the TOP discriminator D is updated by minimizing formula (M) with the parameters unchanged. T is trained.

[0130]

[0131] Step 7.2, updating the TOP perturbation generator G T . The TOP discriminator D is kept T , the DSM perturbation generator G D , the DSM discriminator D D with the gradient descent method, the TOP perturbation generator G is updated by minimizing formula (N) with the parameters unchanged. T .

[0132]

[0133] Step 7.3, updating the DSM discriminator D D . The DSM perturbation generator G is kept D , the TOP discriminator D T , the TOP perturbation generator G T with the gradient descent method, the DSM discriminator D is updated by minimizing formula (O) with the parameters unchanged. D .

[0134]

[0135] Step 7.4, updating the DSM perturbation generator G D . The DSM discriminator D is kept D , the TOP perturbation generator G T , the TOP discriminator D T with the gradient descent method, the DSM perturbation generator G is updated by minimizing formula (P) with the parameters unchanged. D .

[0136]

[0137] Step eight, input the test sample into the multi-modal adversarial attack network in the above step seven, and generate the corresponding test adversarial sample.

[0138] In the above step eight, the specific process of attacking the multi-modal remote sensing image classification network by using the test sample is as follows:

[0139] The TOP test sample is input into the trained TOP perturbation generation network, and the TOP perturbation is output, and the TOP test sample is obtained by adding the TOP perturbation and the input TOP test sample At the same time, the DSM test sample Input into the trained DSM disturbance generation network, output DSM disturbance And add the DSM disturbance to the input DSM test sample, get the adversarial sample of the DSM test sample Complete the adversarial attack of the multi-modal remote sensing image classification network, which is specifically represented as:

[0140]

[0141]

[0142] Step nine is as follows: input the training sample into the multi-modal adversarial attack network, get the adversarial sample of the training sample, and add the adversarial sample of the training sample to the training sample set, and retrain the target model f in step two; repeat steps three to eight.

[0143] Specifically, input the optical remote sensing image training sample x T Into the trained optical remote sensing image disturbance generation network, output the optical remote sensing image disturbance, and add the optical remote sensing image disturbance to the input optical remote sensing image training sample, get the adversarial sample x' of the optical remote sensing image training sample T ;

[0144] Input the digital elevation model image training sample x D Into the trained digital elevation model image disturbance generation network, output the digital elevation model image disturbance, and add the digital elevation model image disturbance to the input digital elevation model image training sample, get the adversarial sample x' of the digital elevation model image training sample D ; x' T And x' D Add to the training sample set to get a new training sample set T', that is {x T , x D , x' T , x' D}∈T', retrain the target model f in step two to enhance the ability of the model to resist attacks.

[0145] The effect of the method in the application can be further illustrated by the following simulation experiment:

[0146] (1) Simulation conditions:

[0147] The hardware conditions of the simulation of the application are: windows10, GPU NVIDIA GeForce RTX3060; the software platform is: MatlabR2016a, Pycharm;

[0148] The source of the selected pictures for simulation is the Potsdam dataset and the Vaihingen dataset. The Potsdam dataset includes 28 6000*6000 pixel unmanned aerial vehicle images, with a resolution of 5 centimeters / pixel, and contains three different channel image data, terrain data, and label data. A pair of DSM data and TOP data is randomly selected as the input of the multi-source remote sensing image network as shown in FIG. 2a. Figure 2 The Vaihingen dataset includes 33 remote sensing images of different sizes, including 3-band TOP data and single-band DSM data, with a resolution of 9 centimeters / pixel. A pair of DSM data and TOP data is also selected as the input of the multi-source remote sensing image classification network as shown in FIG. 2b. In the present application, 10,000 pixel points of each class are randomly selected in the Potsdam dataset to construct training samples, and 800 pixel points of each class are randomly selected in the Vaihingen dataset to construct training samples. Figure 2

[0149] Simulation content and results:

[0150] Simulation 1: The method in the present application and three existing technologies are used to classify the two datasets shown in FIG. 1, and the results are as follows, wherein: Figure 2

[0151] Figure 4 (a) to (d) of FIG. 3 are the classification results of FGSM, C&W, PGD, and the technology in the present application on the test adversarial samples on the target model f on the Potsdam image dataset, and (i) is the classification result of the network on the original test sample. From the figure, it can be seen that, compared with the classification graph of the original test sample on the target model f, the adversarial samples generated by the four technologies can cause the target model to produce obvious misclassification. The classification graphs (a)-(d) and the classification graph (i) have obvious differences, but the difference between the classification graph obtained by the present application and the graph (i) is more obvious, and the misclassification area is more, which indicates that the adversarial samples generated by the technology in the present application are more attackable.

[0152] Figure 5 (a) to (d) of FIG. 4 are the classification results of FGSM, C&W, PGD, and the technology in the present application on the test adversarial samples on the target model f on the Vaihingen image dataset, and (i) is the classification result of the network on the original test sample, Figure 5 Figure 4 the same results as shown in FIG. i, compared with the classification graph obtained by the method in the present application, the misclassification is more serious, which further indicates that the technology in the present application is more attackable.

[0153] ​​​The numerical comparison of classification results of the method in the present application and prior art FGSM, C&W, PGD before and after adversarial training on the Potsdam dataset is shown in Table 4 and Table 5:

[0154] Table 4 is the numerical comparison of classification results of the method in the present application and prior art FGSM, C&W, PGD before and after adversarial training on the Potsdam dataset

[0155]

[0156] Table 5 is the numerical comparison of classification results of the method in the present application and prior art FGSM, C&W, PGD before and after adversarial training on the Vaihingen dataset

[0157]

[0158] The data in Table 4 and Table 5 show that the present application has shorter test time and higher attack time efficiency. Obviously, the present method has obvious advantages in processing multi-modal data attacks.

Claims

1. A method for adversarial attack on a multi-modal remote sensing image classification network, characterized in that, The method comprises the following steps: Step one, constructing a multi-modal training sample set T and a test sample set S; Step two, inputting the training sample set T and the test sample set S in step one into a target model f to be attacked; Step three, constructing a target attack class t according to the target model f to be attacked in step two; Step four, building a multi-modal adversarial attack network; the multi-modal adversarial attack network is composed of a perturbation generation network of each modality and a discriminator network; Step five, generating a multi-modal adversarial sample; Step six, inputting the multi-modal adversarial sample in step five into the perturbation generation network, the discriminator network and the target model f respectively to construct a multi-modal generation loss function and a multi-modal discrimination loss function; Step seven, alternately training the multi-modal generation loss function and the multi-modal discrimination loss function in step six to update the perturbation generation network and the discriminator network, and completing the training of the multi-modal adversarial attack network; Step eight, inputting a test sample into the multi-modal adversarial attack network in step seven to generate a corresponding test adversarial sample; The multi-modal is an optical remote sensing image and a digital elevation model image; The specific process of constructing the multi-modal generation loss function and the multi-modal discrimination loss function is as follows: Step 6.1, constructing a multi-modal generation loss function, as shown in formula (C): wherein: respectively a multi-modal optical remote sensing image adversarial loss and a digital elevation model image adversarial loss; The hyperparameters α, β and γ are weight coefficients of the perception loss, the deception loss and the cooperative attack loss; L f For a fraud loss, being misclassified as class t, the fraud loss L f is defined as shown in equation (H); L c For synergy loss, it is defined as shown in equation (I); are defined as shown in Formulas (D) and (E): wherein: D T (x′ T ) represents the generated optical remote sensing image adversarial sample x′ T input into the optical remote sensing image discriminator network D T to obtain the output probability; D D (x′ D ) represents the generated digital elevation model image adversarial sample x′ D input into the digital elevation model image discriminator network D D to obtain the output probability; respectively, the optical remote sensing image perception loss and the digital elevation model image perception loss, are defined as shown in equations (F) and (G): where: ε is a hyperparameter that controls the minimum allowed disturbance strength, || || is the l P norm of the disturbance; and P norm of the disturbance; and Formula (H) is: L f = l f (f(x′ T , x′ D ), t) (H). where: f is the target model to be attacked, l f is the cross-entropy loss, and the target label t of the target attack is obtained from step three. Formula (I) is: L C = l C (T(G T (x T )), G D (x D )) (I) ; where: T is a band variation function; l C is a cosine similarity measure function; Step 6.2, constructing a multi-modal discrimination loss function, as shown in formula (J): wherein: respectively are discriminator loss functions for multi-modal optical remote sensing images and digital elevation model images, defined as in Equations (K) and (L), respectively: wherein: D T (x T ) represents inputting the original multi-modal optical remote sensing image training sample x T into the multi-modal optical remote sensing image discriminator network D T to obtain the output probability; D D (x D ) represents inputting the original digital elevation model image training sample x D into the digital elevation model image discriminator network D D to obtain the output probability; D T (x′ T ) represents inputting the generated optical remote sensing image adversarial sample x′ T into the optical remote sensing image discriminator network D T to obtain the output probability; D D (x′ D ) represents inputting the generated digital elevation model image adversarial sample x′ D into the digital elevation model image discriminator network D D to obtain the output probability; In the step four, the multi-modal adversarial attack network is composed of an optical remote sensing image perturbation generation network G T , an optical remote sensing image discriminator network D T , a digital elevation model image perturbation generation network G D , and a digital elevation model image discriminator network D D . In step five, the process of generating a multi-modal adversarial sample is as follows: The optical remote sensing image training sample x T Input to the optical remote sensing image perturbation generation network G T In the optical remote sensing image perturbation generation network G, the optical remote sensing image perturbation is generated, and the generated optical remote sensing image perturbation is added to the input optical remote sensing image training sample to obtain the TOP adversarial sample x′ T As shown in formula (A): x' T = x T + G T (x T ) (A); where: G T (x T ) represents the TOP perturbation generated by the optical remote sensing image perturbation generation network. a digital elevation model image training sample x D input to the digital elevation model image perturbation generation network D T In the digital elevation model image perturbation generation network D, a digital elevation model image perturbation is generated, and the generated perturbation is added to the input digital elevation model image training sample to obtain a digital elevation model image adversarial sample x′ D As shown in formula (B): x' D = x D + G D (x D ) (B); where: G D (x D ) denotes a digital elevation model image perturbation generated by the digital elevation model image perturbation generation network.

2. The method of claim 1, wherein, After step eight, step nine is further included, as follows: inputting a training sample into the multi-modal adversarial attack network to obtain an adversarial sample of the training sample, adding the adversarial sample of the training sample to the training sample set, and retraining the target model f in step two; Repeat steps three to eight.

3. The method of claim 2, wherein, In step two, the target model f is a trained multi-source remote sensing image classification network.

4. The method of claim 1 or 2, wherein, In the step three, the prediction probability scores P = [p1, p2,..., pn] of the sample are calculated by the target model f forwardly. n ] T ; According to the original label of the sample and the number of categories, One-Hot encoding is performed to obtain an encoded vector, for example, if the original sample label is 1, the encoded vector is h = [1, 0,..., 0] T , and the vector length is the number of categories of the sample; Obtaining an inverse mask according to the encoding vector The inverse mask is multiplied by the prediction probability score P to obtain the prediction probability value p' = [0, p2,..., p n ] T Difference processing is performed on p', a large value is subtracted from the position where the original label is located, that is, the difference s = p'- hxl010, and the position number where the maximum value of the difference s is taken is the target class. where: p n denotes the probability score of the nth class of output.

5. The method of claim 1 or 2, wherein, In step eight, the specific process of attacking a multi-modal remote sensing image classification network with a test sample is as follows: The TOP test sample is input into the trained TOP perturbation generation network, and the TOP perturbation is output The TOP test sample is input into the trained TOP perturbation generation network, and the TOP perturbation is output The TOP test sample is input into the trained TOP perturbation generation network, and the TOP perturbation is output The DSM test sample is input into the trained DSM (interpretation) perturbation generation network, and the DSM perturbation is output The DSM test sample is input into the trained DSM (interpretation) perturbation generation network, and the DSM perturbation is output The DSM test sample is input into the trained DSM (interpretation) perturbation generation network, and the DSM perturbation is output The multi-modal remote sensing image classification network is attacked by the adversarial attack, and the specific expression is:

6. The method of claim 2, wherein, The specific process of step nine is as follows: The optical remote sensing image training sample x T The input optical remote sensing image training sample is added to the optical remote sensing image perturbation output by the trained optical remote sensing image perturbation generation network to obtain the adversarial sample x' of the optical remote sensing image training sample T ; The digital elevation model image training sample x D Input into the trained digital elevation model image perturbation generation network, output the digital elevation model image perturbation, and add the digital elevation model image perturbation to the input digital elevation model image training sample to obtain the adversarial sample x' of the digital elevation model image training sample D ; x' T and x' D are added to the training sample set to obtain a new training sample set T', that is, {x T , x D , x' T , x' D}∈T', the target model f of step two is retrained.

Images