Deployable patch adversarial attack method, device and equipment for remote sensing image

Abstract

The application relates to a deployable patch adversarial attack method, device and equipment for a remote sensing image. The method comprises the following steps: determining a corresponding feasibility graph according to the remote sensing image through a feasibility graph module, obtaining an attack effectiveness graph corresponding to the remote sensing image according to gradient information through an attack effectiveness graph module, and obtaining a position mask for patch position sampling according to the feasibility graph and the attack effectiveness graph through a position mask module; obtaining a plurality of patch positions through patch position sampling according to the position mask through a patch attack module, superimposing randomly generated adversarial perturbations at the patch positions to obtain an adversarial sample, iteratively optimizing the adversarial perturbations through a gradient descent algorithm until the adversarial attack succeeds, and outputting a final adversarial sample. The application can not only obtain a higher attack success rate under a smaller patch area, but also reduce time consumption, and achieves a good adversarial patch attack effect in the field of remote sensing image scene recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of remote sensing images, in particular to a deployable patch adversarial attack method, device and equipment for remote sensing images. BACKGROUND

[0002] The existing remote sensing image (RSI) adversarial attack method mainly studies the attack on the scene classifier based on deep neural networks (DNNs). The existing technology fools the DNNs natural image classifier by pasting an adversarial patch on the image. The method of adversarial patch has been widely used in many different fields.

[0003] The method of generating RSI adversarial samples by adding adversarial noise is effective, but it is not realistic to deploy dense micro noise in the physical world. It is known that there is currently no research on physically realizable adversarial patches in the RSI scene recognition task. There are three challenges in directly applying the existing adversarial patch attack method on natural images to RSI: (1) it is more difficult to attack the RSI scene classifier, because RSI usually contains fewer scene categories than natural images; (2) the adversarial patch on RSI should be as small as possible to avoid occupying too large a physical size in real deployment; (3) the adversarial patch can only be deployed in limited areas of the scene, and it is difficult to deploy the adversarial patch on, for example, trees, vehicles and water areas. Based on the above analysis, there is a dilemma in deploying adversarial patches on RSI, that is, a smaller patch is difficult to ensure a high enough attack success rate, and a larger patch is difficult to actually deploy. Therefore, the existing technology has the problem of poor adaptability. SUMMARY

[0004] Therefore, it is necessary to provide a deployable patch adversarial attack method, device, computer equipment and storage medium for remote sensing images, which can effectively attack the RSI scene recognition classifier, aiming at the above technical problems.

[0005] A deployable patch adversarial attack method for remote sensing images, the method comprising:

[0006] obtaining a remote sensing image, and inputting the remote sensing image into a pre-designed multi-patch adversarial attack model; the multi-patch adversarial attack model comprises a cascaded position selection module and a patch attack module; the position selection module comprises a feasibility map module, an attack effectiveness map module and a position mask module;

[0007] determining a corresponding feasibility map according to the remote sensing image through the feasibility map module; the feasibility map contains position information capable of placing an adversarial patch;

[0008] inputting the remote sensing image into a target classifier through the attack effectiveness map module, calculating gradient information of each pixel point under a current classification result through a loss function of the target classifier, and obtaining an attack effectiveness map corresponding to the remote sensing image according to the gradient information; the attack effectiveness map contains probability guidance information for placing a patch at a current position;

[0009] obtaining, through the position mask module, a position mask for patch position sampling according to the feasibility map and the attack effectiveness map;

[0010] sampling a plurality of patch positions according to the position mask through the patch attack module, superimposing randomly generated adversarial perturbations at the patch positions, obtaining an adversarial sample, inputting the adversarial sample into the target classifier, and iteratively optimizing the adversarial perturbations through a gradient descent algorithm until the adversarial attack is successful and the iteration is stopped;

[0011] If the iteration reaches a preset maximum number of times and the attack is still unsuccessful, patch position sampling is performed again according to the position mask until a final adversarial sample is output.

[0012] In one of the embodiments, the method further includes: determining, by the feasibility map module, a corresponding feasibility map according to the remote sensing image based on an object context representation semantic segmentation network.

[0013] In one of the embodiments, the method further includes: adding gradient absolute values of three channels of RGB of each pixel point according to the gradient information to obtain the attack effectiveness map corresponding to the remote sensing image.

[0014] In one of the embodiments, the method further includes: obtaining, by the position mask module, a position mask for patch position sampling according to the feasibility map and the attack effectiveness map.

[0015]

[0016] wherein, the feasibility map is denoted as ε, the attack effectiveness map is denoted as ε, and denotes element-wise multiplication.

[0017] In one of the embodiments, the method further includes: calculating a probability value of a pixel point being selected as a patch position according to the gradient information.

[0018]

[0019] wherein, (i, j) denotes a pixel point coordinate, denotes a sum of gradient values of a region where a patch is located, and (u, v) denotes traversal of image pixel points according to coordinates. denotes the sum of gradient values within the region where the patch with (u, v) as the top-left corner is located, and t is a temperature hyper-parameter, which is set as t=10 by default;

[0020] The patch attack module samples patch positions according to the probability values and the position mask to obtain a plurality of patch positions.

[0021] In one embodiment, the method further comprises: superimposing a randomly generated adversarial perturbation on the patch positions to obtain an adversarial sample as follows:

[0022] x adv (m, p)=(1-m)⊙x+m⊙p

[0023] where p denotes an adversarial patch, m has the same size as x, denotes a corresponding 0, 1 mask matrix, is used to determine the position and shape of the patch, and denotes a Hadamard product.

[0024] In one embodiment, the method further comprises: iteratively optimizing the adversarial perturbation according to a preset optimization objective function by using a gradient descent algorithm; and wherein the preset optimization objective function is as follows:

[0025]

[0026] where p denotes an adversarial patch, m has the same size as x, denotes a corresponding 0, 1 mask matrix, is used to determine the position and shape of the patch, f(·) denotes the target classifier, y denotes the true label of the image, S p and S x respectively denote the area of a single patch and the area of the entire image, L denotes a loss function of f(·), and ∈ denotes a limit parameter for the proportion of a single patch in the total area of the image.

[0027] In one embodiment, the method further comprises: randomly generating an initial value of the adversarial perturbation in the interval [0, 1].

[0028] A deployable patch adversarial attack device for remote sensing images, the device comprising:

[0029] A data input module configured to acquire a remote sensing image and input the remote sensing image into a pre-designed multi-patch adversarial attack model; the multi-patch adversarial attack model comprises a cascaded position selection module and a patch attack module; the position selection module comprises a feasibility map module, an attack effectiveness map module, and a position mask module;

[0030] A feasibility map determination module configured to determine a corresponding feasibility map according to the remote sensing image by using the feasibility map module; the feasibility map contains position information of positions where an adversarial patch can be placed;

[0031] An attack effectiveness map determination module is configured to input the remote sensing image into a target classifier through the attack effectiveness map determination module, calculate gradient information of each pixel point under a current classification result through a loss function of the target classifier, and obtain an attack effectiveness map corresponding to the remote sensing image according to the gradient information; the attack effectiveness map contains probability guidance information of placing a patch at a current position;

[0032] A position mask determination module is configured to obtain a position mask for patch position sampling according to the feasibility map and the attack effectiveness map through the position mask determination module;

[0033] An adversarial attack module is configured to obtain a plurality of patch positions by performing patch position sampling according to the position mask through the patch attack module, superimpose randomly generated adversarial perturbations at the patch positions, obtain an adversarial sample, input the adversarial sample into the target classifier, and perform iterative optimization on the adversarial perturbations through a gradient descent algorithm until an adversarial attack succeeds, and stop iteration;

[0034] A result output module is configured to perform patch position sampling again according to the position mask if the number of iterations reaches a preset maximum number and the attack is not successful, and output a final adversarial sample.

[0035] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0036] A remote sensing image is obtained, and the remote sensing image is input into a pre-designed multi-patch adversarial attack model; the multi-patch adversarial attack model includes a cascaded position selection module and a patch attack module; the position selection module includes a feasibility map module, an attack effectiveness map module, and a position mask module;

[0037] A corresponding feasibility map is determined according to the remote sensing image through the feasibility map module; the feasibility map contains position information of an adversarial patch that can be placed;

[0038] The remote sensing image is input into a target classifier through the attack effectiveness map determination module, gradient information of each pixel point under a current classification result is calculated through a loss function of the target classifier, and an attack effectiveness map corresponding to the remote sensing image is obtained according to the gradient information; the attack effectiveness map contains probability guidance information of placing a patch at a current position;

[0039] A position mask for patch position sampling is obtained according to the feasibility map and the attack effectiveness map through the position mask determination module;

[0040] The patch attack module samples patch positions according to the position mask to obtain a plurality of patch positions, superimposes randomly generated adversarial perturbations on the patch positions to obtain an adversarial sample, inputs the adversarial sample into the target classifier, and iteratively optimizes the adversarial perturbations by using a gradient descent algorithm until the adversarial attack is successful, and the iteration is stopped.

[0041] If the iteration reaches a preset maximum number of times and the attack is still unsuccessful, patch position sampling is performed again according to the position mask until a final adversarial sample is output.

[0042] A computer-readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the following steps:

[0043] A remote sensing image is obtained, and the remote sensing image is input into a pre-designed multi-patch adversarial attack model; the multi-patch adversarial attack model includes a cascaded position selection module and a patch attack module; the position selection module includes a feasibility map module, an attack effectiveness map module, and a position mask module;

[0044] The feasibility map module determines a corresponding feasibility map according to the remote sensing image; the feasibility map contains position information of positions where an adversarial patch can be placed;

[0045] The attack effectiveness map module inputs the remote sensing image into a target classifier, calculates gradient information of each pixel point under a current classification result by using a loss function of the target classifier, and obtains an attack effectiveness map corresponding to the remote sensing image according to the gradient information; the attack effectiveness map contains probability guidance information of placing a patch at a current position;

[0046] The position mask module obtains a position mask for patch position sampling according to the feasibility map and the attack effectiveness map;

[0047] The patch attack module samples patch positions according to the position mask to obtain a plurality of patch positions, superimposes randomly generated adversarial perturbations on the patch positions to obtain an adversarial sample, inputs the adversarial sample into the target classifier, and iteratively optimizes the adversarial perturbations by using a gradient descent algorithm until the adversarial attack is successful, and the iteration is stopped.

[0048] If the iteration reaches a preset maximum number of times and the attack is still unsuccessful, patch position sampling is performed again according to the position mask until a final adversarial sample is output.

[0049] The deployable patch adversarial attack method, device, computer equipment and storage medium for remote sensing images, determine a corresponding feasibility map according to the remote sensing image through a feasibility map module, input the remote sensing image into a target classifier through an attack effectiveness map module, calculate the gradient information of each pixel point under the current classification result through the loss function of the target classifier, obtain the attack effectiveness map corresponding to the remote sensing image according to the gradient information, and obtain the position mask for patch position sampling according to the feasibility map and the attack effectiveness map through a position mask module; the patch position sampling is performed according to the position mask through a patch attack module to obtain a plurality of patch positions, the adversarial perturbation generated randomly is superimposed at the patch positions to obtain an adversarial sample, the adversarial sample is input into the target classifier, the gradient descent algorithm is used to iteratively optimize the adversarial perturbation until the adversarial attack succeeds, and the iteration is stopped; if the iteration times reach the preset maximum number and the attack is still unsuccessful, the patch position sampling is performed again according to the position mask until the final adversarial sample is output. The present application uses a plurality of small and imperceptible adversarial patches to replace a single adversarial patch to perform a physically feasible adversarial attack, which can not only obtain a higher attack success rate under a smaller patch area, but also reduce the time consumption, and achieves a good adversarial patch attack effect in the field of remote sensing image scene recognition. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A flowchart of a deployable patch adversarial attack method for remote sensing images in an embodiment;

[0051] Figure 2 A multi-patch adversarial attack algorithm framework in an embodiment;

[0052] Figure 3 Adversarial sample comparison graphs with and without the addition of FRSNet in an embodiment, wherein, Figure 3 (a), Figure 3 (b) and Figure 3 (c) are adversarial sample result graphs of three scenes with the addition of FRSNet, respectively, Figure 3 (d), Figure 3 (e) and Figure 3 (f) are adversarial sample result graphs of three scenes without the addition of FRSNet, respectively;

[0053] Figure 4 Adversarial sample comparison graphs with and without the addition of a TV loss function in an embodiment, wherein, Figure 4 (a), Figure 4 (b) and Figure 4 (c) are adversarial sample result graphs of three scenes with the addition of a TV loss function, respectively, Figure 4 (d), Figure 4 (e) and Figure 4(f) is the result map of the adversarial sample without adding the TV loss function for three scenes respectively;

[0054] Figure 5 Fig. 4 is a heat map of adversarial samples for different patch numbers in an embodiment, wherein Figure 5 (a) is a heat map of adversarial samples for clean images, Figure 5 (b) is a heat map of adversarial samples for patch number 4, Figure 5 (c) is a heat map of adversarial samples for patch number 8, Figure 5 (d) is a heat map of adversarial samples for patch number 16;

[0055] Figure 6 Fig. 5 is a structural block diagram of a deployable patch adversarial attack device for remote sensing images in an embodiment;

[0056] Figure 7 Fig. 6 is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0057] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0058] In an embodiment, as shown in Figure 1 , a deployable patch adversarial attack method for remote sensing images is provided, comprising the following steps:

[0059] Step 102, acquiring a remote sensing image, inputting the remote sensing image into a pre-designed multi-patch adversarial attack model.

[0060] The present application proposes a multi-patch adversarial attack (Multi-patch Adversarial Attack, MPAA) method.

[0061] As shown in Figure 2 , the multi-patch adversarial attack model comprises a cascaded position selection module and a patch attack module; the position selection module comprises a feasibility map module, an attack effectiveness map module and a position mask module.

[0062] The principle of the present application is as follows:

[0063] In order to realize a feasible and robust RSI scene classification adversarial patch attack, the present application proposes to generate an adversarial sample x adv (m, p) by optimizing n patches on n feasible positions of a benign image x. The optimization objective of MPAA can be expressed as:

[0064]

[0065] x adv (m, p) = (1 - m) 0 x + m 0 p. (2)

[0066] where denotes Hadamard product, p denotes adversarial patch, m has the same size as x, denotes the corresponding 0, 1 mask matrix, which is used to determine the position and shape of the patch. f(·) denotes the target classifier, y denotes the real label of the image, denotes the feasibility map, S p and S x denote the area of a single patch and the whole image, respectively. L denotes the loss function of f(·), which is calculated using the output confidence of f(·) with respect to y:

[0067] L = Pr(y | f(x adv (m, p)). (3)

[0068] As can be seen from formula (1), the objective function contains two variables that affect each other, patch position m and patch pattern p. This is a double-layer optimization problem, which is difficult to optimize directly. The present application proposes a search and optimization method to solve this problem, that is, using a feasible and effective location selection (FELS) module to determine m, and a patch attack (PA) module to optimize p.

[0069] Step 104, determining the corresponding feasibility map according to the remote sensing image through the feasibility map module.

[0070] The feasibility map contains the position information of the adversarial patch that can be placed.

[0071] The feasible and effective location selection module simultaneously considers the physical feasibility map and the attack effectiveness map ε to optimize the mask m. The position map is obtained by the element-wise product of and ε, which is used for sampling the patch position.

[0072]

[0073] The feasibility map is obtained by a feasible region selection network (FRSNet) that applies the structure of an object-contextual representation (OCR) semantic segmentation network trained on the DLRSD dataset with 17 classes. The network is used to exclude locations that are not suitable for patch placement, where the present embodiment selects bare soil, docks, fields, grass, sidewalks, and beaches as regions where adversarial patches can be placed. The pixels in the output that can place patches are assigned a value of 1, and other pixels are assigned a value of 0. In order to avoid boundary overlap, the results of the region segmentation are expanded relative to the size of the patch.

[0074] At step 106, the remote sensing image is input into the target classifier through the attack effectiveness map module, the gradient information of each pixel point under the current classification result is calculated through the loss function of the target classifier, and the attack effectiveness map corresponding to the remote sensing image is obtained according to the gradient information. The attack effectiveness map contains probability guidance information for placing a patch at the current position.

[0075] Unlike methods based on a single patch, the present application requires the deployment of n small adversarial patches, which is a difficult combinatorial optimization problem. The present application proposes a fast and effective position selection algorithm, the random gradient (Rand-Grad) method.

[0076] The present application generates an effectiveness map ε to represent the influence of the pixel points on the image on the classification result. Specifically, x is input into the classifier, and the gradient with respect to x is calculated using the loss function L. It is noted that the greater the gradient of each pixel point, the more likely it is to affect the classification result by changing the pixel point. Therefore, the absolute values of the gradients of the three channels of each pixel point RGB are added to calculate ε:

[0077]

[0078] At step 108, the position mask for patch position sampling is obtained by the position mask module according to the feasibility map and the attack effectiveness map.

[0079] As shown in equation (4), the position mask is obtained by the element-wise product of ε, which is used for patch position sampling.

[0080] At step 110, the patch attack module samples a plurality of patch positions according to the position mask, superimposes a randomly generated adversarial perturbation on the patch positions to obtain an adversarial sample, inputs the adversarial sample into the target classifier, and iteratively optimizes the adversarial perturbation by a gradient descent algorithm until the adversarial attack succeeds, and stops iteration.

[0081] The patch attack module samples a plurality of patch positions according to the position mask. It is mentioned above that the greater the gradient of each pixel point, the more likely it is to affect the classification result by changing the pixel point. However, directly selecting the largest n gradient values cannot guarantee the optimal solution, so randomness is introduced in the selection. Specifically, these gradient values are regarded as the probability guide for selecting patch positions, and the larger the gradient, the easier the region is to be selected. The Rand-Grad algorithm uses a softmax algorithm with a temperature coefficient to smooth the weight of the gradient. The specific probability calculation is as follows:

[0082]

[0083] where (i, j) represents the coordinates of the upper left corner of the patch, represents the sum of gradient values in the region where the patch is located, p i，j represents the probability of selecting the (i, j) coordinates, (u, v) represents the traversal of the image pixel points according to the coordinates, represents the sum of gradient values in the region where the patch is located, p represents the sum of gradient values in the region where the patch is located, p

[0084] After the FELS module gives the mask m, the patch attack (PA) module optimizes the adversarial perturbation p to mislead the target classifier. The initial value of p is randomly generated in the interval [0, 1], and then the adversarial sample picture xadv can be obtained by formula 2. After m is determined, only one variable p needs to be optimized in formula 1, and the gradient descent method is used to update and optimize the value of p. The specific optimizer is set to Adam, and the learning rate is 2 / 255.

[0085] The maximum number of iterations in the optimization is set to 2000. If the confidence of the classifier output y is less than 10%, it is considered that the adversarial attack is successful and the iteration is stopped. If the sampling position does not attack successfully after 2000 iterations, the patch position will be resampled according to the probability. By using this patch position resampling strategy, MPAA can obtain better patch positions to improve the attack success rate.

[0086] At step 112, if the number of iterations reaches the preset maximum number and the attack is not successful, the patch position is resampled according to the position mask until the final adversarial sample is output.

[0087] In the above deployable patch adversarial attack method for remote sensing images, the feasibility map module determines the corresponding feasibility map according to the remote sensing image, the attack effectiveness map module inputs the remote sensing image into the target classifier, the gradient information of each pixel point under the current classification result is calculated by the loss function of the target classifier, the attack effectiveness map corresponding to the remote sensing image is obtained according to the gradient information, and the position mask module obtains the position mask for patch position sampling according to the feasibility map and the attack effectiveness map; the patch attack module performs patch position sampling according to the position mask to obtain a plurality of patch positions, superimposes randomly generated adversarial perturbations on the patch positions to obtain an adversarial sample, inputs the adversarial sample into the target classifier, and iteratively optimizes the adversarial perturbations by the gradient descent algorithm until the adversarial attack is successful and the iteration is stopped; if the iteration number reaches the preset maximum number and the attack is still unsuccessful, the patch position sampling is performed again according to the position mask until the final adversarial sample is output. The present application proposes a method of using multiple small adversarial patches for attack, solves the problem of too large physical size of the patch in real application, introduces the restriction of the feasibility map in position selection, solves the problem of unreasonable patch placement position in real scene, and obtains a more optimal patch placement position by the Rand-Grad position selection algorithm, thereby improving the success rate of adversarial attack.

[0088] It should be understood that, although Figure 1 The steps in the flowchart of the present application are displayed in sequence according to the arrows, but these steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, Figure 1 At least part of the steps in the flowchart of the present application can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or sub-steps or stages of other steps.

[0089] In one specific embodiment, the method of the present application is experimentally verified, and the experiment is as follows:

[0090] Dataset and target model: The proposed method is evaluated using the Aerial Image Dataset (AID). AID has 30 classes and 10,000 images, all with a resolution of 600*600. The dataset is randomly divided into training and test sets in a ratio of 7:3. A pre-trained ResNet50 is used as the target model, which is fine-tuned on the training set to achieve a top-1 error rate of 3.83%. ResNet101 and Densenet121 are also fine-tuned, achieving top-1 error rates of 4.43% and 3.80%, respectively. All experiments are performed on an NVIDIA RTX 3090 GPU with 24G of video memory.

[0091] Evaluation metrics: The attacking success rate (ASR) and the time consumed are used as evaluation metrics to determine the effectiveness and efficiency of the method. During testing, images that cannot be correctly classified or do not have enough space to place patches are discarded.

[0092] Baseline: To verify the effectiveness of the proposed MPAA method, it is compared with three patch attack methods: Advp, GDPA, and Pfool. To ensure fairness, all methods are evaluated on the AID dataset, performing untargeted attacks with a total patch area of 1% of the image area. For MPAA, the number of patches is set to 16, and the maximum number of resampling is set to 3.

[0093] Comparison results:

[0094] To evaluate the effectiveness of the proposed method, the MPAA method and the baseline methods are compared on three different classifiers. The main focus of this paper is on non-targeted attacks, and for the sake of fair comparison, the constraint of position feasibility is not considered when evaluating the MPAA method, as in the baseline methods. Table 1 shows the adversarial attack performance of different methods. It can be seen that the MPAA method of the present application achieves the highest ASR for all classifiers, and the convergence speed is also faster. The ASR of the three comparative methods is lower, especially when attacking the classifier at a deeper level. The AdvP method uses one patch and randomly selects the placement position, resulting in a huge gap in ASR compared with the MPAA method using multiple patches to attack (the ASR gap is more than 25% on three classifiers), although the total area size of the patch is the same. GDPA uses a generator instead of an optimizer to generate adversarial patches, so it also has a lower time complexity for complex classifiers, but the ASR is the lowest among all methods. The PFool method divides the image into a fixed number of blocks according to the size of the patch, and then selects blocks to place the patch using a saliency map, which may limit the flexibility of patch placement. The comparison results also show that the ASR of PFool is lower than the proposed MPAA method using gradient information as a probability guide for patch position selection.

[0095] Table 1 Attack performance of different methods

[0096]

[0097] *: To compare fairly, remove the constraint of FRSNet

[0098] Ablation experiments:

[0099] To study the performance of the method, this embodiment performs experiments on the performance of each component on ResNet50. As shown in Table 2, the proposed patch position selection algorithm Rand-Grad indeed obtains better ASR than the random position selection method without any guidance. If FRSNet is not used, the final ASR will be slightly improved, but this means that the adversarial patch is only optimized for adversarial performance, and will be placed at an infeasible position (such as trees, vehicles, water) with a high probability, as shown in Figure 3 It is also interesting to note that by increasing the number of patches, MPAA not only obtains a higher ASR, but also can reach convergence in a shorter time. This verifies the effectiveness and efficiency of the proposed patch adversarial attack. Considering the difficulties in actual deployment, the number of patches n = 16 is set as the default value.

[0100] Table 2 Attack success rate (ASR%) of MPAA method without different components

[0101] Patch number 1 2 4 8 16 No Rand-Grad 74.49 81.84 87.79 89.74 92.34 No FRSNet 79.05 87.08 92.23 93.42 96.61 No resampling 69.72 77.62 84.49 89.01 91.37 Full MPAA 76.19 84.44 90.21 92.74 94.80

[0102] In another embodiment, to illustrate the scalability of the method of the present application in imperceptibility. The present application can add a total variation (TV) loss function to reduce the visibility of the adversarial patch, which can be expressed as:

[0103]

[0104] The total loss function can be expressed as:

[0105] L total = L(f(x adv ), y) + λL TV , # (8)

[0106] Where λ is a hyperparameter for balancing imperceptibility and ASR, and the default setting is λ = 10. The TV loss function can help generate more smooth pattern adversarial patches, thus improving the imperceptibility of the adversarial samples. The visual effect of the adversarial samples with and without TV loss function is shown in Figure 4 . It should be noted that although adding TV loss can effectively increase the imperceptibility of the adversarial samples, as a trade-off, when using 16 patches, the attack success rate also decreases from 94.80% to 82.92%, but compared with the traditional method, it is still a relatively excellent attack success rate.

[0107] In another embodiment, the Grad-CAM method is used to visualize the attention shift of the classifier between benign images and adversarial images. The visualization result is shown in Figure 5 . It can be seen that in the clean image, the focus of the classifier is mainly on the oil tank, while in the adversarial image generated by MPAA, almost all the attention is shifted to the adversarial patch. With the increase of the number of patches, the attention becomes more scattered, and it can be considered that this is the reason why the attack can become more successful with the increase of the number of patches.

[0108] In one embodiment, as shown in Figure 6 , a deployable patch adversarial attack device for remote sensing images is provided, comprising a data input module 602, a feasibility map determination module 604, an attack effectiveness map determination module 606, a position mask determination module 608, an adversarial attack module 610 and a result output module 612, wherein:

[0109] The data input module 602 is configured to acquire a remote sensing image and input the remote sensing image into a pre-designed multi-patch adversarial attack model; the multi-patch adversarial attack model comprises a cascaded position selection module and a patch attack module; the position selection module comprises a feasibility map module, an attack effectiveness map module and a position mask module;

[0110] The feasibility map determination module 604 is configured to determine a corresponding feasibility map according to the remote sensing image by the feasibility map module; the feasibility map contains position information of a position capable of placing an adversarial patch;

[0111] The attack effectiveness map determination module 606 is configured to input the remote sensing image into a target classifier by the attack effectiveness map module, calculate gradient information of each pixel point under a current classification result by a loss function of the target classifier, and obtain an attack effectiveness map corresponding to the remote sensing image according to the gradient information; the attack effectiveness map contains probability guidance information of placing a patch at a current position;

[0112] The position mask determination module 608 is configured to obtain a position mask for patch position sampling according to the feasibility map and the attack effectiveness map by the position mask module;

[0113] The adversarial attack module 610 is configured to obtain a plurality of patch positions by patch position sampling according to the position mask by the patch attack module, superimpose randomly generated adversarial perturbations at the patch positions, obtain an adversarial sample, input the adversarial sample into the target classifier, and perform iterative optimization on the adversarial perturbations by a gradient descent algorithm until the adversarial attack is successful, and stop iteration;

[0114] The result output module 612 is configured to re-perform patch position sampling according to the position mask if the iteration times reach a preset maximum number and the attack is not successful, until a final adversarial sample is output.

[0115] The feasibility map determination module 604 is further configured to determine a corresponding feasibility map according to the remote sensing image by the feasibility map module based on an object context representation semantic segmentation network.

[0116] The attack effectiveness map determination module 606 is further configured to add absolute values of gradients of three channels of RGB of each pixel point to obtain an attack effectiveness map corresponding to the remote sensing image according to the gradient information.

[0117] The position mask determination module 608 is further configured to obtain a position mask for patch position sampling according to the feasibility map and the attack effectiveness map by the position mask module.

[0118]

[0119] wherein f represents the feasibility map, and ε represents the attack effectiveness map. The feasibility map is represented by f, and the attack effectiveness map is represented by ε. represents element-wise multiplication.

[0120] The anti-attack module 610 is further configured to calculate a probability value of a pixel point being selected as a patch position according to gradient information:

[0121]

[0122] wherein (i, j) represents a pixel point coordinate, represents a sum of gradient values of a region where the patch is located, (u, v) represents traversal of image pixel points according to coordinates, represents a sum of gradient values in a region where the patch with the top-left vertex (u, v) is located, and t is a temperature hyperparameter, and the default setting is t = 10;

[0123] The patch attack module samples a plurality of patch positions according to the probability value and the position mask.

[0124] The anti-attack module 610 is further configured to superimpose a randomly generated adversarial perturbation on the patch position to obtain an adversarial sample as:

[0125] x adv (m, p) = (1 - m) O x + m O p

[0126] wherein p represents an adversarial patch, m is the same size as x, represents a corresponding 0, 1 mask matrix, and is used to determine the position and shape of the patch, and O represents Hadamard product.

[0127] For specific limitations of the deployable patch adversarial attack device for remote sensing images, refer to the limitations of the deployable patch adversarial attack method for remote sensing images in the foregoing, which will not be repeated here. Each module in the above deployable patch adversarial attack device for remote sensing images can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0128] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram thereof can be as shown in Figure 7As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a deployable patch attack method for remote sensing images. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0129] Those skilled in the art can understand that, Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0130] In one embodiment, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0131] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0133] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0134] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A deployable patch adversarial attack method for remote sensing images, characterized in that, The method comprises: acquiring a remote sensing image, and inputting the remote sensing image into a pre-designed multi-patch adversarial attack model; the multi-patch adversarial attack model comprises cascaded position selection modules and patch attack modules; the position selection modules comprise a feasibility map module, an attack effectiveness map module and a position mask module; determining a corresponding feasibility map from the remote sensing image through the feasibility map module; the feasibility map contains position information capable of placing an adversarial patch; inputting the remote sensing image into a target classifier through the attack effectiveness map module, calculating gradient information of each pixel point under a current classification result through a loss function of the target classifier, and obtaining an attack effectiveness map corresponding to the remote sensing image according to the gradient information; the attack effectiveness map contains probability guidance information of placing a patch at a current position; the probability guidance information is a gradient value of a pixel point, and the greater the gradient value, the greater the probability of the pixel point being selected; obtaining a position mask for patch position sampling according to the feasibility map and the attack effectiveness map through the position mask module; sampling a plurality of patch positions according to the position mask through the patch attack module, superimposing randomly generated adversarial perturbations at the patch positions to obtain an adversarial sample, inputting the adversarial sample into the target classifier, and iteratively optimizing the adversarial perturbations through a gradient descent algorithm until the adversarial attack is successful, and stopping iteration; if the iteration number reaches a preset maximum number and the attack is still not successful, re-sampling patch positions according to the position mask until a final adversarial sample is output.

2. The method of claim 1, wherein, determining a corresponding feasibility map from the remote sensing image through the feasibility map module, comprising: determining a corresponding feasibility map from the remote sensing image through the feasibility map module based on an object context representation semantic segmentation network.

3. The method of claim 2, wherein, obtaining an attack effectiveness map corresponding to the remote sensing image according to the gradient information, comprising: adding gradient absolute values of three channels of each pixel point RGB to obtain an attack effectiveness map corresponding to the remote sensing image according to the gradient information.

4. The method of claim 3, wherein, obtaining a position mask for patch position sampling according to the feasibility map and the attack effectiveness map through the position mask module, comprising: obtaining a position mask for patch position sampling according to the feasibility map and the attack effectiveness map through the position mask module is: ； wherein, represents the feasibility map, represents the attack effectiveness map, represents element-wise multiplication.

5. The method of claim 4, wherein, sampling a plurality of patch positions according to the position mask through the patch attack module, comprising: calculating a probability value of a pixel point being selected as a patch position according to the gradient information: ； wherein, represents the pixel point coordinate, represents the sum of gradient values in the region where the patch is located, represents the traversal of image pixel points by coordinates, represents the sum of gradient values in the region where the patch is located with the top left corner as the top left corner, is a temperature hyperparameter, and the default setting is ; sampling a plurality of patch positions according to the probability value and the position mask through the patch attack module.

6. The method of claim 5, wherein, superimposing randomly generated adversarial perturbations at the patch positions to obtain an adversarial sample, comprising: superimposing randomly generated adversarial perturbations at the patch positions to obtain an adversarial sample is: ； wherein, represents the adversarial patch, and of the same size, represents the corresponding 0, 1 mask matrix, which determines the position and shape of the patch, represents the Hadamard product.

7. The method of claim 6, wherein, iteratively optimizing the adversarial perturbations through a gradient descent algorithm, comprising: iteratively optimizing the adversarial perturbations through a gradient descent algorithm according to a preset optimization objective function; wherein the preset optimization objective function is: ； wherein, represents the adversarial patch, with same size, represents the corresponding 0, 1 mask matrix, used to determine the location and a shape of the patch, represents the target classifier, represents the image ground truth label, and respectively represent the area of a single patch and the whole picture, represents the loss function of, represents the restriction parameter on the proportion of a single patch in the total area of the picture.

8. The method according to any one of claims 1 to 7, characterized in that, The initial value of the adversarial perturbation is randomly generated in the interval [0, 1].

9. A deployable patch adversarial attack device for remote sensing images, characterized in that, The device comprises: a data input module configured to acquire a remote sensing image and input the remote sensing image into a pre-designed multi-patch adversarial attack model; the multi-patch adversarial attack model comprises a cascaded position selection module and a patch attack module; the position selection module comprises a feasibility map module, an attack effectiveness map module and a position mask module; a feasibility map determination module configured to determine a corresponding feasibility map according to the remote sensing image by the feasibility map module; the feasibility map contains position information of positions capable of placing adversarial patches; an attack effectiveness map determination module configured to input the remote sensing image into a target classifier by the attack effectiveness map module, calculate gradient information of each pixel point under a current classification result by a loss function of the target classifier, and obtain an attack effectiveness map corresponding to the remote sensing image according to the gradient information; the attack effectiveness map contains probability guidance information of placing patches at current positions; the probability guidance information is a gradient value of a pixel point, and the greater the gradient value, the greater the probability of selecting the pixel point; a position mask determination module configured to obtain a position mask for patch position sampling according to the feasibility map and the attack effectiveness map by the position mask module; an adversarial attack module configured to perform patch position sampling according to the position mask by the patch attack module to obtain a plurality of patch positions, superimpose randomly generated adversarial perturbations at the patch positions to obtain an adversarial sample, input the adversarial sample into the target classifier, and perform iterative optimization on the adversarial perturbations by a gradient descent algorithm until the adversarial attack is successful, and stop iteration; a result output module configured to, if the iteration reaches a preset maximum number and the attack is still unsuccessful, re-perform patch position sampling according to the position mask until a final adversarial sample is output.

10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor implements the steps of the method in any one of claims 1 to 8 when executing the computer program.

Images