An adaptive defense method and device against patch attack of ViT

Abstract

This invention discloses an adaptive defense method and apparatus against ViT adversarial patch attacks, belonging to the field of adversarial defense technology. The method includes: based on a dynamic sliding window strategy, using an attack detection module to identify adversarial patches from the original image to obtain a patch mask; based on the patch mask, using an attack classifier to determine the attack nature from the original image to obtain an attack classification result; and based on the attack classification result and the patch mask, using an adaptive defense strategy module to perform cleanup and re-inference from the original image to obtain image confidence. This invention provides an efficient, accurate, and robust adaptive defense method against ViT adversarial patch attacks.

Description

Technical Field

[0001] This invention relates to the field of adversarial defense technology, and in particular to an adaptive defense method and apparatus for ViT adversarial patch attacks. Background Technology

[0002] With the rapid development of artificial intelligence technology, the Vision Transformer (ViT) has achieved remarkable results in tasks such as image classification and object detection. However, with the adjustment of model architecture and the increasing complexity of application scenarios, its security issues have gradually been exposed, especially when facing adversarial attacks. Studies have shown that ViT is more vulnerable to adversarial patch attacks. The stealth and efficiency of patch attacks make them a more threatening attack method. Attackers can significantly alter the attention distribution of ViT by adding carefully designed adversarial patches to local regions of an image, thereby misleading the model's predictions. This attack method is not only difficult for the human eye to detect, but it can also disrupt the model's decision-making logic without affecting the overall semantics of the image. Since ViT's global modeling capability relies on a multi-head self-attention mechanism, patch attacks, by manipulating the features of local regions, can amplify their influence globally, significantly weakening or diverting the model's attention to key regions. The flexibility and destructiveness of this attack method pose a serious threat to the security of models in practical applications.

[0003] Research on defense against ViT adversarial patch attacks mainly falls into two categories: training-time defense and inference-time defense. Traditional adversarial training methods face significant cost and effectiveness bottlenecks. As a classic defense strategy, adversarial training's core idea is to continuously generate adversarial examples during training and integrate them into the training set, forcing the model to learn more robust feature representations. However, when applied to patch attack scenarios against ViT models, its limitations become particularly pronounced. Adversarial training is computationally extremely expensive. The ViT model itself has a large number of parameters, and generating effective patch adversarial examples requires multiple rounds of optimization for each training sample, a process whose computational complexity far exceeds that of standard training. Therefore, various defense methods have been proposed for inference-time defense strategies to address these threats. Inference-time defense strategies against ViT patch attacks can be divided into two main categories: input detection methods and representation optimization methods.

[0004] Input detection methods identify potential adversarial patches by analyzing the feature distribution of the input image and the distribution changes of the model output, and then remove or repair the identified patches. This type of defense strategy treats the model as a black box and is applicable to various models such as ViT and convolutional neural networks. Researchers also found that adversarial patches can cause incorrect predictions because, within the model's internal computation, the feature representations of adversarial patches exhibit unusually significant attention weights. This phenomenon deepens with layer stacking, causing the model to incorrectly focus on the feature representations of adversarial patches, leading to inconsistencies between predicted and true labels.

[0005] Existing defense methods have significant shortcomings in dealing with patch attacks. First, most methods do not perform attack detection on input samples, but instead apply a uniform defense strategy to all input samples. This approach not only increases computational overhead but may also negatively impact the classification performance of clean samples. Since clean samples have not been attacked, directly applying a defense strategy to them may destroy their original features, leading to a decrease in model performance. The lack of an attack detection mechanism makes it difficult for existing methods to strike a balance between defense effectiveness and computational efficiency. Second, existing methods often ignore the differential impact of attack location on model decisions. Patch attacks may occur in areas of the image such as high-attention regions or background regions, and the impact of these two situations on the model is obviously different. However, existing defense methods often do not distinguish between attack locations, but instead apply the same defense strategy to all attacked samples. This neglect of the differences in attack location may lead to insufficient targeting of the defense strategy. For example, when the attack occurs in a non-critical region, excessive defense may destroy the overall features of the image; when the attack occurs in a critical region, the defense strategy may not be able to effectively mitigate the impact of the attack. Therefore, how to design a strategy that can distinguish the attack location and implement targeted defense has become a key issue in improving defense effectiveness.

[0006] Furthermore, the diversity and complexity of patch attacks further exacerbate the difficulty of defense. In real-world scenarios, attackers may evade detection by adjusting the size, shape, and position of patches, and leverage ViT's attention mechanism to optimize the perturbation effect of the patches. This diversity of attack forms makes it difficult for a single defense strategy to provide comprehensive coverage. Patch attacks may also be combined with other adversarial techniques, such as pixel-level perturbation or occlusion attacks, to further enhance the stealth and destructiveness of the attacks. These issues not only challenge existing defense methods but also highlight the necessity of designing more efficient and robust defense strategies.

[0007] In the existing technology, there is a lack of an efficient, accurate and robust adaptive defense method against ViT anti-patch attacks. Summary of the Invention

[0008] To address the technical problems of high computational cost, insufficient targeting, and limited adaptability in existing technologies, this invention provides an adaptive defense method and apparatus against ViT anti-patch attacks. The technical solution is as follows:

[0009] On the one hand, an adaptive defense method against ViT anti-patch attacks is provided. This method is implemented by an adaptive defense device and includes:

[0010] Obtain the original image containing the adversarial patch;

[0011] Based on a dynamic sliding window strategy, an attack detection module is used to identify adversarial patches and obtain patch masks from the original image.

[0012] Based on the patch mask, an attack classifier is used to determine the nature of the attack based on the original image, and the attack classification result is obtained.

[0013] Based on the attack classification results and patch mask, the image confidence is obtained by using the adaptive defense strategy module to perform sanitization and re-inference based on the original image.

[0014] On the other hand, an adaptive defense device for ViT anti-patch attacks is provided, which is applied to an adaptive defense method against ViT anti-patch attacks. The device includes:

[0015] Image acquisition unit, used to acquire the original image containing the adversarial patch;

[0016] The attack detection unit is used to identify adversarial patches based on the original image using the attack detection module, based on a dynamic sliding window strategy, and to obtain the patch mask.

[0017] The attack classification unit is used to determine the nature of the attack based on the patch mask and the original image, and obtain the attack classification result.

[0018] The image re-inference unit is used to perform sanitization re-inference based on the attack classification results and patch mask, according to the original image, and obtain the image confidence score.

[0019] On the other hand, an adaptive defense device is provided, comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for adaptive defense against ViT anti-patch attacks.

[0020] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described adaptive defense methods against ViT anti-patch attacks.

[0021] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0022] This invention proposes an adaptive defense device against ViT adversarial patch attacks. By combining an attack detector module with the Sobel operator to calculate the gradient and distance scores of image patches, and using a dynamic window size strategy to scan the image, it can accurately identify and locate samples containing adversarial patches and clean samples. Defense operations are only performed on adversarial samples, which avoids the damage to the classification performance of clean samples caused by the existing "one-size-fits-all" defense strategy, and significantly reduces unnecessary computational overhead, thus solving the problem of excessively high computational cost of traditional defense methods.

[0023] The attack classifier module calculates the average pairwise similarity of the model output after replacing the adversarial region with adjacent image blocks, effectively distinguishing between catastrophic attacks (adversarial blocks covering key semantic regions) and non-catastrophic attacks (adversarial blocks located in low semantic importance regions), thus making up for the shortcomings of existing methods that ignore the differences in attack locations, resulting in insufficient defense targeting.

[0024] The adaptive defense strategy module designs adaptation schemes for two types of attacks. For non-catastrophic attacks, image reconstruction is used (using masks to remove adversarial blocks and replace them with adjacent benign blocks) to preserve key semantic features. For catastrophic attacks, an image patch-aware attention suppression algorithm (TAS) is proposed. By excluding adversarial image blocks, the average attention value of clean patches is calculated, and the mean is calculated independently for each attention head. This perfectly adapts to the characteristics of the ViT multi-head self-attention mechanism and significantly improves the robustness and accuracy of the model in adversarial scenarios.

[0025] The dynamic window strategy and TAS image patch awareness logic can cope with adversarial patch attacks of different sizes and positions, effectively avoiding the problem of attackers circumventing defenses by adjusting patch parameters, and enhancing the applicability of the method in complex real-world scenarios. This invention is an efficient, accurate, and robust adaptive defense method against ViT adversarial patch attacks. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of an adaptive defense method against ViT patch attacks provided by an embodiment of the present invention;

[0028] Figure 2 This is a block diagram of an adaptive defense device against ViT patch attacks provided in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of an adaptive defense device provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0031] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0032] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0033] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0034] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0035] This invention provides an adaptive defense method against ViT anti-patch attacks. This method can be implemented by an adaptive defense device, which can be a terminal or a server. Figure 1 The flowchart shown illustrates an adaptive defense method against ViT anti-patch attacks. The processing flow of this method may include the following steps:

[0036] S1. Obtain the original image containing the anti-patch;

[0037] In one feasible implementation, an adversarial patch is a deliberately designed, pixel-continuous small patch of image that misleads a deep learning model into making incorrect judgments by replacing or overlaying it onto local areas of an image. For example, an adversarial patch overlaid on a normal image, such as a regular street scene photograph (original image), might prevent the object detector from recognizing "pedestrians" or "stop signs" after an adversarial patch is placed in the upper left corner. In infrared images, the adversarial patch might appear as a hotspot pattern, causing pedestrian detectors to fail. These images are deceptive to the model and are therefore widely used to test the robustness of models.

[0038] S2. Based on the dynamic sliding window strategy, the attack detection module is used to identify adversarial patches according to the original image and obtain the patch mask.

[0039] Optionally, based on a dynamic sliding window strategy, an attack detection module is used to identify adversarial patches from the original image to obtain a patch mask, including:

[0040] Based on the sliding scan method with dynamically reduced window size, the Sobel operator is used to calculate the local neighborhood gradient difference on a scale-by-scale according to the original image to obtain the distance score;

[0041] Based on a preset distance score threshold, the distance scores are filtered to obtain distance scores that exceed the threshold.

[0042] Based on a preset multi-scale sliding window, the window region corresponding to the maximum value of the distance score exceeding the threshold is selected as the adversarial patch region according to the original image, and the position index and patch side length of the adversarial patch region are generated.

[0043] The patch mask is constructed based on the position index and the patch side length.

[0044] In one feasible implementation, the motivation for designing the attack detector in this invention stems from the reduced pixel continuity within adversarial image patches. The generation process of adversarial image patches ignores the natural local correlations between pixels, thus generating abrupt changes in pixel values ​​within the image patch, i.e., large local gradients. These high gradients can be detected by classical edge detectors, particularly the Sobel operator. It uses horizontal and vertical convolution kernels to convolve the image to estimate local derivatives, effectively highlighting regions with drastic pixel changes.

[0045] Adversarial image patches exhibit the most significant pixel gradients, while clean image patches show relatively smaller pixel gradients. This contrast can be used for adversarial patch detection. Since the size of the attack patch patch is proportional to the size of the model's input image patch, a Sobel score is calculated for each pixel, and then the average Sobel score within each image patch is calculated, denoted as AvgSobel. Image patches with higher AvgSobel values ​​are more likely to contain adversarial patches. However, some clean image patches may also exhibit high AvgSobel values, making detection based solely on this metric unreliable.

[0046] To alleviate this problem, this invention introduces a distance score d, which considers not only the AvgSobel value of the current image patch but also the AvgSobel values ​​of its neighboring image patches, denoted as da. ,in This represents the set of K adjacent image patches.

[0047] For each image patch, samples are taken from adjacent image patches in the following four directions (up, down, left, right), with the same sample size as the current image patch, and the AvgSobel average of its own and that of its neighboring image patches is calculated. The distance between them is given by the following formula (1):

[0048] (1);

[0049] Since the location of adversarial blocks is unknown, a sliding window strategy is used to scan the image to find suspected adversarial regions. The window size is set to be equal to the assumed size of the adversarial block. For each window, its neighboring adversarial blocks are sampled, and the corresponding distance scores are calculated. After sliding through the entire image, a set of distance values ​​is obtained, and the maximum distance is determined. The process is as follows (2):

[0050] (2);

[0051] Where z represents the total number of sliding windows. If Exceeding the predefined threshold If the condition is met, the adversarial patch has been detected and located, and a corresponding mask M has been generated. Otherwise, the input sample is considered clean.

[0052] The size of the adversarial patch is usually unknown and may differ from the size of the input image patch for the model. To accurately locate these image patches, a dynamic window sizing strategy is employed during the sliding process.

[0053] Use a larger window (window height) = Window width = 4 × input image patch size), and perform a full slide for each window size. After completing one full slide for each window size, the maximum distance defined in the above formula is obtained. .if Not exceeding the maximum distance If an adversarial image patch is detected, the window size is gradually reduced and the process is repeated. Otherwise, it is assumed that an adversarial image patch has been detected, and the window sliding stops. If no adversarial image patches are detected after the last round (when the window size matches the input image patch size of the model), the sample is considered clean.

[0054] S3. Based on the patch mask, use an attack classifier to determine the nature of the attack according to the original image and obtain the attack classification result;

[0055] Optionally, based on the patch mask, an attack classifier is used to determine the nature of the attack according to the original image, and the attack classification result is obtained, including:

[0056] Based on the patch mask, adjacent image blocks of the adversarial patch region are determined from the original image;

[0057] Based on adjacent image patches, local replacement of the adversarial patch region is performed on the original image to obtain candidate reconstructed images;

[0058] Based on the candidate reconstructed images, the ViT model is used to predict the confidence level and obtain the confidence level of the reconstructed images.

[0059] Based on a preset similarity calculation function, the average pairwise similarity is calculated according to the confidence level of the reconstructed image.

[0060] Based on a preset similarity threshold, attack classification is determined according to the average pairwise similarity, and the attack classification result is obtained.

[0061] In one feasible implementation, the module in this invention aims to quickly identify the attack type of adversarial patches in an input image, namely non-catastrophic attacks and catastrophic attacks, through a simple and effective classification strategy.

[0062] Non-catastrophic attacks involve injecting adversarial image patches into semantically less important regions of the input image, regions that have little impact on the model's decisions. Conversely, catastrophic attacks involve adversarial image patches overlapping with important semantic regions (e.g., key object features like eyes and ears). These two types of attacks require different defensive approaches. For non-catastrophic attacks, adversarial signals can be removed to improve robustness while avoiding the loss of key features. Conversely, removing or covering adversarial regions in catastrophic attacks destroys information crucial for accurate prediction, inevitably impairing prediction accuracy.

[0063] This invention designs a simple and efficient method to distinguish between catastrophic and non-catastrophic attacks. Its key principle is as follows: If the attack is catastrophic, replacing the adversarial region with an adjacent benign region will cause a significant change in the model's prediction results, because the removal of key features reduces the model's confidence in the correct category and makes the model's prediction probability unknowable. Conversely, in non-catastrophic attacks, replacing the adversarial region with adjacent content can preserve key features, and the model can still consistently predict the original label.

[0064] Based on this idea, each adversarial region is replaced with four different adjacent image patches using a patch mask M, thereby reconstructing four candidate images. These reconstructed images are then input into the model to obtain the corresponding output. The stability of the prediction is measured by calculating the average pairwise similarity of the output, as shown in equation (3).

[0065] (3);

[0066] in, and These represent the model outputs corresponding to two different reconstructed images. If the average similarity... Below a predefined threshold If the attack is catastrophic, it will be classified as a catastrophic attack; otherwise, it will be classified as a non-catastrophic attack.

[0067] S4. Based on the attack classification results and patch mask, the image confidence is obtained by using the adaptive defense strategy module to perform purification and re-inference based on the original image.

[0068] Optionally, when the attack classification result is a non-catastrophic attack, based on the attack classification result and the patch mask, and according to the original image, the adaptive defense strategy module performs sanitization re-inference to obtain the image confidence, including:

[0069] Based on the patch mask, adversarial patch image patches are determined from the original image;

[0070] Using the adversarial patch image patch as the midpoint, search clockwise on the original image for the first valid neighbor image patch, which is then considered a benign neighbor image patch.

[0071] The original image is replaced and repaired based on benign neighboring image patches to obtain a repaired image;

[0072] Based on the restored image, the ViT model is used to predict the confidence level and obtain the first image confidence level.

[0073] In one feasible implementation, when a non-catastrophic attack occurs, the key semantic regions of the image remain intact, which is sufficient for the model to make correct predictions. Furthermore, neighboring regions typically exhibit strong semantic similarity, making them well-suited for reconstructing regions corrupted by adversarial patches. Based on this characteristic, adversarial image patches are removed using a mask M that locates the damaged region. Starting with the upper neighbor, neighboring image patches are searched clockwise until an existing neighbor is found that can be used to reconstruct the image (e.g., the upper left corner is selected from the right neighbor). The adversarial effects on the reconstructed image are eliminated while key features are preserved. The reconstructed image is then fed into the original ViT architecture to generate the final prediction result.

[0074] Optionally, when the attack classification result is a catastrophic attack, based on the attack classification result and the patch mask, and according to the original image, the adaptive defense strategy module performs sanitization re-inference to obtain the image confidence, including:

[0075] Based on the original image, attention maps are obtained by performing attention-per-attention head calculation using the ViT model.

[0076] Based on the patch mask, mean pooling of clean regions is performed according to the attention map to obtain the clean average attention value;

[0077] Based on the patch mask, mean pooling of the adversarial patch regions is performed according to the attention map to obtain the average attention contribution value of the patch.

[0078] Based on the clean average attention value and the patch average attention contribution value, an improved attention suppression algorithm is used to perform suppressive reweighting to obtain the suppressed attention weights.

[0079] Based on the suppressed attention weights, the ViT model is used to predict the confidence of the second image based on the original image.

[0080] In one feasible implementation, the present invention excludes adversarial image patches when calculating the average attention value, thereby more effectively suppressing adversarial effects. Furthermore, instead of using a global value applicable to all attention heads, the present invention calculates the average attention value independently for each attention head, matching the suppression mechanism to its different representation subspaces. The detailed calculation process is shown in equations (4) and (5):

[0081] (4);

[0082] (5);

[0083] Among them, using Let represent the attention weights of the m-th layer of ViT, and h represent the indices of different attention heads. i, j represent the indices of the attention weight values; This represents the average attention contribution of the j-th patch. This represents the suppression coefficient. The average attention weight of all clean patches in this module is expressed as... , where N represents the total number of patches. It is the side length of the attack patch. It is the location index of the adversarial patch detected by the attack detector.

[0084] Among them, the improved attention suppression algorithm is an attention suppression algorithm based on clean image patch perception; the improved attention suppression algorithm is used to suppress the attention dominance of adversarial patches in catastrophic attacks.

[0085] In one feasible implementation, this invention proposes an improved attention suppression algorithm (Token Attention Suspension, TAS) for catastrophic attack defense: attention suppression methods such as Restricting Token Attention (RTA) can effectively defend against catastrophic attacks. RTA works by using a uniform global attention value (the average of the attention weights of all image patches) to constrain the attention weights of each image patch, thereby preventing the model from being dominated by adversarial patches. However, RTA has two limitations. First, since adversarial patches often acquire abnormally high attention, including them in the averaging calculation distorts the global value, reducing the ability to suppress adversarial patches. Second, different attention heads capture different representation subspaces; therefore, applying a single uniform global attention mean to all attention heads is not optimal. To address these issues, this invention proposes an improved attention suppression algorithm.

[0086] This invention proposes an adaptive defense device against ViT adversarial patch attacks. By combining an attack detector module with the Sobel operator to calculate the gradient and distance scores of image patches, and using a dynamic window size strategy to scan the image, it can accurately identify and locate samples containing adversarial patches and clean samples. Defense operations are only performed on adversarial samples, which avoids the damage to the classification performance of clean samples caused by the existing "one-size-fits-all" defense strategy, and significantly reduces unnecessary computational overhead, thus solving the problem of excessively high computational cost of traditional defense methods.

[0087] The attack classifier module calculates the average pairwise similarity of the model output after replacing the adversarial region with adjacent image blocks, effectively distinguishing between catastrophic attacks (adversarial blocks covering key semantic regions) and non-catastrophic attacks (adversarial blocks located in low semantic importance regions), thus making up for the shortcomings of existing methods that ignore the differences in attack locations, resulting in insufficient defense targeting.

[0088] The adaptive defense strategy module designs adaptation schemes for two types of attacks. For non-catastrophic attacks, image reconstruction is used (using masks to remove adversarial blocks and replace them with adjacent benign blocks) to preserve key semantic features. For catastrophic attacks, an image patch-aware attention suppression algorithm (TAS) is proposed. By excluding adversarial image blocks, the average attention value of clean patches is calculated, and the mean is calculated independently for each attention head. This perfectly adapts to the characteristics of the ViT multi-head self-attention mechanism and significantly improves the robustness and accuracy of the model in adversarial scenarios.

[0089] The dynamic window strategy and TAS image patch awareness logic can cope with adversarial patch attacks of different sizes and positions, effectively avoiding the problem of attackers circumventing defenses by adjusting patch parameters, and enhancing the applicability of the method in complex real-world scenarios. This invention is an efficient, accurate, and robust adaptive defense method against ViT adversarial patch attacks.

[0090] Figure 2 This is a block diagram of an adaptive defense device against ViT anti-patch attacks provided by an embodiment of the present invention. This device is used in an adaptive defense method against ViT anti-patch attacks. (Refer to...) Figure 2 The device includes an image acquisition unit 210, an attack detection unit 220, an attack classification unit 230, and an image re-inference unit 240. Wherein:

[0091] Image acquisition unit 210 is used to acquire the original image containing the adversarial patch;

[0092] The attack detection unit 220 is used to identify adversarial patches based on the original image using the attack detection module, based on a dynamic sliding window strategy, and obtain the patch mask.

[0093] The attack classification unit 230 is used to determine the nature of the attack based on the patch mask and the original image using an attack classifier, and obtain the attack classification result.

[0094] Image re-inference unit 240 is used to perform sanitization re-inference based on the attack classification results and patch mask, according to the original image, and obtain image confidence by using the adaptive defense strategy module.

[0095] Optionally, the attack detection unit 220 is further used for:

[0096] Based on the sliding scan method with dynamically reduced window size, the Sobel operator is used to calculate the local neighborhood gradient difference on a scale-by-scale according to the original image to obtain the distance score;

[0097] Based on a preset distance score threshold, the distance scores are filtered to obtain distance scores that exceed the threshold.

[0098] Based on a preset multi-scale sliding window, the window region corresponding to the maximum value of the distance score exceeding the threshold is selected as the adversarial patch region according to the original image, and the position index and patch side length of the adversarial patch region are generated.

[0099] The patch mask is constructed based on the position index and the patch side length.

[0100] Optionally, the attack classification unit 230 is further used for:

[0101] Based on the patch mask, adjacent image blocks of the adversarial patch region are determined from the original image;

[0102] Based on adjacent image patches, local replacement of the adversarial patch region is performed on the original image to obtain candidate reconstructed images;

[0103] Based on the candidate reconstructed images, the ViT model is used to predict the confidence level and obtain the confidence level of the reconstructed images.

[0104] Based on a preset similarity calculation function, the average pairwise similarity is calculated according to the confidence level of the reconstructed image.

[0105] Based on a preset similarity threshold, attack classification is determined according to the average pairwise similarity, and the attack classification result is obtained.

[0106] Optionally, the attack classification unit 230 is further used for:

[0107] Based on the patch mask, adversarial patch image patches are determined from the original image;

[0108] Using the adversarial patch image patch as the midpoint, search clockwise on the original image for the first valid neighbor image patch, which is then considered a benign neighbor image patch.

[0109] The original image is replaced and repaired based on benign neighboring image patches to obtain a repaired image;

[0110] Based on the restored image, the ViT model is used to predict the confidence level and obtain the first image confidence level.

[0111] Optionally, the attack classification unit 230 is further used for:

[0112] Based on the original image, attention maps are obtained by performing attention-per-attention head calculation using the ViT model.

[0113] Based on the patch mask, mean pooling of clean regions is performed according to the attention map to obtain the clean average attention value;

[0114] Based on the patch mask, mean pooling of the adversarial patch regions is performed according to the attention map to obtain the average attention contribution value of the patch.

[0115] Based on the clean average attention value and the patch average attention contribution value, an improved attention suppression algorithm is used to perform suppressive reweighting to obtain the suppressed attention weights.

[0116] Based on the suppressed attention weights, the ViT model is used to predict the confidence of the second image based on the original image.

[0117] Among them, the improved attention suppression algorithm is an attention suppression algorithm based on clean image patch perception; the improved attention suppression algorithm is used to suppress the attention dominance of adversarial patches in catastrophic attacks.

[0118] This invention proposes an adaptive defense device against ViT adversarial patch attacks. By combining an attack detector module with the Sobel operator to calculate the gradient and distance scores of image patches, and using a dynamic window size strategy to scan the image, it can accurately identify and locate samples containing adversarial patches and clean samples. Defense operations are only performed on adversarial samples, which avoids the damage to the classification performance of clean samples caused by the existing "one-size-fits-all" defense strategy, and significantly reduces unnecessary computational overhead, thus solving the problem of excessively high computational cost of traditional defense methods.

[0119] The attack classifier module calculates the average pairwise similarity of the model output after replacing the adversarial region with adjacent image blocks, effectively distinguishing between catastrophic attacks (adversarial blocks covering key semantic regions) and non-catastrophic attacks (adversarial blocks located in low semantic importance regions), thus making up for the shortcomings of existing methods that ignore the differences in attack locations, resulting in insufficient defense targeting.

[0120] The adaptive defense strategy module designs adaptation schemes for two types of attacks. For non-catastrophic attacks, image reconstruction is used (using masks to remove adversarial blocks and replace them with adjacent benign blocks) to preserve key semantic features. For catastrophic attacks, an image patch-aware attention suppression algorithm (TAS) is proposed. By excluding adversarial image blocks, the average attention value of clean patches is calculated, and the mean is calculated independently for each attention head. This perfectly adapts to the characteristics of the ViT multi-head self-attention mechanism and significantly improves the robustness and accuracy of the model in adversarial scenarios.

[0121] The dynamic window strategy and TAS image patch awareness logic can cope with adversarial patch attacks of different sizes and positions, effectively avoiding the problem of attackers circumventing defenses by adjusting patch parameters, and enhancing the applicability of the method in complex real-world scenarios. This invention is an efficient, accurate, and robust adaptive defense method against ViT adversarial patch attacks.

[0122] Figure 3 This is a schematic diagram of the structure of an adaptive defense device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the adaptive defense device may include the above-mentioned Figure 2The diagram illustrates an adaptive defense device against ViT anti-patch attacks. Optionally, the adaptive defense device 310 may include a first processor 2001.

[0123] Optionally, the adaptive defense device 310 may also include a memory 2002 and a transceiver 2003.

[0124] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0125] The following is combined Figure 3 A detailed introduction to each component of the adaptive defense device 310:

[0126] The first processor 2001 is the control center of the adaptive defense device 310. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0127] Optionally, the first processor 2001 can perform various functions of the adaptive defense device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0128] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.

[0129] In a specific implementation, as one example, the adaptive defense device 310 may also include multiple processors, for example... Figure 3 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0130] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0131] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the adaptive defense device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0132] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0133] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0134] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the adaptive defense device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0135] It should be noted that, Figure 3 The structure of the adaptive defense device 310 shown does not constitute a limitation on the router. Actual adaptive defense devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0136] Furthermore, the technical effectiveness of the adaptive defense device 310 can be referenced from the technical effectiveness of the adaptive defense method against ViT anti-patch attacks described in the above method embodiments, and will not be repeated here.

[0137] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0138] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0139] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0140] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0141] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0142] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0144] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0145] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0148] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An adaptive defense method against ViT anti-patch attacks, characterized in that, The method includes: Obtain the original image containing the adversarial patch; Based on a dynamic sliding window strategy, an attack detection module is used to identify adversarial patches and obtain patch masks from the original image. Based on the patch mask, an attack classifier is used to determine the nature of the attack based on the original image, and the attack classification result is obtained. Based on the attack classification results and patch mask, the image confidence is obtained by using the adaptive defense strategy module to perform sanitization and re-inference based on the original image.

2. The adaptive defense method against ViT anti-patch attacks according to claim 1, characterized in that, The dynamic sliding window strategy, based on the original image, uses an attack detection module to identify adversarial patches and obtain patch masks, including: Based on the sliding scan method with dynamically reduced window size, the Sobel operator is used to calculate the local neighborhood gradient difference on a scale-by-scale according to the original image to obtain the distance score; Based on a preset distance score threshold, the distance scores are filtered to obtain distance scores that exceed the threshold. Based on a preset multi-scale sliding window, the window region corresponding to the maximum value of the distance score exceeding the threshold is selected as the adversarial patch region according to the original image, and the position index and patch side length of the adversarial patch region are generated. The patch mask is constructed based on the position index and the patch side length.

3. The adaptive defense method against ViT anti-patch attacks according to claim 1, characterized in that, The process of determining the nature of an attack based on a patch mask and the original image using an attack classifier to obtain attack classification results includes: Based on the patch mask, adjacent image blocks of the adversarial patch region are determined from the original image; Based on adjacent image patches, local replacement of the adversarial patch region is performed on the original image to obtain candidate reconstructed images; Based on the candidate reconstructed images, the ViT model is used to predict the confidence level and obtain the confidence level of the reconstructed images. Based on a preset similarity calculation function, the average pairwise similarity is calculated according to the confidence level of the reconstructed image. Based on a preset similarity threshold, attack classification is determined according to the average pairwise similarity, and the attack classification result is obtained.

4. The adaptive defense method against ViT anti-patch attacks according to claim 1, characterized in that, When the attack classification result is a non-catastrophic attack, the process of performing cleanup and re-inference based on the attack classification result and patch mask, according to the original image, using the adaptive defense strategy module to obtain image confidence, includes: Based on the patch mask, adversarial patch image patches are determined from the original image; Using the adversarial patch image patch as the midpoint, search clockwise on the original image for the first valid neighbor image patch, which is then considered a benign neighbor image patch. The original image is replaced and repaired based on benign neighboring image patches to obtain a repaired image; Based on the restored image, the ViT model is used to predict the confidence level and obtain the first image confidence level.

5. The adaptive defense method against ViT anti-patch attacks according to claim 1, characterized in that, When the attack classification result is a catastrophic attack, the process of performing cleanup and re-inference based on the attack classification result and patch mask, according to the original image, using the adaptive defense strategy module to obtain image confidence, includes: Based on the original image, attention maps are obtained by performing attention-per-attention head calculation using the ViT model. Based on the patch mask, mean pooling of clean regions is performed according to the attention map to obtain the clean average attention value; Based on the patch mask, mean pooling of the adversarial patch regions is performed according to the attention map to obtain the average attention contribution value of the patch. Based on the clean average attention value and the patch average attention contribution value, an improved attention suppression algorithm is used to perform suppressive reweighting to obtain the suppressed attention weights. Based on the suppressed attention weights, the ViT model is used to predict the confidence of the second image based on the original image.

6. The adaptive defense method against ViT anti-patch attacks according to claim 5, characterized in that, The improved attention suppression algorithm is an attention suppression algorithm based on clean image patch perception; The improved attention suppression algorithm is used to suppress the attention-dominant role of adversarial patches in catastrophic attacks.

7. An adaptive defense device against ViT anti-patch attacks, wherein the adaptive defense device against ViT anti-patch attacks is used to implement the adaptive defense method against ViT anti-patch attacks as described in any one of claims 1-6, characterized in that, The device includes: Image acquisition unit, used to acquire the original image containing the adversarial patch; The attack detection unit is used to identify adversarial patches based on the original image using the attack detection module, based on a dynamic sliding window strategy, and to obtain the patch mask. The attack classification unit is used to determine the nature of the attack based on the patch mask and the original image, and obtain the attack classification result. The image re-inference unit is used to perform sanitization re-inference based on the attack classification results and patch mask, according to the original image, and obtain the image confidence score.

8. The adaptive defense device against ViT patch attacks according to claim 7, characterized in that, The attack detection unit is further used for: Based on the sliding scan method with dynamically reduced window size, the Sobel operator is used to calculate the local neighborhood gradient difference on a scale-by-scale according to the original image to obtain the distance score; Based on a preset distance score threshold, the distance scores are filtered to obtain distance scores that exceed the threshold. Based on a preset multi-scale sliding window, the window region corresponding to the maximum value of the distance score exceeding the threshold is selected as the adversarial patch region according to the original image, and the position index and patch side length of the adversarial patch region are generated. The patch mask is constructed based on the position index and the patch side length.

9. An adaptive defense device, characterized in that, The adaptive defense device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images