An adversarial patch removal method based on an adversarial patch location model
By designing an adversarial patch localization model, which combines image reconstruction and multi-scale feature fusion, the semantic information loss caused by adversarial patch removal in existing methods is solved, achieving more efficient adversarial patch defense and improved robustness of downstream models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2024-05-17
- Publication Date
- 2026-07-17
AI Technical Summary
Existing methods employ simple removal strategies after adversarial patch detection, resulting in downstream object detection models lacking image semantic information and affecting detection performance. Furthermore, adversarial training methods suffer from high time and computational costs, requiring the retraining of robust models.
An adversarial patch localization model was designed, which includes a training sample generation module, an edge extraction module, a feature extraction module, a multi-scale feature fusion module, and an adversarial patch removal module. Semantic information is restored through an image reconstruction model, and multi-scale feature fusion and mask completion algorithms are used to improve localization accuracy and recall.
It effectively defends against adversarial patch attacks, reduces the loss of semantic information in images, and improves the robustness of downstream neural networks and the accuracy of target detection.
Smart Images
Figure CN118506074B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to an adversarial patch removal method based on an adversarial patch localization model. Background Technology
[0002] When facing security threats posed by adversarial patch attacks in object detection, adversarial training can often provide a simple and efficient defense. However, adversarial training methods have inherent drawbacks, primarily including high time and computational costs, impact on the prediction accuracy of clean samples, and the need to retrain robust models for each specific dataset and model. In certain scenarios, we need to avoid these drawbacks. Researchers have attempted to defend against adversarial patch attacks on object detection models based on sample cleanup, providing more diverse adversarial defense strategies. Among existing research, SAC and PatchZero have both successfully detected and removed adversarial patches in images using deep neural networks, improving the detection accuracy of downstream object detection models.
[0003] These methods study relatively simple adversarial patches, using random noise initialization and limiting their shapes to rectangles or other regular shapes. Furthermore, existing methods employ simplistic removal strategies after adversarial patch detection: SAC uses 0 pixels to fill the adversarial patch region, and PatchZero uses the average pixels within the region. In this case, downstream object detection models lose significant semantic information, impacting final detection performance. Image reconstruction is a mature field; introducing image reconstruction models into the post-processing stage of adversarial patch removal to complete and reconstruct the adversarial patch region can effectively recover semantic losses in the image. However, current methods have not yet combined image reconstruction models with adversarial patch removal. Summary of the Invention
[0004] In view of the above problems, this invention provides an adversarial patch removal method based on an adversarial patch localization model. This invention designs an adversarial patch removal model, comprising a training sample generation module, an edge extraction module, a feature extraction module, a multi-scale feature fusion module, and an adversarial patch removal module. This model is used to implement adversarial patch defense, more effectively defending against adversarial patch attacks and improving the robustness of downstream neural networks. It acquires edge information features of the image under test to improve localization accuracy; assigns different weights to each channel of the feature map, fusing feature information from different resolutions; completes the adversarial patch localization mask to improve the recall rate of the prediction mask; detects and locates patches based on the texture feature differences between the patch region and the original image region, avoiding the model focusing on adversarial patches with a specific texture; and reduces the loss of image semantic information caused by adversarial patch removal.
[0005] This invention provides a method for removing adversarial patches based on an adversarial patch location model, comprising:
[0006] S1. Construct an initial adversarial patch localization model; establish a test image dataset; input the test image dataset into the initial patch localization model for training to obtain the adversarial patch localization model;
[0007] S2. Design a mask completion module and an image reconstruction module, and embed the mask completion module and the image reconstruction module into the adversarial patch localization model to obtain an updated adversarial patch localization model;
[0008] S3. Input the image to be tested into the adversarial patch localization model to obtain the initial prediction mask b'. k ;
[0009] Preferably, the mask completion algorithm takes a certain pixel of the image as the center and determines whether the sum of the predicted values of the mask in the window around the pixel is greater than a certain threshold. If it is greater, then the point is considered to be a point in the adversarial patch area.
[0010] S4. The initial prediction mask b' is... k Input the updated adversarial patch location model and remove the adversarial patch.
[0011] Preferably, the specific steps for the test image dataset in step S1 include:
[0012] A target detection image dataset is established based on multiple target detection image data obtained from downstream target detection tasks.
[0013] An adversarial patch is added to each target detection image in the target detection image dataset to obtain a test image dataset;
[0014] Furthermore, each target detection image data corresponds to annotation information;
[0015] The annotation information includes the bounding box coordinates and classification labels of objects within the target detection image;
[0016] Furthermore, each test image corresponds to annotation information; the annotation information corresponding to each test image includes the mask of the adversarial patch, the coordinates of the rectangular box of the object in the image, and the classification label;
[0017] Preferably, the specific steps for constructing the patch location model in step S1 include:
[0018] S11. Establish the initial patch location model framework;
[0019] Let k = 1, where k = 1 represents the first object detection image;
[0020] S12. Design a feature extraction backbone network A in the initial patch localization model framework. k The patch removes model one;
[0021] S13. Construct a multi-scale fusion feature module Z in patch localization model one. k The second patch localization model is obtained; the multi-scale fusion feature module E k The input is the view feature r of the corresponding RGB image. k and edge view feature g′ k The output is the multi-scale fusion feature E′. k ;
[0022] S14. Design a pixel classification convolutional layer D in the patch positioning model two. k An initial patch localization model is obtained; the input of the initial patch localization model is the multi-scale fusion feature E. k The output is the prediction mask b. k and losses;
[0023] S15. Input multiple test images from the test image dataset into the initial patch localization model sequentially for training to obtain the patch localization model;
[0024] Preferably, the feature extraction backbone network A in step S12 k The input is the RGB image R of the target detection image k. k and image g with edge features k The output is the corresponding RGB view feature R′. k and edge view feature g′ k ;
[0025] Furthermore, step S12 obtains the edge view feature g′ k The specific steps include:
[0026] Select image x in the test image dataset k and random image x' k ; the image x k Characterized as a clean image c i According to image x k The clean image is obtained by annotating the object's border (b). i The corresponding patch mask m k ;
[0027] Based on the random image x' k The mask for the clean image c described in step S3 k The corresponding patch mask is used for data augmentation to obtain the augmented training samples. k and the corresponding mask;
[0028] The training samples ^x were augmented using the Laplacian operator. k High-pass filtering is performed to obtain an image g with edge features. k ;
[0029] Preferably, S12 further includes obtaining an updated feature extraction backbone network, specifically including the following steps:
[0030] Let k = 1, where k = 1 represents the first object detection image;
[0031] Get image x k RGB image;
[0032] The image x k RGB images and images with edge features g k Input feature extraction backbone network A k Obtain the view features r of the RGB image k and edge view feature g′ k And update the feature extraction backbone network;
[0033] Compare the size of k with K, where K represents the total number of target detection images. If k < K, increment the value of k by 1 and return to step S3. If k = K, end the training and obtain the updated feature extraction backbone network A.
[0034] Preferably, step S13 further includes specific steps for obtaining the updated multi-scale fusion feature module Z, including:
[0035] The command is k=1, where k=1 represents the first target detection image;
[0036] The view features r of the RGB image k and edge view feature g′ k The features are stitched together to obtain the stitched multi-scale feature map e. k ;
[0037] The stitched multi-scale feature map e k Input multi-scale fusion feature model Z k Based on multi-head depthwise convolution transpose attention, the concatenated multi-scale feature map e k Multi-scale cross-channel feature fusion is performed to obtain multi-scale fused feature E. k And update the multi-scale fusion feature module;
[0038] Compare the size of k with K, where K represents the total number of target detection images. If k < K, increment the value k by 1 and return to step S3. If k = K, end the training and obtain the updated multi-scale fusion feature module Z.
[0039] Furthermore, step S13 also includes obtaining multi-head depthwise convolution transpose attention, specifically including the following steps:
[0040] Let the input tensor Standardize the tensor X to obtain the standardized tensor.
[0041] Extract normalized tensors using 1×1 ordinary convolution. Pixel-level cross-channel context information is used to obtain an updated normalized tensor;
[0042] The spatial context information at the channel level of the updated normalized tensor is encoded using 3×3 depth-direction convolution to obtain the initial query Q matrix, the initial key K matrix, and the initial numerical V matrix.
[0043] The initial query Q matrix, initial key K matrix, and initial numerical V matrix are dimensionality transformed to obtain multi-head depthwise convolutional transpose attention.
[0044] Preferably, step S14 further includes obtaining the updated pixel classification convolutional layer D, specifically including:
[0045] S141. Let k = 1, where k = 1 is the first target detection image;
[0046] S142. The multi-scale fusion feature E obtained in step S7 is... k D in the input pixel classification convolutional layer k The prediction mask b is obtained after processing based on the activation function. k Loss L total,k And update the pixel classification convolutional layer;
[0047] S143. Compare the size of k with K, where K represents the total number of target detection images. If k < K, increment the value k by 1 and return to step S151. If k = K, end the training and obtain the updated pixel classification convolutional layer D.
[0048] In step S142, the predicted mask b k The loss expression is:
[0049] L total,k =L BCE,k +λL DICE,k ;
[0050] Among them, L total,k Let be the loss for the prediction mask of image k, where λ is a hyperparameter used to adjust the weights of the Dice loss; L DICE,k Dice loss for the prediction mask of image k; L BCE,k Let be the cross-entropy loss of the prediction mask for image k.
[0051] Furthermore, the expression for the cross-entropy loss is:
[0052]
[0053] Among them, y k It is the patch mask of image k, x k The prediction mask for image k is the patch mask y. k In this task, the probability is calculated by averaging the cross-entropy loss for each of the n pixels of the image; further, the Dice loss function L... DICE，k The expression is:
[0054]
[0055] Where, x k，j Let y be the prediction mask for pixel j in the k-th image. k，j Let be the patch mask for pixel j in the k-th image; j = 1, 2, 3...J, where J is the total number of pixels in the k-th image.
[0056] Specifically, step S4, which involves removing the anti-patch, includes the following steps:
[0057] The initial prediction mask b′ k Input the updated adversarial patch localization model, and obtain the completed prediction mask b based on the mask completion module. k ;
[0058] The completion prediction mask b k The corresponding region is represented as the patch removal region;
[0059] The patch removal area is input into the image reconstruction module to obtain the original image area of the image to be tested; the texture of the patch removal area is removed based on the original image area of the image to be tested to achieve anti-patch removal.
[0060] Furthermore, the obtained complete prediction mask b k The specific steps include:
[0061] a. Obtain test image A k The original mask M o ; Preset window width w and height h, preset adversarial patch mask threshold z;
[0062] b. Convert the original mask M o Input to the adversarial patch mask completion module, initialize with the original mask M o An array M of the same size containing zero values c ;
[0063] c. The original mask M oIt includes multiple pixels; each h×w window includes multiple pixel values;
[0064] d. Set a window of size h×w at the center of pixel p, and calculate the sum of multiple pixel values within the window;
[0065] e. Compare the sum of multiple pixel values within the window with the adversarial patch mask threshold z. If the sum of multiple pixel values within the window is greater than the adversarial patch mask threshold z, then the array M corresponding to pixel p is... c The value is set to 255; if it is less than the adversarial patch mask threshold z, the value of pixel p is not modified.
[0066] f. Repeat step de until the original mask M has been traversed. o From multiple pixels in the image, we obtain the complete prediction mask b. k .
[0067] This invention also provides an adversarial patch location model, comprising:
[0068] The module includes a training sample generation module, an edge extraction module, a feature extraction module, a multi-scale feature fusion module, and an adversarial patch removal module.
[0069] Furthermore, the training sample generation module uses an image tampering "copy-move" method to construct a pseudo-adversarial patch on a clean image to obtain a training image, and uses a basic geometric shape stacking method to generate a mask of random shape to perform data augmentation on the clean image;
[0070] The edge extraction module uses the Laplacian operator to construct a high-pass filter, performs high-pass filtering on the input image, and extracts information of the image with edge features.
[0071] The feature extraction module uses the data output by the training sample generation module to train the backbone network and extract feature information from RGB images and images with edge features;
[0072] The multi-scale feature fusion module uses a multi-head deep convolution transpose attention module to fuse the feature information output by the feature extraction module at the skip connections of encoders and decoders at each level of the backbone network. It applies a cross-channel attention mechanism on feature maps at different scales and finally provides an adversarial patch localization mask.
[0073] The adversarial patch removal module is used to complete the adversarial patch localization mask output by the multi-scale feature fusion module, while reducing the loss of image semantic information caused by adversarial patch removal.
[0074] Furthermore, the adversarial patch removal module includes a mask completion module and an image reconstruction module;
[0075] The mask completion module uses classification information around the pixel for assistance to count whether the number of pixels classified as adversarial patch pixels within the window centered on the target pixel is greater than a preset threshold, and decides whether to correct the classification result of the target pixel.
[0076] The image reconstruction module uses an image reconstruction neural network to restore semantic information in the image after removing the patch, fill in the texture details in the patch removal area, and improve the detection accuracy of the downstream object detection network.
[0077] Furthermore, the feature extraction module uses a fully convolutional neural network, and the multi-scale feature fusion module uses a cross-channel attention mechanism.
[0078] Furthermore, the feature extraction module includes four levels of downsampling and upsampling, and the upsampling process of the feature extraction module is based on bilinear interpolation.
[0079] Furthermore, the image size processed by the feature extraction module and the multi-scale feature fusion module is 512x512 pixels.
[0080] The technical solution of this invention includes a training sample generation module to avoid the model focusing on adversarial patches with a specific texture, and to detect and locate patches by the difference in texture features between the patch region and the original image region; an edge extraction module to improve localization accuracy by utilizing edge information features; a feature extraction module to extract features from the RGB view and edge view images using an encoder-decoder network; a multi-scale feature fusion module to assign different weights to each channel of the feature map according to the correlation of features, and to fully fuse feature information of different resolutions using multi-scale fusion; and an adversarial patch removal module to complete the adversarial patch localization mask output by the multi-scale feature fusion module, while reducing the loss of image semantic information caused by adversarial patch removal. This invention designs an end-to-end adversarial patch removal method for adversarial patch defense tasks, which can more effectively defend against adversarial patch attacks and improve the robustness of downstream neural networks.
[0081] Compared with the prior art, the present invention has at least the following beneficial effects:
[0082] (1) The masking algorithm of the present invention uses information around the pixel for assistance. When a pixel that is not judged as an adversarial patch is surrounded by a large number of pixels that are judged as adversarial patches, the algorithm will correct this pixel to the adversarial patch class, thereby improving the recall rate of adversarial patch mask prediction.
[0083] (2) The image reconstruction network of the present invention effectively recovers the semantic features of the adversarial patch removal region and improves the prediction accuracy of the downstream target detection model;
[0084] (3) Based on the adversarial patch location network, this invention achieves robust adversarial patch defense through mask completion and image reconstruction. Attached Figure Description
[0085] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.
[0086] Figure 1 This is a schematic diagram of the anti-patch mask completion algorithm in an embodiment of the present invention;
[0087] Figure 2 This is a schematic diagram illustrating the effect of the mask completion algorithm in an embodiment of the present invention;
[0088] Figure 3 This is a schematic diagram of the image reconstruction network in an embodiment of the present invention;
[0089] Figure 4 This is a schematic diagram illustrating the effect of the image reconstruction network in an embodiment of the present invention. Detailed Implementation
[0090] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.
[0091] A specific embodiment of the present invention, such as Figure 1-4 A method for removing adversarial patches based on an adversarial patch location model is disclosed.
[0092] To illustrate the effectiveness of the method proposed in this invention, the following detailed description of the above technical solution is provided through a specific embodiment. The specific implementation steps are as follows:
[0093] This invention provides a method for removing adversarial patches based on an adversarial patch location model, comprising:
[0094] S1. Construct an initial adversarial patch localization model; establish a test image dataset; input the test image dataset into the initial patch localization model for training to obtain the adversarial patch localization model;
[0095] S2. Design a mask completion module and an image reconstruction module, and embed the mask completion module and the image reconstruction module into the adversarial patch localization model to obtain an updated adversarial patch localization model;
[0096] S3. Input the image to be tested into the adversarial patch localization model to obtain the initial prediction mask b'. k ;
[0097] Preferably, the mask completion algorithm takes a certain pixel of the image as the center and determines whether the sum of the predicted values of the mask in the window around the pixel is greater than a certain threshold. If it is greater, then the point is considered to be a point in the adversarial patch area.
[0098] S4. The initial prediction mask b' is... k Input the updated adversarial patch location model and remove the adversarial patch.
[0099] Preferably, the specific steps for the test image dataset in step S1 include:
[0100] A target detection image dataset is established based on multiple target detection image data obtained from downstream target detection tasks.
[0101] An adversarial patch is added to each target detection image in the target detection image dataset to obtain a test image dataset;
[0102] Furthermore, each target detection image data corresponds to annotation information;
[0103] The annotation information includes the bounding box coordinates and classification labels of objects within the target detection image;
[0104] Furthermore, each test image corresponds to annotation information; the annotation information corresponding to each test image includes the mask of the adversarial patch, the coordinates of the rectangular box of the object in the image, and the classification label;
[0105] Preferably, the specific steps for constructing the patch location model in step S1 include:
[0106] S11. Establish the initial patch location model framework;
[0107] Let k = 1, where k = 1 represents the first object detection image;
[0108] S12. Design a feature extraction backbone network A in the initial patch localization model framework. k The patch removes model one;
[0109] S13. Construct a multi-scale fusion feature module Z in patch localization model one. k The second patch localization model is obtained; the multi-scale fusion feature module E k The input is the view feature r of the corresponding RGB image. k and edge view feature g′ k The output is the multi-scale fusion feature E′. k ;
[0110] S14. Design a pixel classification convolutional layer D in the patch positioning model two. k An initial patch localization model is obtained; the input of the initial patch localization model is the multi-scale fusion feature E. k The output is the prediction mask b. k and losses;
[0111] S15. Input multiple test images from the test image dataset into the initial patch localization model sequentially for training to obtain the patch localization model;
[0112] Preferably, the feature extraction backbone network A in step S12 k The input is the RGB image R of the target detection image k. k and image g with edge features k The output is the corresponding RGB view feature R′. k and edge view feature g′ k ;
[0113] Furthermore, step S12 obtains the edge view feature g′ k The specific steps include:
[0114] Select image x in the test image dataset k and random image x' k ; the image x k Characterized as a clean image c i According to image x k The clean image is obtained by annotating the object's border (b). i The corresponding patch mask m k ;
[0115] Based on the random image x' k The mask for the clean image c described in step S3 k The corresponding patch mask is used for data augmentation to obtain the augmented training samples. k and the corresponding mask;
[0116] The training samples ^x were augmented using the Laplacian operator. k High-pass filtering is performed to obtain an image g with edge features. k ;
[0117] Preferably, S12 further includes obtaining an updated feature extraction backbone network, specifically including the following steps:
[0118] Let k = 1, where k = 1 represents the first object detection image;
[0119] Get image x kRGB image;
[0120] The image x k RGB images and images with edge features g k Input feature extraction backbone network A k Obtain the view features r of the RGB image k and edge view feature g′ k And update the feature extraction backbone network;
[0121] Compare the size of k with K, where K represents the total number of target detection images. If k < K, increment the value of k by 1 and return to step S3. If k = K, end the training and obtain the updated feature extraction backbone network A.
[0122] Preferably, step S13 further includes specific steps for obtaining the updated multi-scale fusion feature module Z, including:
[0123] The command is k=1, where k=1 represents the first target detection image;
[0124] The view features r of the RGB image k and edge view feature g′ k The features are stitched together to obtain the stitched multi-scale feature map e. k ;
[0125] The stitched multi-scale feature map e k Input multi-scale fusion feature model Z k Based on multi-head depthwise convolution transpose attention, the concatenated multi-scale feature map e k Multi-scale cross-channel feature fusion is performed to obtain multi-scale fused feature E. k And update the multi-scale fusion feature module;
[0126] Compare the size of k with K, where K represents the total number of target detection images. If k < K, increment the value k by 1 and return to step S3. If k = K, end the training and obtain the updated multi-scale fusion feature module Z.
[0127] Furthermore, step S13 also includes obtaining multi-head depthwise convolution transpose attention, specifically including the following steps:
[0128] Let the input tensor Standardize the tensor X to obtain the standardized tensor.
[0129] Extract normalized tensors using 1×1 ordinary convolution. Pixel-level cross-channel context information is used to obtain an updated normalized tensor;
[0130] The spatial context information at the channel level of the updated normalized tensor is encoded using 3×3 depth-direction convolution to obtain the initial query Q matrix, the initial key K matrix, and the initial numerical V matrix.
[0131] The initial query Q matrix, initial key K matrix, and initial numerical V matrix are dimensionality transformed to obtain multi-head depthwise convolutional transpose attention.
[0132] Preferably, step S14 further includes obtaining the updated pixel classification convolutional layer D, specifically including:
[0133] S141. Let k = 1, where k = 1 is the first target detection image;
[0134] S142. The multi-scale fusion feature E obtained in step S7 is... k D in the input pixel classification convolutional layer k The prediction mask b is obtained after processing based on the activation function. k Loss L total,k And update the pixel classification convolutional layer;
[0135] S143. Compare the size of k with K, where K represents the total number of target detection images. If k < K, increment the value k by 1 and return to step S151. If k = K, end the training and obtain the updated pixel classification convolutional layer D.
[0136] In step S142, the predicted mask b k The loss expression is:
[0137] L total,k =L BCE,k +λL DICE,k ;
[0138] Among them, L total,k Let be the loss for the prediction mask of image k, where λ is a hyperparameter used to adjust the weights of the Dice loss; L DICE,k Dice loss for the prediction mask of image k; L BCE,k Let be the cross-entropy loss of the prediction mask for image k.
[0139] Furthermore, the expression for the cross-entropy loss is:
[0140]
[0141] Among them, y k It is the patch mask of image k, x k The prediction mask for image k is the patch mask y. k The probability is calculated by averaging the cross-entropy loss of the classification results of n pixels of the image in this task.
[0142] Furthermore, the Dice loss function L DICE，k The expression is:
[0143]
[0144] Where, x k，j Let y be the prediction mask for pixel j in the k-th image. k，j Let be the patch mask for pixel j in the k-th image; j = 1, 2, 3...J, where J is the total number of pixels in the k-th image.
[0145] Specifically, step S4, which involves removing the anti-patch, includes the following steps:
[0146] The initial prediction mask b′ k Input the updated adversarial patch localization model, and obtain the completed prediction mask b based on the mask completion module. k ;
[0147] The completion prediction mask b k The corresponding region is represented as the patch removal region;
[0148] The patch removal area is input into the image reconstruction module to obtain the original image area of the image to be tested; the texture of the patch removal area is removed based on the original image area of the image to be tested to achieve anti-patch removal.
[0149] Furthermore, the obtained complete prediction mask b k The specific steps include:
[0150] a. Obtain test image A k The original mask M o ; Preset window width w and height h, preset adversarial patch mask threshold z;
[0151] b. Convert the original mask M o Input to the adversarial patch mask completion module, initialize with the original mask M o An array M of the same size containing zero values c ;
[0152] c. The original mask M o It includes multiple pixels; each h×w window includes multiple pixel values;
[0153] d. Set a window of size h×w at the center of pixel p, and calculate the sum of multiple pixel values within the window;
[0154] e. Compare the sum of multiple pixel values within the window with the adversarial patch mask threshold z. If the sum of multiple pixel values within the window is greater than the adversarial patch mask threshold z, then the array M corresponding to pixel p is... cThe value is set to 255; if it is less than the adversarial patch mask threshold z, the value of pixel p is not modified.
[0155] f. Repeat step de until the original mask M has been traversed. o From multiple pixels in the image, we obtain the complete prediction mask b. k .
[0156] This invention also provides an adversarial patch location model, comprising:
[0157] The module includes a training sample generation module, an edge extraction module, a feature extraction module, a multi-scale feature fusion module, and an adversarial patch removal module.
[0158] Furthermore, the training sample generation module uses an image tampering "copy-move" method to construct a pseudo-adversarial patch on a clean image to obtain a training image, and uses a basic geometric shape stacking method to generate a mask of random shape to perform data augmentation on the clean image;
[0159] The edge extraction module uses the Laplacian operator to construct a high-pass filter, performs high-pass filtering on the input image, and extracts information of the image with edge features.
[0160] The feature extraction module uses the data output by the training sample generation module to train the backbone network and extract feature information from RGB images and images with edge features;
[0161] The multi-scale feature fusion module uses a multi-head deep convolution transpose attention module to fuse the feature information output by the feature extraction module at the skip connections of encoders and decoders at each level of the backbone network. It applies a cross-channel attention mechanism on feature maps at different scales and finally provides an adversarial patch localization mask.
[0162] The adversarial patch removal module is used to complete the adversarial patch localization mask output by the multi-scale feature fusion module, while reducing the loss of image semantic information caused by adversarial patch removal.
[0163] Furthermore, the adversarial patch removal module includes a mask completion module and an image reconstruction module;
[0164] The mask completion module uses classification information around the pixel for assistance to count whether the number of pixels classified as adversarial patch pixels within the window centered on the target pixel is greater than a preset threshold, and decides whether to correct the classification result of the target pixel.
[0165] The image reconstruction module uses an image reconstruction neural network to restore semantic information in the image after removing the patch, fill in the texture details in the patch removal area, and improve the detection accuracy of the downstream object detection network.
[0166] Furthermore, the feature extraction module uses a fully convolutional neural network, and the multi-scale feature fusion module uses a cross-channel attention mechanism.
[0167] Furthermore, the feature extraction module includes four levels of downsampling and upsampling, and the upsampling process of the feature extraction module is based on bilinear interpolation.
[0168] Furthermore, the image size processed by the feature extraction module and the multi-scale feature fusion module is 512x512 pixels.
[0169] The technical solution of this invention includes a training sample generation module to avoid the model focusing on adversarial patches with a specific texture, and to detect and locate patches by the difference in texture features between the patch region and the original image region; an edge extraction module to improve localization accuracy by utilizing edge information features; a feature extraction module to extract features from the RGB view and edge view images using an encoder-decoder network; a multi-scale feature fusion module to assign different weights to each channel of the feature map according to the correlation of features, and to fully fuse feature information of different resolutions using multi-scale fusion; and an adversarial patch removal module to complete the adversarial patch localization mask output by the multi-scale feature fusion module, while reducing the loss of image semantic information caused by adversarial patch removal. This invention designs an end-to-end adversarial patch removal method for adversarial patch defense tasks, which can more effectively defend against adversarial patch attacks and improve the robustness of downstream neural networks.
[0170] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for removing adversarial patches based on an adversarial patch location model, characterized in that, include S1. Construct an adversarial patch localization model; the adversarial patch localization model includes a training sample generation module, an edge extraction module, a feature extraction module, a multi-scale feature fusion module, and an adversarial patch removal module; S2. Design a mask completion module and an image reconstruction module, and embed the mask completion module and the image reconstruction module into the adversarial patch localization model to obtain an updated adversarial patch localization model; S3. Input the image to be tested into the adversarial patch localization model to obtain the initial prediction mask; S4. Input the initial prediction mask into the updated adversarial patch localization model to remove the adversarial patch. The specific steps include: The initial prediction mask is input into the updated adversarial patch localization model, and the completed prediction mask is obtained based on the mask completion module. The specific steps include: a. Obtain the test image A k raw mask ; Preset window width and height Preset anti-patch mask threshold z; b. The original mask Input to the adversarial patch mask completion module, initialize with the original mask. Arrays of zero values of the same size ; c. The original mask Includes multiple pixels; the The window contains multiple pixel values; d. Set the center size of pixel p to [value]. The window is used to calculate the sum of multiple pixel values within the window; e. Compare the sum of multiple pixel values within the window with the adversarial patch mask threshold z. If the sum of multiple pixel values within the window is greater than the adversarial patch mask threshold z, then the array corresponding to pixel p is... The value is set to 255; if it is less than the adversarial patch mask threshold z, the value of pixel p is not modified. f. Repeat step de until the original mask has been traversed. Multiple pixels in the image are used to obtain the complete prediction mask; The region corresponding to the completion prediction mask is represented as the patch removal region; The patch removal area is input into the image reconstruction module to obtain the original image area of the image to be tested; the texture of the patch removal area is removed based on the original image area of the image to be tested to achieve anti-patch removal. The training sample generation module uses the image tampering "copy-move" method to construct a pseudo-adversarial patch on a clean image to obtain a training image, and uses the basic geometric shape stacking method to generate a mask of random shape to perform data augmentation on the clean image; The edge extraction module uses the Laplacian operator to construct a high-pass filter, performs high-pass filtering on the input image, and extracts information of the image with edge features; The feature extraction module uses the data output by the training sample generation module to train the backbone network and extract feature information from RGB images and images with edge features. The multi-scale feature fusion module uses a multi-head deep convolutional transposed attention module to fuse the feature information output by the feature extraction module at the skip connections of the encoder-decoder at each level of the backbone network. It applies a cross-channel attention mechanism on feature maps at different scales and finally provides an adversarial patch localization mask.
2. The method for removing anti-patch according to claim 1, characterized in that, The mask completion module is used to obtain the target pixels of the image under test, count whether the number of pixels classified as adversarial patch pixels in the window centered on the target pixel is greater than a preset threshold, and decide whether to correct the classification result of the target pixel.
3. The method for removing anti-patch according to claim 2, characterized in that, The image reconstruction module uses an image reconstruction neural network to restore semantic information in the image after the patch is removed, and fills in the texture details of the patch removal area.