Adversarial patch localization method based on self-supervised learning and multi-view feature fusion

Abstract

The present application relates to an adversarial patch positioning method based on self-supervised learning and multi-view feature fusion, and belongs to the technical field of image processing. The present application designs an end-to-end adversarial patch positioning method for realizing adversarial patch positioning, which can more effectively position the adversarial patch in the image, and can improve the robustness of the downstream neural network in combination with the removal method. The patch is detected and positioned through the texture feature difference between the patch region and the original image region, so as to avoid the model from focusing on the adversarial patch of a specified texture. The edge information feature is used to improve the positioning accuracy. An encoder-decoder network is used to extract features from the images of the RGB view and the edge view. According to the correlation of the features, different weights are given to each channel of the feature map, and multi-scale fusion is used to fully fuse the feature information of different resolutions. The adversarial patch positioning mask output by the multi-scale feature fusion module is completed, and the loss of image semantic information caused by the removal of the adversarial patch is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an adversarial patch positioning method based on self-supervised learning and multi-view feature fusion. Background Art

[0002] Adversarial patch attacks modify pixels within a specified area of ​​an image, causing the neural network to alter its predictions. Adversarial patch attack technology has become increasingly robust, maintaining a high success rate despite changes in environmental factors such as lighting and angle. Adversarial patch attacks pose serious security risks in the physical world. For example, criminals can use adversarial attacks on traffic signs, preventing self-driving cars from correctly recognizing them and potentially causing traffic accidents.

[0003] In current research, adversarial training methods are often used to defend against adversarial patch attacks. However, adversarial training methods have inherent drawbacks, primarily due to their high time and computational overhead, impact on clean sample prediction accuracy, and the need to retrain robust models for each specific dataset and model. Another defense option is adversarial defense methods based on sample purification, which locate and remove adversarial patches during image preprocessing, also providing effective defense. Current adversarial patch removal methods effectively defend against a certain range of adversarial patch attacks. However, the adversarial patches considered in existing removal methods are relatively simple, while adversarial patches using more complex attack strategies have emerged, increasing the success rate and stealth of attacks. Existing defense methods struggle to locate complex adversarial patches, resulting in weaker defenses. Summary of the Invention

[0004] In view of the above problems, the present invention discloses an adversarial patch localization method based on self-supervised learning and multi-view feature fusion, designs an end-to-end adversarial patch localization method for realizing adversarial patch localization, which can more effectively locate adversarial patches in images, and can improve the robustness of downstream neural networks when combined with a removal method; detects and locates patches by comparing the texture feature differences between the patch area and the original image area, avoiding the model's focus on adversarial patches of a certain specified texture; utilizes edge information features to improve positioning accuracy; uses an encoder-decoder network to extract features from images of RGB views and edge views; assigns different weights to each channel of the feature map according to the correlation of the features, and uses multi-scale fusion to fully fuse feature information of different resolutions; designs an adversarial patch localization mask output by the multi-scale feature fusion module for completion, while reducing the loss of image semantic information caused by the removal of adversarial patches.

[0005] The present invention provides an adversarial patch localization method based on self-supervised learning and multi-view feature fusion, comprising:

[0006] S1, constructing a target detection image dataset and a test image dataset;

[0007] The specific steps of constructing the dataset in step S1 include:

[0008] S11, obtaining a plurality of target detection image data based on a downstream target detection task, and establishing a target detection image dataset;

[0009] Further, each target detection image data corresponds to annotation information respectively;

[0010] The annotation information includes the rectangular box coordinates and classification labels of the objects in the target detection image; it can be understood that the objects are the targets to be detected, such as people and vehicles in various targets in the COCO dataset;

[0011] S12, adding an adversarial patch to each target detection image in the target detection image dataset to obtain a test image dataset;

[0012] Further, each test image corresponds to annotation information respectively; the annotation information corresponding to each test image respectively includes the mask of the adversarial patch, the rectangular box coordinates and the classification labels of the objects in the image;

[0013] S2, let i = 1, where i = 1, x i is the first target detection image, A i is an initial feature extraction backbone network, B i is an initial multi-scale fusion feature model, C i is an initial pixel classification convolutional layer;

[0014] S3, selecting an image x i and a random image x' i from the target detection image dataset; the image x i is represented as a clean image c i ; the patch mask m = f(b) corresponding to the clean image c i is obtained according to the object bounding box annotation b of the image x i ;

[0015] S4, obtaining the mask of the random image x' i ; based on the mask of the random image x' i , data augmentation is performed on the patch mask corresponding to the clean image c i in step S3 to obtain a data-augmented training sample x i and the corresponding mask;

[0016] The expression of the data-augmented training sample x i is:

[0017] ^x i =x i *(1-m i )+x' i *m i

[0018] Preferably, the training sample ^x obtained after data enhancement in step S4 is i The specific steps of the corresponding mask include:

[0019] Generate random images x' using basic geometric shapes stacking i 's mask;

[0020] The clean image c i The corresponding patch mask includes multiple rectangular regions;

[0021] Based on the random image x' i The mask of the clean image c i Each rectangular area of ​​the corresponding patch mask is transformed to obtain the training sample ^x after data enhancement i and the corresponding mask.

[0022] S5, using the Laplace operator to perform the training sample ^x after the data enhancement in step S4 i Perform high-pass filtering to obtain an image with edge features d i ;

[0023] Furthermore, in step S5, a Laplace operator is used to perform high-pass filtering, and edge detection is achieved by applying second-order partial derivatives to the image.

[0024] S6. Get image x i RGB image; based on feature extraction backbone network A i Get the view feature r of the RGB image i ; Based on the feature extraction backbone network, the view feature d′ of the image with edge features is obtained i and updating the feature extraction backbone network;

[0025] Preferably, the view feature r of the RGB image is obtained i and the view feature d′ of the image with edge features i The specific steps include:

[0026] Using feature extraction backbone network A i Extract image x separately i The features of the RGB image and the image with edge features d i characteristics;

[0027] The image xi The features of the RGB image are downsampled and upsampled step by step to obtain feature maps r with multiple resolutions. i ;

[0028] The image d with edge features i The features of the d′′ are obtained by downsampling and upsampling step by step to obtain a feature map d′′ with multiple edge features. i ;

[0029] The feature map r with multiple resolutions i and a feature map d″ with multiple edge features i ;

[0030] Establish a skip connection between the downsampling layer and the upsampling layer at the same level to obtain the corresponding RGB image view feature r i and the view feature d′ of the image with edge features i and feature extraction backbone network.

[0031] Furthermore, the feature extraction backbone network in step S6 is an encoder-decoder structure with 4 layers of downsampling and 4 layers of upsampling; in the encoder-decoder structure, jump connections are set between encoders and decoders of the same level; the feature extraction backbone network includes an output layer, which uses a single-category convolutional layer and uses a Sigmoid activation function to limit the output value between 0 and 1.

[0032] S7, the view feature r of the RGB image i and the view feature d′ of the image with edge features i After splicing, input the multi-scale fusion feature model B i Perform fusion processing to obtain multi-scale fusion features E i and update the multi-scale fusion feature model;

[0033] Preferably, the multi-scale fusion feature E obtained in step S6 is i The specific steps include:

[0034] The view feature r of the RGB image i and the view feature d′ of the image with edge features i Splice and get the spliced ​​multi-scale feature map e i ;

[0035] The spliced ​​multi-scale feature map e i Input multi-scale fusion feature model B i , based on the multi-head deep convolution transposed attention to the spliced ​​multi-scale feature map e i Perform multi-scale cross-channel feature fusion to obtain multi-scale fusion feature Ei and updating the multi-scale fusion feature model.

[0036] Further, the step S7 further comprises obtaining a multi-head deep convolution transpose attention, and the specific steps comprise:

[0037] Let the input tensor be X Standardize the tensor X to obtain a standardized tensor

[0038] Use a 1x1 normal convolution to extract the standardized tensor Pixel-level cross-channel context information is obtained to update the standardized tensor;

[0039] Use a 3x3 deep direction convolution to encode the spatial context information of the updated standardized tensor channel level to obtain an initial query Q matrix, an initial key value K matrix, and an initial value V matrix;

[0040] Respectively, the dimensions of the initial query Q matrix, the initial key value K matrix, and the initial value V matrix are transformed to obtain a multi-head deep convolution transpose attention.

[0041] Further, the multi-head deep convolution transpose attention expression is:

[0042]

[0043] Wherein, is the output of the multi-head deep convolution transpose attention, W p is a 1x1 convolution operation; alpha is a learnable parameter for controlling the dot product result size of the K matrix and the Q matrix; Softmax(·) is an activation function; X represents the feature map input into the attention module.

[0044] S8, the multi-scale fusion feature E i is obtained by processing the pixel classification convolution layer Ci to obtain the predicted mask b i and the loss of the predicted mask b i .

[0045] Preferably, the loss of the predicted mask b i of step S8 comprises the following specific steps:

[0046] The multi-scale fusion feature E i obtained by step S7 is input into the pixel classification convolution layer, and after processing based on an activation function, the predicted mask b i is obtained, and the loss of the predicted mask b i and the patch mask obtained in S3 is calculated.

[0047] Further, the activation function is a Sigmoid function.

[0048] Preferably, the prediction mask b i The value of the middle mask ranges from 0 to 1, where the closer it is to 1, the greater the probability that the pixel is an adversarial patch;

[0049] Preferably, the loss expression in step S8 is:

[0050] L total,i =L BCE,i +λL DICE,i ;

[0051] Among them, λ is a hyperparameter used to adjust the weight of Dice loss; L total,i is the loss of the i-th image, L BCE,i is the cross entropy loss of the i-th image, L DICE,i is the Dice loss of the i-th image.

[0052] Furthermore, the cross entropy loss expression is:

[0053]

[0054] Among them, y i is the patch mask of the i-th image, x i is the probability that the predicted mask of the i-th image is a patch mask. The cross entropy loss is calculated for the classification results of n images and the average is taken.

[0055] Furthermore, the Dice loss function L DICE,i The expression is:

[0056]

[0057] Among them, x i,j is the predicted mask of pixel j in the i-th image, y i,j is the patch mask of pixel j in the i-th image; j = 1, 2, 3...J, where J is the total number of pixels.

[0058] S9. Compare i and I, where I represents the total number of target detection images. If i < I, add 1 to the value i and return to step S3. If i = I, end the training to obtain an updated feature extraction backbone network, an updated multi-scale fusion feature model, and an updated pixel classification convolutional layer.

[0059] Furthermore, in order to verify the effectiveness and practicality of the present invention, the COCO sub-dataset was used as the training data set (10,000 images), and the model was trained according to steps S1 to S9. RMSprop was used as the model optimizer, the learning rate was set to 0.00001, 80% of the training data was used to train the model, 10% was used as the validation model, and 10% was used as the test model. A total of 20 iterations were trained, and the model with the best evaluation index of the validation set was finally saved as the final result.

[0060] The COCO sub-dataset is used for model evaluation. The evaluation dataset contains 1,000 samples with complex adversarial patch attacks added. The trained model is used to perform evaluation according to the above step S9 and compared with the true labels. The mAP index of the COCO dataset is 46.5 when no adversarial patches are added, and the mAP index of the COCO dataset after adding the adversarial patches is 18.9. The pixel-level F1 value of the adversarial patch positioning mask is 0.917. After combining the adversarial patch removal method, the mAP index of the COCO dataset is 39.5, which is a good result, indicating that the present invention is effective and feasible.

[0061] S10. Construct a patch positioning model based on the trained feature extraction backbone network, the trained multi-scale fusion feature model, and the trained pixel classification convolutional layer; and perform patch positioning based on the patch positioning model.

[0062] The present invention also provides a method for locating and removing countermeasure patches, comprising:

[0063] S11. Build a patch localization model based on the updated feature extraction backbone network, the updated multi-scale fusion feature model, and the updated pixel classification convolutional layer;

[0064] The patch positioning model is updated to obtain a patch positioning removal model, and the image to be tested is input into the patch removal model to perform adversarial patch removal to obtain an image after the adversarial patch is removed.

[0065] Preferably, the specific steps of obtaining the image after the adversarial patch is removed include:

[0066] S111. Construct a patch localization model based on an updated feature extraction backbone network, an updated multi-scale fusion feature model, and an updated pixel classification convolutional layer;

[0067] S112: Input the image to be tested into the adversarial patch positioning model to obtain the initial prediction mask b′ i ;

[0068] S113, designing a mask completion module and an image reconstruction module; embedding the mask completion module and the image reconstruction module into the patch positioning model to obtain a patch positioning removal model;

[0069] S114、obtaining the initial prediction mask b i inputting the patch positioning removal model, and obtaining a completion prediction mask b based on the mask completion module i ;

[0070] obtaining the completion prediction mask b i The corresponding area is represented as a patch removal area.

[0071] S115, the patch removal area uses an image reconstruction module to obtain a to-be-tested image original area; and based on the to-be-tested image original area, texture of the patch removal area is removed to obtain an image after patch removal.

[0072] Further, in the step S114, the completion prediction mask b i The specific steps include:

[0073] S1141, obtaining a test image A i and an original mask M o ; a preset window width w and height h, and a preset adversarial patch mask threshold z;

[0074] S1142, inputting the original mask M o to an adversarial patch mask completion module, initializing a 0-value array M o with the same size as the original mask M c ;

[0075] S1143, the original mask M o includes a plurality of pixels; each h x w window includes a plurality of pixel values.

[0076] S1144, setting the center of the pixel p as a window with a size of h x w, and calculating the sum of the plurality of pixel values in the window.

[0077] S1145, comparing the sum of the plurality of pixel values in the window with the adversarial patch mask threshold z, if the sum of the plurality of pixel values in the window is greater than the adversarial patch mask threshold z, setting the value of the array M c corresponding to the position of the pixel p to 255; if it is less than the adversarial patch mask threshold z, the value of the pixel p is not modified.

[0078] S1146, repeating steps S1144-1145 until the plurality of pixels in the original mask M o are traversed, to obtain the completion prediction mask b i .

[0079] Compared with the prior art, the present application has at least the following beneficial effects:

[0080] (1) The self-supervised learning strategy of the present invention can enhance the robustness and generalization of the defense model while saving the computing power resources required to calculate the adversarial patch samples;

[0081] (2) The present invention introduces edge view features into adversarial patch localization and uses the attention mechanism to fuse features from different views to improve the accuracy of adversarial patch localization;

[0082] (3) The present invention can remove the adversarial patches in the image and reconstruct the texture of the patch area. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The drawings are only for purposes of illustrating particular embodiments and are not to be considered limiting of the invention.

[0084] Figure 1 Schematic diagram of the structure of an adversarial patch localization method based on self-supervised learning and multi-view feature fusion in an embodiment of the present invention;

[0085] Figure 2 Schematic diagram of the training process of the adversarial patch localization method based on self-supervised learning and multi-view feature fusion in an embodiment of the present invention;

[0086] Figure 3 This is a structural diagram of a feature fusion module in an embodiment of the present invention;

[0087] Figure 4 Schematic diagram of the testing process of the adversarial patch localization method based on self-supervised learning and multi-view feature fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0088] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited by the specific embodiments disclosed below.

[0089] A specific embodiment of the present invention, as Figure 1-4 , discloses an adversarial patch localization method based on self-supervised learning and multi-view feature fusion. To illustrate the effectiveness of the method proposed in the present invention, the above technical solution of the present invention is described in detail through a specific embodiment below. The specific implementation steps are as follows:

[0090] The present invention provides an adversarial patch localization method based on self-supervised learning and multi-view feature fusion, comprising:

[0091] S1. Build target detection image dataset and test image dataset;

[0092] The specific steps of constructing the data set in step S1 include:

[0093] S11. Obtain multiple target detection image data based on the downstream target detection task and establish a target detection image dataset;

[0094] Furthermore, each target detection image data corresponds to annotation information;

[0095] The annotation information includes the rectangular frame coordinates and classification labels of the objects in the target detection image; it can be understood that the objects are targets to be detected, for example, people, vehicles, etc. among the various targets in the COCO dataset;

[0096] S12, adding an adversarial patch to each target detection image in the target detection image dataset to obtain a test image dataset;

[0097] Furthermore, each of the test images corresponds to annotation information; the annotation information corresponding to each of the test images includes a mask of the adversarial patch, a rectangular frame coordinate of an object in the image, and a classification label;

[0098] S2, let i = 1, where i = 1, x i For the first target detection image, A i For the initial feature extraction backbone network, B i is the initial multi-scale fusion feature model, C i is the initial pixel classification convolution layer;

[0099] S3. Select an image x in the target detection image dataset. i and a random image x' i ; The image x i Represented as a clean image c i ; According to image x i The object bounding box annotation b obtains the clean image c i The corresponding patch mask m = f(b);

[0100] S4. Get a random image x' i mask; based on the random image x' i The mask of the clean image c in step S3 i The corresponding patch mask is used for data enhancement to obtain the training sample ^x after data enhancement i and the corresponding mask;

[0101] The training sample after data enhancement ^x i The expression is:

[0102] ^x i =xi *(1-m i )+x' i *m i

[0103] Preferably, the training sample ^x obtained after data enhancement in step S4 is i The specific steps of the corresponding mask include:

[0104] Generate random images x' using basic geometric shapes stacking i 's mask;

[0105] The clean image c i The corresponding patch mask includes multiple rectangular regions;

[0106] Based on the random image x' i The mask of the clean image c i Each rectangular area of ​​the corresponding patch mask is transformed to obtain the training sample ^x after data enhancement i and the corresponding mask.

[0107] S5, using the Laplace operator to perform the training sample ^x after the data enhancement in step S4 i Perform high-pass filtering to obtain an image with edge features d i ;

[0108] Furthermore, in step S5, a Laplace operator is used to perform high-pass filtering, and edge detection is achieved by applying second-order partial derivatives to the image.

[0109] S6. Get image x i RGB image; based on feature extraction backbone network A i Get the view feature r of the RGB image i ; Based on the feature extraction backbone network, the view feature d′ of the image with edge features is obtained i and updating the feature extraction backbone network;

[0110] Preferably, the view feature r of the RGB image is obtained i and the view feature d′ of the image with edge features i The specific steps include:

[0111] Using feature extraction backbone network A i Extract image x separately i The features of the RGB image and the image with edge features d i characteristics;

[0112] The image x iThe features of the RGB image are downsampled and upsampled step by step to obtain feature maps r with multiple resolutions. i ;

[0113] The image d with edge features i The features of the d′′ are obtained by downsampling and upsampling step by step to obtain a feature map d′′ with multiple edge features. i ;

[0114] The feature map r with multiple resolutions i and a feature map d″ with multiple edge features i ;

[0115] Establish a skip connection between the downsampling layer and the upsampling layer at the same level to obtain the corresponding RGB image view feature r i and the view feature d′ of the image with edge features i and feature extraction backbone network.

[0116] Furthermore, the feature extraction backbone network in step S6 is an encoder-decoder structure with 4 layers of downsampling and 4 layers of upsampling; in the encoder-decoder structure, jump connections are set between encoders and decoders of the same level; the feature extraction backbone network includes an output layer, which uses a single-category convolutional layer and uses a Sigmoid activation function to limit the output value between 0 and 1.

[0117] S7, the view feature r of the RGB image i and the view feature d′ of the image with edge features i After splicing, input the multi-scale fusion feature model B i Perform fusion processing to obtain multi-scale fusion features E i and update the multi-scale fusion feature model;

[0118] Preferably, the multi-scale fusion feature E obtained in step S6 is i The specific steps include:

[0119] The view feature r of the RGB image i and the view feature d′ of the image with edge features i Splice and get the spliced ​​multi-scale feature map e i ;

[0120] The spliced ​​multi-scale feature map e i Input multi-scale fusion feature model B i , based on the multi-head deep convolution transposed attention to the spliced ​​multi-scale feature map e i Perform multi-scale cross-channel feature fusion to obtain multi-scale fusion feature E iand update the multi-scale fusion feature model.

[0121] Furthermore, step S7 also includes obtaining transposed attention based on multi-head deep convolution, and the specific steps include:

[0122] Let the input tensor be Normalize the tensor X to get the normalized tensor

[0123] Extract the normalized tensor using a 1×1 ordinary convolution Pixel-level cross-channel context information is used to update the normalized tensor;

[0124] Using a 3×3 depthwise convolution to encode the spatial context information of the updated normalized tensor channel level, an initial query Q matrix, an initial key value K matrix, and an initial value V matrix are obtained;

[0125] The initial query Q matrix, initial key value K matrix and initial value V matrix are transformed in dimension respectively to obtain multi-head deep convolution transposed attention.

[0126] Furthermore, the multi-head depth convolution transposed attention expression is:

[0127]

[0128] in, The output of the multi-head deep convolution transposed attention, W p is a 1×1 convolution operation; α is a learnable parameter used to control the size of the dot product result of the K matrix and the Q matrix; Softmax(·) is the activation function; X represents the feature map of the input attention module.

[0129] S8, the multi-scale fusion feature E obtained in step S7 i After processing through the pixel classification convolution layer Ci, the predicted mask b is obtained i And the prediction mask b i losses;

[0130] Preferably, the prediction mask b in step S8 i The specific steps for loss include:

[0131] The multi-scale fusion feature E obtained in step S7 i In the input pixel classification convolution layer, the prediction mask b is obtained after processing based on the activation function i , calculate the predicted mask b i The loss of the patch mask obtained in S3;

[0132] Furthermore, the activation function is a Sigmoid function;

[0133] Preferably, the prediction mask b i The value of the middle mask ranges from 0 to 1, where the closer it is to 1, the greater the probability that the pixel is an adversarial patch;

[0134] Preferably, the loss expression in step S8 is:

[0135] L total,i =L BCE,i +λL DICE,i ;

[0136] Among them, λ is a hyperparameter used to adjust the weight of Dice loss; L total,i is the loss of the i-th image, L BCE,i is the cross entropy loss of the i-th image, L DICE,i is the Dice loss of the i-th image.

[0137] Furthermore, the cross entropy loss expression is:

[0138]

[0139] Among them, y i is the patch mask of the i-th image, x i is the probability that the predicted mask of the i-th image is a patch mask. The cross entropy loss is calculated for the classification results of n images and the average is taken.

[0140] Furthermore, the Dice loss function L DICE,i The expression is:

[0141]

[0142] Among them, x i,j is the predicted mask of pixel j in the i-th image, y i,j is the patch mask of pixel j in the i-th image; j = 1, 2, 3...J, where J is the total number of pixels.

[0143] S9. Compare i and I, where I represents the total number of target detection images. If i < I, add 1 to the value i and return to step S3. If i = I, end the training to obtain an updated feature extraction backbone network, an updated multi-scale fusion feature model, and an updated pixel classification convolutional layer.

[0144] Furthermore, in order to verify the effectiveness and practicality of the present invention, the COCO sub-dataset was used as the training data set (10,000 images), and the model was trained according to steps S1 to S9. RMSprop was used as the model optimizer, the learning rate was set to 0.00001, 80% of the training data was used to train the model, 10% was used as the validation model, and 10% was used as the test model. A total of 20 iterations were trained, and the model with the best evaluation index of the validation set was finally saved as the final result.

[0145] The COCO sub-dataset is used for model evaluation. The evaluation dataset contains 1,000 samples with complex adversarial patch attacks added. The trained model is used to perform evaluation according to the above step S9 and compared with the true labels. The mAP index of the COCO dataset is 46.5 when no adversarial patches are added, and the mAP index of the COCO dataset after adding the adversarial patches is 18.9. The pixel-level F1 value of the adversarial patch positioning mask is 0.917. After combining the adversarial patch removal method, the mAP index of the COCO dataset is 39.5, which is a good result, indicating that the present invention is effective and feasible.

[0146] S10. Construct a patch positioning model based on the trained feature extraction backbone network, the trained multi-scale fusion feature model, and the trained pixel classification convolutional layer; and perform patch positioning based on the patch positioning model.

[0147] The present invention also provides a method for locating and removing countermeasure patches, comprising:

[0148] S11. Build a patch localization model based on the updated feature extraction backbone network, the updated multi-scale fusion feature model, and the updated pixel classification convolutional layer;

[0149] The patch positioning model is updated to obtain a patch positioning removal model, and the image to be tested is input into the patch removal model to perform adversarial patch removal to obtain an image after the adversarial patch is removed.

[0150] Preferably, the specific steps of obtaining the image after the adversarial patch is removed include:

[0151] S111. Construct a patch localization model based on an updated feature extraction backbone network, an updated multi-scale fusion feature model, and an updated pixel classification convolutional layer;

[0152] S112: Input the image to be tested into the adversarial patch positioning model to obtain the initial prediction mask b′ i ;

[0153] S113, designing a mask completion module and an image reconstruction module; embedding the mask completion module and the image reconstruction module into the patch positioning model to obtain a patch positioning removal model;

[0154] S114, the initial prediction mask b' i Input the patch positioning removal model and obtain the completed prediction mask b based on the mask completion module i ;

[0155] The completed prediction mask b i The corresponding area is characterized as the patch removal area;

[0156] S115 , the patch-removed area uses an image reconstruction module to obtain an original image area of ​​the image to be tested; and based on the original image area of ​​the image to be tested, textures of the patch-removed area are removed to obtain an image after the adversarial patch is removed.

[0157] Furthermore, the completed prediction mask b obtained in step S114 is i The specific steps include:

[0158] S1141. Obtain test image A i The original mask M o ; Preset the window width w and height h, and preset the adversarial patch mask threshold z;

[0159] S1142, the original mask M o Input to the adversarial patch mask completion module, initialized with the original mask M o Array M of zero values ​​of the same size c ;

[0160] S1143: the original mask M o comprising a plurality of pixels; wherein each h×w window comprises a plurality of pixel values;

[0161] S1144, setting a window of size h×w at the center of pixel p, and calculating the sum of multiple pixel values ​​in the window;

[0162] S1145, compare the sum of the pixel values ​​in the window with the adversarial patch mask threshold z, if the sum of the pixel values ​​in the window is greater than the adversarial patch mask threshold z, then the array M corresponding to the pixel p is c The value of is set to 255; if it is less than the adversarial patch mask threshold z, the value of pixel p is not modified;

[0163] S1146, repeat steps S1144-1145 until the original mask M is traversed o Multiple pixels in the , get the completed prediction mask b i .

[0164] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. Adversarial patch localization method based on self-supervised learning and multi-view feature fusion, characterized by: include: S1. Build target detection image dataset and test image dataset; S2, let i = 1, where i = 1, x i For the first target detection image, A i For the initial feature extraction backbone network, B i is the initial multi-scale fusion feature model, C i is the initial pixel classification convolution layer; S3. Select an image x in the target detection image dataset. i and a random image x' i ; The image x i Represented as a clean image c i ; According to image x i The object bounding box annotation b obtains the clean image c i The corresponding patch mask m i ; S4. Get a random image x' i mask; based on the random image x' i The mask of the clean image c in step S3 i The corresponding patch mask is used for data enhancement to obtain the training sample ^x after data enhancement i and the corresponding mask; S5, obtain the training sample ^x after the data enhancement in step S4 i Image d with edge features i ; S6. Get image x i RGB image; the image d with edge features i and image x i RGB image input feature extraction backbone network A i Get the view feature r of the RGB image i and the view feature d′ of the image with edge features i And update the feature extraction backbone network A i ; S7, the view feature r of the RGB image i and the view feature d′ of the image with edge features i After splicing, input the multi-scale fusion feature model B i , get the multi-scale fusion feature E i And update the multi-scale fusion feature model B i ; S8, the multi-scale fusion feature E obtained in step S7 i Input pixel classification convolution layer C i After processing, the predicted mask b is obtained i , loss, and update pixel classification convolution layer C i ; S9, compare the size of i and I, I represents the total number of target detection images, if i < I, add 1 to the value i and return to step S3, if i = I, then end the training and get the updated feature extraction backbone network A i , update the multi-scale fusion feature model B i and update the pixel classification convolution layer C i ; S10, building a patch localization model based on the updated feature extraction backbone network, the updated multi-scale fusion feature model, and the updated pixel classification convolutional layer; Performing patch positioning based on the patch positioning model; S11, updating the patch positioning model to obtain a patch positioning removal model, inputting the image to be tested into the patch removal model, performing adversarial patch removal, and obtaining an image after the adversarial patch removal. The specific steps include: S111: Input the image to be tested into the adversarial patch positioning model to obtain the initial prediction mask b' i ; S112, designing a mask completion module and an image reconstruction module; embedding the mask completion module and the image reconstruction module into the patch positioning model to obtain a patch positioning removal model; S113, the initial prediction mask b' i Input the patch positioning removal model and obtain the completed prediction mask b based on the mask completion module i ; The completed prediction mask b i The corresponding area is characterized as the patch removal area; S114 , using an image reconstruction module to obtain an original image region of the image to be tested in the patch-removed region; removing textures of the patch-removed region based on the original image region to obtain an image after the adversarial patch is removed.

2. The method for locating adversarial patches according to claim 1, wherein: The specific steps of constructing the data set in step S1 include: S11. Based on the target detection task, multiple target detection image data are obtained and a target detection image dataset is established; S12. Add an adversarial patch to each target detection image in the target detection image dataset to obtain a test image dataset.

3. The method for locating an adversarial patch according to claim 1, wherein: The training sample after data enhancement ^x i The expression is: ^x i＝ x i *(1-m i )+x' i *m i 。 4. The method for locating an adversarial patch according to claim 1, wherein: The view feature r of the RGB image is obtained based on the feature extraction backbone network i and the view feature d′ of the image with edge features i The specific steps include: Use the feature extraction backbone network to extract the image x i The features of the RGB image and the image with edge features d i characteristics; The image x i The features of the RGB image are downsampled and upsampled step by step to obtain feature maps with multiple resolutions The image d with edge features i The features of the d′′ are obtained by downsampling and upsampling step by step to obtain a feature map d′′ with multiple edge features. i ; The feature maps with multiple resolutions and a feature map d″ with multiple edge features i Establish a skip connection between the downsampling layer and the upsampling layer at the same level to obtain the corresponding RGB image view feature r i and the view feature d′ of the image with edge features i .

5. The method for locating an adversarial patch according to claim 1, wherein: Use the Laplace operator to perform the training sample ^x after the data enhancement in step S4 i Perform high-pass filtering to obtain an image with edge features d i .

6. The method for locating an adversarial patch according to claim 1, wherein: The multi-scale fusion feature E is obtained in step S7 i The specific steps include: The view feature r of the RGB image i and the view feature d′ of the image with edge features i Splice and get the spliced ​​multi-scale feature map e i ; The spliced ​​multi-scale feature map e i Input multi-scale fusion feature model, based on multi-head deep convolution transposed attention to splice multi-scale feature map e i Perform multi-scale cross-channel feature fusion to obtain multi-scale fusion feature E i .

7. The method for locating an adversarial patch according to claim 6, wherein: Step S6 also includes obtaining transposed attention based on multi-head deep convolution, and the specific steps include: Let the input tensor be Normalize the tensor X to get the normalized tensor Extracting the normalized tensor Pixel-level cross-channel context information is used to update the normalized tensor; Encoding the spatial context information of the updated normalized tensor channel level to obtain an initial query Q matrix, an initial key value K matrix and an initial value V matrix; After dimensionality transformation of the initial query Q matrix, the initial key value K matrix and the initial value V matrix respectively, the multi-head deep convolution transposed attention is obtained.

8. The method for locating an adversarial patch according to claim 1, wherein: The expression of the loss in step S8 is: THE total,i =L BCE,i +λL DICE,i ; Among them, L total,i is the loss of image i, L BCE,i is the cross entropy loss of the intersection image i, λ is a hyperparameter; L DICE,i is the Dice loss of image i.

9. The method for locating an adversarial patch according to any one of claims 1 to 8, wherein: The adversarial patch is removed using the patch localization result based on the patch localization model.

Images