Adversarial patch positioning method based on self-supervised learning and multi-view feature fusion

Abstract

The invention relates to an adversarial patch positioning method based on self-supervised learning and multi-view feature fusion, belongs to the technical field of image processing, and designs an end-to-end adversarial patch positioning method which is used for realizing adversarial patch positioning, can more effectively position an adversarial patch in an image, and improves the positioning accuracy of the adversarial patch. The robustness of the downstream neural network can be improved in combination with a removal method; the patch is detected and positioned through the texture feature difference between the patch area and the original image area, so that the model is prevented from paying attention to an adversarial patch with a certain specified texture; the positioning accuracy is improved by utilizing edge information features; performing feature extraction on the images of the RGB view and the edge view by using an encoder-decoder network; different weights are given to all channels of the feature map according to the correlation of the features, and feature information of different resolutions is fully fused through multi-scale fusion; and designing an adversarial patch positioning mask output by the multi-scale feature fusion module for complementing, and meanwhile, reducing image semantic information loss caused by adversarial patch removal.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an adversarial patch positioning method based on self-supervised learning and multi-view feature fusion. Background Art

[0002] Adversarial patch attacks refer to modifying pixels in a specified area of ​​an image to change the neural network's prediction of the image. With the advancement of adversarial patch attack technology, the robustness of adversarial patch attacks has gradually improved, and a high attack success rate can be maintained in the face of changes in environmental factors such as light and angle. Adversarial patch attacks have brought serious security risks to the physical world. For example, criminals can use adversarial attack methods on traffic signs, causing driverless cars to be unable to correctly identify relevant signs, thereby causing traffic safety accidents.

[0003] In current research work, adversarial training methods are often used to defend against adversarial patch attacks. However, adversarial training methods have inherent disadvantages, mainly reflected in the large time and computational overhead, the impact on the prediction accuracy of clean samples, and the need to retrain robust models for each specific data set and model. Another defense option is the adversarial defense method based on sample purification. The adversarial patch location and removal in the image preprocessing stage can also play a good defense effect. The current adversarial patch removal method effectively defends against a certain range of adversarial patch attacks. However, the adversarial patches considered in the existing removal methods are relatively simple, but adversarial patches that use more complex attack strategies have emerged, which improve the success rate and concealment of the attack. Existing defense methods are difficult to locate complex adversarial patches, and the defense effect is weak. Summary of the invention

[0004] In view of the above problems, the present invention discloses an adversarial patch localization method based on self-supervised learning and multi-view feature fusion, designs an end-to-end adversarial patch localization method for realizing adversarial patch localization, which can more effectively locate adversarial patches in images, and can improve the robustness of downstream neural networks in combination with removal methods; detects and locates patches by using the texture feature differences between the patch area and the original image area, avoiding the model from focusing on adversarial patches of a certain specified texture; utilizes edge information features to improve positioning accuracy; uses an encoder-decoder network to extract features from images of RGB views and edge views; assigns different weights to each channel of the feature map according to the correlation of the features, and uses multi-scale fusion to fully fuse feature information of different resolutions; designs an adversarial patch localization mask output by a multi-scale feature fusion module for completion, while reducing the loss of image semantic information caused by the removal of adversarial patches.

[0005] The present invention provides an adversarial patch localization method based on self-supervised learning and multi-view feature fusion, comprising:

[0006] S1. Build target detection image dataset and test image dataset;

[0007] The specific steps of constructing the data set in step S1 include:

[0008] S11, obtaining a plurality of target detection image data based on a downstream target detection task, and establishing a target detection image data set;

[0009] Furthermore, each target detection image data corresponds to annotation information respectively;

[0010] The annotation information includes the rectangular frame coordinates and classification labels of the object in the target detection image; it can be understood that the object is a target to be detected, for example, a person, a vehicle, etc. among the various targets in the COCO dataset;

[0011] S12, adding an adversarial patch to each target detection image in the target detection image dataset to obtain a test image dataset;

[0012] Furthermore, each of the test images corresponds to annotation information; the annotation information corresponding to each of the test images includes a mask of the adversarial patch, a rectangular frame coordinate of an object in the image, and a classification label;

[0013] S2, let i = 1, where i = 1, x i For the first target detection image, A i is the initial feature extraction backbone network, B i is the initial multi-scale fusion feature model, C i is the initial pixel classification convolution layer;

[0014] S3, select image x in the target detection image dataset i and a random image x' i ; The image x i Characterized as a clean image c i ; According to the image x i The object border annotation b obtains the clean image c i The corresponding patch mask m = f(b);

[0015] S4. Get a random image x' i Mask; Based on the random image x' i The mask of the clean image c in step S3 i The corresponding patch mask is used for data enhancement to obtain the training sample ^x after data enhancement i and the corresponding mask;

[0016] The training sample after data enhancement ^x i The expression is:

[0017] ^x i =x i *(1-m i )+x' i *m i

[0018] Preferably, the training sample ^x obtained after data enhancement in step S4 is i And the corresponding mask specific steps include:

[0019] Generate a random image x' using basic geometric shapes stacking i 's mask;

[0020] The clean image c i The corresponding patch mask includes multiple rectangular regions;

[0021] Based on the random image x' i The mask of the clean image c i Each rectangular area of ​​the corresponding patch mask is transformed to obtain the training sample ^x after data enhancement i and the corresponding mask.

[0022] S5, using the Laplace operator to enhance the training sample ^x after the data in step S4 i Perform high-pass filtering to obtain an image with edge features d i ;

[0023] Furthermore, in step S5, a Laplace operator is used for high-pass filtering, and edge detection is achieved by applying second-order partial derivatives in the image.

[0024] S6. Get image x i RGB image; based on feature extraction backbone network A i Get the view feature r of the RGB image i ; Based on the feature extraction backbone network, the view feature d′ of the image with edge features is obtained i and updating the feature extraction backbone network;

[0025] Preferably, the view feature r of the RGB image is obtained i and the view feature d′ of the image with edge features i The specific steps include:

[0026] Using feature extraction backbone network A i Extract image x separately i The features of the RGB image and the image with edge features d i Features;

[0027] The image xi The features of the RGB image are downsampled and upsampled step by step to obtain feature maps r with multiple resolutions. i ;

[0028] The image d with edge features i The features of the d″ are obtained by downsampling and upsampling step by step to obtain a feature map d″ with multiple edge features i ;

[0029] The feature map r with multiple resolutions i and a feature map d″ with multiple edge features i ;

[0030] A skip connection is established between the downsampling layer and the upsampling layer at the same level to obtain the corresponding RGB image view feature r i and the view feature d′ of the image with edge features i And feature extraction backbone network.

[0031] Furthermore, in step S6, the feature extraction backbone network is an encoder-decoder structure with 4 layers of downsampling and 4 layers of upsampling; in the encoder-decoder structure, jump connections are set between encoders-decoders of the same level; the feature extraction backbone network includes an output layer, the output layer uses a single-category convolutional layer, and uses a Sigmoid activation function to limit the output value between 0 and 1.

[0032] S7, transforming the view feature r of the RGB image i and the view feature d′ of the image with edge features i After splicing, input the multi-scale fusion feature model B i Perform fusion processing to obtain the multi-scale fusion feature E i and update the multi-scale fusion feature model;

[0033] Preferably, the multi-scale fusion feature E obtained in step S6 is i The specific steps include:

[0034] The view feature r of the RGB image i and the view feature d′ of the image with edge features i Splice to obtain the spliced ​​multi-scale feature map e i ;

[0035] The spliced ​​multi-scale feature map e i Input multi-scale fusion feature model B i , based on multi-head deep convolution transposed attention to the concatenated multi-scale feature map e i Perform multi-scale cross-channel feature fusion to obtain the multi-scale fusion feature Ei And update the multi-scale fusion feature model.

[0036] Furthermore, step S7 also includes obtaining transposed attention based on multi-head deep convolution, and the specific steps include:

[0037] Let the input tensor be Normalize the tensor X to get the normalized tensor

[0038] Extract the normalized tensor using a 1×1 regular convolution Pixel-level cross-channel context information to get updated normalized tensor;

[0039] Using a 3×3 depth-wise convolution to encode the spatial context information of the updated normalized tensor channel level, an initial query Q matrix, an initial key value K matrix and an initial value V matrix are obtained;

[0040] The initial query Q matrix, the initial key value K matrix and the initial value V matrix are transformed in dimension respectively to obtain multi-head deep convolution transposed attention.

[0041] Furthermore, the multi-head deep convolution transposed attention expression is:

[0042]

[0043] in, Transpose the output of the attention for the multi-head deep convolution, W p is a 1×1 convolution operation; α is a learnable parameter used to control the size of the dot product result of the K matrix and the Q matrix; Softmax(·) is the activation function; X represents the feature map of the input attention module.

[0044] S8, the multi-scale fusion feature E obtained in step S7 i After processing through the pixel classification convolution layer Ci, the predicted mask b is obtained i And the prediction mask b i loss;

[0045] Preferably, the prediction mask b in step S8 i The specific steps of loss include:

[0046] The multi-scale fusion feature E obtained in step S7 i In the input pixel classification convolution layer, the predicted mask b is obtained after processing based on the activation function i , calculate the predicted mask b i The loss of the patch mask obtained in S3;

[0047] Furthermore, the activation function is a Sigmoid function;

[0048] Preferably, the prediction mask b i The value of the middle mask ranges from 0 to 1, where the closer it is to 1, the greater the probability that the pixel is an adversarial patch;

[0049] Preferably, the loss expression in step S8 is:

[0050] L total,i =L BCE,i +λL DICE,i ;

[0051] Among them, λ is a hyperparameter used to adjust the weight of Dice loss; L total,i is the loss of the i-th image, L BCE,i is the cross entropy loss of the i-th image, L DICE,i is the Dice loss of the i-th image.

[0052] Furthermore, the cross entropy loss expression is:

[0053]

[0054] Among them, y i is the patch mask of the ith image, x i is the probability that the predicted mask of the i-th image is a patch mask. The cross entropy loss is calculated for the classification results of n images and the average is taken.

[0055] Furthermore, the Dice loss function L DICE,i The expression is:

[0056]

[0057] Among them, x i,j is the predicted mask of pixel j in the i-th image, y i,j is the patch mask of pixel j of the ith image; j=1,2,3...J, where J is the total number of pixels.

[0058] S9, compare the size of i and I, I represents the total number of target detection images, if i < I, add 1 to the value i, and return to step S3, if i = I, then end the training, and obtain an updated feature extraction backbone network, an updated multi-scale fusion feature model, and an updated pixel classification convolution layer;

[0059] Further, in order to verify the effectiveness and practicality of the present invention, the COCO sub-dataset is used as the training data set (10,000 images), the model is trained according to steps S1 to S9, RMSprop is used as the optimizer of the model, the learning rate is set to 0.00001, 80% of the training data is used to train the model, 10% is used as the verification model, and 10% is used as the test model. A total of 20 iterations are trained, and finally the model with the best evaluation index of the verification set is saved as the final result;

[0060] The COCO sub-dataset is used to evaluate the model. The evaluation data set contains 1000 samples with complex adversarial patch attacks added. The trained model is used to perform evaluation according to the above step S9 and compared with the true label. The mAP index of the COCO dataset is 46.5 when no adversarial patches are added, and the mAP index of the COCO dataset after adding the adversarial patches is 18.9. The pixel-level F1 value of the adversarial patch positioning mask is 0.917. After combining the adversarial patch removal method, the mAP index of the COCO dataset is 39.5, which is a good result, indicating that the present invention is effective and feasible.

[0061] S10, constructing a patch localization model based on the trained feature extraction backbone network, the trained multi-scale fusion feature model and the trained pixel classification convolutional layer; and performing patch localization based on the patch localization model.

[0062] The present invention also provides a method for positioning and removing an adversarial patch, comprising:

[0063] S11, constructing a patch localization model based on updating the feature extraction backbone network, updating the multi-scale fusion feature model and updating the pixel classification convolution layer;

[0064] The patch positioning model is updated to obtain a patch positioning removal model, and the image to be tested is input into the patch removal model to perform adversarial patch removal to obtain an image after the adversarial patch is removed.

[0065] Preferably, the specific steps of obtaining the image after the adversarial patch is removed include:

[0066] S111, constructing a patch localization model based on updating the feature extraction backbone network, updating the multi-scale fusion feature model and updating the pixel classification convolution layer;

[0067] S112, input the image to be tested into the adversarial patch positioning model to obtain an initial prediction mask b′ i ;

[0068] S113, designing a mask completion module and an image reconstruction module; embedding the mask completion module and the image reconstruction module into the patch positioning model to obtain a patch positioning removal model;

[0069] S114, the initial prediction mask b′ i Input the patch positioning removal model, and obtain the completed prediction mask b based on the mask completion module i ;

[0070] The completion prediction mask b i The corresponding area is characterized as the patch removal area;

[0071] S115, the patch removal area uses an image reconstruction module to obtain the original image area to be tested; based on the original image area to be tested, the texture of the patch removal area is removed to obtain an image after the adversarial patch is removed.

[0072] Furthermore, the completed prediction mask b obtained in step S114 is i The specific steps include:

[0073] S1141, obtain test image A i The original mask M o ; Preset the window width w and height h, and preset the adversarial patch mask threshold z;

[0074] S1142, the original mask M o Input to the adversarial patch mask completion module, initialized with the original mask M o Array M of zero values ​​of the same size c ;

[0075] S1143, the original mask M o comprising a plurality of pixels; each h×w window comprising a plurality of pixel values;

[0076] S1144, setting a window of size h×w at the center of pixel p, and calculating the sum of multiple pixel values ​​in the window;

[0077] S1145, comparing the sum of the pixel values ​​in the window with the adversarial patch mask threshold z, if the sum of the pixel values ​​in the window is greater than the adversarial patch mask threshold z, then c The value of is set to 255; if it is less than the adversarial patch mask threshold z, the value of pixel p is not modified;

[0078] S1146, repeat steps S1144-1145 until the original mask M is traversed o Multiple pixels in the image are used to obtain the completed prediction mask b. i .

[0079] Compared with the prior art, the present invention has at least the following beneficial effects:

[0080] (1) The self-supervised learning strategy of the present invention can enhance the robustness and generalization of the defense model while saving the computing resources required to calculate the adversarial patch samples;

[0081] (2) The present invention introduces edge view features into adversarial patch localization and uses the attention mechanism to fuse features of different views to improve the accuracy of adversarial patch localization;

[0082] (3) The present invention can remove adversarial patches in an image and reconstruct the texture of the patch area. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0084] Figure 1 It is a schematic diagram of a structure diagram of an adversarial patch localization method based on self-supervised learning and multi-view feature fusion in an embodiment of the present invention;

[0085] Figure 2 Schematic diagram of the training process of the adversarial patch localization method based on self-supervised learning and multi-view feature fusion in an embodiment of the present invention;

[0086] Figure 3 It is a structural diagram of a feature fusion module in an embodiment of the present invention;

[0087] Figure 4 Schematic diagram of the testing process of the adversarial patch localization method based on self-supervised learning and multi-view feature fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0088] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0089] A specific embodiment of the present invention, as Figure 1-4 , discloses an adversarial patch localization method based on self-supervised learning and multi-view feature fusion. In order to illustrate the effectiveness of the method proposed in the present invention, the above technical solution of the present invention is described in detail through a specific embodiment below. The specific implementation steps are as follows:

[0090] The present invention provides an adversarial patch localization method based on self-supervised learning and multi-view feature fusion, comprising:

[0091] S1. Build target detection image dataset and test image dataset;

[0092] The specific steps of constructing the data set in step S1 include:

[0093] S11, obtaining a plurality of target detection image data based on a downstream target detection task, and establishing a target detection image data set;

[0094] Furthermore, each target detection image data corresponds to annotation information respectively;

[0095] The annotation information includes the rectangular frame coordinates and classification labels of the object in the target detection image; it can be understood that the object is a target to be detected, for example, a person, a vehicle, etc. among the various targets in the COCO dataset;

[0096] S12, adding an adversarial patch to each target detection image in the target detection image dataset to obtain a test image dataset;

[0097] Furthermore, each of the test images corresponds to annotation information; the annotation information corresponding to each of the test images includes a mask of the adversarial patch, a rectangular frame coordinate of an object in the image, and a classification label;

[0098] S2, let i = 1, where i = 1, x i For the first target detection image, A i is the initial feature extraction backbone network, B i is the initial multi-scale fusion feature model, C i is the initial pixel classification convolution layer;

[0099] S3, select image x in the target detection image dataset i and a random image x' i ; The image x i Characterized as a clean image c i ; According to the image x i The object border annotation b obtains the clean image c i The corresponding patch mask m = f(b);

[0100] S4. Get a random image x' i Mask; Based on the random image x' i The mask of the clean image c in step S3 i The corresponding patch mask is used for data enhancement to obtain the training sample ^x after data enhancement i and the corresponding mask;

[0101] The training sample after data enhancement ^x i The expression is:

[0102] ^x i =xi *(1-m i )+x' i *m i

[0103] Preferably, the training sample ^x obtained after data enhancement in step S4 is i And the corresponding mask specific steps include:

[0104] Generate a random image x' using basic geometric shapes stacking i 's mask;

[0105] The clean image c i The corresponding patch mask includes multiple rectangular regions;

[0106] Based on the random image x' i The mask of the clean image c i Each rectangular area of ​​the corresponding patch mask is transformed to obtain the training sample ^x after data enhancement i and the corresponding mask.

[0107] S5, using the Laplace operator to enhance the training sample ^x after the data in step S4 i Perform high-pass filtering to obtain an image with edge features d i ;

[0108] Furthermore, in step S5, a Laplace operator is used for high-pass filtering, and edge detection is achieved by applying second-order partial derivatives in the image.

[0109] S6. Get image x i RGB image; based on feature extraction backbone network A i Get the view feature r of the RGB image i ; Based on the feature extraction backbone network, the view feature d′ of the image with edge features is obtained i and updating the feature extraction backbone network;

[0110] Preferably, the view feature r of the RGB image is obtained i and the view feature d′ of the image with edge features i The specific steps include:

[0111] Using feature extraction backbone network A i Extract image x separately i The features of the RGB image and the image with edge features d i Features;

[0112] The image x iThe features of the RGB image are downsampled and upsampled step by step to obtain feature maps r with multiple resolutions. i ;

[0113] The image d with edge features i The features of the d″ are obtained by downsampling and upsampling step by step to obtain a feature map d″ with multiple edge features i ;

[0114] The feature map r with multiple resolutions i and a feature map d″ with multiple edge features i ;

[0115] A skip connection is established between the downsampling layer and the upsampling layer at the same level to obtain the corresponding RGB image view feature r i and the view feature d′ of the image with edge features i And feature extraction backbone network.

[0116] Furthermore, in step S6, the feature extraction backbone network is an encoder-decoder structure with 4 layers of downsampling and 4 layers of upsampling; in the encoder-decoder structure, jump connections are set between encoders-decoders of the same level; the feature extraction backbone network includes an output layer, the output layer uses a single-category convolutional layer, and uses a Sigmoid activation function to limit the output value between 0 and 1.

[0117] S7, transforming the view feature r of the RGB image i and the view feature d′ of the image with edge features i After splicing, input the multi-scale fusion feature model B i Perform fusion processing to obtain the multi-scale fusion feature E i and update the multi-scale fusion feature model;

[0118] Preferably, the multi-scale fusion feature E obtained in step S6 is i The specific steps include:

[0119] The view feature r of the RGB image i and the view feature d′ of the image with edge features i Splice to obtain the spliced ​​multi-scale feature map e i ;

[0120] The spliced ​​multi-scale feature map e i Input multi-scale fusion feature model B i , based on multi-head deep convolution transposed attention to the concatenated multi-scale feature map e i Perform multi-scale cross-channel feature fusion to obtain the multi-scale fusion feature E iAnd update the multi-scale fusion feature model.

[0121] Furthermore, step S7 also includes obtaining transposed attention based on multi-head deep convolution, and the specific steps include:

[0122] Let the input tensor be Normalize the tensor X to get the normalized tensor

[0123] Extract the normalized tensor using a 1×1 regular convolution Pixel-level cross-channel context information to get updated normalized tensor;

[0124] Using a 3×3 depth-wise convolution to encode the spatial context information of the updated normalized tensor channel level, an initial query Q matrix, an initial key value K matrix and an initial value V matrix are obtained;

[0125] The initial query Q matrix, the initial key value K matrix and the initial value V matrix are transformed in dimension respectively to obtain multi-head deep convolution transposed attention.

[0126] Furthermore, the multi-head deep convolution transposed attention expression is:

[0127]

[0128] in, Transpose the output of the attention for the multi-head deep convolution, W p is a 1×1 convolution operation; α is a learnable parameter used to control the size of the dot product result of the K matrix and the Q matrix; Softmax(·) is the activation function; X represents the feature map of the input attention module.

[0129] S8, the multi-scale fusion feature E obtained in step S7 i After processing through the pixel classification convolution layer Ci, the predicted mask b is obtained i And the prediction mask b i loss;

[0130] Preferably, the prediction mask b in step S8 i The specific steps of loss include:

[0131] The multi-scale fusion feature E obtained in step S7 i In the input pixel classification convolution layer, the predicted mask b is obtained after processing based on the activation function i , calculate the predicted mask b i The loss of the patch mask obtained in S3;

[0132] Furthermore, the activation function is a Sigmoid function;

[0133] Preferably, the prediction mask b i The value of the middle mask ranges from 0 to 1, where the closer it is to 1, the greater the probability that the pixel is an adversarial patch;

[0134] Preferably, the loss expression in step S8 is:

[0135] L total,i =L BCE,i +λL DICE,i ;

[0136] Among them, λ is a hyperparameter used to adjust the weight of Dice loss; L total,i is the loss of the i-th image, L BCE,i is the cross entropy loss of the i-th image, L DICE,i is the Dice loss of the i-th image.

[0137] Furthermore, the cross entropy loss expression is:

[0138]

[0139] Among them, y i is the patch mask of the ith image, x i is the probability that the predicted mask of the i-th image is a patch mask. The cross entropy loss is calculated for the classification results of n images and the average is taken.

[0140] Furthermore, the Dice loss function L DICE,i The expression is:

[0141]

[0142] Among them, x i,j is the predicted mask of pixel j in the i-th image, y i,j is the patch mask of pixel j of the ith image; j=1,2,3...J, where J is the total number of pixels.

[0143] S9, compare the size of i and I, I represents the total number of target detection images, if i < I, add 1 to the value i, and return to step S3, if i = I, then end the training, and obtain an updated feature extraction backbone network, an updated multi-scale fusion feature model, and an updated pixel classification convolution layer;

[0144] Further, in order to verify the effectiveness and practicality of the present invention, the COCO sub-dataset is used as the training data set (10,000 images), the model is trained according to steps S1 to S9, RMSprop is used as the optimizer of the model, the learning rate is set to 0.00001, 80% of the training data is used to train the model, 10% is used as the verification model, and 10% is used as the test model. A total of 20 iterations are trained, and finally the model with the best evaluation index of the verification set is saved as the final result;

[0145] The COCO sub-dataset is used to evaluate the model. The evaluation data set contains 1000 samples with complex adversarial patch attacks added. The trained model is used to perform evaluation according to the above step S9 and compared with the true label. The mAP index of the COCO dataset is 46.5 when no adversarial patches are added, and the mAP index of the COCO dataset after adding the adversarial patches is 18.9. The pixel-level F1 value of the adversarial patch positioning mask is 0.917. After combining the adversarial patch removal method, the mAP index of the COCO dataset is 39.5, which is a good result, indicating that the present invention is effective and feasible.

[0146] S10, constructing a patch localization model based on the trained feature extraction backbone network, the trained multi-scale fusion feature model and the trained pixel classification convolutional layer; and performing patch localization based on the patch localization model.

[0147] The present invention also provides a method for positioning and removing an adversarial patch, comprising:

[0148] S11, constructing a patch localization model based on updating the feature extraction backbone network, updating the multi-scale fusion feature model and updating the pixel classification convolution layer;

[0149] The patch positioning model is updated to obtain a patch positioning removal model, and the image to be tested is input into the patch removal model to perform adversarial patch removal to obtain an image after the adversarial patch is removed.

[0150] Preferably, the specific steps of obtaining the image after the adversarial patch is removed include:

[0151] S111, constructing a patch localization model based on updating the feature extraction backbone network, updating the multi-scale fusion feature model and updating the pixel classification convolution layer;

[0152] S112, input the image to be tested into the adversarial patch positioning model to obtain an initial prediction mask b′ i ;

[0153] S113, designing a mask completion module and an image reconstruction module; embedding the mask completion module and the image reconstruction module into the patch positioning model to obtain a patch positioning removal model;

[0154] S114, the initial prediction mask b′ i Input the patch positioning removal model, and obtain the completed prediction mask b based on the mask completion module i ;

[0155] The completion prediction mask b i The corresponding area is characterized as the patch removal area;

[0156] S115, the patch removal area uses an image reconstruction module to obtain the original image area to be tested; based on the original image area to be tested, the texture of the patch removal area is removed to obtain an image after the adversarial patch is removed.

[0157] Furthermore, the completed prediction mask b obtained in step S114 is i The specific steps include:

[0158] S1141, obtain test image A i The original mask M o ; Preset the window width w and height h, and preset the adversarial patch mask threshold z;

[0159] S1142, the original mask M o Input to the adversarial patch mask completion module, initialized with the original mask M o Array M of zero values ​​of the same size c ;

[0160] S1143, the original mask M o comprising a plurality of pixels; each h×w window comprising a plurality of pixel values;

[0161] S1144, setting a window of size h×w at the center of pixel p, and calculating the sum of multiple pixel values ​​in the window;

[0162] S1145, comparing the sum of the pixel values ​​in the window with the adversarial patch mask threshold z, if the sum of the pixel values ​​in the window is greater than the adversarial patch mask threshold z, then c The value of is set to 255; if it is less than the adversarial patch mask threshold z, the value of pixel p is not modified;

[0163] S1146, repeat steps S1144-1145 until the original mask M is traversed o Multiple pixels in the image are used to obtain the completed prediction mask b. i .

[0164] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An adversarial patch localization method based on self-supervised learning and multi-view feature fusion, characterized in that: include: S1. Build target detection image dataset and test image dataset; S2, let i = 1, where i = 1, x i For the first target detection image, A i is the initial feature extraction backbone network, B i is the initial multi-scale fusion feature model, C i is the initial pixel classification convolution layer; S3, select image x in the target detection image dataset i and a random image x' i ; The image x i Characterized as a clean image c i ; According to the image x i The object border annotation b obtains the clean image c i The corresponding patch mask m i ; S4. Get a random image x' i Mask; Based on the random image x' i The mask of the clean image c in step S3 i The corresponding patch mask is used for data enhancement to obtain the training sample ^x after data enhancement i and the corresponding mask; S5, obtain the training sample ^x after the data enhancement in step S4 i The image with edge features d i ; S6. Get image x i RGB image; the image d with edge features i and image x i RGB image input feature extraction backbone network A i Get the view feature r of the RGB image i and the view feature d′ of the image with edge features i And update the feature extraction backbone network A i ; S7, transforming the view feature r of the RGB image i and the view feature d′ of the image with edge features i After splicing, input the multi-scale fusion feature model B i , and obtain the multi-scale fusion feature E i And update the multi-scale fusion feature model B i ; S8, the multi-scale fusion feature E obtained in step S7 i Input pixel classification convolution layer C i After processing, the predicted mask b is obtained i , loss, and update pixel classification convolution layer C i ; S9, compare the size of i and I, I represents the total number of target detection images, if i < I, add 1 to the value i, and return to step S3, if i = I, then end the training and obtain the updated feature extraction backbone network A i , Update the multi-scale fusion feature model B i and update the pixel classification convolutional layer C i ; S10, constructing a patch localization model based on updating the feature extraction backbone network, updating the multi-scale fusion feature model and updating the pixel classification convolution layer; Patch positioning is performed based on the positioning patch model.

2. The method for locating an adversarial patch according to claim 1, characterized in that: The specific steps of constructing the data set in step S1 include: S11, obtaining a plurality of target detection image data based on the target detection task, and establishing a target detection image data set; S12. Add an adversarial patch to each target detection image in the target detection image dataset to obtain a test image dataset.

3. The method for locating an adversarial patch according to claim 1, characterized in that: The training sample after data enhancement ^x i The expression is: ^x i =x i *(1-m i )+x' i *m i 。 4. The method for locating an adversarial patch according to claim 1, characterized in that: Based on the feature extraction backbone network, the view feature r of the RGB image is obtained i and the view feature d′ of the image with edge features i The specific steps include: Use the feature extraction backbone network to extract the image x i The features of the RGB image and the image with edge features d i Features; The image x i The features of the RGB image are downsampled and upsampled step by step to obtain feature maps with multiple resolutions The image d with edge features i The features of the d″ are obtained by downsampling and upsampling step by step to obtain a feature map d″ with multiple edge features i ; The feature maps with multiple resolutions and a feature map d″ with multiple edge features i A skip connection is established between the downsampling layer and the upsampling layer at the same level to obtain the corresponding RGB image view feature r i and the view feature d′ of the image with edge features i .

5. The method for locating an adversarial patch according to claim 1, characterized in that: Use the Laplace operator to perform the training sample ^x after the data enhancement in step S4 i Perform high-pass filtering to obtain an image with edge features d i .

6. The method for locating an adversarial patch according to claim 1, characterized in that: Step S7 obtains the multi-scale fusion feature E i The specific steps include: The view feature r of the RGB image i and the view feature d′ of the image with edge features i Splice to obtain the spliced ​​multi-scale feature map e i ; The spliced ​​multi-scale feature map e i Input multi-scale fusion feature model, based on multi-head deep convolution transposed attention to spliced ​​multi-scale feature map e i Perform multi-scale cross-channel feature fusion to obtain the multi-scale fusion feature E i .

7. The method for locating an adversarial patch according to claim 6, characterized in that: Step S6 also includes obtaining transposed attention based on multi-head deep convolution, and the specific steps include: Let the input tensor be Normalize the tensor X to get the normalized tensor Extract the normalized tensor Pixel-level cross-channel context information to get updated normalized tensor; Encoding the spatial context information of the updated normalized tensor channel level to obtain an initial query Q matrix, an initial key value K matrix and an initial value V matrix; After dimensionality transformation of the initial query Q matrix, the initial key value K matrix and the initial value V matrix respectively, the multi-head deep convolution transposed attention is obtained.

8. The method for locating an adversarial patch according to claim 1, characterized in that: The expression of the loss in step S8 is: THE total,i =L BCE,i +λL DICE，i ; Among them, L total,i is the loss of image i, L BCE,i is the cross entropy loss of the intersection image i, λ is a hyperparameter; L DICE,i is the Dice loss of image i.

9. The method for locating an adversarial patch according to any one of claims 1 to 8, characterized in that: The adversarial patch is removed using the positioning result of patch positioning based on the positioning patch model.

Images