Positioning decoupling background adversarial patch generation method

Abstract

The invention belongs to the field of computer vision, and relates to a positioning and decoupling background adversarial patch generation method, which comprises the following steps: S1, preprocessing a clean image data set to obtain target annotation information and a non-target mask of a clean image; S2, training a target detection agent model according to the clean image and the target annotation information, and randomly initializing an adversarial patch; s3, generating a shape mask according to the adversarial patch, and generating an effective patch positioning array according to the shape mask and the target labeling information; s4, generating an effective multi-patch image according to the non-target mask, the effective patch positioning array and the shape mask; s5, generating an adversarial image according to the effective multi-patch image; inputting the confrontation image into the target detection agent model to obtain a prediction result; s6, calculating loss and updating an adversarial patch; and S7, circularly executing the steps S3-S6 until a final adversarial patch is obtained. According to the method, the positioning association between the adversarial patch and the target is decoupled, and the unknown property and aggressiveness of the adversarial patch are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to a method for generating background adversarial patches with localization decoupling. Background Technology

[0002] As a fundamental computer vision task, object detection has been extensively studied and applied in real-world scenarios. Especially with the rapid development of deep learning, object detection proxy models based on convolutional neural networks have continuously surpassed traditional machine learning detection methods, achieving highly accurate results. In recent years, the Transformer architecture has received considerable attention, leading to a surge in research and further advancements in object detection.

[0003] However, rapid development brings with it potential threats. Current research has shown that, through careful design, even minor modifications to images, without interfering with human vision, can degrade detector performance or even render them ineffective. These modified images are called adversarial images, and their existence poses new challenges and threats to the robustness and security of object detection. Therefore, adversarial robustness, i.e., the model's ability to correctly handle adversarial examples, has become one of the key indicators for evaluating the performance of object detection proxy models. Unlike the initial adversarial noise, many current studies are beginning to focus on adversarial patches. This adversarial approach generates special patches and then places them in images to produce an attack effect. Adversarial patches have appeared in many fields such as face detection, object recognition, and remote sensing detection. Among these, remote sensing detection, as a widely used remote sensing technology, has received considerable attention for its robustness and security.

[0004] Adversarial patches targeting remote sensing detection have achieved considerable success, but current methods typically place the patch on top of the target or in the background near the target. This localized coupling between the patch and the target hinders the patch's effectiveness. Furthermore, patches are mostly presented as fixed rectangles or combinations of rectangles, which may appear unnatural in certain scenarios. Summary of the Invention

[0005] To address the aforementioned problems in the prior art, this invention employs a background adversarial patch generation method with location decoupling, comprising:

[0006] S1: Obtain a clean image dataset, preprocess each clean image in the dataset to obtain the target annotation information and non-target mask image for each clean image:

[0007] S2: Obtain the object detection proxy model. Train the object detection proxy model using each clean image and its object annotation information in the clean image dataset to obtain the trained object detection proxy model; randomly initialize the adversarial patch;

[0008] S3: Generate a shape mask image based on the adversarial patch, select a clean image from the clean image dataset, and generate an effective patch localization array for the clean image based on the shape mask image and the target annotation information of the clean image;

[0009] S4: Generate a valid multi-patch image of the clean image based on the non-target mask image of the clean image, the valid patch location array, and the shape mask image;

[0010] S5: Generate an adversarial image of the clean image based on the effective multi-patch images of the clean image; input the adversarial image into the trained object detection proxy model to obtain the prediction result of the adversarial image;

[0011] S6: Calculate the loss based on the prediction results of the adversarial patch, the effective multi-patch image, and the adversarial image, and update the adversarial patch based on the loss;

[0012] S7: Repeat steps S3-S6 until the predetermined number of iterations is reached, and the final anti-patch is obtained.

[0013] The beneficial effects of this invention are:

[0014] To address the issue that the location coupling between patches and targets hinders the effectiveness of patch attacks, this invention utilizes target annotation information from a clean image to calculate the base size of the adversarial patch, calculates the patch spacing in the clean image based on the base size of the adversarial patch, and then calculates the patch spacing based on the patch spacing. The number of adversarial patches that can be placed in the clean image in the horizontal and vertical directions is calculated. The coordinates of the adversarial patches are randomly initialized. Based on the initialized adversarial patch coordinates, patch spacing, and patch quantity, the coordinates of each adversarial patch in the clean image are generated. The minimum distance between each adversarial patch in the clean image and the bounding boxes of all targets is calculated. Adversarial patches with a minimum distance less than the preset effective placement range are selected, thereby decoupling the positioning association between the adversarial patch and the attack target, so that the positioning of the patch is not limited by the attack target, and the unknownness and attackability of the patch are improved. Attached Figure Description

[0015] Figure 1 A flowchart of a background adversarial patch generation method for positioning decoupling provided in an embodiment of the present invention;

[0016] Figure 2 Example diagrams of effective multi-patch images and adversarial images provided in embodiments of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 , Figure 2 As shown, this embodiment of the invention employs a background adversarial patch generation method with location decoupling, comprising:

[0019] S1: Obtain a clean image dataset, preprocess each clean image in the clean image dataset to obtain the target annotation information and non-target mask image of each clean image;

[0020] Preprocessing of clean images in the image dataset includes:

[0021] S11: Annotate the clean image to obtain the annotation information for each target in the clean image; the target annotation information includes: the target's bounding box and the ground truth class; the coordinates of the bounding box include the coordinates of the top left corner. lower right corner coordinates , where i is the index of the target;

[0022] S12: Segment and label the clean image, and convert the segmented and labeled clean image into a target mask image;

[0023] Segmentation annotation refers to the annotation of the target edge. The target mask image is a binary mask, that is, the pixel value of the target area is 1, and the value of the other pixels is 0.

[0024] In one embodiment, a saliency detector can also be used to process the clean image and directly output the target mask image of the clean image.

[0025] S13: Invert the target mask image of the clean image, that is, set the target region pixels to 0 and the other pixels to 1, to obtain the non-target mask image of the clean image.

[0026] S2: Obtain the object detection proxy model. Train the object detection proxy model using each clean image in the image dataset and its object annotation information to obtain the trained object detection proxy model; randomly initialize the adversarial patch;

[0027] S3: Generate a shape mask image based on the adversarial patch, select a clean image from the clean image dataset, and generate an effective patch localization array for the clean image based on the shape mask image and the target annotation information of the clean image;

[0028] The shape mask image is a binary mask, where the area with a pixel value of 1 is the effective area of ​​the adversarial patch, i.e., the shape of the patch, and the area with a value of 0 is the invalid area of ​​the adversarial patch.

[0029] The effective patch location array for calculating a clean image includes:

[0030] S31: Calculate the base size of the adversarial patch based on the target annotation information of the clean image;

[0031] The formula is:

[0032]

[0033] in, Indicates the base size of the anti-patch. Indicates the number of targets in a clean image. and These represent the first and second parts of the clean image, respectively. The width and height of the label box for each target. To adjust the parameters of the patch base size and the target annotation box size.

[0034] Preferably, parameters Set it to 0.8.

[0035] S32: Based on the base size of the adversary patch Calculate the patch spacing of a clean image According to the patch spacing Calculate the number of adversarial patches that can be placed in the clean image in the horizontal and vertical directions. , ;

[0036] The specific formula is as follows:

[0037]

[0038] in, This represents the coefficient between the patch spacing and the patch size. and These represent the width and height of the clean image, respectively. This indicates that the floor function is being performed.

[0039] Preferably, parameters Set it to 4.

[0040] S33: Randomly initialize the coordinates of the adversary patches, based on the initialized adversary patch coordinates and patch spacing. Number of patches and the number of patches The coordinates of each adversarial patch in the generated clean image are obtained, resulting in a patch location array. , ;

[0041] The specific formula is as follows:

[0042]

[0043] in, Represents the final coordinates of each adversarial patch on the clean image. This indicates the location of the patch at the top left corner of the clean image, i.e., the coordinates of the adversarial patch that are randomly initialized.

[0044] S34: Calculate the effective placement range of the adversarial patch based on the base size of the adversarial patch, calculate the minimum distance between each adversarial patch in the clean image and the bounding boxes of all targets, select the adversarial patch whose minimum distance is less than the effective placement range, and obtain the effective patch location array of the clean image.

[0045] The following are adversarial patches selected where the minimum distance is less than the effective placement range:

[0046]

[0047] in, Indicates the effective placement range of the anti-patch. For the effective placement range coefficient, This indicates taking the minimum value, which is used to determine the patch coordinates in the above formula. The minimum distance to any target in the corresponding clean image. Let be the coordinates of the bounding box of the i-th target in the clean image.

[0048] Preferably, the coefficient Set it to 5.

[0049] S4: Generate a valid multi-patch image of the clean image based on the non-target mask image of the clean image, the valid patch location array, and the shape mask image;

[0050] Valid multi-patch images for generating clean images include:

[0051] S41: Clone the corresponding number of adversarial patches according to the effective patch location array of the clean image to obtain the patch array P of the clean image;

[0052] S42: Randomly adjust the contrast and brightness of each adversarial patch in the patch array P, and add zero-mean Gaussian noise to obtain a clean image-processed patch array. ;

[0053] The specific formula is as follows:

[0054]

[0055] in, and These represent the patch arrays before and after processing, respectively. , and These represent contrast adjustment, brightness adjustment, and Gaussian noise, respectively. This represents element-wise multiplication.

[0056] S43: Array of patches after cleaning the image Each adversarial patch in the image is element-wise multiplied with the shape mask, thereby cropping each adversarial patch to the desired shape, resulting in a clean image with a fixed-shape patch array. ;

[0057] S44: Initialize an image where all pixels have a value of 0. Based on the patch positioning array of the clean image, each adversarial patch in the patch array P of the clean image is placed in the image. In the process, a clean image is obtained from multiple patch images;

[0058] In one embodiment, the size of each patch in a multi-patch image is adjusted based on the base size of the patch to further enhance patch robustness.

[0059] S45: Perform element-wise multiplication between the non-target mask image of the clean image and the multi-patch image, remove the areas in the multi-patch image that should be occluded by the target, and obtain the effective multi-patch image of the clean image.

[0060] S5: Generate an adversarial image for each clean image based on the effective multi-patch images of the clean image, input the adversarial image of the clean image into the trained object detection proxy model, and obtain the prediction result of the adversarial image of the clean image.

[0061] The prediction results for adversarial images include: the predicted bounding box, the predicted category of the predicted bounding box, and its confidence level;

[0062] Generating adversarial images from valid multi-patch images of a clean image includes:

[0063] S51: Binarize the effective multi-patch image of the clean image to obtain the effective multi-patch mask image of the clean image;

[0064] S52: Invert the effective multi-patch mask image of the clean image to obtain the non-patch mask image of the clean image;

[0065] S53: Perform element-wise multiplication of the clean image with its unpatched mask image, and then perform element-wise addition with the valid multi-patched image to obtain the adversarial image of the clean image.

[0066] The specific formula is as follows:

[0067]

[0068] Where X represents a clean image. Represents adversarial images, and These represent valid multi-patch images and valid multi-patch mask images, respectively. This represents element addition. This represents element-wise multiplication.

[0069] S6: Calculate the loss based on the adversarial patch, the adversarial image, and the prediction results of the adversarial image; update the adversarial patch through backpropagation based on the loss.

[0070] loss ;in, To counteract the confidence loss in the prediction results of the image, To combat image quality loss, For the total variational loss, To combat the unprintable loss of patches, , Hyperparameters are used to balance various losses.

[0071] Confidence loss for predictions of adversarial images:

[0072]

[0073] in, This indicates the number of predicted boxes whose confidence level is higher than a preset confidence threshold (used to determine which predicted boxes should be retained; its value can be 0.4, 0.5, etc.). This represents the confidence level of the predicted category for prediction box k.

[0074] Combating image quality loss:

[0075]

[0076] in, This indicates the calculation of the NIQE score (Natural Image Quality Evaluation Score), which assesses the natural quality of an image, making adversarial images appear relatively natural and less prone to detecting patches.

[0077] To avoid failure in the physical world, the patch's color cannot have overly drastic changes, but should be relatively gradual; furthermore, printing devices cannot print all colors, so the patch's color needs to be limited to ensure that the patch is generated in the physical world. Therefore, a total variational loss against the patch and an unprintable loss are added:

[0078]

[0079]

[0080] in, and These represent the width and height of the anti-patch, respectively. Indicates the anti-patch Okay, number Column pixel values, Represents a physically printable color set. One of the color elements within;

[0081] The pixel values ​​of the adversarial patch are updated through backpropagation based on the loss, including:

[0082]

[0083]

[0084] in, Indicates the first The anti-patch for the wheel, This represents the gradient of the loss with respect to the adversarial patch. This is the learning rate.

[0085] S7: Repeat steps S3-S6 until the predetermined number of iterations is reached, and the final anti-patch is obtained.

[0086] Using the adversarial patch generated by this invention can further test the adversarial robustness of the target detection proxy model, providing fundamental yet crucial technical support for the study of the robustness of target detection. In addition, adding the adversarial image generated by this invention to the training of the target detector can improve the robustness of the target detection model through adversarial training.

[0087] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating background adversarial patches with location decoupling, characterized in that, include: S1: Obtain a clean image dataset, preprocess each clean image in the dataset to obtain the target annotation information and non-target mask image for each clean image: S2: Obtain the object detection proxy model. Train the object detection proxy model using each clean image and its object annotation information in the clean image dataset to obtain the trained object detection proxy model; randomly initialize the adversarial patch; S3: Generate a shape mask image based on the adversarial patch, select a clean image from the clean image dataset, and generate an effective patch localization array for the clean image based on the shape mask image and the target annotation information of the clean image; S4: Generate a valid multi-patch image of the clean image based on the non-target mask image of the clean image, the valid patch location array, and the shape mask image; S5: Generate an adversarial image of the clean image based on the effective multi-patch images of the clean image; input the adversarial image into the trained object detection proxy model to obtain the prediction result of the adversarial image; S6: Calculate the loss based on the adversarial patch, the adversarial image, and the prediction results of the adversarial image, and update the adversarial patch based on the loss; S7: Repeat steps S3-S6 until the predetermined number of iterations is reached, and the final anti-patch is obtained.

2. The background adversarial patch generation method for location decoupling according to claim 1, characterized in that, Preprocessing of clean images includes: S11: Annotate the clean image to obtain the annotation information for each target in the clean image; the annotation information for the target includes: the target's bounding box and the true class; S12: Segment and label the clean image, and convert the segmented and labeled clean image into a target mask image; S13: Invert the target mask image of the clean image to obtain the non-target mask image of the clean image.

3. The background adversarial patch generation method for location decoupling according to claim 1, characterized in that, The target annotation information of a clean image includes the bounding box of each target in the clean image; The effective patch location array for calculating a clean image includes: S31: Calculate the base size of the adversarial patch based on the target annotation information of the clean image; S32: Calculate the patch spacing of the clean image based on the base size of the adversarial patch. According to the patch spacing Calculate the number of adversarial patches that can be placed in the clean image in the horizontal and vertical directions. , ; S33: Randomly initialize the coordinates of the adversary patches, based on the initialized adversary patch coordinates and patch spacing. Number of patches and the number of patches The coordinates of each adversarial patch used to generate a clean image; S34: Calculate the effective placement range of the adversarial patch based on the base size of the adversarial patch, calculate the minimum distance between each adversarial patch in the clean image and the bounding boxes of all targets, select the adversarial patch whose minimum distance is less than the effective placement range, and obtain the effective patch location array of the clean image.

4. The background adversarial patch generation method for positioning decoupling according to claim 3, characterized in that, Calculate the base size of the adversarial patch include: in, Indicates the number of objects in a clean image. and These represent the first and second parts of the clean image, respectively. The width and height of the label box for each target. To adjust the parameters of the patch base size and the target annotation box size.

5. The method for generating a background adversarial patch for location decoupling according to claim 3, characterized in that, Coordinates of each adversarial patch used to generate a clean image include: in, Represents the coordinates of a randomly initialized adversarial patch, where j and k are the indices of adversarial patches that can be placed in the clean image in the horizontal and vertical directions, respectively.

6. The background adversarial patch generation method for positioning decoupling according to claim 3, characterized in that, Calculating the minimum distance between each adversarial patch in the clean image and the bounding boxes of all targets includes: in, Indicates the effective placement range of the anti-patch. For the effective placement range coefficient, To counter the base size of the patch, This indicates taking the minimum value. The coordinates of the adversarial patch representing the clean image. Let be the coordinates of the bounding box of the i-th target in the clean image, and j and k be the indices of the adversarial patches that can be placed in the clean image in the horizontal and vertical directions, respectively.

7. The method for generating a background adversarial patch for location decoupling according to claim 1, characterized in that, Valid multi-patch images for generating clean images include: S41: Clone the corresponding number of adversarial patches based on the effective patch location array of the clean image to obtain the patch array; S42: Randomly adjust the contrast and brightness of each adversarial patch in the patch array, and add zero-mean Gaussian noise to obtain the processed patch array; S43: Perform element-wise multiplication between each adversarial patch in the processed patch array and the shape mask to obtain a patch array with a fixed shape; S44: Initialize an image where all pixels have a value of 0. Based on the patch positioning array, each adversarial patch in the fixed-shape patch array is placed on the image. In the process, multiple patch images are obtained; S45: Perform element-wise multiplication between the non-target mask image and the multi-patch image to obtain the valid multi-patch image.

8. The method for generating a background adversarial patch for location decoupling according to claim 1, characterized in that, Adversarial images that generate clean images include: S51: Binarize the effective multi-patch image of the clean image to obtain the effective multi-patch mask image; S52: Invert the valid multi-patch mask image to obtain the non-patch mask image of the clean image; S53: Perform element-wise multiplication of the clean image with its unpatched mask image, and then perform element-wise addition with the valid multi-patched image to obtain the adversarial image of the clean image.

9. The method for generating a background adversarial patch for location decoupling according to claim 1, characterized in that, loss ;in, To combat the confidence loss in the prediction results of the image, To combat image quality loss, For the total variational loss, To combat the unprintable loss of patches, , Hyperparameters are used to balance various losses.

10. The method for generating a background adversarial patch for location decoupling according to claim 9, characterized in that, The quality loss of the adversarial image is: in, This indicates the calculation of a natural image quality score. To counteract images.

Images