A vehicle adversarial patch attack method based on model perception and attention
By optimizing the generation process of vehicle adversarial patches through discrete cosine transform and gradient-weighted class activation mapping, the problems of ignoring the physical world patch coverage location and poor mobility in existing technologies are solved, and the adversarial patches are effectively attacked in the physical world.
Patent Information
- Application Number
- CN202310885079.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-18
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-07-18
AI Technical Summary
Existing vehicle-based anti-patch attack methods ignore the impact of patch coverage location in the physical world and have poor mobility, making them ineffective at attacking other detectors.
High-frequency information is preserved by discrete cosine transform as an initial adversarial patch. Adversarial examples are generated by combining brightness, contrast, noise, and rotation transformations. Gradient-weighted class activation mapping is used to obtain the model's region of interest. The coverage position of the patch in the physical world is optimized. The model is trained using printing loss, change loss, and confidence loss functions.
This improves the feasibility and effectiveness of countermeasures patches from the digital world to the physical world, ensuring the effectiveness and portability of patches in the physical world.
Smart Images

Figure CN116994088B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of network security technology, specifically relating to a vehicle adversarial patch attack method based on model perception and attention. Background Technology
[0002] Artificial intelligence (AI) technology is increasingly being applied across various fields, and it will undoubtedly play a crucial role in the future world. Deep learning, a key pillar of AI development, has been widely applied in recent years to areas such as image recognition, natural language processing, and speech recognition. While AI has brought convenience and possesses immense potential, it also presents numerous challenges. Research has revealed that deep learning models based on multi-layered neural networks exhibit certain vulnerabilities, and adversarial examples are a major source of threats to the integrity and usability of deep learning systems.
[0003] Adversarial examples are generated by adding specific perturbations to clean samples, causing classifiers to misclassify with high confidence. Currently, visible light adversarial example attacks can be divided into two main categories based on the size of the perturbation range: global subtle perturbations and local obvious perturbations. Global perturbations typically involve adding noise to the entire image, such as FGSM, PGD, and CW. The adversarial examples after adding noise are very similar to the original image, virtually indistinguishable to the human eye, but the neural network will give different judgment results. In adversarial patch attacks, attackers are no longer limited to imperceptible changes. Adversarial patch attacks generate a patch unrelated to the image. While the human eye can see the difference between the patched adversarial example and the original sample, this adversarial example can attack the neural network, causing it to output incorrect or misclassified target categories. Compared to global pixel-level adversarial examples, local adversarial patches do not pursue visual concealment, making it easier to achieve attack effects in the real world and causing damage to the physical world.
[0004] The existing countermeasures against patch attacks are as follows: Step 1: Obtain the vehicle dataset; Step 2: Initialize the anti-patch; Step 3: Modify the patch; Step 4: Design the loss function; Step 5: Add the transformed patch to the clean samples of the training dataset. Do not change the model during training; instead, optimize and update the patch according to the loss function. Step 6: Add the final adversarial patch to the clean sample to generate adversarial samples, causing the detector to make incorrect decisions. If physical factors were considered during the patch optimization process, the final adversarial patch can be applied to the physical target of the attack through printing or pasting, causing the model to make incorrect decisions and thus achieving the attack effect in the physical world.
[0005] In the above steps, the difference in loss function, the difference in initial patch, the size of the patch, and the location of its addition can all affect the final effectiveness of the anti-patch attack.
[0006] Disadvantages of existing technology: Existing adversarial patch attacks targeting vehicles mostly consider environmental changes such as lighting, but neglect the impact of patch coverage location in the physical world on attack effectiveness. Some methods have not undergone real-world patch attack experiments, and attack effectiveness testing remains confined to the digital world. Furthermore, the patch optimization process does not consider the attack's impact when transferred to other models.
[0007] In summary: (1) The impact of the patch coverage location in the physical world on the attack effect was ignored, or no patch attack experiments were conducted in the physical world; (2) Poor mobility, unable to effectively attack other detectors outside the patch generation model. Summary of the Invention
[0008] To overcome the shortcomings of existing technologies, this invention provides a vehicle adversarial patching attack method based on model perception and attention. The method involves acquiring a vehicle image dataset; selecting samples as pre-selected patches; processing the pre-selected patches using Discrete Cosine Transform (DCT) to retain their high-frequency information as initial adversarial patches; processing the initial adversarial patches; adding the transformed patches to the target bounding boxes in the training set to generate an adversarial sample dataset; sending the adversarial sample dataset to a vehicle target detector for training; and finally, after the model reaches convergence, obtaining the optimized patch as the final adversarial patch. This invention improves the feasibility of adversarial patches from the digital world to the physical world and ensures the attack effectiveness of adversarial patches in the physical world.
[0009] The technical solution adopted by this invention to solve its technical problem includes the following steps: Step 1: Obtain the vehicle image dataset; Step 2: Select samples that other models cannot detect correctly as pre-selected patches. The other models are detectors other than the attack patch generation model. Step 3: Use Discrete Cosine Transform (DCT) to process the pre-selected patch and retain its high-frequency information as the initial adversarial patch; Step 4: Adjust the brightness and contrast of the initial adversarial patch, add random noise, random rotation, and scale transformation. Step 5: Add the transformed patch to the region of the target box in the training set to generate the adversarial example dataset; Step 6: Send the adversarial example dataset to the vehicle target detector for training. During the training process, do not change the vehicle target detector, but optimize and update the patch according to the loss function. Step 7: After the model reaches the convergence condition, the optimized patch obtained is the final adversarial patch. Test the attack effect of the adversarial patch in the digital world. Step 8: Obtain the region of interest of the model for a vehicle at a specified angle in the physical world through gradient-weighted class activation mapping. This region is the coverage location of the adversarial patch in the physical world. Step 9: Print and paste the adversarial patch at the selected coverage area of the attack target. When the model detects the attack target at that angle, it will make an incorrect decision.
[0010] Furthermore, step 3 specifically includes: The pre-selected patch is subjected to a global discrete cosine transform. Based on the coefficient diagram, the low-frequency part is set to zero, while the high-frequency part is retained. Then, a global inverse discrete cosine transform is performed, and the high-frequency information is used as the initial adversarial patch.
[0011] Furthermore, step 4 specifically includes: When performing transformation operations on the initial adversarial patch, the brightness transformation range is [-0.1, 0.1], the contrast transformation range is [0.8, 1.2], the random noise factor is 0.1, the rotation transformation range is [-20, 20], and the patch scale adjustment factor is... ,in , These are the width and height of the vehicle's bounding box, respectively. This is the original size of the patch.
[0012] Furthermore, the loss function in step 6 is specifically as follows: The loss function includes printing loss. Change loss and confidence loss :
[0013] in, , As a weighting factor; Print loss function This reflects the extent to which patches are unprintable from the digital world to the physical world:
[0014] in, Indicates counter-patch P One pixel in , Represents a color in the printable color set C; Change loss function This reflects the smoothness of the patch:
[0015] in, Indicates the first Line 1 The pixel values of the column; Confidence loss function It reflects the probability that the detection frame contains the target.
[0016] Furthermore, the target detector is YOLOv2.
[0017] The beneficial effects of this invention are as follows: This invention takes into account environmental changes such as patch printing loss and illumination, and provides the coverage location of the patch in the physical world based on the model attention pattern. Based on the model's perception capability, it improves the portability of adversarial patches, enhances the feasibility of adversarial patches from the digital world to the physical world, and ensures the attack effect of adversarial patches in the physical world. Attached Figure Description
[0018] Figure 1 This is a framework diagram of the method of the present invention. Detailed Implementation
[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0020] This invention proposes a vehicle adversarial patch attack method based on model perception and attention. While considering patch printing loss, lighting and other environmental changes, it uses high-frequency information of images that the model cannot correctly classify as the initial adversarial patch based on the model's perception capability to improve the transfer attack power of the patch. Based on the model's attention pattern, it obtains the region of interest of the model for the image of the physical world attack object from a specific angle through gradient weighted class activation mapping. Covering the patch in this region makes it easier to achieve the attack, thus ensuring the attack effect of the adversarial patch in the physical world.
[0021] like Figure 1 As shown, a vehicle adversarial patch attack method based on model perception and attention is implemented through the following steps: Step 1: Obtain the vehicle image dataset; Step 2: Select samples that other models cannot detect correctly as pre-selected patches. Here, other models are detectors other than those generated by the attack patch. Step 3: Use Discrete Cosine Transform (DCT) to process the pre-selected patch and retain its high-frequency information as the initial adversarial patch; Step 4: Adjust the brightness and contrast of the initial adversarial patch, add random noise, random rotation, and scale transformation; Step 5: Add the transformed patch to the region of the target box in the training set to generate the adversarial example dataset; Step 6: Send the adversarial example dataset to the vehicle target detector for training. Do not change the model during training. Instead, optimize and update the patch according to the loss function. Step 7: After the model reaches the convergence condition, the resulting optimized patch is the final adversarial patch, which can then be used to test the attack effect of the adversarial patch in the digital world. Step 8: Obtain the region of interest of the model for a specific angle image of a vehicle in the physical world through gradient weighted activation mapping. This region is the coverage location of the adversarial patch in the physical world. Step 9: Print and paste the adversarial patch at the selected coverage area of the attack target. When the model detects the attack target at that angle, it will make an incorrect decision.
[0022] Step 3 specifically involves applying a global discrete cosine transform to the pre-selected patch, setting the low-frequency part to zero according to the coefficient diagram, retaining its high-frequency part, and then performing a global inverse discrete cosine transform to use the high-frequency information as the initial adversarial patch.
[0023] Step 4 specifically involves the following transformation operations on the normalized patch: Brightness transformation range is [-0.1, 0.1], contrast transformation range is [0.8, 1.2], random noise factor is 0.1, rotation transformation range is [-20, 20], and patch scale adjustment factor is... ,in , These are the width and height of the vehicle's bounding box, respectively. This is the original size of the patch.
[0024] Step 6 specifically involves the loss function, which includes print loss, change loss, and confidence loss.
[0025] in, , These are weighting factors determined empirically.
[0026] Print loss function This reflects the extent to which patches are unprintable from the digital world to the physical world:
[0027] in, This represents a pixel in the anti-patch P. Let C represent a color from the set of printable colors. Optimizing the printing loss function can make adversarial patches print more accurately, thereby improving their attack capabilities in the physical world.
[0028] Change loss function This reflects the smoothness of the patch:
[0029] in, Indicates the first Line 1 The column pixel values and the variation loss function ensure that the model prefers images with smooth color transitions and avoids noisy images. Optimizing the variation loss function makes the patches smoother and more natural, improving their robustness against attacks.
[0030] Confidence loss function This reflects the probability that the detection box contains a target. The purpose of optimizing the confidence loss function is to minimize the target confidence of the model output, so that the detector cannot detect the target, thereby achieving a targetless attack. Specific implementation examples: Step 1: Obtain the vehicle dataset, specifically: The dataset used is the BIT-Vehicle dataset, whose images are all top-down frontal views of road vehicles. Samples of the categories Sedan and SUV were selected and their labels were changed to Car to generate the BIT-Car dataset. 1830 images containing only complete vehicle targets were selected from the BIT-Car dataset as the dataset to be used, of which the training set contains 1280 images and the test set contains 550 images. Step 2: Select images that the Faster R-CNN detector cannot detect correctly as pre-patch; Step 3: Discrete Cosine Transform (DCT) adopts global transformation instead of block transformation. Global Discrete Cosine Transform is performed on the pre-selected patch. According to the coefficient diagram, the low-frequency part in the upper left corner is set to zero, while the high-frequency part is retained. Then, global inverse Discrete Cosine Transform is performed, and the high-frequency information is used as the initial adversarial patch. Step 4: Adjust the brightness and contrast of the initial adversarial patch, add random noise, random rotation, and scale transformation. Specifically: 4-1: Randomly initialize the brightness transformation vector, contrast transformation vector, and noise vector. The brightness transformation range is [-0.1, 0.1], the contrast transformation range is [0.8, 1.2], and the random noise factor is 0.1. The normalized patch, after brightness transformation, contrast transformation, and noise addition operations, is as follows:
[0032] Then, adjust the patch pixel values to between [0, 1] to ensure that the patch can be processed normally in the future; 4-2: The rotation transformation range is [-20, 20], and the patch scale adjustment factor is... ,in , These are the width and height of the vehicle's bounding box, respectively. The default size for the patch is 300. 300. After rotation transformation and scale adjustment, adjust the patch pixel value to between [0, 1] to ensure that the patch can be processed normally in the future.
[0033] Step 5: Add the transformed patch to the region of the target box in the training set to generate the adversarial example dataset; Step 6: Train the vehicle target detector using the adversarial example dataset. During training, do not change the model; instead, optimize and update the patches based on the loss function. Specifically: 6-1: The detector used is YOLOv2, which was trained on the MS COCO dataset and can detect 80 categories, including Car. 6-2: The loss function includes print loss, change loss, and confidence loss:
[0034] in, , The weighting factors are determined empirically. , .
[0035] 6-3: Print the loss function This reflects the extent to which patches are unprintable from the digital world to the physical world:
[0036] in, This represents a pixel in the anti-patch P. Let C represent a color from the set of printable colors. Optimizing the printing loss function can make adversarial patches print more accurately, thereby improving their attack capabilities in the physical world.
[0037] 6-4: Change Loss Function This reflects the smoothness of the patch:
[0038] in, Indicates the first Line 1 The column pixel values and the variation loss function ensure that the model prefers images with smooth color transitions and avoids noisy images. Optimizing the variation loss function makes the patches smoother and more natural, improving their robustness against attacks.
[0039] 6-5: Confidence Loss Function This reflects the probability that the detection box contains a target. The purpose of optimizing the confidence loss function is to minimize the target confidence of the model output, so that the detector cannot detect the target, thereby achieving a targetless attack.
[0040] Step 7: After the model reaches convergence, the resulting optimized patch is the final adversarial patch. The effectiveness of this adversarial patch in the digital world can then be tested. Specifically: Adversarial examples on the test set caused YOLOv2's detection accuracy to drop from 80.99% to 17.84%. Step 8: Obtain the region of interest of the model for a specific angle of the image of the physical world attack object through gradient weighted class activation mapping. This region is the coverage location of the adversarial patch in the physical world. Step 9: Print and paste the adversarial patch at the selected coverage area of the attack target. When the model detects the attack target at this angle, it will make an incorrect decision, specifically: At a specific angle in Step 8, keeping the angle, distance, lighting, and shooting pixels unchanged, vehicle images before and after the adversarial patch are captured and sent to the model for detection. Faster R-CNN detects the car with confidence from 98% for clean samples to 47% for adversarial samples. YOLOv2 detects the clean samples as car and the adversarial samples as bowl.
[0041] This scheme should only be used under legal circumstances.
Claims
1. A vehicle adversarial patch attack method based on model perception and attention, characterized in that, Includes the following steps: Step 1: Obtain the vehicle image dataset; Step 2: Select samples that other models cannot detect correctly as pre-selected patches. The other models are detectors other than the attack patch generation model. Step 3: Use Discrete Cosine Transform (DCT) to process the pre-selected patch and retain its high-frequency information as the initial adversarial patch; Step 4: Adjust the brightness and contrast of the initial adversarial patch, add random noise, random rotation, and scale transformation; Step 5: Add the transformed patch to the region of the target box in the training set to generate the adversarial example dataset; Step 6: Send the adversarial example dataset to the vehicle target detector for training. During the training process, do not change the vehicle target detector, but optimize and update the patch according to the loss function. Step 7: After the model reaches the convergence condition, the optimized patch obtained is the final adversarial patch. Test the attack effect of the adversarial patch in the digital world. Step 8: Obtain the region of interest of the model for a vehicle at a specified angle in the physical world through gradient-weighted class activation mapping. This region is the coverage location of the adversarial patch in the physical world. Step 9: Print and paste the adversarial patch at the selected coverage area of the attack target. When the model detects the attack target at that angle, it will make an incorrect decision.
2. The vehicle adversarial patching attack method based on model perception and attention as described in claim 1, characterized in that, Step 3 specifically involves: The pre-selected patch is subjected to a global discrete cosine transform. Based on the coefficient diagram, the low-frequency part is set to zero, while the high-frequency part is retained. Then, a global inverse discrete cosine transform is performed, and the high-frequency information is used as the initial adversarial patch.
3. The vehicle adversarial patch attack method based on model perception and attention according to claim 2, characterized in that, Step 4 specifically involves: When performing transformation operations on the initial adversarial patch, the brightness transformation range is [-0.1, 0.1], the contrast transformation range is [0.8, 1.2], the random noise factor is 0.1, the rotation transformation range is [-20, 20], and the patch scale adjustment factor is... ,in , These are the width and height of the vehicle's bounding box, respectively. This is the original size of the patch.
4. The vehicle adversarial patch attack method based on model perception and attention as described in claim 3, characterized in that, The loss function in step 6 is as follows: The loss function includes printing loss. Change loss and confidence loss : in, , As a weighting factor; Print loss function This reflects the extent to which patches are unprintable from the digital world to the physical world: in, Indicates counter-patch P One pixel in , This represents a color from the printable color set C; Change loss function This reflects the smoothness of the patch: in, Indicates the first Line 1 The pixel values of the column; Confidence loss function It reflects the probability that the detection frame contains the target.
5. The vehicle adversarial patch attack method based on model perception and attention according to claim 1, characterized in that, The target detector is YOLOv2.
