A physical adversarial attack method for drone target detection system
By using pre-trained generative adversarial network and YOLOv5 target detector to optimize adversarial patches, the problem of reducing recognition accuracy of the drone aerial target detection system under adversarial sample attacks is solved, and effective physical adversarial attacks are achieved in high-altitude aerial photography scenarios are improved, and the robustness and naturalness of the attacks are improved.
Patent Information
- Application Number
- CN202310084989.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-01
- Publication Date
- 2025-05-06
- Estimated Expiration
- 2043-02-01
AI Technical Summary
UAV aerial target detection systems are vulnerable to attacks when facing adversarial samples, resulting in a reduced recognition accuracy. Due to factors such as weather and long-distance imaging, physical adversarial attacks are less studied and more difficult.
The pre-trained generative adversarial network is used to generate fixed-size adversarial patches, and optimize and enhance it through the YOLOv5 target detector and patch conversion module to generate adversarial samples that can effectively attack in drone high-altitude aerial photography scenarios.
Under the influence of atmospheric factors and the altitude of the drone, the generated adversarial patch can effectively avoid drone target detection, improving the robustness and nature of the adversarial attack, making the attack difficult to be identified by the observer.
Smart Images
Figure CN116091462B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence security, and in particular relates to a physical confrontation attack method for an unmanned aerial vehicle target detection system. Background Art
[0002] UAV aerial target detection has been a research hotspot in the field of computer vision in recent years, and it is also a difficult point in the field of target detection. It uses drones equipped with cameras to take aerial photos and collect image data in real time, and then identifies and locates the target ground objects in the image. It has been widely used in many fields such as agricultural irrigation and military reconnaissance. At present, most of the algorithms related to UAV aerial target detection use deep learning technology, which has achieved outstanding results in many fields such as image classification, target detection, face recognition, and autonomous driving.
[0003] Recent studies have shown that deep neural networks are vulnerable to adversarial examples. Adversarial examples are samples obtained by adding some carefully designed and imperceptible perturbations to the input image, which can induce the deep neural network to output incorrect prediction results. Physical adversarial attacks refer to inserting adversarial examples into the physical world to cause errors in the reasoning of deep neural networks. Due to the widespread application of deep learning technology in the physical world, adversarial attacks in the physical world are more meaningful.
[0004] In the UAV aerial target detection scenario, extremely high recognition accuracy is required. If attackers use adversarial attack techniques in real scenarios to make ground targets evade detection in some scenarios, serious consequences may occur. Therefore, applying deep neural network models to environments with strict security requirements and dealing with model vulnerabilities caused by adversarial samples has become an important task and a hot topic in the current research of deep learning security. At the same time, due to the influence of factors such as changeable weather, long-distance imaging, and distortion of the equipment imaging process, there is little research on physical adversarial attacks on UAV aerial target detection systems, and the difficulty is relatively high. Summary of the invention
[0005] To solve the above problems, the present invention provides a physical adversarial attack method for a drone target detection system, using a high-definition picture set of a drone aerial vehicle as a training data set, and a target detection algorithm using a YOLOv5 network, including the following steps:
[0006] S1. Use a pre-trained generative adversarial network to generate adversarial patches of fixed size;
[0007] S2. Preprocessing the training samples in the training data set;
[0008] S3. Input the batch of preprocessed training samples into the Yolov5 target detector to obtain the label file corresponding to the training sample, which includes the location information and category label information of all vehicles in the training sample;
[0009] S4. Input the adversarial patch, training sample and its corresponding label file into the patch conversion module to obtain the adversarial sample P-Images;
[0010] S5. Input the adversarial sample P-Images into the Yolov5 target detector and use the overall loss function L all Calculate overall losses;
[0011] S6. Design a target optimization function based on the overall loss calculated in step S5, and use the Adam optimizer to update the adversarial patch;
[0012] S7. Repeat steps S3-S6 until the number of iterations reaches a preset value or the loss function converges;
[0013] S8. Print out the trained adversarial patch and set it on the roof of the vehicle to perform a physical adversarial attack.
[0014] Furthermore, the generative adversarial network is a StyleGAN2 network pre-trained on ImageNet, and the size of its output adversarial patch is 300×300.
[0015] Furthermore, the patch conversion module is used to physically enhance the adversarial patch. The specific operation process includes:
[0016] S41. Performing a first processing on the adversarial patch, the first processing includes scaling, rotating, adding Gaussian noise, adding brightness, and changing contrast;
[0017] S42. Generate a MASK matrix according to the label file corresponding to the training sample, and the MASK matrix determines the shape and position of the added adversarial patch;
[0018] S43. Add the adversarial sample after the first processing to the training sample through the MASK matrix to obtain a training patch sample;
[0019] S44. Perform a second processing on the training patch sample to obtain an adversarial sample. The second processing includes adding motion blur, adding ISO device noise, adding optical distortion, and adding weather changes.
[0020] Furthermore, in step S43, an adaptive patch placement function is used, and the adaptive patch placement function can adaptively limit the adversarial patch to the range of the label box Ground Truth, which is expressed as:
[0021] S pnew =ε h ×S pori
[0022] Among them, S pnew is the scaled patch size, ε h Indicates the scaling factor when the shooting height is h, S pori is the original size of the adversarial patch.
[0023] Furthermore, the overall loss function L used in step S5 is all Including the average prediction box confidence, the non-printing loss of the adversarial patch, and the total variation loss of the adversarial patch, the overall loss function L all The calculation formula is:
[0024] L all =αL conf +βL nps +γL tv
[0025] Among them, α represents the average prediction box confidence weight coefficient, β represents the non-printing loss weight coefficient, γ represents the total variation loss weight coefficient, and L conf represents the average prediction box confidence, L nps denotes the non-printing loss of the adversarial patch, L tv represents the total variation loss of the adversarial patch.
[0026] Furthermore, the average prediction box confidence refers to calculating the average confidence of all prediction boxes in an image, and its calculation formula is:
[0027]
[0028] Among them, N represents the number of prediction boxes of the image, confidence i Represents the confidence of the i-th prediction box.
[0029] Furthermore, the calculation formula of the non-printing loss of the adversarial patch is:
[0030]
[0031] Among them, p patch represents the pixel value in the adversarial patch P, c print Represents a set of pixel values for printable colors C.
[0032] Furthermore, the total variation loss of the adversarial patch is calculated as:
[0033]
[0034] Among them, pi,j Represents the RGB value of the pixel (i, j) in the adversarial patch.
[0035] Furthermore, the target optimization function calculation formula in step S6 is:
[0036]
[0037] k(p)={p i |min(max(p i ,-τ),τ),p i ~p}
[0038] Among them, p t represents the optimization target of the tth iteration, η is the learning rate, is the gradient of the overall loss, k is the defined clipping function, and p i is the i-th element of the adversarial patch p, and τ is the threshold of the patch cropping function.
[0039] Beneficial effects of the present invention:
[0040] The present invention provides a physical adversarial attack method for a drone target detection system. The high-definition aerial vehicles collected by the drone are used as a data set. The pre-trained generative adversarial network is used to optimize the adversarial patch. In addition to integrating the traditional EOT transformation, the adversarial patch is enhanced by motion blur, ISO noise, weather change, etc. to adapt to the drone high-altitude aerial photography scene, and non-printing loss and total variation loss are used. While maintaining strong physical adversarial robustness, a more natural-looking adversarial patch is generated, making the adversarial patch difficult to be recognized by the observer. Under the influence of atmospheric factors (light, weather, season) and the flight altitude of the drone, the adversarial patch installed on the roof of the car can still enable the vehicle to avoid drone target detection, thereby performing safety detection on the drone aerial target detection neural network. In addition, in order to ensure that a large number of objects in the aerial pictures can be attacked, the average prediction box confidence is used as one of the optimization losses; in order to ensure that the visual effect of the adversarial patch is more natural, a special target optimization function is designed to limit the optimization direction of the patch. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 A flow chart of a physical countermeasure attack method for a drone aerial target detection system provided by the present invention;
[0042] Figure 2 The optimization and attack schematic diagram of the physical confrontation attack method for the UAV aerial target detection system provided by the present invention;
[0043] Figure 3 A schematic diagram of an attack with weather changes added provided by the present invention;
[0044] Figure 4 This is a schematic diagram of the attack provided by the present invention with added motion blur, optical distortion, and camera sensor noise;
[0045] Figure 5 A specific flow chart of the second physical enhancement of the training patch sample provided by the present invention;
[0046] Figure 6 This is a schematic diagram of the real weather adding module provided by the present invention. DETAILED DESCRIPTION
[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] Existing physical adversarial attack scenarios against target detection systems are all on the ground, such as face recognition, pedestrian detection, unmanned driving, etc., which are relatively close and less affected by the weather; while drones operate at high altitudes, and the aerial photography distance of drones is tens of meters or even hundreds of meters, which is farther away and more seriously affected by weather and imaging equipment, making them more difficult to attack. Currently, there is little research on logistics adversarial attacks on drone target detection; in addition, existing methods rarely add constraints to the appearance of adversarial patches, which makes the patches easily recognizable by observers.
[0049] To this end, the present invention provides a physical adversarial attack method for a drone target detection system. The method uses a high-definition picture set of a drone aerial vehicle as a training data set, and the target detection algorithm adopts a YOLOv5 network, such as Figure 1 , Figure 2 As shown, the following steps are included:
[0050] S1. Use a pre-trained generative adversarial network to generate adversarial patches of fixed size;
[0051] S2. Preprocessing the training samples in the training data set;
[0052] S3. Input the batch of preprocessed training samples into the Yolov5 target detector to obtain the label file corresponding to the training sample, which includes the location information and category label information of all vehicles in the training sample;
[0053] S4. Input the adversarial patch, training sample and its corresponding label file into the patch conversion module for physical robustness data enhancement to obtain adversarial sample P-Images;
[0054] S5. Input the adversarial sample P-Images into the Yolov5 target detector, and use the overall loss function L according to the prediction results of the Yolov5 target detector all Calculate overall losses;
[0055] S6. Design the target optimization function, combine the overall loss calculated in step S5, and use the Adam optimizer to update the adversarial patch;
[0056] S7. Repeat steps S3-S6 until the number of iterations reaches a preset value or the loss function converges;
[0057] S8. Print out the obtained adversarial patch and set it on the roof of the vehicle, and use a drone to move and shoot at different heights to perform a physical adversarial attack.
[0058] Specifically, in this embodiment, high-definition pictures taken by a drone at an altitude of 40-60 meters are mainly used.
[0059] Specifically, the generative adversarial network used is the StyleGAN2 network pre-trained on ImageNet, and the size of the output adversarial patch is 300×300.
[0060] Specifically, the patch conversion module is used to physically enhance the adversarial patch. The physical enhancement methods it adopts include scaling, rotating, adding contrast transformation and brightness transformation, adding motion blur, adding ISO device noise, adding random Gaussian noise, adding optical distortion, and adding weather changes to the adversarial patch.
[0061] Specifically, the specific operation process of the patch conversion module includes:
[0062] S41. Performing a first processing on the adversarial patch, the first processing including scaling, rotating, adding noise, adding brightness, and changing contrast;
[0063] S42. Generate a MASK matrix according to the label file corresponding to the training sample, and the MASK matrix determines the shape and position of the added adversarial patch;
[0064] S43. Add the adversarial sample after the first processing to the training sample through the MASK matrix to obtain a training patch sample;
[0065] S44. Perform a second processing on the training patch sample to obtain an adversarial sample. The second processing includes adding motion blur, adding ISO device noise, adding optical distortion, and adding weather changes, such as Figure 4 In addition, various weather factors are added, such as Figure 3 shown.
[0066] Specifically, in step S44, the training patch samples are processed for the second time to obtain adversarial samples; wherein adding motion blur, ISO device noise, and optical distortion are all differentiable operations, such as Figure 5 As shown, the training patch samples are first copied and converted to RGB format, and then enhanced by the Albumentations function library, in which motion blur, ISO device noise and optical distortion are added with a certain probability; the output of the Albumentations function library is converted to Tensor format and subtracted from the input training patch sample to obtain a first enhanced patch sample, and the first enhanced patch sample is then added to the input training patch sample to obtain a second enhanced patch sample.
[0067] Next, the process of adding weather changes to the second enhanced patch sample is as follows Figure 6 As shown, the training samples are taken by the camera lens in real weather. The weather addition module captures the natural weather influence at the time of shooting the training samples, extracts the weather pattern to generate the corresponding mask, and attaches the mask to the second enhanced patch sample to obtain the adversarial sample.
[0068] Specifically, in step S43, an adaptive patch placement function is used, and the adaptive patch placement function adaptively limits the adversarial patch to within the range of the label box Ground Truth, that is, to within the specific annotation box (real box) in each image annotation information.
[0069] The height range of drone photography varies greatly, resulting in large changes in the area of objects in the image. To this end, the present invention designs an adaptive patch placement function that can adapt to multi-scale objects, ensuring that the adversarial patch can be adaptively scaled and placed on the target vehicle according to the height of the drone photography. The formula is:
[0070] S pnew =ε h ×S pori
[0071] Among them, S pnew is the scaled patch size, ε h S represents the scaling factor for the shooting height h, which is obtained according to the image size at different heights in actual measurement; pori is the original size of the adversarial patch.
[0072] Specifically, the overall loss function L used in step S6 is all Including the average prediction box confidence, the non-printing loss of the adversarial patch, and the total variation loss of the adversarial patch, the overall loss function L all The calculation formula is:
[0073] Lall =αL conf +βL nps +γL tv
[0074] Among them, α represents the average prediction box confidence weight coefficient, β represents the non-printing loss weight coefficient, γ represents the total variation loss weight coefficient, and L conf represents the average prediction box confidence, L nps denotes the non-printing loss of the adversarial patch, L tv represents the total variation loss of the adversarial patch.
[0075] Aerial images taken by drones usually contain a large number of objects, and the higher the aerial photography altitude, the more objects are contained; a large number of objects increases the difficulty of counterattack. In order to ensure that more objects can be attacked, the present invention relates to an average prediction box confidence, which refers to the average value of all prediction box confidences in an image. The calculation formula is:
[0076]
[0077] Among them, N represents the number of prediction boxes of the image, confidence i Represents the confidence of the i-th prediction box.
[0078] The non-printing loss of the adversarial patch is calculated as:
[0079]
[0080] Among them, p patch represents the pixel value in the adversarial patch P, c print Represents a set of pixel values for printable colors C.
[0081] The total variation loss of the adversarial patch is calculated as:
[0082]
[0083] Among them, p i,j Represents the RGB value of the pixel (i, j) in the adversarial patch.
[0084] Specifically, in step S6, the target optimization function processes the gradient of the overall loss function, which serves to constrain the pixels of the adversarial patch to balance the physical attack performance and visual effect of the adversarial patch.
[0085] Without adding constraints to the adversarial patch, the generated adversarial patch P will not contain the potential data distribution of the generator, that is, the generated adversarial patch will not have a real visual effect. In order to limit the adversarial patch to the data distribution of the generative adversarial network, a target optimization function is designed, and its formula is:
[0086]
[0087] k(p)={p i |min(max(p i ,-τ),τ),p i ~p}
[0088] Where t is the number of iterations, η is the learning rate, is the gradient of the overall loss, k is the defined clipping function, and p i is the i-th element of the adversarial patch p, and τ is the threshold of the patch cropping function.
[0089] In addition, the present invention hopes that the trained adversarial patches have better migration ability, that is, they still have attack ability on detectors other than Yolov5. The present invention collects hard instances in the verification set, that is, difficult samples, and fine-tunes the adversarial patches on other detectors such as Yolov3, SSD, and Faster-Rcnn.
[0090] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.
[0091] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A physical adversarial attack method for a drone target detection system, characterized in that: A high-definition picture set of drone aerial vehicles is used as a training data set. The target detection algorithm adopts the YOLOv5 network, which includes the following steps: S1. Use a pre-trained generative adversarial network to generate adversarial patches of fixed size; S2. Preprocessing the training samples in the training data set; S3. Input the batch of preprocessed training samples into the Yolov5 target detector to obtain the label file corresponding to the training sample, which includes the location information and category label information of all vehicles in the training sample; S4. Input the adversarial patch, training sample and its corresponding label file into the patch conversion module to obtain the adversarial sample P-Images; The patch conversion module is used to physically enhance the adversarial patch. The specific operation process includes: S41. Performing a first processing on the adversarial patch, the first processing including scaling, rotating, adding noise, adding brightness, and changing contrast; S42. Generate a MASK matrix according to the label file corresponding to the training sample, and the MASK matrix determines the shape and position of the added adversarial patch; S43. Add the adversarial sample after the first processing to the training sample through the MASK matrix to obtain a training patch sample; S44. Perform a second processing on the training patch sample to obtain an adversarial sample, where the second processing includes adding motion blur, adding ISO device noise, adding optical distortion, and adding weather changes; S5. Input the adversarial sample P-Images into the Yolov5 target detector and use the overall loss function L all Calculate overall losses; S6. Design a target optimization function based on the overall loss calculated in step S5, and use the Adam optimizer to update the adversarial patch; The target optimization function calculation formula in step S6 is: k(p)={p i |min(max(p i ,-τ),τ),p i ~p} Among them, p t represents the optimization target of the tth iteration, η is the learning rate, is the gradient of the overall loss, k is the defined clipping function, and p i is the i-th element of the adversarial patch p, τ is the threshold of the patch cropping function; S7. Repeat steps S3-S6 until the number of iterations reaches a preset value or the overall loss function converges; S8. Print out the trained adversarial patch and set it on the roof of the vehicle to perform a physical adversarial attack.
2. According to claim 1, a physical counterattack method for a drone target detection system is characterized in that: The generative adversarial network used in step S1 is the StyleGAN2 network pre-trained on ImageNet, and the size of the output adversarial patch is 300×300.
3. The physical counterattack method for a drone target detection system according to claim 1 is characterized in that: In step S43, an adaptive patch placement function is used, and the adaptive patch placement function adaptively limits the adversarial patch to the range of the label box Ground Truth, which is expressed as: S pnew =e h ×S pori Among them, S pnew is the scaled patch size, ε h Indicates the zoom factor when the shooting height is h, S pori is the original size of the adversarial patch.
4. The physical counterattack method for a drone target detection system according to claim 1, characterized in that: The overall loss function L used in step S5 all Including the average prediction box confidence, the non-printing loss of the adversarial patch, and the total variation loss of the adversarial patch, the overall loss function L all The calculation formula is: L all =αL conf +βL nps +γL tv Among them, α represents the average prediction box confidence weight coefficient, β represents the non-printing loss weight coefficient, γ represents the total variation loss weight coefficient, and L conf represents the average prediction box confidence, L nps denotes the non-printing loss of the adversarial patch, L tv represents the total variation loss of the adversarial patch.
5. The physical counterattack method for a drone target detection system according to claim 4 is characterized in that: The average prediction box confidence refers to calculating the average confidence of all prediction boxes in an image. The calculation formula is: Among them, N represents the number of prediction boxes of the image, confidence i Represents the confidence of the i-th prediction box.
6. A physical counterattack method for a drone target detection system according to claim 4, characterized in that: The non-printing loss of the adversarial patch is calculated as: Among them, p patch represents the pixel value in the adversarial patch P, c print Represents a set of pixel values for printable colors C.
7. The physical counterattack method for a drone target detection system according to claim 4, characterized in that: The total variation loss of the adversarial patch is calculated as: Among them, p i,j Represents the RGB value of the pixel (i, j) in the adversarial patch.