Target detection confrontation sample generation method based on space-frequency domain information mask

Abstract

The invention discloses a target detection confrontation sample generation method based on a space-frequency domain information mask. According to the method, on the basis of space loss and frequency domain loss of a backbone network or an RPN confidence coefficient branch, the intermediate feature map gradient of the backbone network / RPN is directly utilized to guide generation of countermeasure disturbance, the technical limitation that an existing method depends on back propagation of a detection head is broken through, and effective attacks under the condition that the detection head cannot be accessed are achieved. In order to generate a transferable adversarial sample when a detection head of the dual-stage target detection model is inaccessible, the invention designs a space-frequency domain information mask method, and by randomly masking partial information in a space domain and a frequency domain, adversarial disturbance is prevented from over-fitting the white-box dual-stage target detection model, so that the transferable adversarial sample is generated. And the mobility of the target detection confrontation sample is further enhanced, and corresponding loss functions are respectively designed for the RPN feature layer and the backbone network feature layer.

Description

Technical Field

[0001] This invention belongs to the field of deep learning adversarial attacks, specifically involving a method for generating adversarial examples for target detection based on spatial-frequency domain information masks. Background Technology

[0002] Object detection technology focuses on identifying specific objects in a visual scene and determining their category. Essentially, this technology requires object detection algorithms to accurately locate the region containing the target object (usually marked with a rectangular bounding box) and identify its type within a single image or consecutive video frames. As a core component of fundamental computer vision capabilities, object detection integrates both object recognition and spatial localization. Object detection technology provides crucial technical support for many high-level vision tasks (such as autonomous driving), and its performance directly impacts the overall inference results of these downstream tasks. Thanks to the breakthrough development of deep learning, the technological path in this field has been revolutionized. Deep learning's powerful feature learning capabilities and fast inference advantages effectively address the significant shortcomings of traditional methods in terms of real-time response speed, detection accuracy, and computational efficiency. Therefore, a new generation of deep learning-based object detection methods is rapidly gaining popularity and playing an increasingly important supporting role in various application fields.

[0003] Recent studies have shown that object detection technology has security flaws and is vulnerable to adversarial examples. Many studies have proposed adversarial attack methods for object detection tasks, the key to which lies in using the object detection model's head to calculate a classification loss function to guide the generation of adversarial examples. However, in resource-constrained edge computing scenarios, when deploying a two-stage object detection model, the detection head can be decoupled and deployed in the cloud, while only the backbone network for feature extraction and the RPN for feature preprocessing are deployed on edge devices. In this situation, where attackers cannot directly access the object detection model's head and therefore cannot use it for backpropagation, the aforementioned head-based attack methods become inapplicable. Furthermore, these methods suffer from low portability. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for generating adversarial examples for target detection based on spatial-frequency domain information masks.

[0005] In a first aspect, the present invention provides a method for generating adversarial examples for target detection based on a spatial-frequency domain information mask, the method comprising: Initialization and Clean Image x Same size counter-perturbation Initialize the adversarial sample at time 0. ; Adversarial examples at time t Add anti-perturbation Generate adversarial examples for the current iteration phase. ; adversarial examples The image is transformed into a first enhanced adversarial sample image using a spatial information masking method. ; adversarial examples The image is transformed into a second enhanced adversarial sample image using a frequency domain information masking method. ; Calculate the first enhanced adversarial sample image respectively Second enhanced adversarial sample image With clean images x Spatial loss function of feature layer in white-box object detection model and frequency domain loss function The feature layer is either the confidence branch or backbone of the Regional Proposal Network (RPN). Space loss function and frequency domain loss function The summation is used as the overall loss function, and the backpropagation algorithm is executed to calculate the gradient against the perturbation. Then utilize the gradient to counteract the perturbation. Update counter-disturbance ; Updated to counter disturbances As the next moment's counter-disturbance Repeat the iteration until the maximum number of iterations is reached. And at the end of each iteration, - Norm, limiting adversarial perturbations The maximum number of pixels does not exceed the threshold .

[0006] Preferably, the space loss function Represented as:

[0007] in, This indicates that the input is a clean image. The first feature layer output Each feature map This indicates that the input is the first enhanced adversarial sample image. The first feature layer output Each feature map The total number of feature maps. for -norm.

[0008] Preferably, the frequency domain loss function Represented as:

[0009] in, This indicates that the input is a clean image. The first feature layer output Each feature map This indicates that the input is the second enhanced adversarial sample image. The first feature layer output Each feature map The total number of feature maps. for -norm.

[0010] Preferably, the gradient against disturbance Represented as:

[0011] in, Indicates to Find the partial derivative.

[0012] Preferably, the gradient utilizing the anti-perturbation method Update counter-disturbance Specifically:

[0013] in, Indicates the learning rate. express -norm.

[0014] Preferably, the adversarial sample The image is transformed into a first enhanced adversarial sample image using a spatial information masking method. The implementation process is as follows:

[0015] in Represents element-wise product. Represents a binary mask matrix. and They represent the first A real bounding box Height and width, Indicates the first A real bounding box The corresponding proportionality coefficient.

[0016] Preferably, the adversarial sample The image is transformed into a second enhanced adversarial sample image using a frequency domain information masking method. The implementation process is as follows:

[0017] in Represents element-wise product. Indicates amplitude spectrum, Represents a binary mask matrix. and Indicates the second enhanced adversarial sample image Height and width, The scale factor is randomly selected. IDCT stands for Inverse Discrete Cosine Transform, which is used to transform the amplitude spectrum after the high-frequency information of the random mask back to the spatial domain.

[0018] Preferably, the white-box target detection model is a two-stage target detection model or a single-stage target detection model.

[0019] In a second aspect, the present invention provides an electronic device, characterized in that it includes a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor executing the machine-executable instructions to implement the method.

[0020] Thirdly, the present invention provides a machine-readable storage medium, characterized in that the machine-readable storage medium stores machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the method.

[0021] The beneficial effects of this invention are at least as follows: 1. To address the attack surface deficiency caused by the decoupled deployment of detection heads in edge computing, this invention proposes a method based on the spatial and frequency domain losses of the backbone network or RPN confidence branch. It directly utilizes the gradient of the intermediate feature map of the backbone network / RPN to guide the generation of adversarial perturbations, breaking through the technical limitation of existing methods that rely on backpropagation of the detection head, and achieving effective attacks in scenarios where the detection head is inaccessible.

[0022] 2. This invention proposes the Spatial-Frequency Information Erasure (SFIE) method, which forces perturbation to learn attack patterns with stronger generalization by synergistic effect of random mask partial information in the spatial and frequency domains, avoids overfitting white-box models, and further improves the transferability of adversarial examples.

[0023] 3. This invention adopts a dual-path design that attacks the backbone network and attacks the RPN confidence branch, which can flexibly adapt to different edge deployment architectures.

[0024] 4. This invention ensures visual imperceptibility by constraining the perturbation amplitude, and controls computational overhead by using gradient normalization updates and a limited number of iterations, thus satisfying the feasibility under the resource constraints of edge devices. Attached Figure Description

[0025] Figure 1 This is a flowchart of the target detection adversarial example generation method of the present invention. Detailed Implementation

[0026] The present invention will be further analyzed below with reference to specific embodiments, taking the SFIE-B method as an example.

[0027] This embodiment provides a method for generating adversarial examples for target detection based on spatial-frequency domain information masks. It is applicable to two-stage target detection models containing a Region Proposal Network (RPN) and a Backbone network, and also to single-stage target detection models containing a Backbone network. Figure 1 Specifically, the method includes: Step 1: Initialization and Cleaning the Image x Same size counter-perturbation Initialize the adversarial sample at time 0. ; Step 2: Analyze the adversarial sample at time t. Add anti-perturbation Generate adversarial examples for the current iteration phase. ; Step 3: Prepare adversarial examples The image is transformed into a first-enhanced adversarial sample image using the Spatial Information Masking (SIE) method. ; One implementation method, the Spatial Information Masking (SIE) method, is as follows: (1) For those containing A real bounding box Input image Construct a binary mask matrix of the same size consisting entirely of 1s. , and They are the first A real bounding box Height and width.

[0028] (2) In each iteration, for each true bounding box Randomly select the proportional coefficient ,use Determine the height of a rectangular mask region and width . For fixed hyperparameters, the possible values ​​are... .

[0029] (3) For each true bounding box From the top left area Randomly select one pixel as the top-left corner of the rectangular mask area to ensure that the rectangular mask area is always within the true bounding box. within, within The value in this area is set to 0.

[0030] (4) Finally, process the input image Each channel (total) (one channel) and Perform element-wise multiplication to mask the actual bounding box. Part of the information.

[0031] (5) Processes (1) to (4) can be described by the following formulas:

[0032] in This represents element-wise (Hadamard) product. Represents a binary mask matrix. and They represent the first A real bounding box Height and width, Indicates the first A real bounding box The corresponding proportionality coefficient.

[0033] Step 4: Prepare adversarial examples The image is transformed into a second enhanced adversarial sample image using the Frequency Domain Information Masking (FIE) method. ; In one implementation, the frequency domain information masking method (FIE) is as follows: (1) Input image Each channel undergoes a two-dimensional discrete cosine transform (DCT) to transform it from the spatial domain to the frequency domain, yielding the corresponding amplitude spectrum. .

[0034] (2) Construction and input image Binary mask matrices of the same size, all 1s .

[0035] (3) In A rectangular area is defined at the top left corner, with a height and width of [missing information]. and , For in the interval The proportionality coefficient is randomly selected in the equation, and b is a hyperparameter that can take a value of 0.8.

[0036] (4) In The values ​​outside the rectangular region are set to 0, and then the amplitude spectrum is... Each channel (total) (one channel) and Element-wise multiplication is performed to randomly mask the input image. High-frequency information.

[0037] (5) Processes (1) to (4) can be described by the following formulas:

[0038] in Represents element-wise product. Indicates amplitude spectrum, Represents a binary mask matrix. and Indicates the second enhanced adversarial sample image Height and width, The scale factor is randomly selected. IDCT stands for Inverse Discrete Cosine Transform, which is used to transform the amplitude spectrum after the high-frequency information of the random mask back to the spatial domain.

[0039] Step 5: Calculate the first enhanced adversarial sample image With clean images x Spatial loss function of feature layer in white-box object detection model ; Calculate the second enhanced adversarial sample image With clean images x Frequency domain loss function of feature layer in white-box object detection model The feature layer is either the confidence branch or backbone of the Regional Proposal Network (RPN). Space loss function Represented as:

[0040] in, This indicates that the input is a clean image. The first feature layer output Each feature map This indicates that the input is the first enhanced adversarial sample image. The first feature layer output Each feature map The total number of feature maps. for -norm.

[0041] Frequency domain loss function Represented as:

[0042] in, This indicates that the input is a clean image. The first feature layer output Each feature map This indicates that the input is the second enhanced adversarial sample image. The first feature layer output Each feature map The total number of feature maps. for -norm.

[0043] Space loss function and frequency domain loss function The summation is used as the overall loss function, and the backpropagation algorithm is executed to calculate the gradient against the perturbation. ,express:

[0044] in, Indicates to Find the partial derivative; Gradients that resist perturbations Perform normalization and accumulation, then update the anti-perturbation algorithm. :

[0045] in, This represents the learning rate, which can take a value of 0.5. express -norm; Step Six: Update the anti-disturbance measures As the next moment's counter-disturbance Repeat steps two through eight. The iteration stops when the maximum number of iterations is reached. The value used in this embodiment At the end of each iteration, adopt - Norm, limiting adversarial perturbations The maximum number of pixels does not exceed the threshold ( ).

[0046] To test the effectiveness of adversarial example transfer attacks, this embodiment selects pre-trained object detection models in the MMDetection framework for experiments. These include the two-stage object detection model Faster R-CNN (FR-R50) with ResNet50 as the backbone, the single-stage object detection model YOLOF (YOLOF-R50), and the DETR (DETR-R50) model based on a Transformer architecture. Additionally, the SSD (SSD-VGG16) with VGG16 as the backbone, the single-stage object detection model YOLOv3 (YOLOv3-D53) with DarkNet53 as the backbone, the single-stage object detection model YOLOX (YOLOX-CSPD) with CSPDarkNet53 as the backbone, and the two-stage Mask R-CNN (MR-SWINT) object detection model based on a Swin Transformer backbone were also considered as test subjects. FR-R50 was used as a white-box model to generate adversarial examples, while the other models were used as black-box models to test the transferability of adversarial examples.

[0047] To verify the effectiveness of this invention, five existing adversarial attack methods targeting object detection models—Dense Adversary Generation (DAG), Robust Adversarial Perturbation (RAP), Contextual Adversarial Perturbation (CAP), TOG, and Enhanced Dense Adversary Generation (DAG+)—were compared with the SFIE-B method (which uses a backbone network for its feature layer) and the SFIE-R method (which uses the Region Proposal Network (RPN) confidence branch for its feature layer). The datasets used in the experiments consisted of 2000 images randomly selected from the PASCALVOC 2012 validation set and 2000 images randomly selected from the MS COCO 2017 validation set.

[0048] To verify the effectiveness of the attack method proposed in this invention, the mean Average Precision (mAP) is used as the performance evaluation standard for the target detection model.

[0049] The experimental results are shown in Tables 1 and 2. On the PASCAL VOC 2012 and MS COCO2017 datasets, the SFIE-R method of this invention reduces the mAP evaluation index by 0.1 to 11.4 and 0.1 to 5.2 compared to DAG, RAP, CAP, and TOG on black-box object detection models, respectively, and is slightly lower than the DAG+ method. This can be explained by the fact that the proposed SFIE-R method only interferes with the internal processing of the RPN. Compared to the DAG+ method, which attacks the backbone network feature layer and thus affects all downstream modules, the SFIE-R method of this invention does not rely on the intermediate feature output of the backbone network. This allows the SFIE-R method to remain effective even when attackers cannot directly obtain the intermediate features of the backbone network. The SFIE-B method of this invention performs best compared to the other methods shown in Tables 1 and 2, with the most significant effect on the YOLOX-CSPD and MR-SWINT black-box object detection models.

[0050] Table 1. mAP (%) of different attack methods for migration attacks on different black-box target detection models on the MS COCO 2017 validation set.

[0051]

[0052] Note: The attack occurred when " The results are indicated on the white-box model. The best results are highlighted in bold, and the second best results are marked with an underline.

[0053] Table 2 shows the mAP (%) of different attack methods for migration attacks on different black-box target detection models on the PASCAL VOC 2012 validation set.

[0054]

[0055] Note: The attack occurred when " The results are indicated on the white-box model. The best results are highlighted in bold, and the second best results are marked with an underline.

[0056] The adversarial samples obtained by the method described above, together with the original clean samples, constitute a dataset. The target detection model to be defended is then used for adversarial training. The network parameters are adjusted by training with the maximum loss minimized. Finally, the defense evaluation results are obtained based on the defense effectiveness evaluation, thereby achieving the anti-attack defense of the target detection model.

[0057] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A target detection adversarial example generation method based on spatial-frequency domain information mask, characterized in that, The method includes: Initialization and Clean Image x Same size counter-perturbation Initialize the adversarial sample at time 0. ; Adversarial examples at time t Add anti-perturbation Generate adversarial examples for the current iteration stage. ; adversarial examples The image is transformed into a first enhanced adversarial sample image using a spatial information masking method. ; adversarial examples The image is transformed into a second enhanced adversarial sample image using a frequency domain information masking method. ; Calculate the first enhanced adversarial sample image Second enhanced adversarial sample image Compared with clean images x Spatial loss function of feature layer in white-box object detection model and frequency domain loss function The feature layer is either the confidence branch or backbone of the Regional Proposal Network (RPN). Space loss function and frequency domain loss function The summation is used as the overall loss function, and the backpropagation algorithm is executed to calculate the gradient against the perturbation. Then utilize the gradient to counteract the perturbation. Update counter-disturbance ; Updated to counter disturbances As the next moment's counter-disturbance Repeat the iteration until the maximum number of iterations is reached. And at the end of each iteration, - Norm, limiting adversarial perturbations The maximum number of pixels does not exceed the threshold .

2. The method according to claim 1, characterized in that, The space loss function Represented as: （1） in, This indicates that the input is a clean image. The first feature layer output Each feature map This indicates that the input is the first enhanced adversarial sample image. The first feature layer output Each feature map The total number of feature maps. for - Norm.

3. The method according to claim 1, characterized in that, The frequency domain loss function Represented as: （2） in, This indicates that the input is a clean image. The first feature layer output Each feature map This indicates that the input is the second enhanced adversarial sample image. The first feature layer output Each feature map The total number of feature maps. for - Norm.

4. The method according to claim 1, characterized in that, The gradient against the disturbance Represented as: （3） in, Indicates to Find the partial derivative.

5. The method according to claim 1 or 4, characterized in that, The gradient that utilizes anti-perturbation Update counter-disturbance Specifically: （4） in, Indicates the learning rate. express - Norm.

6. The method according to claim 1, characterized in that, The adversarial examples The image is transformed into a first enhanced adversarial sample image using a spatial information masking method. The implementation process is as follows: （5） in Represents element-wise product. Represents a binary mask matrix. and They represent the first A true bounding box Height and width, Indicates the first A real bounding box The corresponding proportionality coefficient.

7. The method according to claim 1, characterized in that, The adversarial examples The image is transformed into a second enhanced adversarial sample image using a frequency domain information masking method. The implementation process is as follows: （6） in Represents element-wise product. Indicates amplitude spectrum, Represents a binary mask matrix. and Indicates the second enhanced adversarial sample image Height and width, represents the randomly selected scaling factor, and IDCT represents the two-dimensional inverse discrete cosine transform.

8. The method according to claim 1, characterized in that, The white-box target detection model is either a two-stage target detection model or a single-stage target detection model.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method of any one of claims 1-8.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the method of any one of claims 1-8.

Images