A Physical Adversarial Patch Generation Method for Object Detection Models

Abstract

This invention discloses a physical adversarial patch attack method for target detection models. The method first randomly initializes the pixel values ​​of the adversarial patch and applies a random transformation to it. The pixel values ​​of a local region of a digital image are covered with the randomly transformed adversarial patch. The digital image covered with the adversarial patch is then fed into multiple target detection models. Based on the output detection results, a loss function is calculated for each target detection model to quantify the degree of degradation in target recognition capability. The gradient of the loss function of each target detection model with respect to the pixel values ​​of the adversarial patch is calculated to obtain a mixed gradient for all target detection models. The pixel values ​​of the adversarial patch are then optimized and updated until convergence. This method is universally effective against target detection models of various unknown structures, effectively maintains adversarial properties under black-box attacks, and exhibits enhanced practicality in real-world scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection models, and in particular to a method for generating physical adversarial patches for target detection models. Background Art

[0002] With the rise of deep learning technology, object detection algorithms based on convolutional neural networks have seen significant improvements in accuracy and real-time performance, nearly matching the human eye's visual perception capabilities. This technology has also gained widespread application in public security. However, the readily available open-source object detection algorithms and increasingly affordable edge computing devices have exposed object detectors to the risk of malicious use. For example, deploying object detectors in public places to track and identify pedestrians would infringe on people's privacy. Developing ways to attack object detection models and prevent targets from being easily identified has become a new technical challenge.

[0003] Physical adversarial patch attacks exploit the structural characteristics of deep neural networks and generate robust adversarial texture patterns through training. These patterns can be physically shaped in real scenes. The resulting digital images, captured by cameras, will result in incorrect predictions when fed into object detectors. Existing physical adversarial patch attack schemes targeting object detection models are mostly based on white-box assumptions. These schemes require a complete understanding of the model's structure, weights, and hyperparameters. The resulting adversarial patches are only effective against certain types of object detectors, resulting in limitations. Summary of the Invention

[0004] The purpose of this invention is to provide a physical adversarial patch generation method for target detection models, which can effectively maintain adversarial properties under black-box attacks and has better practicality in real scenarios.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for generating physical adversarial patches for an object detection model, the method comprising:

[0007] Step 1: First, randomly initialize the pixel values ​​of the adversarial patch and apply a random transformation to it, and then use the randomly transformed adversarial patch to cover the pixel values ​​of the local area of ​​the digital image;

[0008] Step 2: Input the digital image covered with the adversarial patch into multiple target detection models. Based on the output detection results, calculate the loss function of each target detection model to quantify the degree of degradation of target recognition ability.

[0009] Step 3: Calculate the gradient of the loss function of each target detection model with respect to the pixel value of the adversarial patch, obtain the mixed gradient of all target detection models, and optimize and update the pixel value of the adversarial patch until convergence.

[0010] It can be seen from the technical solution provided by the present invention that the above method can effectively maintain adversarial properties under black box attacks and has better practicality in real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A schematic diagram of a flow chart of a method for generating physical adversarial patches for target detection models provided by an embodiment of the present invention;

[0013] Figure 2 This is a schematic diagram of an application of the example of the present invention in a public safety scenario. DETAILED DESCRIPTION

[0014] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments, and do not constitute a limitation of the present invention. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0015] The embodiments of the present invention will be described in further detail below in conjunction with the accompanying drawings. The contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art. In the embodiments of the present invention, if specific conditions are not specified, the conditions are carried out according to conventional conditions in the art or the conditions recommended by the manufacturer. If the manufacturer is not specified for the reagents or instruments used in the embodiments of the present invention, they are all conventional products that can be purchased commercially.

[0016] like Figure 1 FIG2 is a flow chart of a method for generating physical adversarial patches for a target detection model according to an embodiment of the present invention. The method includes:

[0017] Step 1: First, randomly initialize the pixel values ​​of the adversarial patch and apply a random transformation to it, and then use the randomly transformed adversarial patch to cover the pixel values ​​of the local area of ​​the digital image;

[0018] In this step, a square image is filled with random pixel values ​​as the initial adversarial patch. To enhance the robustness of the adversarial patch, the distortion factors in the real scene are modeled as a series of random transformations. Random transformations are applied to the initial adversarial patch, and the pixel values ​​of the local area of ​​the digital image are covered with the randomly transformed adversarial patch to embed it into the digital image. The applied random transformations include:

[0019] 1) Gaussian noise

[0020] Additive Gaussian noise and multiplicative Gaussian noise are applied to the adversarial patch to simulate changes in hue, brightness and contrast; multiplicative Gaussian noise G m and additive Gaussian noise G a is a Gaussian noise matrix with mean 1 and 0 respectively, and the matrix size is the same as the adversarial patch P; the multiplicative Gaussian noise G m Multiply the pixel value at the corresponding position of the adversarial patch P, and then add the additive Gaussian noise G a The values ​​at the corresponding positions are added together, and the result is used as the pixel value at the corresponding position. The above operation is shown in formula (1):

[0021] P←G m ×P+G a (1)

[0022] 2) Geometric transformation

[0023] To simulate changes in imaging perspective and distance, geometric transformations are applied to the adversarial patches in sequence: random enlargement or reduction by up to 30%, and random rotation by up to 20°.

[0024] Step 2: Input the digital image covered with the adversarial patch into multiple target detection models. Based on the output detection results, calculate the loss function of each target detection model to quantify the degree of degradation of target recognition ability.

[0025] In this step, the loss function is used to quantify the impact of the adversarial patch on the performance of the target detection model, so that the adversarial patch is optimized and updated in the direction of enhancing adversarial resistance. The loss function calculation is divided into three parts, as shown in formula (2):

[0026] L=αL tv +βL sal +L obj (2)

[0027] Among them L tv 、L sal 、L obj They are smoothness loss, conspicuousness loss and target detection loss respectively; α and β are L tv 、L sal The corresponding hyperparameters;

[0028] 1) Smoothness loss L tv

[0029] In order to ensure the smoothness of the adversarial patch pattern texture, the smoothness loss L is defined tv , the calculation method is shown in formula (3):

[0030]

[0031] Where s represents the side length of the square adversarial patch; P h,w represents the pixel value of the adversarial patch P at position (h, w); Formula (3) measures the overall smoothness of the image by calculating the difference between adjacent pixel values ​​in rows and columns;

[0032] 2) Loss of conspicuousness L sal

[0033] In order to make the physical adversarial patch more concealed and less likely to be noticed by the human eye, a conspicuousness loss term L is added to the loss function of the adversarial patch. sal , the calculation method is as follows:

[0034]

[0035] Where p is the pixel value in the adversarial patch P; the pixel values ​​in the image dataset are clustered using the K-means method to obtain the main color set C main , c is the main color set C main Color in; D CIE (·) represents the distance between two colors in the CIE color space, which is used to quantify the difference between the adversarial patch and the background color under human perception;

[0036] Formula (4) calculates the pixel value of the adversarial patch and the main color set C in the CIE color space. main The distance between the closest colors in the image is used to quantify the strikingness of the adversarial patch color under human observation;

[0037] 3) Target detection loss L obj

[0038] For the target detection model, the target detection loss L obj The calculation method is as follows:

[0039]

[0040] Where conf and obj represent the confidence score and target score in the detection results respectively; N is the number of samples in a batch during training; is the image containing the adversarial patch, j is a count from 1 to N; M is the target detection model; y is the true detection result of the image;

[0041] Formula (5) is used to quantify the detection effect of the target detection model on the input image.

[0042] Step 3: Calculate the gradient of the loss function of each target detection model with respect to the pixel value of the adversarial patch, obtain the mixed gradient of all target detection models, and optimize and update the pixel value of the adversarial patch until convergence.

[0043] In this step, a set of target detection models M1, M2, ..., M n In the process of optimizing and updating the adversarial patch, the gradient of the loss function L with respect to the pixel value at each position of the adversarial patch P is calculated respectively.

[0044]

[0045] in is the target detection model M i The calculated loss function; n is the number of target detection models;

[0046] Mix the gradients of all models to get a mixed gradient

[0047]

[0048] Let the pixel value of each position of the adversarial patch P be subtracted from the mixed gradient The product of and the learning rate updates the pixel value of the adversarial patch, expressed as:

[0049]

[0050] Where η is the learning rate, which is adjusted using the Adam optimizer during the adversarial patch update process.

[0051] It should be noted that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0052] The method of the embodiment of the present invention is described below with specific examples. Figure 2 The figure shows an application diagram of the example of the present invention in a public security scenario. For autonomous reconnaissance and perception equipment, a physical adversarial patch is generated and attached to the target to be protected; the collected image will cause the target detection model to produce incorrect recognition results.

[0053] Specifically, to generate a physical adversarial patch for malicious unmanned monitoring equipment, the following steps are performed: first, an adversarial patch is randomly initialized, and step 1 of the above method embodiment is executed to obtain an adversarial patch after random transformation; an image dataset of the target scene is constructed, a series of target detection models are selected, and a batch of images containing the adversarial patch are input into the target detection model; then, a loss function is calculated based on the detection results, and the adversarial patch is updated based on the mixed gradient optimization, and this is performed for multiple batches until the pixel values ​​of the adversarial patch converge.

[0054] The obtained countermeasure patch is physically shaped through printing, weaving, etc. to protect the target.

[0055] In summary, most of the attack methods in the existing technology are based on the white-box attack hypothesis, generating adversarial samples for a specific known target detection model, which has reduced the adversarial performance on unknown target detection models. However, the gradient fusion training of this method can effectively mine the common visual features of the target detection model, so that the generated adversarial patches have better generalization under the black-box attack hypothesis.

[0056] In addition, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0057] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A method for generating physical adversarial patches for target detection models, characterized in that: The method comprises: Step 1: First, randomly initialize the pixel values ​​of the adversarial patch and apply a random transformation to it, and then use the randomly transformed adversarial patch to cover the pixel values ​​of the local area of ​​the digital image; Step 2: Input the digital image covered with the adversarial patch into multiple target detection models. Based on the output detection results, calculate the loss function of each target detection model to quantify the degree of degradation of target recognition ability. In step 2, the loss function calculation is divided into three parts, as shown in formula (2): L=αL tv +βL sal +L obj (2) Among them L tv 、L sal 、L obj They are smoothness loss, conspicuousness loss and target detection loss respectively; α and β are L tv 、L sal The corresponding hyperparameters; Step 3: Calculate the gradient of the loss function of each target detection model with respect to the pixel value of the adversarial patch, obtain the mixed gradient of all target detection models, and optimize and update the pixel value of the adversarial patch until convergence; In step 3, a set of target detection models M1, M2, ..., M n In the process of optimizing and updating the adversarial patch, the gradient of the loss function L with respect to the pixel value at each position of the adversarial patch P is calculated respectively. in is the target detection model M i The calculated loss function; n is the number of target detection models; Mix the gradients of all models to get a mixed gradient Let the pixel value of each position of the adversarial patch P be subtracted from the mixed gradient The product of and the learning rate updates the pixel value of the adversarial patch, expressed as: Where η is the learning rate, which is adjusted using the Adam optimizer during the adversarial patch update process.

2. The method for generating physical adversarial patches for target detection models according to claim 1, characterized in that: In step 1, a square image is filled with random pixel values ​​as the initial adversarial patch. To enhance the robustness of the adversarial patch, the distortion factors in the real scene are modeled as a series of random transformations. Random transformations are applied to the initial adversarial patch, and the pixel values ​​of the local area of ​​the digital image are covered with the randomly transformed adversarial patch to embed it into the digital image. The applied random transformations include: 1) Gaussian noise Additive Gaussian noise and multiplicative Gaussian noise are applied to the adversarial patch to simulate changes in hue, brightness and contrast; multiplicative Gaussian noise G m and additive Gaussian noise G a is a Gaussian noise matrix with mean 1 and 0 respectively, and the matrix size is the same as the adversarial patch P; the multiplicative Gaussian noise G m Multiply the pixel value at the corresponding position of the adversarial patch P, and then add the additive Gaussian noise G a The values ​​at the corresponding positions are added together, and the result is used as the pixel value at the corresponding position. The above operation is shown in formula (1): P←G m ×P+G a (1) 2) Geometric transformation To simulate changes in imaging perspective and distance, geometric transformations are applied to the adversarial patches in sequence: random enlargement or reduction by up to 30%, and random rotation by up to 20°.

3. The method for generating physical adversarial patches for target detection models according to claim 1, characterized in that: In step 2: 1) Smoothness loss L tv In order to ensure the smoothness of the adversarial patch pattern texture, the smoothness loss L is defined tv , the calculation method is shown in formula (3): Where s represents the side length of the square adversarial patch; P h,w represents the pixel value of the adversarial patch P at position (h, w); Formula (3) measures the overall smoothness of the image by calculating the difference between adjacent pixel values ​​in rows and columns; 2) Loss of conspicuousness L sal In order to make the physical adversarial patch more concealed, the conspicuousness loss term L is added to the loss function of the adversarial patch. sal , the calculation method is as follows: Where p is the pixel value in the adversarial patch P; The pixel values ​​in the image dataset are clustered using the K-means method to obtain the main color set C main , c is the main color set C main Color in; D CIE (·) represents the distance between two colors in the CIE color space, which is used to quantify the difference between the adversarial patch and the background color under human perception; Formula (4) calculates the pixel value of the adversarial patch and the main color set C in the CIE color space. main The distance between the closest colors in the image is used to quantify the strikingness of the adversarial patch color under human observation; 3) Target detection loss L obj For the target detection model, the target detection loss L obj The calculation method is as follows: Where conf and obj represent the confidence score and target score in the detection results respectively; N is the number of samples in a batch during training; is the image containing the adversarial patch, j is a count from 1 to N; M is the target detection model; y is the real detection result of the image; Formula (5) is used to quantify the detection effect of the target detection model on the input image.

Images