Physical achievable camera pollution adversarial patch generation method

Abstract

The invention provides a physically realizable camera pollution adversarial patch generation method, and belongs to the technical field of image processing. Comprising the steps of initializing parameters of a camera pollution patch; overlapping the clean picture in the training set with the camera pollution patch to form an adversarial sample; feeding the adversarial sample and the clean picture to a detector based on a deep neural network, and outputting a detection result by the detector; according to an output result of the detector, calculating a loss function, and updating and optimizing various parameters of the camera pollution patch in a gradient back propagation mode; when the maximum training round is reached, the training is ended; overlapping the camera pollution patch after training optimization with the clean picture, and constructing an adversarial sample; the confrontation sample is fed to various black box detectors, the target object in the detection result is not detected, and other targets can still be detected. The camera pollution patch constructed by the invention is natural in appearance, and human observers are difficult to perceive the naturally occurring lens pollution.

Description

Technical Field

[0001] The invention provides a physically realizable camera pollution countermeasure patch generation method, belonging to the technical field of image processing. Background Art

[0002] Deep neural networks are increasingly used in daily life due to their powerful functions, such as autonomous driving, face recognition and other scenarios. However, systems based on deep neural networks are very vulnerable to adversarial samples. Most adversarial samples are carefully designed noises that are invisible to humans but can deceive deep neural networks. Attacking deep neural networks by constructing various adversarial samples will lead to various potential risks. According to the different construction fields of adversarial samples, adversarial attacks can be divided into digital attacks and physical attacks. Compared with digital attacks that can only generate single-pixel perturbations, physical attacks take into account real-life deployment and are more threatening to visual systems. Physical attacks must look natural to avoid attracting people's attention. At the same time, physical attacks should also be easy to deploy in practice.

[0003] In the current existing technology, the target object is made "invisible" in front of the detector by pasting a physical patch on the target object. First, the pattern of the patch is optimized in the digital domain, including color, shape, texture, etc. Then, the optimized patch is printed out and made into a physical patch. Finally, the physical patch is pasted on the target object, such as a pedestrian or a traffic sign, so that the detector misses pedestrians or misjudges traffic signs. However, the appearance of the physical patch is very eye-catching and it is easy for people to detect abnormalities. This kind of physical attack requires approaching and manipulating the target. For example, pasting a patch on the target person will alert the target person. When the attacker wants to attack multiple targets, he needs to paste a patch on each target, which greatly increases the difficulty and cost of deploying the attack, and is stretched when attacking multiple targets.

[0004] Another physical attack focuses on attacking the camera. Translucent plastic sheets of various colors are pasted in front of the camera lens to attack the deep learning-based detector. These translucent plastic sheets of different shapes and colors constitute the camera adversarial patch. First, initialize a set of parameters of the translucent plastic sheet, including color, center position, shape, transparency, etc. Second, superimpose the camera adversarial patch and the clean image to construct an adversarial sample. Then, the adversarial sample and the clean image are fed to the detector at the same time, and the detector will output different detection results. Finally, the loss is calculated based on the detection results, and the loss is back-propagated through the gradient to update and optimize the various parameters of the camera adversarial patch, that is, the parameters of a set of translucent plastic sheets. This camera adversarial patch using translucent colored plastic sheets has a very unnatural appearance and is easily noticed. In addition, in actual deployment, it is very difficult to cut plastic sheets of different sizes and carefully arrange the position and corners of each plastic sheet. Summary of the invention

[0005] In view of the above-mentioned problems of the prior art, the present invention proposes a new physically implementable camera pollution antagonistic patch generation method. First, the present invention is inspired by the daily phenomenon that the vehicle-mounted camera is often stained with stains during daily driving, and designs the camera antagonistic patch to have a polluted appearance, that is, the camera pollution antagonistic patch, which has a more natural appearance and is not easy to arouse people's vigilance. Secondly, once the camera pollution patch designed by the present invention is deployed in front of the camera lens, it can attack all target objects in various scenes without the need to paste patches on the target objects one by one. Then, in actual physical deployment, the camera pollution antagonistic patch of the present invention can use water droplets or mud spots as attack media, avoiding the cutting of plastic sheets of different sizes. Moreover, the present invention uniformly sets the shape of the stain to a circle, avoiding the cutting of plastic sheets of different shapes, and there is no need to carefully adjust the corner of the pollution, because a single pollution point is circular. Finally, the camera pollution patch designed by the present invention is target-oriented, that is, the present invention does not indiscriminately attack all targets in the image, but can achieve the attack of specific categories of targets, such as traffic signs and traffic lights, and the detection effect of other targets is still maintained at a high level.

[0006] The specific technical solution is:

[0007] A physically realizable camera contamination adversarial patch generation method, including a training phase and an inference phase;

[0008] The training phase consists of the following steps:

[0009] Training step 1: Initialize the parameters of the camera contamination patch.

[0010] The camera pollution patch designed by the present invention supports multiple pollution modes, which are divided into semi-transparent patches and transparent patches. The pollution modes supported by the semi-transparent patch include mud spots, stains, and frost. The pollution modes supported by the transparent patch include raindrops and water mist. The parameters of the camera pollution patch include the number of pollution points, the center coordinates of the pollution points, the shape of the pollution points, the transparency of the pollution points, and the color of the pollution points.

[0011] Training step 2: Superimpose the clean images in the training set with the camera contamination patches to form adversarial samples.

[0012] Training step 3: Adversarial samples are fed together with clean images to a deep neural network-based detector, which outputs the detection results.

[0013] Training step 4: Based on the output of the detector, calculate the loss function, and then use gradient backpropagation to update and optimize the parameters of the camera contamination patch.

[0014] Training step 5: When the maximum number of training rounds is reached, the training ends and the parameters of the final camera pollution patch are saved.

[0015] The inference phase consists of the following steps:

[0016] Reasoning step 1: Superimpose the trained and optimized camera contamination patch with the clean image to construct an adversarial sample;

[0017] Reasoning step 2: The adversarial sample is fed to various black-box detectors. The target object is not detected in the detection results, but other targets can still be detected.

[0018] Furthermore, the training step 1 includes the following sub-steps:

[0019] 1.1 Translucent patch construction:

[0020] A semi-transparent patch is composed of n semi-transparent pollution points, where n is a manually set parameter. The parameters of each pollution point include:

[0021] (x0, y0)∈[0, 1]: The center coordinates of the pollution point, with the upper left corner of the image as the origin of the coordinate system;

[0022] r∈[r min , r max ]: Radius of pollution point;

[0023] α∈[0,1]: transparency of pollution points;

[0024] c∈R 3 : Pollution point color;

[0025] For a clean image I, the goal is to generate a camera contamination patch M. I(x, y) represents the pixel value of the clean image I at (x, y). The mask of a single contamination point is calculated as follows:

[0026]

[0027]

[0028] Where d(x, y) is the mask of a single contamination point, α(x, y) generates distance-dependent opacity based on the masked area, β is the attenuation factor that controls the decrease in transparency, and ξ prevents data overflow.

[0029] Alpha blending of clean images and camera contamination patches to construct adversarial samples

[0030]

[0031] 1.2 Transparent patch construction:

[0032] A transparent patch is composed of z semi-transparent pollution points, where z is a manually set parameter. The parameters of each pollution point include:

[0033] (i0, j0)∈[-1, 1]: The center coordinates of the pollution point, with the center of the image as the origin;

[0034] a∈[a min , a max ]: the length of the minor axis of the contamination point; the major axis of the contamination point b = 1.25a;

[0035] Blur_radius∈[5, 25]: controls the transparency of the contaminated points;

[0036] The contamination point mask is calculated using the following formula:

[0037]

[0038]

[0039] Among them, mask(i, j) allows a smooth transition between the contaminated point and the clean image, β is the attenuation factor that controls the decrease in transparency, and ω prevents data overflow.

[0040] Each pixel in the contaminated area undergoes Gaussian blur, making the camera contamination patch look natural to the naked eye and difficult for human observers to detect anomalies:

[0041] I trans (i,j)=I(i,j)·mask(i,j)

[0042] I Gauss (i, j) = GaussianBlur(I trans (i, j), σ)

[0043] Here, σ is the standard deviation of the Gaussian function.

[0044] Finally, alpha blending is also used to superimpose the clean image and the camera pollution patch to construct an adversarial sample:

[0045]

[0046] For the semi-transparent patch construction method of 1.1 and the transparent patch construction method of 1.2, the location of the pollution points is initialized as follows: extract the center points of the bounding boxes of all target objects. For semi-transparent patches, cluster these points into 10 regions, and the center of each region is used as the initial location of the 10 pollution points. For transparent patches, cluster the points into 12 regions, and use the center of each region as the initial location of the 12 pollution points.

[0047] The initial position of the contamination point is fixed at the optimal position, and other parameters are optimized during the training process.

[0048] The detector used in the training step 3 is YOLO v5.

[0049] There are three loss functions used in the training step 4: target confidence loss l conf , target disappearance loss l dis , non-target intersection-over-union loss l IoU .

[0050] Target confidence loss l conf : Extract all bounding boxes with the target object, filter out those objects that undergo non-maximum suppression NMS, and suppress the confidence difference between these results and the confidence threshold; the loss calculation formula is as follows:

[0051]

[0052] Where N is the total number of bounding boxes in an image, conf is the confidence threshold of NMS, and C i，obj Represents the confidence of the target object.

[0053] Target disappearance loss dis : Used to compress the area of ​​the bounding box of the target object so that it disappears. The calculation formula is as follows:

[0054]

[0055] Where S = b w b h is the area of ​​the bounding box, b w and b h are the width and height of the bounding box respectively. M is the number of bounding boxes of the target objects.

[0056] Non-target intersection-over-union loss l IoU : Calculate the intersection over union (IoU) between the non-target object in the adversarial sample and the real object in the clean image. The goal is to make this value close to 1. The calculation formula is as follows:

[0057]

[0058] Where Z represents the number of bounding boxes of non-target objects.

[0059] The technical effects of the technical solution of the present invention are as follows:

[0060] 1. The camera contamination patch constructed by the present invention has a natural appearance, and it is difficult for human observers to detect that this naturally occurring lens contamination is a means of attack.

[0061] 2. The camera pollution patch constructed by the present invention is convenient for physical deployment.

[0062] 3. The camera contamination patch constructed by the present invention is highly aggressive and can reduce the detection accuracy of target objects by about 60%, while the detection accuracy of non-target objects only decreases by 25%.

[0063] 4. The detectors of autonomous vehicles in real scenarios are all black box models, and attackers cannot obtain any relevant knowledge, such as network architecture, parameters, etc. The training optimization framework designed by the present invention is query-based. Only YOLO v5 is used for training in the training phase, and it can be migrated to various other detectors in the inference phase. The attack method of the present invention has good portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0065] The specific technical solutions of the present invention are described in conjunction with the accompanying drawings.

[0066] like Figure 1 As shown, the present invention proposes a physically achievable camera pollution adversarial patch generation method, including a training phase and an inference phase. Specifically, the training phase includes the following steps:

[0067] Training step 1: Initialize the parameters of the camera pollution patch. The camera pollution patch designed by the present invention supports multiple pollution modes, which are divided into semi-transparent patches and transparent patches. The pollution modes supported by the semi-transparent patch include mud spots, stains, and frost. The pollution modes supported by the transparent patch include raindrops and water mist. The parameters of the camera pollution patch include the number of pollution points, the center coordinates of the pollution points, the shape of the pollution points, the transparency of the pollution points, the color of the pollution points, etc.

[0068] Training step 2: Superimpose the clean images in the training set with the camera contamination patches to form adversarial samples.

[0069] Training step 3: Adversarial samples are fed together with clean images to a deep neural network-based detector, which outputs the detection results.

[0070] Training step 4: Based on the output of the detector, calculate the loss function, and then use gradient backpropagation to update and optimize the parameters of the camera contamination patch.

[0071] Training step 5: When the maximum number of training rounds is reached, the training ends and the parameter optimization results of the final camera pollution patch are saved.

[0072] Furthermore, the training step 1 includes:

[0073] 1.1 Translucent patch structure (supports three pollution modes: mud spots, stains, and frost):

[0074] A semi-transparent patch is composed of n semi-transparent pollution points, where n is a parameter that can be set manually. The parameters of each pollution point include:

[0075] (x0, y0)∈[0, 1]: The center coordinates of the pollution point, with the upper left corner of the image as the origin of the coordinate system;

[0076] r∈[r min , r max ]: Radius of pollution point;

[0077] α∈[0,1]: transparency of pollution points;

[0078] c∈R 3 : Pollution point color;

[0079] For a clean image I, the goal of the present invention is to generate a camera contamination patch M. I(x, y) represents the pixel value of the clean image I at (x, y). The mask of a single contamination point is calculated by the following formula:

[0080]

[0081]

[0082] Where d(x, y) is the mask of a single contaminated point. α(x, y) generates distance-dependent opacity based on the mask area, ensuring a smooth transition between the edge of the contaminated point and the clean image. β is the attenuation factor that controls the decrease in transparency, and ξ prevents data overflow.

[0083] Finally, the present invention performs α blending on the clean image and the camera contaminated patch to construct an adversarial sample.

[0084]

[0085] 1.2 Transparent patch construction (supports two pollution modes: raindrops and water mist):

[0086] A transparent patch is composed of z semi-transparent pollution points, where z is a parameter that can be set manually. The parameters of each pollution point include:

[0087] (i0, j0)∈[-1, 1]: coordinates of the center of the pollution point, with the center of the image as the origin of the coordinates.

[0088] a∈[a min , a max ]: The length of the minor axis of the contamination point. The major axis of the contamination point b = 1.25a

[0089] Blur_radius∈[5, 25]: controls the transparency of the contaminated points;

[0090] The contamination point mask is calculated using the following formula:

[0091]

[0092]

[0093] Among them, mask(i, j) allows a smooth transition between the contaminated point and the clean image, β is the attenuation factor that controls the decrease in transparency, and ω prevents data overflow.

[0094] Each pixel in the contamination area undergoes Gaussian blurring to mimic the image distortion caused by transparent contamination on the camera lens. This process makes the camera contamination patch look natural to the naked eye and difficult for human observers to detect abnormalities.

[0095] I trans (i,j)=I(i,j)·mask(i,j)

[0096] I Gauss (i, j) = GaussianBlur(I trans (i, j), σ)

[0097] Here, σ is the standard deviation of the Gaussian function.

[0098] Finally, alpha blending is also used to superimpose the clean image and the camera pollution patch to construct an adversarial sample:

[0099]

[0100] In a specific implementation, the present invention uses blur_radius to control the blur degree of the polluted points, which is also the convolution kernel size of the Gaussian blur.

[0101] For the semi-transparent patch construction method of 1.1 and the transparent patch construction method of 1.2, the position of the pollution point is initialized as follows: the present invention extracts the center point of the bounding box of all target objects. For the semi-transparent patch, the present invention clusters these points into 10 areas, and the center of each area is used as the initial position of the 10 pollution points. Similarly, for the transparent patch, the present invention clusters the points into 12 areas, and the center of each area is used as the initial position of the 12 pollution points. During the training process, the present invention observed that making the position of the pollution point learnable will lead to non-convergence because all pollution points tend to move in the same direction at the same time. For example, if the target object is in the upper left corner of the image, all pollution points move to the upper left; if the target object is in the upper right corner, the pollution points will shift accordingly. This constant change in optimization direction hinders stable training. In order to solve this problem, the present invention fixes the initial position of the pollution point at the optimal position and focuses on optimizing other parameters during the training process.

[0102] The detector used in the training step 3 is YOLO v5.

[0103] There are three loss functions used in the training step 4: target confidence loss l conf , target disappearance loss l dis , non-target intersection-over-union loss l IoU .

[0104] Target confidence loss l conf :The present invention extracts all bounding boxes with the target object, filters out those objects that undergo non-maximum suppression (NMS), and suppresses the confidence difference between these results and the confidence threshold. This ensures that the confidence of all target objects does not exceed the threshold. The loss calculation formula is as follows:

[0105]

[0106] Where N is the total number of bounding boxes in an image, conf is the confidence threshold of NMS, and C i，obj Represents the confidence of the target object.

[0107] Target disappearance loss dis : Used to compress the area of ​​the bounding box of the target object so that it disappears. The calculation formula is as follows:

[0108]

[0109] Where S = b w b h is the area of ​​the bounding box, b w and b h are the width and height of the bounding box respectively. M is the number of bounding boxes of the target objects.

[0110] Non-target intersection-over-union loss l IoU : In order to maintain a high detection accuracy for non-target objects, the present invention calculates the intersection over union (IoU) between non-target objects in adversarial samples and real objects in clean images. The goal is to make this value close to 1. The calculation formula is as follows:

[0111]

[0112] Where Z represents the number of bounding boxes of non-target objects.

[0113] The custom loss function of the present invention integrates target object confidence suppression, target object bounding box disappearance and non-target object IoU preservation. Since the entire attack of the present invention is differentiable, the present invention uses the automatic differentiation toolkit to use back propagation to optimize the camera contamination patch parameters.

[0114] The inference phase consists of the following steps:

[0115] Reasoning step 1: Superimpose the trained and optimized camera contamination patch with the clean image to construct an adversarial sample

[0116] Inference step 2: The adversarial sample is fed to various black-box detectors. The target object is not detected in the detection results, but other objects can still be detected. The black-box detectors used in the inference stage are YOLO v2, YOLOv3, YOLOv3tiny, YOLO v4, YOLO v4tiny, Faster R-CNN, SSD, and DETR.

[0117] The present invention can be used in autonomous vehicles to capture image information through cameras. An attacker can put a few drops of water or mud in front of the lens of the vehicle-mounted camera to contaminate the lens. This contamination is carefully designed to make the detector unable to detect the specified target object, while the detection effect of other objects is still good. For example, if the present invention wants to attack the stop sign, a group of camera contamination patches for the stop sign are trained, resulting in the detector not being able to detect the stop sign, but being able to detect other vehicles. In this way, the autonomous vehicle will violate traffic regulations or cause traffic accidents.

Claims

1. A physically realizable camera pollution-resistant patch generation method, characterized in that: It includes training phase and inference phase; The training phase includes the following steps: Training step 1: Initialize the parameters of the camera contamination patch; The camera pollution patch supports multiple pollution modes, which are divided into semi-transparent patches and transparent patches; the pollution modes supported by the semi-transparent patch include mud spots, stains, and frost; the pollution modes supported by the transparent patch include raindrops and water mist; the parameters of the camera pollution patch include the number of pollution points, the center coordinates of the pollution points, the shape of the pollution points, the transparency of the pollution points, and the color of the pollution points; Training step 2: Superimpose the clean images in the training set with the camera contamination patches to form adversarial samples; Training step 3: The adversarial sample and the clean image are fed to a deep neural network-based detector, which outputs the detection result. Training step 4: Calculate the loss function based on the detector output, and then use gradient backpropagation to update and optimize the parameters of the camera contamination patch; Training step 5: When the maximum number of training rounds is reached, the training ends and the parameters of the final camera pollution patch are saved; The inference phase includes the following steps: Reasoning step 1: Superimpose the trained and optimized camera contamination patch with the clean image to construct an adversarial sample; Reasoning step 2: The adversarial sample is fed to various black-box detectors. The target object is not detected in the detection results, but other targets can still be detected.

2. A physically realizable camera contamination resistant patch generation method according to claim 1, characterized in that: The training step 1 includes the following sub-steps: 1.1 Translucent patch construction: A semi-transparent patch is composed of n semi-transparent pollution points, where n is a manually set parameter; the parameters of each pollution point include: (x0,y0)∈[0,1]: The center coordinates of the pollution point, with the upper left corner of the image as the origin of the coordinate system; r∈[r min ,r max ]: Radius of pollution point; α∈[0,1]: transparency of pollution points; c∈R 3 : Pollution point color; For a clean image I, the goal is to generate a camera contamination patch M; I(x, y) represents the pixel value of the clean image I at (x, y); the mask of a single contamination point is calculated by the following formula: Where d(x,y) is the mask of a single contamination point; α(x,y) generates distance-dependent opacity based on the mask area, β is the attenuation factor that controls the decrease in transparency, and ξ prevents data overflow; Alpha blending of clean images and camera contamination patches to construct adversarial samples 1.2 Transparent patch construction: A transparent patch is composed of z semi-transparent pollution points, where z is a manually set parameter; the parameters of each pollution point include: (i0, j0)∈[-1,1]: The center coordinates of the pollution point, with the center of the image as the origin; a∈[a min , a max ]: the length of the minor axis of the contamination point; the major axis of the contamination point b = 1.25a; Blur_radius∈[5,25]: controls the transparency of the contaminated points; The contamination point mask is calculated using the following formula: Among them, mask(i,j) allows a smooth transition between the contaminated point and the clean image, β is the attenuation factor that controls the decrease in transparency, and ω prevents data overflow; Each pixel in the contaminated area undergoes Gaussian blur, making the camera contamination patch look natural to the naked eye and difficult for human observers to detect anomalies: I trans (i,j)=I(i,j)·mask(i,j) I Gauss (i,j)=Gaussian Blur(I trans (i,j),σ) Where σ is the standard deviation of the Gaussian function; Finally, alpha blending is also used to superimpose the clean image and the camera pollution patch to construct an adversarial sample:

3. A physically realizable camera pollution resistant patch generation method according to claim 2, characterized in that: For the semi-transparent patch construction method in substep 1.1 and the transparent patch construction method in 1.2, the location of the contaminated points is initialized as follows: extract the center points of the bounding boxes of all target objects; for semi-transparent patches, cluster these points into 10 regions, and use the center of each region as the initial location of the 10 contaminated points; for transparent patches, cluster the points into 12 regions, and use the center of each region as the initial location of the 12 contaminated points; The initial position of the contamination point is fixed at the optimal position, and other parameters are optimized during the training process.

4. The physically realizable camera pollution resistant patch generation method according to claim 1, characterized in that: The detector used in the training step 3 is YOLO v5.

5. The physically realizable camera pollution resistant patch generation method according to claim 2, characterized in that: There are three loss functions used in the training step 4: target confidence loss l conf , target disappearance loss l dis , non-target intersection-over-union loss l IoU ; Target confidence loss l conf : Extract all bounding boxes with the target object, filter out those objects that undergo non-maximum suppression NMS, and suppress the confidence difference between these results and the confidence threshold; the loss calculation formula is as follows: Where N is the total number of bounding boxes in an image, conf is the confidence threshold of NMS, and C i，obj Represents the confidence of the target object; Target disappearance loss dis : Used to compress the area of ​​the bounding box of the target object so that it disappears; The calculation formula is as follows: Where S = b w b h is the area of ​​the bounding box, b w and b h are the width and height of the bounding box respectively; M is the number of bounding boxes of the target object; Non-target intersection-over-union loss l IoU : Calculate the intersection over union (IoU) between the non-target object in the adversarial sample and the real object in the clean image; the goal is to make this value close to 1; the calculation formula is as follows: Where Z represents the number of bounding boxes of non-target objects.

Images