A physically realizable camera contamination counter patch generation method

Abstract

The application provides a physically realizable camera pollution countermeasure patch generation method, and belongs to the technical field of image processing. The method comprises the following steps: initializing parameters of a camera pollution patch; superimposing clean pictures in a training set and the camera pollution patch to form countermeasure samples; feeding the countermeasure samples and the clean pictures into a detector based on a deep neural network together, and outputting detection results by the detector; calculating a loss function according to the output results of the detector, and updating and optimizing each parameter of the camera pollution patch in a gradient backpropagation mode; ending the training when a maximum training round is reached; superimposing the camera pollution patch after training and optimization and the clean pictures to form countermeasure samples; and feeding the countermeasure samples into various black-box detectors, so that the target object is not detected in the detection results, and other targets can still be detected. The camera pollution patch constructed by the application has a natural appearance, and a human observer cannot detect the naturally occurring lens pollution.

Description

TECHNICAL FIELD

[0001] The application provides a physically realizable camera pollution countermeasure patch generation method, and belongs to the technical field of image processing. BACKGROUND

[0002] Deep neural networks are increasingly applied in daily life due to their powerful functions, such as autonomous driving, face recognition, and the like. However, systems based on deep neural networks are very vulnerable when facing adversarial samples. Adversarial samples are mostly carefully designed noises, which are invisible to humans but can deceive deep neural networks. By constructing various adversarial samples to launch attacks on deep neural networks, various potential risks will be caused. According to different construction fields of adversarial samples, adversarial attacks can be divided into digital attacks and physical attacks. Compared with digital attacks that can only generate single-pixel perturbations, physical attacks consider real deployment and are more threatening to visual systems. Physical attacks must appear natural and avoid attracting people's attention. At the same time, physical attacks should also be convenient for actual deployment.

[0003] At present, in the prior art, a physical patch is pasted on a target object to realize the "invisibility" of the target object in front of a detector. First, the mode of the optimized patch is trained in the digital domain, including color, shape, texture, and the like. Then, the optimized patch is printed out to make a physical patch. Finally, the physical patch is pasted on the target object, such as a pedestrian or a traffic sign, to realize the missed detection of pedestrians or misjudgment of traffic signs by the detector. However, the appearance of the physical patch is very eye-catching and easy to be noticed by people. This kind of physical attack needs to approach and manipulate the target, such as pasting the patch on the target person, which will be noticed by the target person. When the attacker wants to attack multiple targets, the attacker needs to paste a patch on each target, which greatly increases the deployment difficulty and deployment cost of the attack, and is not convenient when attacking multiple targets.

[0004] Another kind of physical attack focuses on attacking cameras, and various colored translucent plastic sheets are pasted in front of the camera lens to attack the detector based on deep learning. These translucent plastic sheets of different shapes and colors constitute camera countermeasures. First, the parameters of a set of translucent plastic sheets are initialized, including color, center position, shape, transparency, and the like. Second, the camera countermeasure and the clean picture are superimposed to construct an adversarial sample. Then, the adversarial sample and the clean picture are fed into the detector at the same time, and the detector outputs different detection results. Finally, the loss is calculated according to the detection results, and the loss is back-propagated through the gradient to update and optimize the parameters of the camera countermeasure, i.e., the parameters of the set of translucent plastic sheets. This camera countermeasure using translucent colored plastic sheets appears very unnatural and is easy to be noticed by people. In addition, when actually deployed, plastic sheets of different sizes need to be cut, and the position and corner of each plastic sheet need to be carefully arranged, which is very difficult to implement. Summary of the Invention

[0005] To address the aforementioned issues of the prior art, the present invention proposes a new, physically feasible method for generating camera contamination-resistant patches. First, inspired by the common phenomenon of vehicle cameras becoming stained during daily driving, the present invention designs a camera contamination-resistant patch that appears to be contaminated. This contamination-resistant patch, therefore, has a more natural appearance and is less likely to arouse alarm. Second, once deployed in front of a camera lens, the camera contamination-resistant patch designed by the present invention can attack all target objects in a variety of scenarios, eliminating the need to apply patches to each target object individually. Furthermore, during physical deployment, the camera contamination-resistant patch of the present invention can use water droplets or mud spots as attack vectors, eliminating the need to cut plastic sheets of varying sizes. Furthermore, the present invention uniformly shapes the contamination into a circular shape, eliminating the need to cut plastic sheets of varying shapes and eliminating the need to carefully adjust the corners of the contamination, as individual contamination points are circular. Finally, the camera contamination-resistant patch designed by the present invention is target-oriented. Rather than indiscriminately attacking all objects in an image, the present invention can attack specific categories of targets, such as traffic signs and traffic lights, while maintaining a high level of detection for other targets.

[0006] The specific technical solutions are:

[0007] A physically realizable camera contamination adversarial patch generation method, including training and inference stages;

[0008] The training phase consists of the following steps:

[0009] Training step 1: Initialize the parameters of the camera contamination patch.

[0010] The camera pollution patch designed in this paper supports multiple pollution modes, including semi-transparent patches and transparent patches. The semi-transparent patch supports pollution modes such as mud spots, stains, and frost. The transparent patch supports pollution modes such as raindrops and water mist. The parameters of the camera pollution patch include the number of pollution points, the center coordinates of the pollution points, the shape of the pollution points, the transparency of the pollution points, and the color of the pollution points.

[0011] Training step 2: Superimpose the clean images in the training set with the camera contamination patches to form adversarial samples.

[0012] Training step 3: The adversarial sample is fed together with the clean image to a detector based on a deep neural network, and the detector outputs the detection result.

[0013] Training step 4: Based on the output of the detector, calculate the loss function, and then use gradient backpropagation to update and optimize the parameters of the camera pollution patch.

[0014] Training step 5: When the maximum training round is reached, the training is finished, and the final camera pollution patch parameters are saved.

[0015] The inference stage includes the following steps:

[0016] Inference step 1: Superimpose the camera pollution patch optimized by training on the clean picture to construct an adversarial sample.

[0017] Inference step 2: Feed the adversarial sample to various black-box detectors, and the target object is not detected in the detection result, while other objects can still be detected.

[0018] Further, the training step 1 includes the following sub-steps:

[0019] 1.1 Semi-transparent patch construction:

[0020] A semi-transparent patch is composed of n semi-transparent pollution points, and n is a human-set parameter. Each pollution point parameter includes:

[0021] (x0, y0) ∈ [0, 1]: Pollution point center coordinates, taking the upper left corner of the image as the origin of the coordinate system;

[0022] r ∈ [r min , r max ]: Pollution point radius;

[0023] α ∈ [0, 1]: Pollution point transparency;

[0024] c ∈ R 3 : Pollution point color;

[0025] For a clean image I, the goal is to generate a camera pollution patch M. I(x, y) represents the pixel value of the clean image I at (x, y). The mask of a single pollution point is calculated by the following formula:

[0026]

[0027]

[0028] Where d(x, y) is the mask of a single pollution point. α(x, y) generates a distance-related opacity according to the mask area, β is an attenuation factor that controls the decrease of transparency, and ξ prevents data overflow.

[0029] Alpha blending is performed on the clean image and the camera pollution patch to construct an adversarial sample

[0030]

[0031] 1.2 Transparent patch construction:

[0032] A transparent patch is composed of z semi-transparent pollution points, z is a parameter set artificially. Each pollution point parameter includes:

[0033] (i0, j0)∈[-1, 1]: pollution point center coordinates, with the center of the image as the coordinate origin;

[0034] a∈[a min , a max ]: the length of the short axis of the pollution point; the long axis of the pollution point b = 1.25a;

[0035] Blur_radius∈[5, 25]: control the transparency of the pollution point;

[0036] The pollution point mask is calculated by the following formula:

[0037]

[0038]

[0039] Where mask(i, j) makes a smooth transition between the pollution point and the clean picture, β is the attenuation factor that controls the decline in transparency, and ω prevents data overflow.

[0040] Each pixel in the pollution point area undergoes Gaussian blur, making the camera pollution patch appear natural to the naked eye and difficult for human observers to detect abnormalities:

[0041] I trans (i, j) = I(i, j)·mask(i, j)

[0042] I Gauss (i, j) = GaussianBlur(I trans (i, j), σ)

[0043] Where σ is the standard deviation of the Gaussian function.

[0044] Finally, the clean image and the camera pollution patch are superimposed using alpha blending to construct the adversarial sample:

[0045]

[0046] For the semi-transparent patch construction of 1.1 and the transparent patch construction method of 1.2, the pollution point position initialization method is as follows: extract the center points of all target object bounding boxes. For semi-transparent patches, these points are clustered into 10 regions, and the center of each region is used as the initial position of the 10 pollution points. For transparent patches, the points are clustered into 12 regions, and the center of each region is used as the initial position of the 12 pollution points.

[0047] The initial position of the pollution point is fixed at the optimal position, and other parameters are optimized during the training process.

[0048] The detector used in the training step 3 is YOLO v5.

[0049] The loss function used in the training step 4 has three: target confidence loss l conf , target disappearance loss l dis , and non-target IoU loss l IoU .

[0050] Target confidence loss l conf : Extract all the bounding boxes of the target objects, filter out those objects that perform non-maximum suppression NMS, and suppress the confidence difference between these results and the confidence threshold; the loss calculation formula is as follows:

[0051]

[0052] Where N is the total number of bounding boxes in an image, conf is the confidence threshold of NMS, and C i，obj represents the confidence of the target object.

[0053] Target disappearance loss l dis : used to compress the area of the bounding box of the target object so that it disappears. The calculation formula is as follows:

[0054]

[0055] Where S = b w · b h is the area of the bounding box, b w and b h are the width and height of the bounding box respectively. M is the number of target object bounding boxes.

[0056] Non-target IoU loss l IoU : Calculate the intersection over union IoU between the non-target object in the adversarial sample and the real object in the clean image. The goal is to make this value close to 1. The calculation formula is as follows:

[0057]

[0058] Where Z represents the number of bounding boxes of non-target objects.

[0059] The technical effects of the technical scheme of the present application are:

[0060] 1. The camera pollution patch constructed in the present application has a natural appearance, and human observers are difficult to detect this naturally occurring lens pollution as an attack means.

[0061] 2. The camera pollution patch constructed by the present invention is convenient for physical deployment.

[0062] 3. The camera contamination patch constructed by the present invention is highly aggressive and can reduce the detection accuracy of target objects by about 60%, while the detection accuracy of non-target objects only decreases by 25%.

[0063] 4. Real-world autonomous vehicle detectors are all black-box models, preventing attackers from gaining access to relevant knowledge, such as network architecture and parameters. Our proposed training optimization framework is query-based, using only YOLO v5 for training during the training phase. This framework can be migrated to other detectors during the inference phase, demonstrating the robust transferability of our attack method. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic flow diagram of the present invention. DETAILED DESCRIPTION

[0065] The specific technical solutions of the present invention are described with reference to the accompanying drawings.

[0066] like Figure 1 As shown, the present invention proposes a physically achievable camera contamination adversarial patch generation method, which includes a training phase and an inference phase. Specifically, the training phase includes the following steps:

[0067] Training Step 1: Initialize the parameters of the camera pollution patch. The camera pollution patch designed in this paper supports multiple pollution modes, including semi-transparent patches and transparent patches. The pollution modes supported by the semi-transparent patch include mud spots, stains, and frost. The pollution modes supported by the transparent patch include raindrops and water mist. The parameters of the camera pollution patch include the number of pollution points, the center coordinates of the pollution points, the pollution point shape, the pollution point transparency, and the pollution point color.

[0068] Training step 2: Superimpose the clean images in the training set with the camera contamination patches to form adversarial samples.

[0069] Training step 3: The adversarial sample is fed together with the clean image to a detector based on a deep neural network, and the detector outputs the detection result.

[0070] Training step 4: Based on the output of the detector, calculate the loss function, and then use gradient backpropagation to update and optimize the parameters of the camera pollution patch.

[0071] Training step 5: When the maximum number of training rounds is reached, the training ends and the parameter optimization results of the final camera pollution patch are saved.

[0072] Furthermore, the training step 1 includes:

[0073] 1.1 Semi-transparent patch construction (supporting three pollution modalities of mud spots, stains, and frost):

[0074] A semi-transparent patch is composed of n semi-transparent pollution spots, n is a parameter that can be set artificially. Each pollution spot parameter includes:

[0075] (x0, y0) ∈ [0, 1]: pollution spot center coordinates, taking the upper left corner of the image as the origin of the coordinate system;

[0076] r ∈ [r min , r max ]: pollution spot radius;

[0077] α ∈ [0, 1]: pollution spot transparency;

[0078] c ∈ R 3 : pollution spot color;

[0079] For a clean image I, the goal of the present invention is to generate a camera pollution patch M. I(x, y) represents the pixel value of the clean image I at (x, y). The mask of a single pollution spot is calculated by:

[0080]

[0081]

[0082] where d(x, y) is the mask of a single pollution spot. α(x, y) generates a distance-dependent opacity according to the mask area, ensuring a smooth transition between the pollution spot edge and the clean image. β is the attenuation factor that controls the decrease in transparency, and ξ prevents data overflow.

[0083] Finally, the present invention performs alpha blending on the clean image and the camera pollution patch to construct the adversarial sample

[0084]

[0085] 1.2 Transparent patch construction (supporting two pollution modalities of raindrops and water mist):

[0086] A transparent patch is composed of z semi-transparent pollution spots, z is a parameter that can be set artificially. Each pollution spot parameter includes:

[0087] (i0, j0) ∈ [-1, 1]: pollution spot center coordinates, taking the center of the image as the origin of the coordinate system.

[0088] a ∈ [a min , a max ]: length of the short axis of the pollution spot. The long axis of the pollution spot b = 1.25a

[0089] Blur_radius∈[5, 25]: control the transparency of the pollution point;

[0090] The pollution point mask is calculated using the following formula:

[0091]

[0092]

[0093] where mask(i, j) makes a smooth transition between the pollution point and the clean picture, β is an attenuation factor that controls the transparency drop, and ω prevents data overflow.

[0094] Each pixel in the pollution point area will undergo Gaussian blur, which is to simulate the image distortion caused by the transparent pollution attached to the camera lens. This processing makes the camera pollution patch look natural to the naked eye, and it is difficult for human observers to detect abnormalities.

[0095] I trans (i, j) = I(i, j) * mask(i, j)

[0096] I Gauss (i, j) = GaussianBlur(I trans (i, j), σ)

[0097] where σ is the standard deviation of the Gaussian function.

[0098] Finally, the clean picture and the camera pollution patch are also superimposed using alpha blending to construct the adversarial sample:

[0099]

[0100] In a specific implementation, the present application uses blur_radius to control the blur degree of the pollution point, which is also the convolution kernel size of Gaussian blur.

[0101] For the semi-transparent patch construction of 1.1 and the transparent patch construction method of 1.2, the position initialization of the pollution points is as follows: the present application extracts the center points of the bounding boxes of all target objects. For the semi-transparent patch, the present application clusters these points into 10 regions, and the center of each region is taken as the initial position of the 10 pollution points. Similarly, for the transparent patch, the present application clusters the points into 12 regions, and the center of each region is taken as the initial position of the 12 pollution points. During the training process, the present application observes that making the position of the pollution points learnable will lead to non-convergence, because all the pollution points tend to move in the same direction at the same time. For example, if the target object is in the upper left corner of the image, all the pollution points move to the upper left; if the target object is in the upper right corner, the pollution points will be offset accordingly. This constant change in optimization direction hinders stable training. In order to solve this problem, the present application fixes the initial position of the pollution points at the optimal position and focuses on optimizing other parameters during the training process.

[0102] The detector used in the training step 3 is YOLO v5.

[0103] The loss function used in the training step 4 has three: target confidence loss l conf , target disappearance loss l dis , and non-target IoU loss l IoU .

[0104] Target confidence loss l conf : the present application extracts all the bounding boxes of the target objects, filters out those objects that have been subjected to non-maximum suppression (NMS), and suppresses the confidence difference between these results and the confidence threshold. This ensures that the confidence of all target objects does not exceed the threshold. The loss calculation formula is as follows:

[0105]

[0106] where N is the total number of bounding boxes in a picture, conf is the confidence threshold of NMS, and C i，obj represents the confidence of the target object.

[0107] Target disappearance loss l dis : used to compress the area of the bounding box of the target object so that it disappears. The calculation formula is as follows:

[0108]

[0109] where S = b w · b h is the area of the bounding box, b w and b h are the width and height of the bounding box respectively. M is the number of target object bounding boxes.

[0110] Non-targeted IoU loss l IoU : In order to maintain high detection accuracy for non-target objects, the present application calculates the intersection over union (IoU) between non-target objects in the adversarial sample and real objects in the clean image. The goal is to make this value close to 1. The calculation formula is as follows:

[0111]

[0112] Where Z represents the number of bounding boxes of non-target objects.

[0113] The custom loss function of the present application integrates target object confidence suppression, target object bounding box disappearance and non-target object IoU maintenance. Since the entire attack of the present application is differentiable, the present application uses the automatic differentiation toolkit to optimize the camera pollution patch parameters using backpropagation.

[0114] The inference stage includes the following steps:

[0115] Inference step 1: superimpose the camera pollution patch optimized by training with the clean picture to construct an adversarial sample

[0116] Inference step 2: feed the adversarial sample to various black box detectors, and the target object is not detected in the detection result, but other objects can still be detected. The black box detector used in the inference stage is YOLO v2, YOLOv3, YOLOv3tiny, YOLO v4, YOLO v4tiny, Faster R-CNN, SSD, DETR.

[0117] The present application can be used in an autonomous vehicle that captures image information through a camera. An attacker can point a few drops of water or mud on the lens in front of the vehicle-mounted camera, causing the lens to be contaminated. This contamination is carefully designed to achieve the effect that the detector cannot detect the specified target object, but the detection effect on other objects is still good. For example, if the present application wants to attack the stop sign, a set of camera pollution patches for the stop sign will be trained, which will cause the detector to fail to detect the stop sign, but other vehicles can be detected. In this way, the autonomous vehicle will violate traffic rules or cause traffic accidents.

Claims

1. A physically feasible camera contamination adversarial patch generation method, comprising a training phase and an inference phase; The training phase includes the following steps: Training step 1: Initialize the parameters of the camera pollution patch; The camera pollution patch supports multiple pollution modes, including semi-transparent patches and transparent patches. The pollution modes supported by the semi-transparent patch include mud spots, stains, and frost; the pollution modes supported by the transparent patch include raindrops and water mist. The parameters of the camera pollution patch include the number of pollution points, the center coordinates of the pollution points, the shape of the pollution points, the transparency of the pollution points, and the color of the pollution points. The following sub-steps are included: 1.1 Translucent patch construction: A translucent patch consists of n translucent pollution points, where n is a manually set parameter; the parameters of each pollution point include: (x0,y0)∈[0,1]: The center coordinates of the pollution point, with the upper left corner of the image as the origin of the coordinate system; r∈[r min ,r max ]: Radius of pollution point; α∈[0,1]: transparency of pollution points; c∈R 3 : pollution point color; For a clean image I, the goal is to generate a camera contamination patch M; I(x,y) represents the pixel value of the clean image I at (x,y); the mask of a single contamination point is calculated as follows: Where d(x,y) is the mask of a single contamination point; α(x,y) generates distance-dependent opacity based on the mask area, β is the attenuation factor that controls the decrease in transparency, and ξ prevents data overflow; Alpha blending of clean images and camera contamination patches to construct adversarial samples 1.2 Transparent patch construction: A transparent patch is composed of z semi-transparent pollution points, where z is a manually set parameter; the parameters of each pollution point include: (i0,j0)∈[-1,1]: coordinates of the center of the pollution point, with the center of the image as the origin; a∈[a min , a max ]: the length of the minor axis of the contamination point; the major axis of the contamination point b = 1.25a; Blur_radius∈[5,25]: controls the transparency of the contaminated points; The contamination point mask is calculated using the following formula: Among them, mask(i,j) allows a smooth transition between the contaminated point and the clean image, β is the attenuation factor that controls the decrease in transparency, and ω prevents data overflow; Each pixel within the contamination area undergoes Gaussian blurring, making the camera contamination patch appear natural to the naked eye and difficult for human observers to detect anomalies: I trens (i,j)=I(i,j)·mask(i,j) I Gauss (i,j)=GaussianBlur(I trens (i,j),σ) Where σ is the standard deviation of the Gaussian function; Finally, we also use alpha blending to superimpose the clean image and the camera contamination patch to construct an adversarial sample: Training step 2: Superimpose the clean images in the training set with the camera contamination patches to form adversarial samples; Training step 3: The adversarial sample and the clean image are fed into a deep neural network-based detector, which outputs the detection result. Training step 4: Based on the detector output, calculate the loss function, and then use gradient backpropagation to update and optimize the parameters of the camera contamination patch; Training step 5: When the maximum number of training rounds is reached, the training ends and the parameters of the final camera pollution patch are saved; The inference phase includes the following steps: Reasoning step 1: Superimpose the trained and optimized camera contamination patch with the clean image to construct an adversarial example; Reasoning step 2: The adversarial sample is fed to various black-box detectors. The target object is not detected in the detection results, but other targets can still be detected.

2. A physically achievable camera contamination resistant patch generation method according to claim 1, characterized in that: For the semi-transparent patch construction method in substep 1.1 and the transparent patch construction method in 1.2, the locations of the contamination points are initialized as follows: extract the center points of the bounding boxes of all target objects; for semi-transparent patches, cluster these points into 10 regions, and use the center of each region as the initial location of the 10 contamination points; for transparent patches, cluster the points into 12 regions, and use the center of each region as the initial location of the 12 contamination points; The initial position of the contamination point is fixed at the optimal position, and other parameters are optimized during the training process.

3. The method for generating a physically realizable camera contamination-resistant patch according to claim 1, wherein: The detector used in the training step 3 is YOLO v5.

4. The method for generating a physically realizable camera contamination-resistant patch according to claim 1, wherein: There are three loss functions used in the training step 4: target confidence loss l conf , target disappearance loss l dis , non-target intersection-over-union loss l IoU ; Target confidence loss l conf : Extract all bounding boxes of the target object, filter out those objects that undergo non-maximum suppression (NMS), and suppress the confidence difference between these results and the confidence threshold; the loss calculation formula is as follows: Where N is the total number of bounding boxes in an image, conf is the confidence threshold of NMS, and C i,obj Represents the confidence of the target object; Target disappearance loss l dis : Used to compress the area of ​​the bounding box of the target object so that it disappears; The calculation formula is as follows: Where S = b w b h is the area of ​​the bounding box, b w and b h are the width and height of the bounding box respectively; M is the number of bounding boxes of the target object; Non-target intersection-over-union loss l IoU : Calculate the intersection over union (IoU) between the non-target object in the adversarial sample and the real object in the clean image; the goal is to make this value close to 1; the calculation formula is as follows: Where Z represents the number of bounding boxes of non-target objects.

Images