An adversarial attack method and system for deceiving thermal infrared detectors using cold and hot patches

Abstract

An adversarial attack method for deceiving thermal infrared detectors using heat and cold patches, which relates to the field of image recognition technology, includes the steps of: S1. Obtaining an original image sample set, where the original image contains one or more people; S2. Inputting the original image into the HotCold Block model with heat and cold patches as patches, and based on the particle swarm optimization algorithm, through the adversarial optimization algorithm for SSP, optimizing the size, shape, and position of the patches; S3. Replacing the local area of the original image with the patches to obtain an attack image; S4. Inputting the attack image into a DNNs-based thermal infrared detector to achieve an adversarial attack on the thermal infrared detector. The present invention uses the adversarial optimization algorithm for SSP to explore the influence of the size, shape, and position of the patches on the attack effect, improving the attack effect, concealment, and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to an adversarial attack method and system for deceiving a thermal infrared detector using hot and cold patches. Background Art

[0002] Deep neural networks have achieved great success in many fields. They not only work well under visible light but also solve challenging tasks in thermal infrared imaging, such as thermal infrared detection systems, which are widely used in safety-related fields such as autonomous driving, night surveillance, and temperature measurement. However, adversarial attacks occurring in laboratory environments and real physical environments have exposed vulnerabilities in deep neural networks, raising concerns about the security of related applications. The security under visible light has attracted great attention, but the security in thermal infrared imaging has not been fully studied.

[0003] Currently, there is very little research on the security of thermal infrared imaging. There are methods such as using a light-emitting small bulb to create special infrared images and using a new type of aerogel material to design an infrared stealth cloak. Although reasonable attack effects are achieved, they do not look natural enough and are more noticeable when implementing attacks in the real physical environment. Both of these methods belong to adversarial patch attacks. Adversarial patch attacks usually generate a carefully designed patch and place the patch in a specific area of the image, so that the deep neural network makes misjudgments. This attack method replaces the local area of the attacked image with a patch without considering perturbation constraints, which has greater flexibility and is commonly used in physical attacks. In recent years, researchers have tried to balance the effectiveness and concealment of adversarial patch attacks, mainly by designing special structures and textures for the patches, ignoring the influence of the size, shape, and position of the patches on the attack effect. Summary of the Invention

[0004] Aiming at the defects existing in the prior art, the purpose of the present invention is to provide an adversarial attack method and system for deceiving a thermal infrared detector using hot and cold patches, making the attack on the thermal infrared detector more concealed and improving the effectiveness of the patches.

[0005] To achieve the above purpose, on the one hand, an adversarial attack method for deceiving a thermal infrared detector using hot and cold patches is adopted, including the steps of:

[0006] S1. Obtain an original image sample set under thermal infrared imaging, where the original image contains one or more people;

[0007] S2. Input the original image into the HotCold Block model with hot and cold patches as patches, and based on the particle swarm optimization algorithm, optimize the size, shape, and position of the patches through the adversarial optimization algorithm oriented to SSP.

[0008] S3. Replace the corresponding region of the original image with the patch to obtain the attack image;

[0009] S4. Input the attack image into the thermal infrared detector based on DNNs to achieve adversarial attacks on the thermal infrared detector in the digital space;

[0010] S5. In the physical space, construct an entity hot and cold patch according to S1 - S4, place the entity hot patch at the specified position on the human body, and launch an adversarial attack on the thermal infrared detector under thermal infrared imaging.

[0011] Based on the above - mentioned embodiments, in S2, the particle swarm optimization algorithm includes:

[0012] S201. Initialize the patch, randomly initialize the shape M and position P of the patch; the position P is represented by a coordinate set;

[0013] S202. Obtain the attack image according to the input original image and the initial patch, and input the attack image into the thermal infrared detector to obtain the objective function L of the optimization algorithm obj ;

[0014] S203. Based on the particle swarm optimization algorithm, with the principle of obtaining the minimum objective function L obj optimize the size, shape and position of the patch.

[0015] Based on the above - mentioned embodiments, the SSP adversarial optimization algorithm includes:

[0016] Initialize the particle swarm according to the preset number of iterations epoch;

[0017] In each epoch, traverse each image in the image sample set, add the simulation color blocks of the hot and cold patch in the digital space to the image, and then input it into the thermal infrared detector to obtain L obj 1; change the shape M and position P of the simulation color block, add the changed simulation color block to the image, input it into the thermal infrared detector to obtain L obj 2; compare the sizes of L obj 1 and L obj 2, if L obj 1 > L obj 2, then update the particle best position and the particle swarm best position, if L obj 1 < L obj 2, do not process;

[0018] Each time the particle best position is updated, continuously adjust the speed and position of the particle to make the particle move in the direction of reducing L obj

[0019] Based on the above embodiments, optimizing the size of the patch through the adversarial optimization algorithm for the SSP includes:

[0020] Use a nine - grid to simulate the patch in the digital space. The size of the patch depends on the number of patches m, the number of squares n occupied, and the side length v of the nine - grid. The attack effect of the adversarial attack enhances with the increase of the number of patches m and the side length v of the nine - grid.

[0021] Based on the above embodiments, the growing part of the patch size is used as a penalty term to limit the size of the patch, and the objective function L obj is expressed as:

[0022]

[0023] where is the penalty term, V obj represents the confidence of the existence of an object at the pedestrian position; λ is a hyperparameter to prevent V obj from covering the penalty term; Δ ↑ represents positive growth. When it decreases, the penalty term is set to 0.

[0024] Based on the above embodiments, use a 3×3 matrix M to represent the shape of the patch, 0 and 1 represent the state of each square, and the position P = {(x1, y1), (x2, y2), …, (x n , y n )} to determine the position of the upper - left vertex of each patch. n is the number of patches. Take the shape M and position P of the patch as optimization parameters to optimize the size of the patch.

[0025] Based on the above embodiments, in S4, based on the thermal infrared detector f: I → y, a predicted label matching the true label y is derived That is where I represents the original image without adding patches, V pos represents the pedestrian position, V obj represents the confidence of the existence of an object at this position, V cls represents the class confidence of the detected object, and when the attack image is I adv , the objective description is arg min V obj = arg min f(I adv ).

[0026] Based on the above embodiments, in S1, the original image sample set is obtained after filtering by the data set. Each original image contains a person, and the height of the person exceeds 120 pixels.

[0027] Based on the above embodiments, the hot and cold patches refer to warm babies and cooling patches.

[0028] On the other hand, the present invention also provides a counterattack system using hot and cold patches to deceive a thermal infrared detector, including:

[0029] A sample acquisition module for acquiring an original image sample set under thermal infrared imaging, where the original image contains one or more people;

[0030] The HotCold Block model uses hot and cold patches as patches, which is used to input the original image and, based on the particle swarm optimization algorithm, optimize the size, shape, and position of the patches through the adversarial optimization algorithm for SSP;

[0031] An image generation module for replacing the corresponding area of the original image with the patch to obtain an attack image;

[0032] A detection module for inputting the attack image into a thermal infrared detector based on DNNs to achieve an adversarial attack on the thermal infrared detector in the digital space.

[0033] One of the above technical solutions has the following beneficial effects:

[0034] By using wearable hot and cold patches as adversarial patches, a flexible and concealed adversarial attack on the thermal infrared detector can be achieved. The hot and cold patches will affect the infrared imaging and appear as solid color blocks under the thermal infrared detector, which can effectively avoid the human eye and the thermal infrared detector. And the hot and cold patches can be directly attached to the body, making the attack implementation more convenient and with lower cost. At the same time, by using the adversarial optimization algorithm for SSP, the influence of the size, shape, and position of the patches on the attack effect is explored, improving the effectiveness, concealment, and robustness of the attack. Description of the Drawings

[0035] Figure 1 It is a flowchart of the adversarial attack method using hot and cold patches to deceive a thermal infrared detector in an embodiment of the present invention;

[0036] Figure 2 is Figure 1 The specific flowchart of step S2 in Detailed Embodiments

[0037] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] Such as Figure 1As shown in the figure, an embodiment of the countermeasure attack method for deceiving a thermal infrared detector using a hot and cold patch provided by the present invention includes the steps:

[0039] S1. Obtain a set of original image samples, where the original images contain one or more people, and the set of original image samples is captured under a thermal infrared imaging camera.

[0040] S2. Input the original image into the HotCold Block model. The HotCold Block model uses a hot and cold patch as a patch, and based on the particle swarm optimization algorithm, through the adversarial optimization algorithm for SSP, optimizes the size, shape, and position of the patch to improve the success rate of the attack.

[0041] S3. Replace the corresponding area of the original image with the patch to obtain an attack image.

[0042] S4. Input the attack image into the thermal infrared detector based on DNNs to achieve a countermeasure attack on the thermal infrared detector in the digital space;

[0043] S5. In the physical space, by pasting the hot and cold patch on the corresponding area of the human body, launch a countermeasure attack on the thermal infrared detector under thermal infrared imaging.

[0044] It can be understood that the hot and cold patch in the above steps refers to a warming patch and a cooling patch, and can also be other products with similar functions. Since the hot and cold patch forms a black and white block pure color pattern under a thermal infrared camera, this physical phenomenon can be utilized to interfere with the detection of the thermal infrared detector by pasting the hot and cold patch on the human body, achieving the effect of invisibility under the thermal infrared camera. Among them, the warming patch continuously generates heat and can maintain a temperature of 53°C for 6 hours; the cooling patch continuously cools down and can maintain a temperature of 24°C for 4 hours. While providing a stable attack ability, both are highly operable and convenient to launch an attack in the physical space.

[0045] Further, in the above step S1, the performance of this method is evaluated on the FLIR ADAS dataset. In order to better fit the adversarial patch attack, the dataset is filtered, and images that meet the conditions of containing a person in the image and the height of the person exceeding 120 pixels are selected as the original images for testing, and each image is attacked using this method.

[0046] In the above step S2, it specifically includes the following steps:

[0047] S201. Initialize the patch, randomly initialize the shape M and position P of the patch, and the position P is represented by a coordinate set.

[0048] S202. Obtain an attack image according to the input original image and the initial patch, and input the attack image into the thermal infrared detector to obtain the objective function L of the optimization algorithmobj .

[0049] S203. Based on the particle swarm optimization algorithm, in the principle of obtaining the minimum objective function L obj , to optimize the size, shape and position of the patch.

[0050] In the above S203, since each particle in the particle population constantly adjusts the speed and position of the particle, making the particle move in the direction of reducing L obj , the position of each particle and the position of the particle swarm can be updated, so as to optimize the shape and position of the patch.

[0051] Specifically, the SSP adversarial optimization algorithm includes: initializing the particle swarm, according to the preset number of iterations epoch. In each epoch, traverse each image in the image sample set, add the simulated color patches of the hot and cold stickers in the digital space to the image, and then input the image into the thermal infrared detector to obtain L obj 1; change the shape M and position P of the simulated color patch, add the changed simulated color patch to the image, and then input the image into the thermal infrared detector to obtain L obj 2; compare the sizes of L obj 1 and L obj 2. If L obj 1 > L obj 2, update the optimal position of the particle and the optimal position of the particle swarm. If L obj 1 < L obj 2, do not process; until the optimization parameters are stable, obtain the patch with the optimized shape and position.

[0052] In the physical world, it is unreasonable to allow the optimization algorithm to fit any shape, because the shapes of some patches cannot be realized in the physical world, and the area calculation of some shapes is complex. Therefore, a nine-square grid can be used to simulate the patch in the digital space. Optimizing the size of the patch includes: taking the area of the patch as the size of the patch, then the size of the whole patch depends on the number m of patches, the number n of squares occupied, and the side length v of the nine-square grid. Since the smaller the patch, the more beneficial it is to the concealment of the attack, a mechanism is designed to balance the size of the patch and the attack effect.

[0053] Take the growth part of the patch size as the penalty term, and the penalty term is used to limit the size of the patch to make the patch more concealed. The objective function L obj is expressed as:

[0054]

[0055] Among them, is the penalty term, V obj represents the confidence that there is an object at the pedestrian position; λ is a hyperparameter to prevent Vobj Coverage penalty term; Δ ↑ Indicates positive growth. When it decreases, the penalty term is set to 0. A large number of experiments in the digital and physical spaces have proven that the attack effect increases with the increase in the number m of patches and the side length v; when the number is set to 4, the side length is set to 12, and λ is set to 3, the experimental effect is the best.

[0056] In this embodiment, a 3×3 matrix M is used to represent the shape of the patch, and 0 and 1 represent the states of each grid. A large number of patch shapes can be obtained using various combinations. The position P = {(x1,y1),(x2,y2),…,(x n ,y n )}, where n is the number of patches, to determine the position of the upper left vertex of each patch. The shape M and position P of the patch are used as optimization parameters to optimize the size of the patch.

[0057] In step S4 above, YOLOv5 can be used as the target model, which is a fast, accurate, and widely used detector. First, use the pre-trained weights on the MSCOCO dataset, and then fine-tune on the FLIR ADAS dataset.

[0058] Based on the thermal infrared detector f: I→y, a predicted label matching the true label y can be derived That is where I represents the original image without added patches, V pos represents the pedestrian position, V obj represents the confidence that there is an object at this position, V cls represents the class confidence of the detected object, and the attack image is I adv When, the target can be described as arg min V obj = arg min f(I adv ).

[0059] In this embodiment, the average precision AP and the attack success rate ASR are used to evaluate the effectiveness of the adversarial attack method of using hot and cold patches to deceive the thermal infrared detector. Among them, the more AP decreases, the better the attack effect. In addition, ASR is defined as:

[0060]

[0061] where, N is the number of images that actually contain people in the test set, and L pre is the set of people detected by the detector when under attack.

[0062] As Figure 2 shown, an embodiment of applying hot and cold patches to deceive the thermal infrared detector in the physical space is provided, specifically including the steps:

[0063] A10. Obtain the original image sample set under thermal infrared imaging. Specifically, a FLIR ONE Pro camera with a thermal resolution of 160x120 can be used to capture infrared images. During the capture process, the camera is connected to a Xiaomi mobile phone for real-time image display. Record 8 videos in different scenarios (shooting distances from 0 to 4 meters), and then extract one frame per second. A total of 112 pictures are captured, and 224 labels are annotated using LabelImg, which are used as the original image sample set.

[0064] A20. Determine the size, shape, and position of the patch according to the above steps S1 - S4, and construct the solid hot and cold patch.

[0065] A30. Place the solid hot patch on the designated position on the human body, and launch an adversarial attack on the thermal infrared detector under thermal infrared imaging. Specifically, the designated position can be determined based on experience according to different actual situations. Wear another piece of clothing over the hot and cold patch to achieve a hidden effect that cannot be recognized by the human eye. At the same time, under the thermal infrared camera, the hot and cold patch can still be clearly imaged without affecting the attack effect.

[0066] The present invention also provides an embodiment of an adversarial attack system using a hot and cold patch to deceive a thermal infrared detector, which can be used to implement the above method embodiments.

[0067] The above system includes a sample acquisition module, a HotCold Block model, an image generation module, and a detection module. The sample acquisition module is used to obtain the original image sample set under thermal infrared imaging, and the original image contains one or more people; preferably, the height of the person exceeds 120 pixels.

[0068] The HotCold Block model uses the hot and cold patch as a patch, which is used to input the original image, and based on the particle swarm optimization algorithm, through the adversarial optimization algorithm for SSP, optimize the size, shape, and position of the patch.

[0069] The image generation module is used to replace the corresponding area of the original image with the patch to obtain the attack image;

[0070] The detection module is used to input the attack image into the thermal infrared detector based on DNNs to achieve an adversarial attack on the thermal infrared detector.

[0071] Specifically, in the HotCold Block model, initialize the patch, randomly initialize the shape M and position P of the patch, obtain the attack image according to the input original image and the initial patch, and input the attack image into the thermal infrared detector to obtain the objective function L of the optimization algorithm obj , continuously adjust the speed and position of the particle to make the particle move towards reducing L objMove in the direction, update the position of each particle and the position of the particle swarm, so as to optimize the shape and position of the patch until the optimization parameters are stable, and obtain the patch with the optimal shape and position.

[0072] Since the smaller the patch, the more beneficial it is to the concealment of the attack, a mechanism is designed to balance the size of the patch and the attack effect.

[0073] Use a nine-square grid to simulate the patch in the digital space. Take the area of the patch as the size of the patch. Then the overall size of the patch depends on the number m of patches, the number n of squares occupied, and the side length v of the nine-square grid. Take the growing part of the patch size as the penalty term to limit the size of the patch, and the objective function L obj is expressed as:

[0074]

[0075] where is the penalty term, V obj represents the confidence that there is an object at the pedestrian position; λ is a hyperparameter to prevent V obj from covering the penalty term; Δ ↑ represents positive growth. When it decreases, the penalty term is set to 0.

[0076] The position P = {(x1, y1), (x2, y2), …, (x n , y n )}, n is the number of patches, determine the position of the upper left vertex of each patch, and take the shape M and position P of the patch as the optimization parameters to optimize the size of the patch.

[0077] The above are only the embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. An adversarial attack method for deceiving thermal infrared detectors using cold and hot patches, characterized in that, Including the steps: S1. Obtain the original image sample set under thermal infrared imaging, where the original image contains one or more persons; S2. Input the original image into the HotCold Block model with hot and cold patches as patches, and based on the particle swarm optimization algorithm, through the adversarial optimization algorithm for SSP, optimize the size, shape and position of the patches; S3. Replace the corresponding region of the original image with the patches to obtain the attack image; S4. Input the attack image into the thermal infrared detector based on DNNs to achieve adversarial attacks on the thermal infrared detector in the digital space; S5. In the physical space, construct physical hot and cold patches according to S1 - S4, place the physical hot patches at the specified positions on the person, and launch adversarial attacks on the thermal infrared detector under thermal infrared imaging; In the S2, the particle swarm optimization algorithm includes: S201. Initialize the patches, randomly initialize the shape M and position P of the patches; the position P is represented by a coordinate set; S202. Obtain an attack image based on the input original image and the initial patch, and input the attack image into a thermal infrared detector to obtain the objective function of the optimization algorithm ; S203. Based on the particle swarm optimization algorithm, in principle of obtaining the minimum objective function to optimize the size, shape and position of the patch; The SSP adversarial optimization algorithm includes: Initialize the particle swarm according to the preset number of iterations epoch; In each epoch, traverse each image in the image sample set, add the simulated color patches of the hot and cold patches in the digital space to the image, and then input it into the thermal infrared detector to obtain L obj 1; change the shape M and position P of the simulated color patch, add the changed simulated color patch to the image, input it into the thermal infrared detector, and obtain L obj 2; compare L obj 1 and L obj 2 in size. If L obj 1 > L obj 2, update the optimal position of the particle and the optimal position of the particle swarm. If L obj 1 < L obj 2, do not process; Each time the optimal position of the particle is updated, the velocity and position of the particle are continuously adjusted to make the particle move in the direction of decreasing ; In the S2, optimizing the size of the patches through the adversarial optimization algorithm for SSP includes: Using a nine - grid to simulate patches in the digital space, the size of the patch depends on the number of patches , the number of squares occupied and the side length of the nine - grid . The attack effect of the adversarial attack increases with the increase in the number of patches \(m\) and the side length of the nine - grid .

2. The countermeasure attack method for deceiving a thermal infrared detector using a cold-hot patch as claimed in claim 1, wherein, Take the growing part of the patch size as a penalty term to limit the size of the patch, and the objective function is expressed as: ， Among them, is a penalty term, representing the confidence that there is an object at the pedestrian position; is a hyperparameter to prevent the coverage penalty term; represents positive growth. When it decreases, the penalty term is set to 0.

3. The countermeasure attack method for deceiving a thermal infrared detector by using a cold and hot patch according to claim 1, characterized in that Use a 3×3 matrix M to represent the shape of the patch, where 0 and 1 represent the state of each square, and position P , determine the position of the upper left vertex of each patch. Let n be the number of patches. Take the shape M and position P of the patch as optimization parameters to optimize the size of the patch.

4. The countermeasure attack method for deceiving a thermal infrared detector by using a cold-hot patch as claimed in claim 2, wherein In S4, based on the thermal infrared detector derive the predicted label matched with the true label , that is , where represents the original image without added patches, represents the pedestrian position, represents the confidence that there is an object at this position, represents the class confidence of the detected object, and when the attack image is , the target description is .

5. The countermeasure attack method for deceiving a thermal infrared detector by using a cold-hot patch as claimed in claim 1, wherein, In the S1, the original image sample set is obtained after filtering by the data set. Each original image contains a person, and the height of the person exceeds 120 pixels.

6. The adversarial attack method for deceiving a thermal infrared detector using a cold-hot patch as described in any one of claims 1-5, characterized in that, The hot and cold patches refer to warm babies and ice cool patches.

7. An adversarial attack system for deceiving thermal infrared detectors using cold and hot patches, characterized in that, Including: A sample acquisition module for obtaining the original image sample set under thermal infrared imaging, where the original image contains one or more persons; The HotCold Block model with hot and cold patches as patches, which is used to input the original image and, based on the particle swarm optimization algorithm, through the adversarial optimization algorithm for SSP, optimize the size, shape and position of the patches; An image generation module for replacing the corresponding region of the original image with the patches to obtain the attack image; A detection module for inputting the attack image into the thermal infrared detector based on DNNs to achieve adversarial attacks on the thermal infrared detector in the digital space; The particle swarm optimization algorithm includes: S201. Initialize the patches, randomly initialize the shape M and position P of the patches; the position P is represented by a coordinate set; S202. Obtain an attack image based on the input original image and the initial patch, and input the attack image into a thermal infrared detector to obtain the objective function of the optimization algorithm ; S203. Based on the particle swarm optimization algorithm, and in the principle of obtaining the minimum objective function to optimize the size, shape and position of the patch; The SSP adversarial optimization algorithm includes: Initialize the particle swarm according to the preset number of iterations epoch; In each epoch, traverse each image in the image sample set, add the simulated color patches of the cold and hot patches in the digital space to the image, and then input it into the thermal infrared detector to obtain L obj 1; change the shape M and position P of the simulated color patch, add the changed simulated color patch to the image, input it into the thermal infrared detector, and obtain L obj 2; compare L obj 1 and L obj 2 in size. If L obj 1 > L obj 2, update the optimal position of the particle and the optimal position of the particle swarm. If L obj 1 < L obj 2, do not process; Each time the optimal position of the particle is updated, the velocity and position of the particle are continuously adjusted to make the particle move in the direction of decreasing ; Optimizing the size of the patches through the adversarial optimization algorithm for SSP includes: Use a nine - grid to simulate patches in the digital space. The size of the patch depends on the number of patches , the number of squares occupied and the side length of the nine - grid . The attack effect of the adversarial attack increases with the increase in the number m of patches and the side length of the nine - grid .

Images