Infrared adversarial patch generation method based on evolutionary optimization

Abstract

The invention discloses a method for generating an infrared adversarial patch (Compatite Block Patch Generation Method) based on evolutionary optimization, and the method is used for interfering an infrared target detector. The method comprises the following steps that firstly, a population is initialized, each individual is a particle and comprises parameters such as the position, the rotation angle and the color, and the parameters are used for representing the spatial form and texture characteristics of a patch; secondly, generating an adversarial patch in a complex shape by superposing a plurality of basic square patch units, mapping the adversarial patch to the surface of a three-dimensional vehicle model, generating an infrared image through a three-dimensional rendering engine (such as Pytorch3D), then detecting the rendered image by using a target detection model (such as YOLOv5), calculating a loss function formed by detection confidence and patch smoothness, and finally obtaining a target detection result; taking as a fitness evaluation index; then, mutation, crossover and selection operations are carried out by adopting a genetic algorithm, the population is iteratively optimized, and the adversarial performance of the patch is gradually improved; and finally, when a set termination condition (such as the maximum number of iterations or a fitness threshold) is reached, outputting the optimal patch individual, and realizing the strongest interference on the detector. According to the method, the interference capability to an infrared target detector is remarkably improved, and an efficient infrared confrontation sample generation scheme suitable for a three-dimensional model is provided.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and computer vision, and specifically to an automatic generation method for adversarial patches for infrared three-dimensional target models. Background Art

[0002] With the development of deep learning technology, object detectors based on convolutional neural networks (CNNs) have been widely used in the field of infrared image object recognition, especially showing good performance in key scenarios such as security, military, and autonomous driving. However, such detection systems generally face the risk of adversarial attacks, especially being misled by adversarial patches to cause incorrect recognition results or missed detections.

[0003] Current research on adversarial patches mainly focuses on the 2D image level, lacking systematic research on the attack effects of three-dimensional real objects and their multi-view and multi-scale renderings. Especially in the infrared image scenario, the thermal characteristics and imaging methods of targets are different from visible light images, and the attack effects of traditional 2D patches are limited in infrared images. In addition, existing research mostly optimizes patches based on static images, and fails to fully consider the rotation, occlusion, and lighting changes of targets in real scenarios. Therefore, there is an urgent need for a 3D adversarial patch generation method that combines three-dimensional model rendering and infrared imaging characteristics, and has strong migration and robustness to effectively interfere with target detection systems in infrared images.

[0004] The present invention proposes a new infrared adversarial patch generation method, which is a combined block patch generation method. Complex-shaped adversarial patches are generated by superimposing square patches within a limited area, calculating a set of attack losses, and the mutation, crossover, and selection processes of genetic algorithms are used to continuously optimize the adversarial patches, making the attack effects of the patches on infrared object detectors more significant in the three-dimensional field. Summary of the Invention

[0005] The present invention proposes a 3D adversarial patch generation method based on infrared images. By flexibly combining and optimizing multiple square basic patch units, complex-shaped 3D adversarial patches are formed. The patches can adhere to the surface of a three-dimensional vehicle model and have a significant interference effect on depth object detectors (such as YOLOv5) under multi-angle rendering and infrared simulation imaging.

[0006] The present invention is realized through the following technical solutions:

[0007] Step 1: Initialize the population. Each individual is a particle, including parameters such as position, rotation angle, and color, used to characterize the spatial form and texture characteristics of the patch.

[0008] Step 2: Generate adversarial patches with complex shapes by overlaying multiple basic square patch units, map them to the surface of the 3D vehicle model, generate infrared images through a 3D rendering engine (such as Pytorch3D), then use an object detection model (such as YOLOv5) to detect the rendered images, and calculate a loss function composed of detection confidence and patch smoothness as the fitness evaluation metric;

[0009] Step 3: Perform mutation, crossover, and selection operations using a genetic algorithm, iteratively optimize the population, and gradually improve the adversarial performance of the patches;

[0010] Step 4: When the set termination conditions (such as the maximum number of iterations or the fitness threshold) are reached, output the optimal patch individual to achieve the strongest interference to the detector.

[0011] Further, the specific content of Step 1 is as follows: Initialize the patch particle swarm. Each patch is composed of several square segments, defined as: vertex coordinates: [(x1, y1, z1), (x2, y2, z2), (x3, y3, z3)]; color or texture parameter: T; one particle: P = [t_1, t_2,..., t_N], which contains N square patches in total.

[0012] Further, the specific content of Step 2 is as follows: Combine multiple square patches in an overlay manner to form a patch with a complex shape. For region constraint and multi-block overlay, to ensure that the patch is only distributed on the target region $\Omega$, set the center of each block $(x_i, y_i)$ to satisfy:

[0013]

[0014] After mutation and crossover, project the coordinates outside the region back through the projection operator $\P i _{\Omega}$:

[0015]

[0016] The blocks can overlap or be adjacent to each other, and complex shapes are achieved through the overlay of multiple blocks;

[0017] Apply the patch to the 3D vehicle model, use pytorch3D to render infrared images from multiple angles, then perform object detection on the rendered images through the YOLOv5 detector, and calculate the attack loss, that is, the loss against the target detector. The specific calculation expression is as follows:

[0018] The confidence of the model output is S i (vehicle target)

[0019] There are a total of K rendering perspectives for the angle

[0020] The attack target is to minimize the confidence as much as possible from all perspectives:

[0021]

[0022] It can be extended to the maximum confidence loss (more conservative):

[0023]

[0024] The smoothing loss is used to limit the abruptness of the patch, making it more natural and smooth, and reducing artifacts. The specific calculation expression is as follows:

[0025] Assume the surface color or infrared value of the patch is T(u, v)

[0026] For each sampling point (u, v) in the triangular texture region, calculate the first derivative change:

[0027]

[0028] After discretization, use the Sobel operator or difference approximation:

[0029]

[0030] The specific calculation expression of the total loss function is as follows:

[0031]

[0032] Among them, λ is the weight hyperparameter, which balances the attack strength and the patch smoothness.

[0033] Furthermore, the specific content of step three is as follows: Through the optimization of the genetic algorithm, including the mutation, crossover, and selection processes, generate individuals with better fitness to enter the next generation. The specific calculation expression is as follows:

[0034] Mutation: For each individual X i , select three different individuals X r1 , X r2 , X r3 , and generate a mutation vector:

[0035] V i = X r1 + F · (X r2 - X r3 )

[0036] Among them, F ∈ (0, 2) is the scaling factor.

[0037] Crossover: Use the mutation vector V i and the current individual X i to generate a trial vector U i :

[0038]

[0039] where CR ∈ [0, 1] is the crossover probability;

[0040] Selection: If the objective function value of the test individual is better than that of the current individual, then replace it:

[0041]

[0042] Furthermore, the specific steps of Step 4 are as follows: Perform iterations for multiple generations until the termination conditions of the maximum number of iterations or the error threshold are met, and finally obtain the optimal individual. The specific calculation expressions for the termination conditions are as follows:

[0043] Let the population in the g-th generation be {Xgi}Ni = 1, and the corresponding minimum attack loss be:

[0044]

[0045] Let the maximum number of generations be G max , and the loss threshold be ε. The termination condition of the algorithm is:

[0046]

[0047] Once the above conditions are met, exit the iteration;

[0048] Definition of the optimal individual: When the algorithm terminates at generation g * , the optimal individual X * is defined as the one with the minimum loss in this generation:

[0049] BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0051] Figure 1 is the flow chart of the present invention;

[0052] Figure 2 is the specific implementation flow chart of the method proposed by the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0054] Please refer to Figure 1 As shown, the present invention is a multi-scale image processing method integrating an attention mechanism, including the following steps:

[0055] S101: Initialize the population. Each individual is a particle, including parameters such as position, rotation angle, and color, which are used to characterize the spatial morphology and texture characteristics of the patch.

[0056] S102: Generate adversarial patches with complex shapes by superimposing multiple basic square patch units, map them to the surface of the 3D vehicle model, generate an infrared image through a 3D rendering engine (such as Pytorch3D), and then use an object detection model (such as YOLOv5) to detect the rendered image, and calculate a loss function composed of detection confidence and patch smoothness as the fitness evaluation index.

[0057] S103: Use a genetic algorithm to perform mutation, crossover, and selection operations, iteratively optimize the population, and gradually improve the adversarial performance of the patch.

[0058] S104: When the set termination condition (such as the maximum number of iterations or the fitness threshold) is reached, output the optimal patch individual to achieve the strongest interference to the detector.

[0059] As an optimized solution of the above embodiment, the specific step one is: Initialize the patch particle swarm. Each patch is composed of several square segments, defined as: vertex coordinates: [(x1, y1, z1), (x2, y2, z2), (x3, y3, z3)]; color or texture parameter: T; one particle: P = [t_1, t_2,..., t_N], which contains a total of N square patches.

[0060] As an optimized solution of the above embodiment, the specific step two is: Combine multiple square patches in a superimposed manner to form a patch with a complex shape. For area constraint and multi-block superposition, to ensure that the patch is only distributed on the target area $\Omega$, set the center of each block $(x_i, y_i)$ to satisfy:

[0061]

[0062] After mutation and crossover, project the coordinates outside the area back through the projection operator $\P i _{\Omega}$:

[0063]

[0064] The squares can overlap or be adjacent to each other, and complex shapes can be achieved by superimposing multiple squares;

[0065] Apply the patch to the 3D vehicle model, provide infrared images rendered from multiple angles by pytorch3D, and then perform object detection on the rendered images through the YOLOv5 detector to calculate the attack loss, that is, the loss against the object detector. The specific calculation expression is as follows:

[0066] The confidence level output by the model is S i (Vehicle target)

[0067] There are a total of K rendering perspectives for the angles

[0068] The attack target is to make the confidence level as small as possible for all perspectives:

[0069]

[0070] It can be extended to the maximum confidence loss (more conservative):

[0071]

[0072] The smoothing loss is used to limit the abruptness of the patch, making it more natural and smoother, and reducing artifacts. The specific calculation expression is as follows:

[0073] Assume that the surface color or infrared value of the patch is T(u, v)

[0074] Sample points (u, v) for each triangular texture region and calculate the first-order derivative change:

[0075]

[0076] After discretization, use the Sobel operator or difference approximation:

[0077]

[0078] The specific calculation expression of the total loss function is as follows:

[0079]

[0080] Among them, λ is the weight hyperparameter that balances the attack strength and the patch smoothness.

[0081] As an optimized solution to the above embodiment, step three is specifically: through the optimization of the genetic algorithm, including mutation, crossover, and selection processes, generate individuals with better fitness to enter the next generation. The specific calculation expression is as follows:

[0082] For each individual X, a mutation is performed i , and three different individuals X r1 , X r2 , X r3 are selected to generate a mutation vector:

[0083] V i = X r1 + F · (X r2 - X r3 )

[0084] where F ∈ (0, 2) is a scaling factor.

[0085] For crossover, the mutation vector V i and the current individual X i are used to generate a trial vector U i :

[0086]

[0087] where CR ∈ [0, 1] is the crossover probability;

[0088] For selection, if the objective function value of the trial individual is better than that of the current individual, then replace:

[0089]

[0090] As an optimized solution to the above embodiment, step four is specifically: perform iterations for multiple generations until the termination conditions of the maximum number of iterations or the error threshold are met, and finally obtain the optimal individual. The specific calculation expressions for the termination conditions are as follows:

[0091] Let the population in the g-th generation be {Xgi}Ni = 1, and the corresponding minimum attack loss be:

[0092]

[0093] Let the maximum number of generations be G max , and the loss threshold be ε. The termination condition of the algorithm is:

[0094]

[0095] Once the above conditions are met, exit the iteration;

[0096] Definition of the optimal individual. When the algorithm terminates at generation g * , the optimal individual X * is defined as the one with the minimum loss in this generation:

[0097]

Claims

1. An infrared countermeasure patch generation method that combines artificial intelligence adversarial sample generation technology with evolutionary algorithm optimization strategy, comprising the following steps: Step 1: Initialize the population. Each individual is a particle, including parameters such as position, rotation angle, color, etc., which are used to characterize the spatial form and texture characteristics of the patch. Step 2: Generate complex-shaped adversarial patches by superimposing multiple basic square patch units, map them to the surface of a 3D vehicle model, generate infrared images through a 3D rendering engine (such as Pytorch3D), and then use an object detection model (such as YOLOv5) to detect the rendered images, and calculate a loss function composed of detection confidence and patch smoothness as the fitness evaluation index. Step 3: Use the genetic algorithm to perform mutation, crossover, and selection operations, iteratively optimize the population, and gradually improve the adversarial performance of the patch. Step 4: When the set termination condition (such as the maximum number of iterations or fitness threshold) is reached, output the optimal patch individual to achieve the strongest interference to the detector.

2. The method according to claim 1, wherein The specific content of Step 1 is as follows: Initialize the patch particle swarm. Each patch is composed of several square segments, defined as: vertex coordinates: [(x1, y1, z1), (x2, y2, z2), (x3, y3, z3)]; color or texture parameter: T; one particle: P = [t_1, t_2,..., t_N], which contains N square patches in total.

3. The method according to claim 1, characterized in that The specific content of Step 2 is as follows: Combine multiple square patches in a superimposed manner to form a patch with a complex shape. For region constraint and multi-block superposition, to ensure that the patch is only distributed on the target area $\Omega$, it is set that the center of each square $(x_i, y_i)$ satisfies: After mutation and crossover, the coordinates outside the region are projected back by the projection operator $\P i _{\Omega}$: The squares can overlap or be adjacent to each other, and a complex shape is achieved through the superposition of multiple squares. Apply the patch to the 3D vehicle model, use pytorch3D to render infrared images from multiple angles, and then perform object detection on the rendered images through the YOLOv5 detector, and calculate the attack loss, that is, the loss against the target detector. The specific calculation expression is as follows: The confidence of the model output is S i (Vehicle target) There are a total of K rendering perspectives for the angle The attack target is to make the confidence as small as possible under all perspectives: It can be extended to the maximum confidence loss (more conservative): The smooth loss is used to limit the abruptness of the patch, make it more natural and smooth, and reduce artifacts. The specific calculation expression is as follows: Assume that the surface color or infrared value of the patch is T(u, v) For each sampling point (u, v) in the triangular texture region, calculate the change in the first-order derivative: After discretization, use the Sobel operator or differential approximation: The specific calculation expression of the total loss function is as follows: Among them, λ is a weight hyperparameter that balances the attack intensity and patch smoothness.

4. The method according to claim 3, wherein The specific content of Step 3 is as follows: Through the optimization of the genetic algorithm, including mutation, crossover, and selection processes, generate individuals with better fitness to enter the next generation. The specific calculation expression is as follows: Mutation is applied to each individual X i , three different individuals X r1 , X r2 , X r3 are selected to generate a mutation vector: V i = X r1 + F·(X r2 - X r3 ) Among them, F ∈ (0, 2) is the scaling factor. Crossover uses the mutation vector V i and the current individual X i to generate the trial vector U i : Among them, CR ∈ [0, 1] is the crossover probability; Selection: If the objective function value of the trial individual is better than the current individual, then replace it.

5. The method according to claim 1, wherein Step 4 specifically is as follows: Conduct iterations for multiple generations until the termination condition, i.e., the maximum number of iterations or the error threshold, is met, and finally obtain the optimal individual. The specific calculation expression for the termination condition is as follows: Let the population of the g-th generation be {Xgi}Ni = 1, and the corresponding minimum attack loss is: Let the maximum number of generations be G max , the loss threshold be ε, and the termination condition of the algorithm be: g≥G max or Once the above conditions are met, exit the iteration; Definition of the optimal individual when the algorithm terminates at generation g * , the optimal individual X * is defined as the one with the minimum loss in this generation:

Images