Physical omnidirectional antagonistic interference method for 3d infrared target detection system

Abstract

The invention discloses a physical omnidirectional antagonistic interference method for a 3d infrared target detection system, and belongs to the technical field of artificial intelligence safety and infrared image processing. The method comprises the following steps: firstly, carrying out three-dimensional modeling and UV texture expansion on a target vehicle, and constructing a patch based on particle information; then generating a multi-view infrared confrontation sample covering multiple groups of pitch angles, horizontal angles and observation distances by utilizing Pytorch3D rendering; further constructing a multi-target fitness function considering the minimization of the detection confidence and the minimization of the patch area, and jointly optimizing the position, the shape and the size of the patch through a self-adaptive particle swarm optimization algorithm; and finally, introducing an expectation transformation (EOT) mechanism, and simulating imaging noise and patch offset in complex environments such as rain, fog, sand and dust, so as to verify the environment robustness of the scheme. The designed flexible disturbance patch can be physically arranged on the surface of a vehicle, and full-view infrared interference detection is achieved.

Description

Technical Field

[0001] The present invention relates to the cross - field of artificial intelligence security and infrared image processing technology, and specifically to a physical omnidirectional adversarial interference method for a 3D infrared target detection system. This method combines three - dimensional modeling, thermal radiation physical mapping, and intelligent optimization algorithms to achieve target interference on infrared detection systems in complex environments, improving physical realizability and full - perspective robustness. Background Art

[0002] With the wide application of infrared target detection technology in fields such as security monitoring, autonomous driving, and military reconnaissance, its security issues have attracted increasing attention. Most existing adversarial attack technologies generate digital perturbation samples based on two - dimensional infrared images. Although they have certain attack effects in simulation environments, they have the following significant limitations when physically deployed:

[0003] 1. Poor perspective adaptability: Two - dimensional image perturbations based on a single perspective are difficult to meet the multi - angle observation requirements of infrared detection systems, and do not achieve omnidirectional attacks, resulting in unstable interference effects in real scenes. Moreover, existing methods mostly ignore environmental perturbation modeling;

[0004] 2. Physical non - realizability: Traditional perturbations do not consider the radiation characteristics of thermal control materials and the fitting structure constraints, and cannot be directly mapped to the surface of real objects;

[0005] 3. Low environmental robustness: Existing methods lack modeling of infrared characteristics in complex environments such as rain, fog, sand, and dust, and are difficult to resist interference such as physical imaging noise and pasting errors.

[0006] Therefore, there is an urgent need for an adversarial perturbation scheme that combines multi - perspective adaptability, physical deployability, and environmental robustness to improve the security defense capabilities of infrared systems in actual scenarios. Summary of the Invention

[0007] To overcome the above problems, the present invention proposes a physical adversarial interference method based on UV texture mapping - three - dimensional multi - angle rendering - adaptive particle swarm optimization, and constructs an adversarial perturbation patch configuration scheme with thermal physical significance and multi - perspective generality for 3D infrared detection systems. Its technical key points include:

[0008] I. Overview of the Method Process

[0009] The interference method of the present invention mainly includes the following steps:

[0010] S101. Target information collection and pre - processing

[0011] Based on the FLIR infrared dataset and the self - built infrared roof image set, use a pre - trained target detection model (such as YOLOv5) to obtain the position and feature information of the vehicle to be attacked in the image, as the input for subsequent texture mapping;

[0012] S102. 3D Model Disassembly and UV Texture Map Processing

[0013] The infrared photos captured by the camera are all two-dimensional images. The challenge lies in how to "paste" these two-dimensional infrared images onto the three-dimensional car mesh model. First, we flatten all the faces of the 3D car mesh onto a 2D plane called the face map. Then, we use MAYA software to rearrange these faces to divide different regions, such as the roof, doors, front of the car, etc., to obtain the UV texture map. The UV texture map is a representation that maps the information on the surface of the 3D model onto a 2D plane, facilitating the patching operation on the model surface. Mark four key regions, namely the doors, roof, engine, and rear of the vehicle, in the UV texture map. These regions are relatively prominent and easily detectable in the infrared imaging of the vehicle. Interfering with them can effectively reduce the detectability of the vehicle;

[0014] Next, establish a sixteen-grid patching simulation space, and configure shape parameters, size parameters, and position parameters for each candidate patch. The size parameter determines the size of the patch on the UV texture map, and the position parameter determines the specific position of the patch within a specific region. Through this parameterized method, the attributes of the patch can be flexibly adjusted to achieve the best adversarial effect;

[0015] S103. Particle Information Patching Mapping and Adversarial Sample Image Generation

[0016] By initializing particle information, map the candidate patch parameters to specific regions of the UV texture map, and generate multi-view images at different pitch angles, horizontal angles, and distances through the Pytorch3D renderer. Use the 3D renderer to generate a 2D adversarial sample image set containing different pitch angles (0 - 90°), horizontal angles (0 - 360°), and distance parameters. By adjusting these parameters, it is possible to simulate the situation when observing the target vehicle from different angles and distances, ensuring that the adversarial patch can play an effective interference role from multiple perspectives. The generated adversarial sample image set will be used in the subsequent optimization process to evaluate the adversarial effects of different patch configuration schemes;

[0017] S104. Application of Particle Swarm Optimization Algorithm

[0018] Construct a multi-objective fitness function based on minimizing the target confidence, introduce an area penalty term to balance the concealment, and use an adaptive particle swarm optimization algorithm to jointly optimize the patch position, size, and shape parameters, and iteratively output the optimal perturbation configuration.

[0019] Adaptive PSO Algorithm:

[0020] Particle Velocity Update Equation:

[0021]

[0022] Velocity vector of the i-th particle at the d-th iteration

[0023] Adaptive inertia weight coefficient

[0024] c1, c2: Learning factors

[0025] r1, r2: Random numbers uniformly distributed in [0, 1]

[0026] Adaptive inertia weight calculation:

[0027]

[0028] Current fitness value of the i-th particle

[0029] Population average fitness

[0030] w max = 0.9, w min = 0.4: Weight upper and lower bounds

[0031] Then introduce a dynamic adjustment mechanism:

[0032] When the change rate of the optimal fitness ΔF for k = 5 consecutive generations best < 1%, activate weight reset:

[0033]

[0034] γ = 0.05 is the decay coefficient, and d is the current iteration number

[0035] The design of the fitness function is the key to the optimization process. This function takes the weighted combination of the detection confidence decline rate of the model output and the proportion of the total patch area as the evaluation index. By maximizing the fitness function, it is possible to control the total patch area while reducing the detection confidence of the target, so as to achieve a balance between concealment and attack effect;

[0036] In addition, a dynamic inertia weight strategy is adopted to update the particle velocity to achieve adaptive optimization. When the optimization goal is to minimize the detection confidence, the inertia weight is dynamically adjusted according to the comparison between the current fitness value of the particle and the population average fitness, ensuring that the particle can fully explore the parameter space during the search process and quickly converge to the optimal solution. When the change rate of the optimal fitness for multiple consecutive generations is small, activate the weight reset mechanism to avoid the algorithm falling into a local optimum.

[0037] Fitness function:

[0038] Taking the rendered image as the training set, the optimization effect of the patch configuration scheme is calculated through the following multi-objective fitness function:

[0039] arg minV obj =arg min i f(I adv )

[0040] Define the optimization objective of the adversarial attack. By generating an adversarial sample I adv , make the target confidence V obj output by the detector minimized, so that the detector cannot recognize the target.

[0041]

[0042] Attack effectiveness term: V obj , directly minimize the target confidence;

[0043] A penalty term is introduced to limit the size (area) of the patch to balance the attack effect and concealment;

[0044] m: number of patches;

[0045] n: number of grids occupied;

[0046] l: side length of the sixteen-square grid;

[0047] λ: hyperparameter, controlling the weight of the penalty term.

[0048] Define the Attack Success Rate (ASR) to quantify the attack effectiveness:

[0049]

[0050] N: number of true positive samples detected without attack;

[0051] L pre : set of labels predicted by the detector after being attacked;

[0052] If a sample is not detected after the attack (i.e., lable i does not belong to L pre ), it is recorded as a successful attack.

[0053] S105. Evaluation of interference effect and environmental robustness

[0054] Introduce the Expectation Over Transformation (EOT) algorithm to simulate sensor noise and patch offset, evaluate the interference success rate in various infrared environments such as rain, fog, and dust, and deploy and verify its attack effect and robustness in the physical real scene. Perform ±10% random perturbation on the grayscale value of S adv to generate samples conforming to N(μ, σ2 ) A noise matrix of Gaussian distribution, where μ is the original gray value and σ = 0.15μ; a random perturbation of [-5%, +5%] relative offset is applied to the patch position P, and the offset direction follows a uniform distribution in the UV texture map plane; a transformation space T ∼ {noise, displacement} is constructed through Monte Carlo sampling, and the sensor errors in reality (such as infrared imaging noise) and physical deformations (such as patch adhesion deviation) are simulated by randomly sampling the noise intensity (±10% gray perturbation) and position offset (±5% position perturbation).

[0055] Calculate the expected loss function of the adversarial sample: E t ~ T [L obj (t(S adv ))]

[0056] During the particle swarm optimization process, the above expected loss is used as the fitness evaluation benchmark to enhance the robustness of the patch scheme in complex environments.

[0057] In the physical environment verification stage, adversarial patches are actually deployed, and aluminum film is used to change the surface emissivity of the object to further verify its interference effect and environmental adaptability in the real scene. By comparing the output results of the target detection model with and without patches, the interference success rate of the patches is quantified, and its performance in different environmental conditions is evaluated.

[0058] II. Technical Features and Innovations

[0059] The present invention has the following significant advantages compared with the prior art:

[0060] (1) Omnidirectional perspective interference ability: Generate multi-perspective images through 3D rendering to achieve full-angle perturbation adaptation of the infrared system, and make up for the problem that two-dimensional perturbations cannot be migrated across perspectives;

[0061] (2) Strong physical realizability: Construct a patch radiation characteristic mapping model and combine the actual flexible aluminum film material design to achieve an operable path from digital perturbation to physical deployment;

[0062] (3) High-robustness design: Introduce the EOT expected perturbation mechanism to enhance the robustness of the adversarial strategy by simulating environmental and device errors, and significantly enhance the adaptability to the real scene;

[0063] (4) High optimization efficiency: Adopt a dynamic inertia factor and weight reset strategy to improve the global search and convergence performance of the particle swarm algorithm, and effectively support the interference patch search task in the high-dimensional parameter space.

[0064] III. Application Scenarios and Advantages

[0065] (1) Application Scenarios

[0066] The present invention is mainly applied to scenarios that require adversarial interference against infrared target detection systems, such as military defense, privacy protection and other fields. In military defense, by performing infrared stealth interference on targets such as vehicles and weaponry, the probability of the target being detected by enemy infrared detection equipment can be reduced. In terms of privacy protection, it can be used to protect the privacy information of individuals or vehicles under infrared surveillance, preventing unauthorized infrared detection and recognition.

[0067] (II) Advantages

[0068] Strong omnidirectional interference ability

[0069] By constructing a sixteen-grid patch space and multi-view three-dimensional rendering technology, covering the full pitch and horizontal angles, the interference ability of the patch against the infrared detection system in the full-angle field of view is significantly enhanced, solving the problem that traditional 2D perturbations cannot be transferred across perspectives.

[0070] High concealment and naturalness

[0071] By introducing an area penalty term during the optimization process to control the patch volume, and combining with the mapping of the thermal radiation characteristics of physical materials, the patch has a natural camouflage effect in the infrared image, is difficult to be detected by observers, and has strong concealment performance.

[0072] Excellent environmental adaptability

[0073] Introduce the EOT expected transformation mechanism, perform random perturbations on the infrared image grayscale and patch position, simulate natural interference factors such as rain, fog, and dust, and verify through digital simulation and field deployment, effectively enhancing the robustness of the countermeasure strategy in a changing environment.

[0074] Excellent physical realizability

[0075] Select flexible aluminum film materials with controllable emissivity, with a thickness less than 0.1 mm, which are convenient to attach and have a compliant structure. The parameter patches designed by combining three-dimensional modeling and optimization algorithms can be directly physically deployed on the target surface, breaking through the limitation that traditional digital perturbations cannot be implemented.

[0076] In summary, through a series of innovative methods and steps, the present invention provides a countermeasure solution for infrared target detection systems with cross-perspective, high concealment, environmental robustness, and physical realizability, having broad application prospects and important practical value in multiple fields. Description of the Drawings

[0077] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0078] Figure 1 It is a schematic flowchart of the present invention;

[0079] Figure 2 It is a schematic diagram of the model structure of the present invention; Detailed implementation manners

[0080] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the drawings of the embodiments of the present invention. It should be understood that the following description is only an exemplary embodiment of the present invention and does not limit the protection scope of the present invention. Those skilled in the art can perform equivalent transformations or improvements without creative efforts and still fall within the protection scope of the present invention.

[0081] S101: First, construct an infrared image dataset as the input data source. The data includes the publicly available FLIR infrared image set and the self - collected infrared image set of the vehicle roof to ensure coverage of more top - down perspectives in real scenarios.

[0082] Then, use a pre - trained infrared vehicle detection model based on the YOLOv5 architecture to perform inference on the images, and extract the detection boxes (bounding boxes), class labels, and key features of the target vehicles. These information will be used for 3D modeling and patch area positioning.

[0083] S102: In this step, pre - process the 3D model (.obj file format) of the target vehicle and complete the following operations using MAYA tools:

[0084] 1. Unfold the 3D mesh into a 2D plane to obtain a complete UV texture map of the vehicle;

[0085] 2. Select the areas on the outer surface of the vehicle with strong infrared radiation characteristics (such as the roof, doors, hood, and rear of the vehicle), and calibrate them in the form of bounding boxes (bbox) on the UV map;

[0086] 3. Construct a 4×4 sixteen - grid patch space within each calibrated area to define the deployable range of the patches. Each unit can be configured as a rectangular or triangular patch with adjustable parameters such as the center position, size, and rotation angle.

[0087] The above - mentioned UV calibration map will be used as the core input for patch design and simulation rendering.

[0088] S103: In this step, a patch mapping model is constructed by initializing particle patch information:

[0089] Map the constructed patch parameters to the specified positions on the UV map. After synthesizing the patch-texture map, use the Pytorch3D rendering engine to load the 3D vehicle mesh model for rendering. The rendering settings are as follows:

[0090] Pitch angle: 0° - 90°, in 5° increments;

[0091] Horizontal angle: 0° - 360°, in 15° increments;

[0092] Distance: Set three distance levels of near, medium, and far to simulate actual imaging conditions.

[0093] S104: Input the adversarial sample image set into the object detection model, and use the particle swarm optimization algorithm. With the detection suppression effect as the optimization goal, jointly adjust the patch size, shape, and position parameters. After iterative calculation, generate the optimal thermal perturbation patch scheme. Adaptive PSO algorithm:

[0094] Particle velocity update equation:

[0095]

[0096] The velocity vector of the i-th particle at the d-th iteration;

[0097] Adaptive inertia weight coefficient;

[0098] c1, c2: Learning factors;

[0099] r1, r2: Random numbers uniformly distributed in [0, 1].

[0100] Adaptive inertia weight calculation:

[0101] When the optimization goal is to minimize the detection confidence (minV obj ):

[0102]

[0103] The current fitness value of the i-th particle;

[0104] Population average fitness;

[0105] w max = 0.9, w min = 0.4: Weight upper and lower bounds.

[0106] Then introduce a dynamic adjustment mechanism:

[0107] When the change rate of the optimal fitness ΔF for k = 5 consecutive generations best < 1%, the activation weights are reset:

[0108]

[0109] γ = 0.05 is the attenuation coefficient, and d is the current iteration number

[0110] The design of the fitness function is the key to the optimization process. This function takes the weighted combination of the detection target confidence drop rate output by the model and the proportion of the total patch area as the evaluation index. By maximizing the fitness function, while reducing the detection confidence of the target, the total patch area can be controlled to achieve a balance between stealth and attack effect.

[0111] In addition, a dynamic inertia weight strategy is adopted to update the particle velocity for adaptive optimization. When the optimization goal is to minimize the detection confidence, according to the comparison between the current fitness value of the particle and the population average fitness, the inertia weight is dynamically adjusted to ensure that the particle can fully explore the parameter space during the search process and quickly converge to the optimal solution. When the change rate of the optimal fitness for multiple consecutive generations is small, the activation weight reset mechanism is activated to prevent the algorithm from falling into a local optimum.

[0112] Taking the rendered image as the training set, the optimization effect of the patch configuration scheme is calculated through the following multi-objective fitness function:

[0113] arg minV obj = arg min i f(I adv )

[0114] Define the optimization goal of the adversarial attack. By generating an adversarial sample I adv , the target confidence V obj output by the detector is minimized, so that the detector cannot recognize the target (i.e., "hide" the target).

[0115]

[0116] Attack effectiveness term: V obj , directly minimize the target confidence.

[0117] A penalty term is introduced to limit the size (area) of the patch to balance the attack effect and stealth.

[0118] m: number of patches;

[0119] n: number of occupied grids;

[0120] l: side length of the 16-grid (unit: pixel);

[0121] λ: Hyperparameter that controls the weight of the penalty term.

[0122] Define the Attack Success Rate (ASR) to quantify the effectiveness of the attack:

[0123]

[0124] N: The number of true positive samples detected without being attacked;

[0125] L pre : The set of labels predicted by the detector after being attacked;

[0126] If a sample is not detected after the attack (i.e., the label i does not belong to L pre ), then it is recorded as a successful attack.

[0127] S105: To verify the interference effect of the designed patch in a real complex environment, an Expectation over Transformation (EOT) mechanism is introduced to enhance the reliability of the simulation. The EOT process includes:

[0128] (1) Apply ±10% Gaussian perturbation (σ = 0.15μ) to the grayscale value of the patch;

[0129] (2) Add a ±5% random offset to the position of the patch on the UV map to simulate the pasting error;

[0130] (3) Generate a set of transformation space samples T based on Monte Carlo sampling to estimate the expected loss function: E t ~ T [L obj (t(S adv ))]

[0131] Finally, in the real scenario, the optimized patch is printed as a flexible infrared perturbation sticker and deployed on the surface of the real vehicle. Use an infrared camera to collect images, evaluate the change in the output results of the target detection model, quantify the Attack Success Rate (ASR), and record the environmental robustness performance.

Claims

1. A physical adversarial interference method for a 3D infrared target detection system, characterized in that It includes the following steps: S101: Obtain infrared image data, and use a pre-trained infrared target detection model to extract the position and feature information of the target vehicle, where the image data includes the FLIR infrared dataset and self-collected infrared roof images; S102: Perform UV texture unwrapping on the target vehicle, calibrate the key areas of the doors, roof, hood, and rear of the vehicle by region, establish a 4×4 sixteen-grid patch simulation space in the UV texture map, and configure shape, size, and position parameters for each candidate patch; S103: Map the particle initialization patch to the specified area of the UV texture map, and generate multi-view two-dimensional images through a 3D renderer, where the views include multiple sets of pitch angles, horizontal angles, and observation distance parameters; S104: Input the multi-view two-dimensional images into the target detection model, construct a fitness function with the goal of minimizing the detection confidence, and use the particle swarm optimization algorithm to jointly optimize the patch size, shape, and position parameters, and output the optimal perturbed patch configuration scheme; S105: Deploy the perturbed patch in the digital simulation and the on-site physical environment respectively, and evaluate its interference effectiveness and robustness indicators on the target detection system under complex environmental conditions.

2. The method according to claim 1, characterized in that, The processing of the UV texture map in step S2 includes: Perform texture unwrapping on the target vehicle model through 3D modeling software to generate a planar UV texture map; calibrate the door, roof, hood, and rear areas in the UV map; Construct a 4×4 grid structure in each calibrated area to form a patch candidate space, and each grid cell supports rectangular or triangular patch shapes and has adjustable size and position parameters.

3. The method according to claim 1, wherein Particle initialization mapping includes: Map the particle information patch owned by particle initialization to the specified area of the UV texture map. Generate multi-view two-dimensional images through a 3D renderer, where the views include multiple sets of pitch angles, horizontal angles, and observation distance parameters.

4. The method according to claim 1, wherein The implementation method of the particle swarm optimization algorithm includes: Define a multi-dimensional parameter search space, and each particle represents a set of patch configuration schemes, and its dimensions include the center coordinates (x, y) of the patch, the rotation angle θ, the aspect ratio α, and the size scaling factor s; Set the fitness function as a weighted combination between the detection confidence decline rate and the proportion of the total patch area, which is used to measure the balance between the interference effect and the concealment; Adopt a dynamic inertia weight strategy with an adaptive adjustment mechanism to update the particle velocity, enhance the search ability of the particles in the parameter space, and improve the global optimization performance. Particle velocity update equation: The velocity vector of the i-th particle at the d-th iteration; Adaptive inertia weight coefficient; c1, c2: Learning factors; r1, r2: Random numbers uniformly distributed in [0, 1]; Adaptive inertia weight calculation: The current fitness value of the i-th particle; Population average fitness; w max = 0.9, w min = 0.4: Weight upper and lower bounds; Dynamic adjustment mechanism: When the change rate ΔF of the optimal fitness for k = 5 consecutive generations best is less than 1%, the activation weight is reset: γ = 0.05 is the attenuation coefficient, and d is the current iteration number.

5. The method according to claim 1, characterized in that The 3D renderer adopts the Pytorch3D framework, and the rendering parameters include: The pitch angle range is from 0° to 90°, and the horizontal angle range covers a 360° omnidirectional view; The generated rendered images are used as the optimized input image set and participate in the optimization process of the patch parameter configuration; arg minV obj = arg min i f(I adv ) The optimization objective is defined as minimizing the confidence V output by the object detector in the adversarial example generation task, so as to achieve the adversarial interference effect of masking the target. obj ​ Attack effectiveness term: V obj , directly minimize the target confidence; A penalty term is introduced to limit the size (area) of the patch to balance the attack effect and the concealment; m: Number of patches; n: Number of occupied grids; l: Side length of the sixteen-grid λ: Hyperparameter that controls the weight of the penalty term. Define the Attack Success Rate (ASR) to quantify the effectiveness of the attack: N: The number of true positive samples detected without being attacked; Lpre: The set of labels predicted by the detector after being attacked; If a certain sample is not detected after the attack (i.e., the lable i does not belong to L pre ), then it is recorded as a successful attack.

6. The method according to claim 1, wherein The environmental robustness evaluation includes: Construct various complex meteorological conditions during the simulation verification phase, including infrared imaging scenarios in rainy and foggy, snowy, and sandy environments, to evaluate the interference stability of the patch solution; Introduce the Expected Output Transformation (EOT) algorithm to perform random perturbation processing on the generated adversarial patch image S adv including: (1) Apply a random perturbation of ±10% to the grayscale values of the patch images to generate a noise image that conforms to the N(μ, σ 2 ) distribution, where μ is the original grayscale value and σ = 0.15μ; (2) Randomly offset the position P of the patch on the UV texture map within the range of [-5%, +5%], and the offset direction follows a uniform distribution within the texture plane; (3) Construct a transformation space T based on Monte Carlo sampling, including the above noise perturbation and position perturbation, to simulate the sensor imaging error and physical patch deviation, and embed this transformation space into the optimization process as a robustness evaluation index. Calculate the expected loss function of adversarial examples: E t ~ T [L obj (t(S adv ))] Use the above expected loss as the fitness evaluation benchmark during the particle swarm optimization process to enhance the robustness of the patch solution in complex environments.

7. A physical countermeasure device for implementing the method according to any one of claims 1-6, characterized in that, Include: A flexible patch unit composed of an aluminum film material with a specific infrared emissivity. The patch can be attached to the surface of the target vehicle and change its local thermal radiation characteristics to achieve anti-interference; The patch has a thickness of 0.08 mm, with good flexibility and conformability, can adapt to different vehicle body surfaces, is convenient for rapid deployment, and maintains appearance concealment.

Images