Infrared confrontation sample generation method based on material attribute optimization
The infrared adversarial sample generation method with optimized material properties utilizes a differentiable renderer and optimized material properties to generate robust infrared adversarial samples, solving the problem that adversarial samples in existing technologies are easily affected by the environment and achieving more stable adversarial effects.
Patent Information
- Application Number
- CN202510662362.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-22
- Publication Date
- 2025-10-28
Smart Images
Figure CN120852268A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and in particular to a method and system for generating infrared adversarial samples based on material property optimization. Background Technology
[0002] With the continuous development of artificial intelligence technology, the security of intelligent models has become increasingly important, giving rise to the field of adversarial attacks. Adversarial attacks refer to deceiving a model by altering its inputs, causing it to output incorrect results. Based on their domain, adversarial attacks can be divided into digital domain attacks and physical domain attacks. Digital domain attacks involve attackers directly adding minute perturbations to the input data in the digital space to deceive models based on deep neural networks. Physical domain attacks, on the other hand, use physical means (such as applying adversarial patches or interfering with lighting) to alter the target object or environment, making the collected data contain adversarial characteristics. Compared to digital domain attacks, physical domain attacks are more challenging, as they rely on establishing a precise mapping relationship between the physical properties of materials and the imaging characteristics of sensors. In the visible light domain, breakthroughs in differentiable rendering technology have made it possible to construct high-fidelity correlation models of material reflectivity, geometric deformation, and imaging color in the digital domain. This guides the optimized design of physical carriers such as adversarial stickers, significantly promoting the engineering implementation of visible light physical adversarial examples. However, in the infrared domain, due to the lack of differentiable rendering techniques for adversarial example generation, existing methods remain limited to grayscale space optimization. The fundamental flaw of this approach is that the grayscale values in infrared imaging are essentially determined by the target surface's thermal radiation properties (emissivity, temperature field, etc.) and the imaging system, rather than by a virtual texture independent of the material. Therefore, adversarial examples that simply optimize the image's grayscale distribution are susceptible to uncontrollable physical factors: in real-world scenarios, the target's radiation distribution is dynamically affected by the environment (such as sunlight and wind speed) and internal heat sources (such as engines), causing digitally optimized grayscale values to be unable to be stably mapped to the physical world. Furthermore, infrared imaging devices typically incorporate algorithms such as dynamic range compression and contrast enhancement, causing the attack effect to be attenuated or even eliminated during imaging through nonlinear mapping; that is, the adversarial effect is easily affected by preprocessing methods. Summary of the Invention
[0003] To address the problem that existing infrared adversarial sample generation methods based on grayscale optimization are not robust to changes in environmental and target radiation, this invention proposes an infrared adversarial sample generation method and system based on material property optimization.
[0004] The first aspect of this invention discloses a method for generating infrared adversarial samples based on material property optimization, the method comprising:
[0005] S1, Load the 3D model file of the target object, obtain the mesh model information of the target object, and initialize the material attribute information of the target object; wherein, the material attribute information includes: K material attribute values of M bands corresponding to N facets of the target mesh model;
[0006] S2, randomly perturb the emitted radiation and / or reflected radiation of the target object, and render it using a differentiable infrared renderer to obtain a set of adversarial images;
[0007] S3: Input the adversarial image into the target detector, obtain the detection results and calculate the total loss, and backpropagate the total loss to optimize and update the material attribute information; where the total loss is the weighted sum of the detection adversarial loss and the neighbor difference loss;
[0008] S4. Repeat steps S2 and S3 until the total loss drops below the threshold to obtain the final material property information. Based on the final material property information and the mesh model information of the target object, generate robust adversarial examples.
[0009] Optionally, in step S1, the mesh model information includes: object vertex information, facet information, and texture information.
[0010] Optionally, in step S2, if the material property category of the target object is near-infrared band, the reflected radiation of the target object is randomly perturbed.
[0011] If the target object's material property category is mid-infrared band, randomly perturb the target object's emitted and reflected radiation.
[0012] If the target object's material property category is far-infrared band, randomly perturb the target object's emitted radiation.
[0013] Optionally, step S2 further includes: performing an image transformation operation on the rendered image to obtain a set of adversarial images.
[0014] Optionally, the image transformation operation includes one or more combinations of brightness adjustment, contrast adjustment, saturation adjustment, and rotation.
[0015] Optionally, in step S3, the adversarial loss L is detected. detect Represented as:
[0016]
[0017] Where B is the total number of target bounding boxes detected in the i-th band, and S i,j The confidence score for the j-th target bounding box in the i-th band;
[0018] Proximity difference loss L adj Represented as:
[0019]
[0020] Where W represents the total number of adjacent faces of the current face. P represents the value of the k-th material property in the m-th band of the i-th adjacent patch. m,k This represents the value of the k-th material property in the m-th band of the current patch, where abs is the absolute value function.
[0021] A second aspect of this invention discloses an infrared adversarial sample generation system based on material property optimization, the system comprising:
[0022] The first processing module is configured to load the 3D model file of the target object, obtain the mesh model information of the target object, and initialize the material attribute information of the target object; wherein, the material attribute information includes: K material attribute values of M bands corresponding to N facets of the target mesh model;
[0023] The second processing module is configured to randomly perturb the emitted radiation and / or reflected radiation of the target object and render it using a differentiable infrared renderer to obtain a set of adversarial images.
[0024] The third processing module is configured to input the adversarial image into the target detector, obtain the detection result and calculate the total loss, and backpropagate the total loss to optimize and update the material attribute information until the total loss drops below the threshold to obtain the final material attribute information. Based on the final material attribute information and the mesh model information of the target object, robust adversarial samples are generated; where the total loss is the weighted sum of the detection adversarial loss and the neighbor difference loss.
[0025] A third aspect of this invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the infrared adversarial sample generation method based on material property optimization described in the first aspect of this invention.
[0026] A fourth aspect of this invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the infrared adversarial sample generation method based on material property optimization described in the first aspect of this invention.
[0027] In summary, the solution proposed in this invention has the following technical effects:
[0028] The proposed infrared adversarial sample generation algorithm based on material property optimization takes material properties as the optimization target and combines it with a differentiable renderer based on a physical optics model. This enables the simulation of environmental radiation changes and target radiation changes in the digital domain, making the adversarial effect less susceptible to the influence of the physical environment.
[0029] The proposed infrared adversarial sample generation algorithm based on material property optimization applies EOT transform for data augmentation and is optimized by jointly detecting adversarial loss and nearest neighbor difference loss. It can generate more robust and textured adversarial samples and is easy to implement physically. Attached Figure Description
[0030] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0031] Figure 1 This is a schematic diagram of an infrared adversarial sample generation method based on material property optimization according to an embodiment of the present invention;
[0032] Figure 2 This is a schematic diagram of the infrared adversarial sample generation network framework based on material property optimization according to an embodiment of the present invention;
[0033] Figure 3 This is a schematic diagram of diffuse reflection according to an embodiment of the present invention;
[0034] Figure 4 This is a schematic diagram of specular reflection according to an embodiment of the present invention;
[0035] Figure 5 This is a radiation diagram of an embodiment of the present invention;
[0036] Figure 6 This is a schematic diagram illustrating the effect of mid-infrared adversarial sample attacks generated using the method proposed in this invention, as shown in an embodiment of the invention. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0038] The first aspect of this invention discloses a method for generating infrared adversarial samples based on material property optimization. Please refer to [link to relevant documentation]. Figure 1 and Figure 2 The method includes:
[0039] S1, Load the 3D model file of the target object, obtain the mesh model information of the target object, and initialize the material attribute information of the target object; wherein, the material attribute information includes: K material attribute values of M bands corresponding to N facets of the target mesh model;
[0040] Optionally, in step S1, the mesh model information includes: object vertex information, facet information, and texture information.
[0041] Based on material characteristics, material attribute values are randomly initialized. The renderer requires a vector of size [N, M, K] for material attribute information, where N is the number of facets, M is the number of band types, and K is the number of material attribute categories. For example, for an adversarial example generation task with three infrared bands (near-infrared, mid-infrared, and far-infrared), M is set to 3; for each infrared band, if the number of material attribute categories to be optimized (such as emissivity, reflectivity, and roughness) is 3, then K is set to 3. This vector is initialized to different values based on material attribute characteristics. For example, for emissivity, it can be randomly initialized to two different values, corresponding to high infrared emissivity locations and low infrared emissivity locations, respectively.
[0042] S2, randomly perturb the emitted radiation and / or reflected radiation of the target object, and render it using a differentiable infrared renderer to obtain a set of adversarial images;
[0043] Optionally, in step S2, if the material property category of the target object is near-infrared band, the reflected radiation of the target object is randomly perturbed.
[0044] If the target object's material property category is mid-infrared band, randomly perturb the target object's emitted and reflected radiation.
[0045] If the target object's material property category is far-infrared band, randomly perturb the target object's emitted radiation.
[0046] As one implementation method, a differentiable renderer incorporating an optical physics model is constructed for each infrared band. This renderer can calculate and render the texture of an object's surface material properties under specific parameters. Taking the generation of mid-infrared adversarial examples as an example, the effects of reflected and emitted radiation must be considered simultaneously, i.e., C = β1C. reflection +β2C emitted C reflection Represents reflected radiation, C emittedLet C represent the emitted radiation, and β1 and β2 be weighting parameters representing the proportions of reflected and emitted radiation, respectively. The reflected radiation is calculated using the Cook-Torrance BRDF illumination model. reflection =C didfuse +C spec C didfuse C represents diffuse light. spec This represents specular reflection. Calculate diffuse reflection values: Where, k d This is the diffuse reflection weighting parameter, and R is the hemispherical reflectance. Calculate the specular reflection value: Where D(h) is the normal distribution function, it can be expressed as: Where α is the surface roughness, taking a value in the range [0,1], and is related to the properties of the material on the object's surface; G(w i ,n,w r ) is the shadow occlusion function; F(w) i h) represents the Fresnel equation; w i w is the direction vector of the incident light. r is the direction vector of the emitted light; h is the half-range vector, i.e., the midpoint between the incident light and the observation direction; n is the macroscopic normal vector of this surface point. When generating near-infrared adversarial examples, only the effect of reflected radiation needs to be considered, while when generating far-infrared adversarial examples, only the effect of emitted radiation needs to be considered. Please refer to [link to relevant documentation]. Figure 3 , Figure 4 and Figure 5 These are schematic diagrams of diffuse reflection, specular reflection, and radiation, respectively.
[0047] As one implementation method, the random perturbation includes:
[0048] J groups of different camera parameters, such as pitch angle, azimuth angle, distance, temperature, light source intensity, and light source type, were randomly set to randomly perturb the emitted / reflected radiation of the target. Then, differentiable rendering models for each band were used to render the images, resulting in J×M rendered images.
[0049] Optionally, step S2 further includes: performing an image transformation operation on the rendered image to obtain a set of adversarial images.
[0050] Optionally, the image transformation operation includes one or more combinations of brightness adjustment, contrast adjustment, saturation adjustment, and rotation.
[0051] S3: Input the adversarial image into the target detector, obtain the detection results and calculate the total loss, and backpropagate the total loss to optimize and update the material attribute information; where the total loss is the weighted sum of the detection adversarial loss and the neighbor difference loss;
[0052] Optionally, in step S3, the adversarial loss L is detected.detect Represented as:
[0053]
[0054] Where B is the total number of target bounding boxes detected in the i-th band, and S i,j The confidence score for the j-th target bounding box in the i-th band;
[0055] Calculating the neighbor difference loss requires matching adjacent faces. This involves iterating through all faces, considering two faces as adjacent if they share two identical vertices. For each face, its neighbor difference loss L... adj Represented as:
[0056]
[0057] Where W represents the total number of adjacent faces of the current face. P represents the value of the k-th material property in the m-th band of the i-th adjacent patch. m,k This represents the value of the k-th material property in the m-th band of the current patch, where abs is the absolute value function.
[0058] The final loss is a weighted sum of the detection adversarial loss and the neighbor difference loss, i.e.: L total =γL detect +μL adj , where γ and μ are weighting parameters.
[0059] S4. Repeat steps S2 and S3 until the total loss drops below the threshold to obtain the final material property information. Based on the final material property information and the mesh model information of the target object, generate robust adversarial examples.
[0060] Please see Figure 6 This is a schematic diagram illustrating the effect of mid-infrared adversarial sample attacks generated using the method proposed in this invention. Figure 6 It can be seen that the infrared adversarial examples generated by this method have higher robustness and can better solve the problem of performance degradation of adversarial examples caused by digital-physical domain migration.
[0061] This invention proposes an infrared adversarial sample generation method based on material property optimization, comprising the following steps:
[0062] Step 1: Load the 3D model file of the target object and obtain information such as the object's vertices, faces, and textures;
[0063] Step 2: Based on the infrared bands and combined with the physical optical model, generate the differentiable rendering model for the corresponding bands, including near-infrared, mid-infrared, and far-infrared bands.
[0064] Step 3: Initialize the material properties of each facet of the target object;
[0065] Step 4: For each band of the renderer, randomly perturb the target's emitted / reflected radiation and render it to obtain a set of adversarial images;
[0066] Step 5: Perform the EOT (Expectation Over Transformation) operation on the rendered image;
[0067] Step Six: Input the image generated in Step Five into the detection network and obtain the detection results;
[0068] Step 7: Calculate the detection adversarial loss and the difference loss between adjacent patches, and sum them by weight to obtain the total loss.
[0069] Step 8: Backpropagate the total loss and optimize and update the material property information.
[0070] Step 9: Repeat steps 4 to 8 until the loss drops below the threshold to obtain the final material attribute information, i.e., the adversarial example.
[0071] A second aspect of this invention discloses an infrared adversarial sample generation system based on material property optimization, the system comprising:
[0072] The first processing module is configured to load the 3D model file of the target object, obtain the mesh model information of the target object, and initialize the material attribute information of the target object; wherein, the material attribute information includes: K material attribute values of M bands corresponding to N facets of the target mesh model;
[0073] The second processing module is configured to randomly perturb the emitted radiation and / or reflected radiation of the target object and render it using a differentiable infrared renderer to obtain a set of adversarial images.
[0074] The third processing module is configured to input the adversarial image into the target detector, obtain the detection result and calculate the total loss, and backpropagate the total loss to optimize and update the material attribute information until the total loss drops below the threshold to obtain the final material attribute information. Based on the final material attribute information and the mesh model information of the target object, robust adversarial samples are generated; where the total loss is the weighted sum of the detection adversarial loss and the neighbor difference loss.
[0075] A third aspect of this invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the infrared adversarial sample generation method based on material property optimization described in the first aspect of this invention.
[0076] The electronic device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, Near Field Communication (NFC), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.
[0077] A fourth aspect of this invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the infrared adversarial sample generation method based on material property optimization described in the first aspect of this invention.
[0078] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein, and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for generating infrared adversarial samples based on material property optimization, characterized in that, The method includes: S1, Load the 3D model file of the target object, obtain the mesh model information of the target object, and initialize the material attribute information of the target object; wherein, the material attribute information includes: K material attribute values of M bands corresponding to N facets of the target mesh model; S2, randomly perturb the emitted radiation and / or reflected radiation of the target object, and render it using a differentiable infrared renderer to obtain a set of adversarial images; S3: Input the adversarial image into the target detector, obtain the detection results and calculate the total loss, and backpropagate the total loss to optimize and update the material attribute information; where the total loss is the weighted sum of the detection adversarial loss and the neighbor difference loss; S4. Repeat steps S2 and S3 until the total loss drops below the threshold to obtain the final material property information. Based on the final material property information and the mesh model information of the target object, generate robust adversarial examples.
2. The method according to claim 1, characterized in that, In step S1, the mesh model information includes: object vertex information, facet information, and texture information.
3. The method according to claim 1, characterized in that, In step S2, if the material property category of the target object is near-infrared band, the reflected radiation of the target object is randomly perturbed. If the target object's material property category is mid-infrared band, randomly perturb the target object's emitted and reflected radiation. If the target object's material property category is far-infrared band, randomly perturb the target object's emitted radiation.
4. The method according to claim 1, characterized in that, Step S2 also includes: performing image transformation operations on the rendered image to obtain a set of adversarial images.
5. The method according to claim 4, characterized in that, Image transformation operations include one or more combinations of brightness adjustment, contrast adjustment, saturation adjustment, and rotation.
6. The method according to claim 5, characterized in that, In step S3, the adversarial loss L is detected. detect Represented as: Where B is the total number of target bounding boxes detected in the i-th band, and S i,j The confidence score for the j-th target bounding box in the i-th band; Proximity difference loss L adj Represented as: Where W represents the total number of adjacent faces of the current face. P represents the value of the k-th material property in the m-th band of the i-th adjacent patch. m,k This represents the value of the k-th material property in the m-th band of the current patch, where abs is the absolute value function.
7. An infrared adversarial sample generation system based on material property optimization, characterized in that, The system includes: The first processing module is configured to load the 3D model file of the target object, obtain the mesh model information of the target object, and initialize the material attribute information of the target object; wherein, the material attribute information includes: K material attribute values of M bands corresponding to N facets of the target mesh model; The second processing module is configured to randomly perturb the emitted radiation and / or reflected radiation of the target object and render it using a differentiable infrared renderer to obtain a set of adversarial images. The third processing module is configured to input the adversarial image into the target detector, obtain the detection result and calculate the total loss, and backpropagate the total loss to optimize and update the material attribute information until the total loss drops below the threshold to obtain the final material attribute information. Based on the final material attribute information and the mesh model information of the target object, robust adversarial samples are generated; where the total loss is the weighted sum of the detection adversarial loss and the neighbor difference loss.
8. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps in the infrared adversarial sample generation method based on material property optimization according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the infrared adversarial sample generation method based on material property optimization according to any one of claims 1 to 6.