Infrared-visible light target detection shape-color collaborative attack method

Abstract

The present application belongs to the field of artificial intelligence security technology, and relates to a shape-color collaborative attack method for infrared-visible light target detection, comprising: loading an infrared-visible light multi-modal target detection model and locking parameters, obtaining a multi-modal original input sample image, first-stage adversarial patch shape optimization, second-stage adversarial patch internal color optimization, obtaining a multi-modal adversarial patch based on Fourier series according to Fourier coefficients and color parameters obtained through stage optimization; the present application parameterizes modeling of the shape of the adversarial patch through Fourier series, realizes complete representation of a complex shape, and effectively solves the problems of difficulty in collaborative optimization of the shape and color of a multi-modal adversarial patch, and insufficient utilization of adversarial information in the prior art by adopting a mechanism of stage-by-stage optimization of the shape and color.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security technology, specifically relating to a method for shape-color collaborative adversarial attacks against infrared-visible light target detection. Background Technology

[0002] Object detection, as a core task of computer vision, plays a crucial role in key areas such as security monitoring, autonomous driving, and smart healthcare. In recent years, with the development of multimodal sensors, multimodal object detection technology has received widespread attention. On the other hand, deep learning technology, with its powerful learning capabilities, has demonstrated superior performance in modeling complex multimodal associations and has been widely applied in high-precision infrared-visible multimodal object detection scenarios. Existing adversarial attack techniques for infrared-visible multimodal detection mainly revolve around the design of adversarial patches, with the core objective of physically deceiving the infrared-visible multimodal detection system. Current methods can be divided into two categories: one is texture-based multimodal adversarial patching: this technique utilizes the imaging differences of different colors under visible light in infrared imaging systems, generating adversarial patches by optimizing the patch color in the visible light image and the patch grayscale in the infrared image. The other is shape-based multimodal adversarial patching: this technique achieves adversarial attacks in infrared-visible multimodal detection based on the contour shape of the adversarial patch.

[0003] However, existing infrared-visible multimodal adversarial patch characterization and optimization techniques have the following drawbacks: Multimodal adversarial patches fail to fully utilize the potential adversarial information in shape and color: Existing techniques optimize the color of square adversarial patches in visible light scenes and their grayscale in infrared scenes, without considering the adversarial information that may exist in the patch shape. Alternatively, a pure shape optimization strategy is used, which, based on optimizing the shape of the adversarial patch, applies monochrome coloring to the adversarial patches in infrared and visible light modes separately, ignoring the adversarial potential of color texture in multimodal adversarial attacks. Both of these methods do not adequately consider the adversarial information carriers possessed by infrared-visible light multimodal adversarial patches, limiting the convergence upper limit of the final attack performance of multimodal adversarial patches.

[0004] Co-optimization of the external shape and internal fill color of multimodal adversarial patches is challenging: Existing multimodal adversarial patch shape attacks mislead the model by altering the patch's structural features, while internal color attacks achieve their effect by changing the RGB values ​​of each pixel. Although both the shape and color of adversarial patches have the potential to achieve multimodal adversarial attacks, they both directly affect the patch's appearance, leading to conflicts in direct co-optimization. Specifically, during optimization, changes in shape cause pixels at the patch edges to be in an unstable state, fluctuating between participating in and not participating in adversarial optimization. This results in reduced optimization efficiency and may even lead to mutual interference between the two attributes.

[0005] Therefore, it is necessary to propose an adversarial patching optimization method for infrared-visible multimodal target detection models to solve the above-mentioned technical problems. Summary of the Invention

[0006] This invention provides the following technical solution: a method for shape-color collaborative adversarial attacks on infrared-visible light target detection, comprising the following steps: Step S1: Load the infrared-visible multimodal target detection model and lock the parameters to obtain the original multimodal input sample image.

[0007] Step S2: First-stage adversarial patch shape optimization: Initialize Fourier coefficients and iteratively optimize the Fourier coefficients until the set convergence condition or iteration termination condition is reached.

[0008] Step S3: Second-stage adversarial patch internal color optimization: Initialize patch color parameters, iteratively optimize patch color parameters until the set convergence condition or iteration termination condition is reached.

[0009] Step S4: Based on the Fourier coefficients and color parameters obtained from the stage optimization, obtain a multimodal adversarial patch based on Fourier series.

[0010] Preferably, step S2 specifically includes the following sub-steps: S201. Based on the Fourier series formula, map the Fourier coefficients to a closed curve on a two-dimensional plane, and then map this closed curve to a shape patch through differentiability mapping. .

[0011] S202. Will The interior is filled with black and rendered onto the original input infrared image to obtain an infrared adversarial example. ,Will The interior is filled with white and rendered onto the original input visible light image to obtain a visible light adversarial sample. .

[0012] S203. Infrared adversarial samples Samples against visible light Input multimodal detector In the process, inference is performed to obtain the target confidence of all candidate boxes before NMS processing. And calculate the adversarial loss based on the adversarial loss function.

[0013] S204. Update the Fourier coefficients using the Adam optimizer based on the adversarial loss calculated in step S203. This completes the current iteration.

[0014] More preferably, step S3 specifically includes the following sub-steps: S301. Patch Color Parameters The RGB space 3D tensor has the same size as the shape patch in step S201. The adversarial shape patch obtained after optimization in step S2 is used. Binarization is performed, assigning 0 to the outer region and 1 to the inner region; the color tensor is then processed. Perform a Hadamard product operation on the binarized shape patch to fill the inner area of ​​the patch with color, thus obtaining the outer shape and... Consistent, internally colored adversarial patches .

[0015] S302. Countermeasures Patch Rendering to the original input visible light image yields visible light adversarial samples containing color adversarial patches. .

[0016] S303. Visible light countermeasures Input multimodal detector In the process, inference is performed to obtain the target confidence of all candidate boxes before NMS processing. And calculate the adversarial loss based on the adversarial loss function.

[0017] S304. Update the patch color parameters using the Adam optimizer based on the adversarial loss calculated in step S303. This completes the current iteration.

[0018] More preferably, in step S4, a multimodal adversarial patch based on Fourier series is obtained according to the method in step S301.

[0019] Preferably, in steps S2 and S3, the loss function is:

[0020] in, , These are the shape optimization loss function and the color optimization loss function, respectively. To combat the losses, For shape normalization loss, For total variance loss, Hyperparameters for controlling different levels of loss contribution.

[0021] More preferably, the adversarial loss, shape normalization loss, and total variance loss are respectively:

[0022]

[0023]

[0024] in, This represents all candidate boxes before NMS processing. Indicates the candidate box. Indicates the confidence level of the detection; These represent the Fourier series coefficients corresponding to different frequency components. , K represents the fundamental frequency component, and K represents the highest frequency component. This represents the penalty coefficient, used to penalize high-frequency harmonic amplitudes exceeding the total fundamental amplitude. It helps to limit the optimization range of the Fourier series coefficients to the coefficients corresponding to the fundamental frequency as much as possible. , () represents the coordinates of the pixel's location. Indicates the location at coordinates , The RGB values ​​of the pixel.

[0025] The beneficial effects of this invention are: 1. This invention effectively evades detection models by deploying optimized adversarial patches onto the surface of target objects. The shape of the multimodal adversarial patch in this invention is parametrically modeled based on Fourier series, using Fourier series to represent the patch shape boundary. Differentiable mapping transforms the Fourier coefficients into arbitrary closed two-dimensional contours, achieving complete representation of complex shapes. Furthermore, this invention employs an adversarial patch representation method that fills the interior of a special shape with color. Shape and color work together to exert an adversarial effect, achieving a synergistic improvement in adversarial effectiveness across visible and infrared modes. Finally, this invention proposes a phased shape-color optimization mechanism. The first stage optimizes the Fourier coefficients to quickly converge to the optimal shape patch. The second stage optimizes the internal color of the adversarial patch based on the fixed shape patch, significantly reducing the convergence difficulty.

[0026] 2. This invention fully explores the attack potential of the patch shape and utilizes the characteristic that the patch shape can be observed in various complex infrared scenarios to achieve stronger anti-attack robustness.

[0027] 3. This invention fully utilizes the anti-countermeasure potential of color information in the visible light mode, significantly enhancing the anti-countermeasure performance of the anti-countermeasure patch in the visible light mode. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the two-stage multimodal adversarial patching optimization process based on Fourier series for the shape-color collaborative adversarial attack method for infrared-visible light target detection in this invention. Figure 2 This is a schematic diagram of the visual attack results of an embodiment of the present invention, wherein (a) is the attack patch, (b) is the visible light detection result, and (c) is the infrared detection result; Figure 3 This is a schematic diagram of the method steps of the present invention. Detailed Implementation

[0029] The related technologies of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0030] like Figures 1-3 As shown, this embodiment of the multimodal adversarial attack method based on Fourier series uses Fourier series to model the shape of the adversarial patch, and optimizes the external shape and internal color of the adversarial patch in stages to achieve effective attacks in infrared-visible multimodal scenarios. The optimization process is as follows: Figure 1 As shown. The specific steps of the method include: Anti-patch optimization process: This invention optimizes the external shape and internal fill color of the adversarial patch in stages to achieve a cooperative jamming target detection system in both infrared and visible light modes. The steps include: S1. Load the infrared-visible multimodal target detection model And lock the parameters to obtain the original input sample images of the multimodal mode. .

[0031] S2. First-stage adversarial patch shape optimization: Initialize Fourier coefficients The Fourier coefficients are iteratively optimized until the set convergence or iteration termination condition is met, completing the first stage of adversarial patch shape optimization. Specifically, this includes: S201. Based on the Fourier series formula, map the Fourier coefficients to a closed curve on a two-dimensional plane, and then map this closed curve to a shape patch through differentiability mapping. .

[0032] S202. Will The interior is filled with black and rendered onto the original input infrared image to obtain an infrared adversarial example. ,Will The interior is filled with white and rendered onto the original input visible light image to obtain a visible light adversarial sample. .

[0033] S203. Infrared adversarial samples Samples against visible light Input multimodal detector In the process, inference is performed to obtain the target confidence of all candidate boxes before NMS processing. And calculate the adversarial loss based on the adversarial loss function.

[0034] S204. Update the Fourier coefficients using the Adam optimizer based on the adversarial loss calculated in S203. This completes the current iteration.

[0035] S3. Second-stage adversarial patch internal color optimization: Initialize patch color parameters The patch color parameters are iteratively optimized until the set convergence or iteration termination condition is met, completing the second stage of adversarial patch color optimization. Specifically, this includes: S301. Patch Color Parameters This is a 3D tensor in RGB space with the same size as the shape patch in S201. The adversarial shape patch obtained after optimization in stage S2 is... Perform binarization, assigning 0 to the outer region and 1 to the inner region. For the color tensor... Perform a Hadamard product operation on the binarized shape patch to fill the inner area of ​​the patch with color, thus obtaining the outer shape and... Consistent, internally colored adversarial patches .

[0036] S302. Countermeasures Patch Rendering to the original input visible light image yields visible light adversarial samples containing color adversarial patches. .

[0037] S303. Visible light countermeasures Input multimodal detector In the process, inference is performed to obtain the target confidence of all candidate boxes before NMS processing. And calculate the adversarial loss based on the adversarial loss function.

[0038] S304. Based on the adversarial loss calculated in S303, update the patch color parameters using the Adam optimizer. This completes the current iteration.

[0039] S4. Fourier coefficients obtained from optimization in stages S2 and S3. With color parameters Follow the steps in S301 to obtain a multimodal adversarial patch based on Fourier series.

[0040] This invention uses a Fourier series method to characterize and optimize the shape of adversarial patches. Let the parameter t take values ​​ranging from... The shape edge determined by parameter t It can be represented as:

[0041] in, and Let i and K represent the Cartesian coordinates of the edge, i be the imaginary unit, and K be the highest-order frequency component, which determines the complexity of the shape. Under this representation method, the shape of the adversarial patch is determined by a set of Fourier coefficients. The decision can be made by optimizing the complex coefficient set. This indirectly optimizes the shape of adversarial patches.

[0042] This invention adds color-resistance information inside irregular shape contours. Let the shape binary mask obtained by differentiable mapping from Fourier parameters be... The value inside the Fourier shape boundary is 1, and the value outside the boundary is 0. Color tensors of the same size are Then, the adversarial patch with color information can be represented as:

[0043] in The Hadamard product represents the element-wise multiplication of the color tensor with the binary patch.

[0044] The infrared-visible multimodal adversarial patch segmentation optimization is as follows: The optimization process of this invention separates shape and color into two stages for optimization, and the loss function for each stage is defined as follows:

[0045] in, To combat the losses, For shape normalization loss, For total variance loss, Hyperparameters are used to control the contribution of different losses. Detailed explanations of each loss are as follows: Adversarial loss: Let all candidate boxes before NMS processing be... Candidate boxes The corresponding detection confidence level is Then the adversarial loss is defined as:

[0046] Shape normalization loss: in Fourier coefficients In the middle, the fundamental frequency component is .set up The shape normalization loss is defined as:

[0047] Total variance loss: Let Given the RGB values ​​of each pixel within the adversarial patch, the total variance loss is defined as: .

[0048] Compared to previous texture-based multimodal adversarial techniques, this implementation fully exploits the attack potential of the adversarial patch shape. Leveraging the fact that the patch shape is observable in various complex infrared scenes, it achieves stronger adversarial robustness than existing technologies. Compared to previous shape-based multimodal adversarial methods, this invention fully utilizes the adversarial potential of color information in the visible light mode, significantly enhancing the adversarial performance of the adversarial patch in the visible light mode.

[0049] Example To demonstrate the effectiveness and feasibility of the shape-color collaborative adversarial attack method proposed in this invention, this embodiment uses the DroneVehicle aerial photography dataset for attack experiments. This dataset covers various scenes such as urban roads, residential areas, and parking lots, with the training set containing 1469 pairs of infrared-visible light images and the test set containing 8980 pairs. Collaborative patches are trained on the training set, and attack evaluation is performed on the test set.

[0050] In the experiment, the parameters of a multimodal target detector were loaded and fixed. Adversarial patches were optimized in two stages on the training set, and the generated patches were rendered onto the surface of aerial vehicle targets, allowing vehicles covered by the patches to evade detector recognition. The first stage was patch shape training, using the Adam optimizer with a learning rate of 0.0008, 30 iterations, and a maximum frequency component K of 6 to optimize Fourier coefficients and obtain the target patch shape. The second stage was patch color training. Based on the fixed external patch shape, the Adam optimizer was used to optimize the internal color parameters with a learning rate of 0.0008, 30 iterations, and a patch resolution of 64×64 pixels. During evaluation, both the target confidence threshold and the IoU threshold were set to 0.5, and the attack success rate on all targets was calculated. For multimodal attacks, a successful attack was only counted when the target disappeared simultaneously in both infrared and visible light detection results.

[0051] The comparison methods include UNIPatch, which is based on pure shape attacks, and CDUPatch, which is based on color and texture attacks. The attack success rates (ASR, %) on all targets in the test set are shown in Table 1 below:

[0052] The results show that the method of the present invention can simultaneously mine two types of adversarial information, namely patch shape and color, and effectively reduce the optimization conflict caused by their direct collaboration through phased optimization. Therefore, it exhibits a higher attack success rate under different detection frameworks, especially on YOLOv5 and YOLOv8, which verifies the superior attack effect of the present invention in infrared-visible multimodal scenarios. Figure 2 The visual attack results of this embodiment are presented. Figure 2 (a) is the optimized shape-color joint adversarial patch. It can be seen that the patch contains both irregular outer contours and internal color texture information. Figure 2 (b) and Figure 2 (c) Shows the detection results after applying the patch to visible light targets and infrared targets, respectively, where the green boxes represent the detection boxes output by the target detector. Figure 2 As can be seen, in most cases, vehicle targets with anti-countermeasure patches are not output as corresponding detection boxes by the detector, indicating that the patches significantly reduce the detectability of the targets in visible light and infrared modes, resulting in missed detections by the detector.

[0053] In summary, the proposed shape-color collaborative adversarial attack method for infrared-visible light target detection utilizes Fourier series to parameterize the shape of the adversarial patch, achieving a complete representation of complex shapes. Furthermore, it employs a phased optimization mechanism for shape and color, effectively addressing the difficulties in co-optimizing the shape and color of multimodal adversarial patches and the insufficient utilization of adversarial information in existing technologies. This method first optimizes Fourier coefficients in the first stage to determine the optimal shape patch, and then optimizes the internal color based on a fixed shape in the second stage. This not only reduces convergence difficulty but also fully leverages the robustness of shape in infrared scenes and the adversarial potential of color in visible light modes, significantly improving the attack performance of adversarial patches in multimodal target detection scenarios. This provides new ideas and solutions for the development of multimodal adversarial attack technology in the field of artificial intelligence security.

[0054] It should be emphasized that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for shape-color collaborative adversarial attacks in infrared-visible light target detection, characterized in that, Includes the following steps: Step S1: Load the infrared-visible multimodal target detection model and lock the parameters to obtain the original multimodal input sample image; Step S2: First-stage adversarial patch shape optimization: Initialize Fourier coefficients and iteratively optimize the Fourier coefficients until the set convergence condition or iteration termination condition is reached. Step S3: Second-stage adversarial patch internal color optimization: Initialize patch color parameters, iteratively optimize patch color parameters until the set convergence condition or iteration termination condition is reached. Step S4: Based on the Fourier coefficients and color parameters obtained from the stage optimization, obtain a multimodal adversarial patch based on Fourier series.

2. The method for shape-color collaborative adversarial attacks in infrared-visible light target detection according to claim 1, characterized in that, Step S2 specifically includes the following sub-steps: S201. Based on the Fourier series formula, map the Fourier coefficients to a closed curve on a two-dimensional plane, and then map this closed curve to a shape patch through differentiability mapping. ; S202. Will The interior is filled with black and rendered onto the original input infrared image to obtain an infrared adversarial example. ,Will The interior is filled with white and rendered onto the original input visible light image to obtain a visible light adversarial sample. ; S203. Infrared adversarial samples Samples against visible light Input multimodal detector In the process, inference is performed to obtain the target confidence of all candidate boxes before NMS processing. And calculate the adversarial loss based on the adversarial loss function; S204. Update the Fourier coefficients using the Adam optimizer based on the adversarial loss calculated in step S203. This completes the current iteration.

3. The method for shape-color collaborative adversarial attacks in infrared-visible light target detection according to claim 2, characterized in that, Step S3 specifically includes the following sub-steps: S301. Patch Color Parameters A 3D tensor in RGB space with the same size as the shape patch in step S201; The adversarial shape patch obtained after optimization in step S2. Binarization is performed, assigning 0 to the outer region and 1 to the inner region; the color tensor is then processed. Perform a Hadamard product operation on the binarized shape patch to fill the inner area of ​​the patch with color, thus obtaining the outer shape and... Consistent, internally colored adversarial patches ; S302. Countermeasures Patch Rendering to the original input visible light image yields visible light adversarial samples containing color adversarial patches. ; S303. Visible light countermeasures Input multimodal detector In the process, inference is performed to obtain the target confidence of all candidate boxes before NMS processing. And calculate the adversarial loss based on the adversarial loss function; S304. Update the patch color parameters using the Adam optimizer based on the adversarial loss calculated in step S303. This completes the current iteration.

4. The method for shape-color collaborative adversarial attacks in infrared-visible light target detection according to claim 3, characterized in that, In step S4, a multimodal adversarial patch based on Fourier series is obtained according to the method in step S301.

5. The method for shape-color collaborative adversarial attacks against infrared-visible light target detection according to claim 1, characterized in that, In steps S2 and S3, the loss function is: in, , These are the shape optimization loss function and the color optimization loss function, respectively. To combat the losses, For shape normalization loss, For total variance loss, Hyperparameters for controlling different levels of loss contribution.

6. The method for shape-color collaborative adversarial attacks in infrared-visible light target detection according to claim 5, characterized in that, The adversarial loss, shape normalization loss, and total variance loss are respectively: in, This represents all candidate boxes before NMS processing. Indicates the candidate box. Indicates the confidence level of the detection; These represent the Fourier series coefficients corresponding to different frequency components. , K represents the fundamental frequency component, and K represents the highest frequency component. Indicates the penalty coefficient. , Represents the coordinates of the corresponding pixel position. Indicates the location at coordinates , The RGB values ​​of the pixel.

Images