An asymmetric adversarial patch defense method and device for multi-modal images

Abstract

The present application belongs to the field of computer vision and artificial intelligence security technology, and relates to an asymmetric adversarial patch defense method and device for multi-modal images. The main steps of the method include: obtaining a candidate abnormal area, respectively outputting a visible light attack probability and an infrared attack probability for the candidate abnormal area, then respectively generating a visible light repair mask and an infrared repair mask, sequentially performing selective diffusion repair on the image according to the attacked mode, purifying the image and re-detecting, and finally outputting a target detection result. The method solves the problems that the multi-modal shared repair strategy is easy to perform unnecessary repair on the unattacked mode in the single-modal attack scene, is difficult to distinguish different modal attack areas in the dual-modal attack scene, and the repair result does not match the downstream target detection task.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence security technology, and particularly relates to an asymmetric adversarial patch defense method and device for multimodal images. Background Technology

[0002] In deep learning-based multimodal target detection tasks, visible light and infrared dual-modal collaborative sensing, due to its combination of texture and thermal radiation information, has been widely applied in safety-critical scenarios such as autonomous driving, unmanned platform perception, and day-night security. Compared to single-modal detection methods, multimodal fusion detection can provide more stable target perception capabilities under low light, occlusion, weather changes, and complex background conditions. However, with the deployment of deep learning models in real-world systems, adversarial attacks have gradually become a key factor affecting their security and reliability. Among these, adversarial patches are a representative type of localized attack. They typically act on the vicinity of the target or on the target surface in the form of patches, patterns, heat sources, or abnormal regions, misleading the detection model through localized but high-intensity perturbations, causing the model to produce missed detections, false detections, or a significant decrease in detection confidence. Unlike full-image perturbations, adversarial patches are easier to implement in the physical world and have strong attack transferability and deployment stealth, thus posing a higher real-world threat in physical adversarial scenarios.

[0003] Existing defense methods typically revolve around two technical paths: one is adversarial training, which involves injecting attack samples into the model during training to enhance its robustness against known attacks. However, this method usually relies on a specific attack distribution, is prone to overfitting to attack patterns seen during training, and has limited generalization ability to unknown patch patterns and cross-modal perturbation forms. The other is input sanitization or image inpainting, which involves detecting and removing abnormal regions before the image enters the detection network, and then restoring it using generative models, image inpainting methods, or filtering methods. While these methods can mitigate the impact of attacks to some extent, they often employ a uniform processing strategy in multimodal scenarios, performing the same repair or enhancement operations on two modalities simultaneously. This leads to the unaffected modality being repaired even when the attack exists only in a single modality, thereby destroying the original effective semantic information and causing problems such as over-repair, mis-repair, and decreased downstream detection performance. Summary of the Invention

[0004] Therefore, it is necessary to provide an asymmetric adversarial patching defense method and apparatus for multimodal images to address the above-mentioned technical problems, and solve the problems that multimodal shared repair strategies tend to perform unnecessary repairs on unattacked modalities in single-modal attack scenarios, have difficulty distinguishing different modal attack regions in dual-modal attack scenarios, and have mismatches between repair results and downstream target detection tasks.

[0005] In a first aspect, the present invention provides an asymmetric adversarial patch defense method for multimodal images, comprising the following steps: S1: Acquire the image to be detected, which is composed of a fused input of a visible light image and an infrared image, and output one or more candidate abnormal regions according to the preset parameters; S2: Extract regional features and meta-features for each candidate anomaly region, and output the visible light attack probability and infrared attack probability respectively; S3: Based on the two attack probabilities output in the above steps, generate a visible light repair mask and an infrared repair mask for each candidate anomalous region. If a candidate anomalous region is not identified as an attacked mode, the mask corresponding to the candidate anomalous region is empty, and the image is directly bypassed and no repair is performed. If a candidate anomalous region is identified as an attacked mode, the image performs selective diffusion repair. S4: The cleaned image re-detection model receives the repaired visible light and infrared images and outputs the final target detection results.

[0006] In one embodiment, the method for generating the image composed of the fusion input of visible light image and infrared image in step S1 includes: using a four-channel input image formed by stitching together a three-channel visible light image and an enhanced single-channel infrared image.

[0007] In one embodiment, the parameters preset in step S1 include target location information for each region, upper limit of the number of target categories, and detection confidence threshold.

[0008] In one embodiment, the regional features in step S2 include a visible light region, an infrared region, a visible light context region, and an infrared context region, and the meta-features include frame confidence, frame area percentage, and frame aspect ratio.

[0009] In one embodiment, the method for generating the visible light restoration mask and the infrared restoration mask in step S3 includes: obtaining the attribution probability by training an attribution determination network on the visible light attack probability and infrared attack probability independently output for any candidate abnormal region; comparing the corresponding attribution probability of each candidate abnormal region with a preset threshold for the attribution probability; if the visible light attack probability is greater than the preset threshold for the attribution probability, then all pixels in the candidate abnormal region are set to 255 to generate a visible light restoration mask; if the infrared attack probability is greater than the preset threshold for the attribution probability, then all pixels in the candidate abnormal region are set to 255 to generate an infrared restoration mask.

[0010] In one embodiment, the selective diffusion repair in step S3 includes infrared expert diffusion model repair and constraint model repair based on mask conditions.

[0011] In one embodiment, the cleaned image re-detection model in step S4 includes two single-modal target detection models trained for the restored visible light image and the restored infrared image, respectively.

[0012] Secondly, the present invention also provides an asymmetric adversarial patch defense device for multimodal images, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described asymmetric adversarial patch defense method for multimodal images.

[0013] The beneficial effects of this invention are: (1) After outputting candidate abnormal regions, this invention extracts the visible light region, infrared region, visible light context region and infrared context region for each candidate abnormal region, and outputs the visible light attack probability and infrared attack probability independently. This method can distinguish between visible light single-mode attack, infrared single-mode attack and dual-mode attack, and provides a basis for judgment for subsequent asymmetric repair of images.

[0014] (2) The present invention generates visible light repair mask and infrared repair mask respectively. The two masks are independent of each other in spatial position and activation state. If a certain mode is not determined to be attacked, the mask of that mode is empty, and its image is directly bypassed and output without entering the repair. This method can reduce the false repair of unattacked mode images in single-mode attack scenarios.

[0015] (3) Based on selective diffusion repair, this invention introduces mask condition constraint model repair, which enables the repair module to restore the local image content that is blocked or disturbed by the patch, thereby improving the effectiveness of the repair results for the target detection task; it introduces infrared expert diffusion model repair, and by setting thermal radiation related constraints, it reduces the problem that the human body thermal response is difficult to recover after infrared attack. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the asymmetric adversarial patch defense method for multimodal images according to the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below. The following embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit the scope of protection of the present invention.

[0018] like Figure 1 As shown, this invention provides an asymmetric adversarial patch defense method for multimodal images, comprising the following steps: S1: Acquire the image to be detected, which is composed of a fused input of a visible light image and an infrared image, and output one or more candidate abnormal regions according to the preset parameters; S2: Extract regional features and meta-features for each candidate anomaly region, and output the visible light attack probability and infrared attack probability respectively; S3: Based on the two attack probabilities output in the above steps, generate a visible light repair mask and an infrared repair mask for each candidate anomalous region. If a candidate anomalous region is not identified as an attacked mode, the mask corresponding to the candidate anomalous region is empty, and the image is directly bypassed and no repair is performed. If a candidate anomalous region is identified as an attacked mode, the image performs selective diffusion repair. S4: The cleaned image re-detection model receives the repaired visible light and infrared images and outputs the final target detection results.

[0019] This embodiment will describe the details of each step in the order of the above method.

[0020] Step S1 aims to receive an image composed of the fused input of visible light and infrared images and output one or more candidate abnormal regions. This step is only responsible for determining the spatial region where an attack may occur. It does not directly determine which mode the attack belongs to, nor does it directly generate the final repair mask. The candidate abnormal regions are used to limit the spatial range of the image for subsequent repair processing.

[0021] The method for generating an image by fusing visible light and infrared images includes: using a four-channel input image formed by stitching together a three-channel visible light image and an enhanced single-channel infrared image.

[0022] The parameters preset in step S1 include the target location information for each region, the upper limit of the number of target categories, and the detection confidence threshold.

[0023] Specifically: Given a visible light image First, contrast-limited adaptive histogram equalization is applied to the infrared image to enhance its local texture contrast, resulting in an enhanced infrared image. The enhanced infrared image will then be... Performing tensor concatenation along the channel dimension results in the input tensor... ,in For tensor splicing operations performed along the channel dimension, the input tensor is... This is the set of candidate anomaly regions for output, where each candidate anomaly region contains the location coordinates of the anomaly region and the anomaly confidence level.

[0024] Step S2: For each candidate abnormal region, extract the region features respectively, that is, crop the visible light and infrared regions from the visible light image and the infrared image, and expand the cropped region according to the preset ratio to obtain the visible light context region and the infrared context region, as well as the meta-features. The meta-features include the bounding box confidence, the bounding box area ratio and the bounding box width-to-height ratio.

[0025] The extracted regional features and meta-features are input into the attack modality attribution module, which includes lightweight networks such as ResNet, EfficientNet, MobileNet, and custom shallow networks. The attack modality attribution module outputs the visible light attack probability and infrared attack probability for each candidate abnormal region.

[0026] In step S3, a visible light repair mask and an infrared repair mask are generated for each candidate abnormal region based on the two attack probabilities output in the above steps. The method for generating the visible light repair mask and the infrared repair mask includes: training the attribution probability of the independently output visible light attack probability and infrared attack probability of any candidate abnormal region through an attribution determination network (wherein the attribution determination network is a lightweight deep neural network classifier MobileNetV2) to obtain the attribution probability; comparing the corresponding attribution probability of each candidate abnormal region with a preset threshold for the attribution probability; if the visible light attack probability is greater than the preset threshold for the attribution probability, then all pixels in the candidate abnormal region are set to 255 (i.e., marked as a region to be repaired), and a visible light repair mask is generated; if the infrared attack probability is greater than the preset threshold for the attribution probability, then all pixels in the candidate abnormal region are set to 255 (i.e., marked as a region to be repaired), and an infrared repair mask is generated.

[0027] If a candidate anomalous region is not identified as an attacked mode, the mask corresponding to the candidate anomalous region is empty, and the image is directly bypassed and output without repair; if a candidate anomalous region is identified as an attacked mode, the image undergoes selective diffusion repair.

[0028] The specific process is as follows: The visible light inpainting mask and infrared inpainting mask are generated above. When the visible light inpainting mask is empty, the visible light image is directly bypassed and output without inpainting; when the infrared inpainting mask is empty, the infrared image is directly bypassed and output without inpainting. Finally, if a candidate anomalous region is determined to be an attacked mode, the image undergoes selective diffusion inpainting. Selective diffusion inpainting includes infrared expert diffusion model inpainting and constraint model inpainting based on mask conditions. The infrared expert diffusion model inpainting introduces a thermal radiation-related constraint penalty loss. This loss function aims to force the model to generate content with sufficient thermal radiation intensity within the inpainting mask area, preventing the model from inpainting human targets as a cold background in pursuit of pixel smoothness.

[0029] Assume a real, clean infrared image is The image reconstructed by the model is The mask to be repaired is The total loss function of the infrared expert diffusion model. It consists of three weighted components: a global reconstruction constraint, a local enhancement constraint, and an asymmetric thermal erasure penalty. in: This represents element-wise multiplication; Control global pixel consistency; Control the repair accuracy within the mask area; The weighting coefficients for the asymmetric penalty term are... A positive penalty is applied only when the model's predicted value is lower than the actual value.

[0030] The repair of the constraint model based on mask conditions specifically includes adding a mask condition discriminator. and the U-Net generator of the diffusion model Perform repair training. Specifically, the discriminator Employing a fully convolutional structure with 4 input channels, the generator is formed by concatenating the image to be judged and the corresponding repair mask along the channel dimension. (That is, the U-Net part of the diffusion model) In addition to optimizing the original diffusion loss, pixel reconstruction loss, and physical prior regularization loss, an adversarial loss is added to encourage the generator to output realistic images that can fool the discriminator. Specifically, edge perception is transformed into component loss or frequency domain consistency loss, which suppresses artifacts, streaks, or high-frequency noise generated during the restoration process, making the restoration result smoother and more natural while maintaining edge sharpness.

[0031] In step S4, the cleaned image re-detection model receives the repaired visible light image and infrared image, and outputs the final target detection result. The cleaned image re-detection model comprises two single-modal target detection expert models trained separately for the cleaned visible light image and the cleaned infrared image. The specific process can be implemented using target detectors, including the YOLO series of target detectors: YOLOv5, YOLOv8, YOLOv9, YOLOv10, etc.

[0032] The inventiveness of this invention will be further verified below through specific experimental data.

[0033] This invention sets up a systematic digital simulation experiment and compares and analyzes the proposed method with adversarial defense work that can perform multimodal defense.

[0034] Experimental Setup Details: In the digital image attack experiment, this invention uses the publicly available visible light-infrared dual-modal pedestrian detection dataset LLVIP as the basic data source. This dataset contains pairs of visible light and infrared images simultaneously acquired in low-light nighttime scenes. To comprehensively evaluate the effectiveness of the defense method, this experiment applies mainstream adversarial patch attack algorithms such as ADP, AdvCloth, Crossattack, AdvIB, BulbAttack, DM-NAP, GNAP, HBlock, IPatch, LaVAN, and MAP to the original clean images, constructing a diverse set of adversarial sample test sets covering visible light single-modal attacks, infrared single-modal attacks, and dual-modal collaborative attacks. The complete defense system of this invention is implemented based on the PyTorch deep learning framework, and all model training and inference processes are completed on an NVIDIA GeForce RTX 4090 graphics processor. The patch location detection four-channel candidate anomaly region proposal module adopts the YOLOv8l architecture, with an input image resolution of 960×960 pixels. During training, the batch size is 4, the optimizer uses stochastic gradient descent with a momentum coefficient of 0.937, an initial learning rate of 0.01, and a cosine annealing decay strategy with a weight decay coefficient of 0.0005. Training is performed for 200 epochs. For data augmentation, the mosaic augmentation probability is fixed at 1.0, the mixed augmentation probability is 0.15, the copy-paste augmentation probability is 0.6, the rotation angle range is ±15°, the scaling ratio range is 0.5, the perspective transformation coefficient is 0.001, and the random erasure probability is 0.4. The modal attribution module uses a custom shallow convolutional network. The bimodal diffusion inpainting expert model is finely tuned based on the Stable DiffusionInpainting architecture, with an input image resolution of 512×512 pixels. Both the generator and discriminator use the AdamW optimizer with an initial learning rate of 1×10⁻⁶. -5 The discriminator weight decay coefficient is 1×10 -4During training, automatic blending precision and gradient clipping were enabled, and the upper limit of the generator gradient norm was set to 1.0. The discriminator used PatchGAN, and the infrared expert model additionally introduced an asymmetric thermal erasure penalty loss with a weight of 20.0. The edge-aware total variation loss weight was set to 0.005, and the adversarial loss was introduced from the 11th training epoch with a weight of 0.1. The diffusion denoising steps during the inference phase were set to 40, with a guidance strength of 7.5 and a repair strength of 0.85. The single-modal detection expert model was based on the YOLOv8s architecture, with an input image resolution of 640×640 pixels, a batch size of 16, and 100 training epochs. The optimizer used stochastic gradient descent with an initial learning rate of 0.01 and 8 data loading processes. To ensure consistency and comparability in the evaluation, all detection models used a uniform confidence threshold of 0.5 during the inference phase to filter low-confidence detection results, and a non-maximum suppression threshold of 0.4 to eliminate overlapping detection boxes.

[0035] Specifically, the five defense methods involved in the comparison are from published related works: (1) Variation-based image preprocessing defense, which weakens the impact of adversarial perturbations while preserving the semantic information of the image by applying bit depth reduction, JPEG compression, total variation minimization or image tiling to the input image in sequence; (2) JPEG compression-based defense strategy, which eliminates some small adversarial perturbations by using quantization and frequency domain selection in the JPEG compression process; (3) Feature squeezing defense method, which reduces the sensitivity of the model to input changes by reducing pixel color depth, applying median filtering or non-local mean smoothing, and detects adversarial samples based on the difference between the original prediction and the compressed prediction; (4) PixelMask-based data augmentation defense, which randomly masks some pixels during the training phase to generate augmented samples, forcing the model to learn nondeterministic feature dependencies, thereby improving the robustness to adversarial patches; (5) DIFFender defense framework based on diffusion model, which uses diffusion model to accurately locate patches by perceiving the abnormal phenomenon that the adversarial region is inconsistent with the distribution of natural images, and combines text guidance and few-sample prompts to adjust and complete the repair of the patch region.

[0036] In the experimental setup, the five defense methods were applied to test images of the LLVIP pedestrian dataset after being contaminated by both single-modal and multi-modal adversarial patches. For the target category of "person," the mean accuracy (AP) of each defense method under different target detection models was calculated, and the changes in detection performance before and after the defense were further compared. The experimental results are shown in Tables 1, 2, and 3. Table 1 shows the test results of the trained person position detector in a digital world against visible light adversarial patch attacks; Table 2 shows the test results of the trained person position detector in a digital world against infrared adversarial patch attacks; and Table 3 shows the test results of the trained person position detector in a digital world against cross-modal adversarial patch attacks. All tables use AP values ​​to represent defense performance. The left axis lists the defense methods, and the top axis represents all adversarial patch methods. The methods are: w / o Defense (baseline), JPEG compression defense, PixelMask data augmentation defense, Variation-based image preprocessing defense, Feature squeezing defense, Diffender-based diffusion model defense, Ours w / o selective (the method of this invention without selective patching), and Ours complete method of this invention. Clean represents a clean image.

[0037] Experimental results show that the defense framework proposed in this invention achieves the best patch detection and repair performance under different dataset conditions. Compared with the five representative defense methods mentioned above, the method of this invention shows a more significant advantage in final detection accuracy.

[0038] Experimental results show that this method can improve the target detection performance after repair in various visible light, infrared, and cross-modal adversarial patch attack scenarios, and is applicable to safety-critical scenarios requiring visible light-infrared collaborative perception, such as autonomous driving, intelligent security, and unmanned platform perception. Compared with existing defense schemes based on shared mask repair, independent input sanitization, or general adversarial training, this invention has a significant advantage in the average accuracy of target detection under various mainstream adversarial patch attacks. Especially in complex scenarios such as infrared single-modal thermal stealth attacks and visible light and infrared spatial asymmetric bimodal attacks, the defensive stability and semantic preservation capabilities of this invention are significantly superior.

[0039] Table 1 shows the test results of the trained human position detector in the digital world against visible light anti-patch attacks.

[0040] Table 2 shows the test results of the trained human position detector in a digital world against infrared countermeasures patch attacks. Table 3 shows the test results of the trained person location detector in the digital world against cross-modal adversarial patch attacks.

Claims

1. An asymmetric adversarial patch defense method for multimodal images, characterized in that, Includes the following steps: S1: Acquire the image to be detected, which is composed of a fused input of a visible light image and an infrared image, and output one or more candidate abnormal regions according to the preset parameters; S2: Extract regional features and meta-features for each candidate anomaly region, and output the visible light attack probability and infrared attack probability respectively; S3: Generate a visible light repair mask and an infrared repair mask for each candidate anomaly region based on the two attack probabilities output in the above steps; If a candidate anomalous region is not identified as an attacked mode, the mask corresponding to that candidate anomalous region is empty, and the image is directly bypassed and output without repair. If a candidate anomalous region is determined to be an attacked mode, selective diffusion inpainting is performed on the image. S4: The cleaned image re-detection model receives the repaired visible light and infrared images and outputs the final target detection results.

2. The asymmetric adversarial patch defense method for multimodal images according to claim 1, characterized in that, The method for generating the image composed of the fusion input of visible light image and infrared image in step S1 includes: using a four-channel input image formed by stitching together a three-channel visible light image and an enhanced single-channel infrared image.

3. The asymmetric adversarial patch defense method for multimodal images according to claim 1, characterized in that, The parameters preset in step S1 include the target location information for each region, the upper limit of the number of target categories, and the detection confidence threshold.

4. The asymmetric adversarial patch defense method for multimodal images according to claim 1, characterized in that, The regional features in step S2 include visible light region, infrared region, visible light context region and infrared context region, and the meta-features include frame confidence, frame area ratio and frame width-to-height ratio.

5. The asymmetric adversarial patch defense method for multimodal images according to claim 1, characterized in that, The method for generating visible light restoration masks and infrared restoration masks in step S3 includes: obtaining the attribution probability by training an attribution determination network on the visible light attack probability and infrared attack probability independently output for any candidate abnormal region; comparing the corresponding attribution probability of each candidate abnormal region with a preset threshold for the attribution probability; if the visible light attack probability is greater than the preset threshold for the attribution probability, then all pixels in the candidate abnormal region are set to 255 to generate a visible light restoration mask; if the infrared attack probability is greater than the preset threshold for the attribution probability, then all pixels in the candidate abnormal region are set to 255 to generate an infrared restoration mask.

6. The asymmetric adversarial patch defense method for multimodal images according to claim 1, characterized in that, The selective diffusion repair in step S3 includes infrared expert diffusion model repair and constraint model repair based on mask conditions.

7. The asymmetric adversarial patch defense method for multimodal images according to claim 1, characterized in that, In step S4, the cleaned image re-detection model includes two single-modal target detection models trained for the repaired visible light image and the repaired infrared image, respectively.

8. An asymmetric adversarial patch defense device for multimodal images, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images