Zero-training adversarial patch defense method and system based on multi-domain quality difference perception

Abstract

The application belongs to the technical field of artificial intelligence security, and discloses a zero-training adversarial patch defense method and system based on multi-domain quality difference perception, which comprises the following steps: step 1, constructing latent space prior, generating manifold deviation heat map and determining the spatial mask of the suspicious patch region; step 2, based on the manifold deviation heat map, constructing an inverse optimization objective function, and outputting a final patch positioning map by using a pixel-level reconstruction difference map; step 3, performing morphological processing on the final patch positioning map to obtain a refined positioning mask, and combining a context perception repair technology to accurately remove the adversarial patch within the current positioning mask range, so as to ensure that the processed image can be correctly recognized by an object detector. The application separately processes clean samples and adversarial samples, accurately locates the position of the adversarial patch, and effectively removes the adversarial patch, so as to realize efficient defense of the shape, size and position of unknown patches without any pre-training.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence security technology, specifically relating to a zero-training adversarial patch defense method and system based on multi-domain quality difference perception. Background Technology

[0002] Object detection technology has made significant breakthroughs in recent years and has been widely applied in various safety-critical scenarios in the real world, such as autonomous driving and video surveillance. However, the rise of adversarial attacks has brought serious challenges to the deployment and security of object detectors, especially adversarial patch attacks. Current mainstream defense strategies against adversarial patch attacks can be categorized into adversarial training methods, provable defense methods, and methods focused on interfering with or blocking adversarial patches from the input image.

[0003] In addition to the three methods mentioned above, researchers have recently begun exploring adversarial patch defense based on generative models, attempting to utilize the prior knowledge of natural image distributions from generative models to identify and repair patch regions. In recent years, latent space manifold analysis has seen rapid development in the field of anomaly detection. Researchers have found that the latent space constructed by generative models trained on clean data can effectively encode the distribution patterns of natural images; regions in the image that do not conform to the natural distribution will exhibit detectable deviations in the latent space. This characteristic provides a new technical perspective for the detection and localization of adversarial patches—adversarial patches, as artificially optimized non-natural patterns, inevitably deviate from the natural image manifold in their latent space representation. However, no research has yet systematically applied this technology to adversarial patch defense tasks, particularly lacking effective technical solutions in areas such as how to accurately locate patch boundaries, how to efficiently distinguish patch regions from natural high-texture regions, and how to achieve universal defense across attack types without retraining.

[0004] From the above analysis, the existing adversarial patch defense methods have the following shortcomings: (1) Strong patch dependency. Most existing methods rely on prior learning to formulate effective defense methods, which are dependent on training data and cannot effectively deal with unseen patch types and unknown patch shapes, sizes and locations. Generative model-based methods also need to be adapted to specific attack data, and their generalization ability is limited. (2) High computational cost. Existing methods process clean samples and adversarial samples in the same way, wasting a lot of time and computational resources. Some methods have huge time costs and are difficult to meet the needs of real-time detection tasks. (3) Insufficient generalization ability. When faced with varied attack methods and unknown adversarial patches, the defense effect of existing methods often drops significantly. Especially when the patch has a natural appearance or is similar to the background texture, the accuracy of existing localization methods degrades sharply. (4) It is difficult to balance localization accuracy and efficiency. Methods based on denoising or mask generation are difficult to accurately delineate patch boundaries, while methods based on iterative optimization can improve accuracy but have slow inference speed. There is an inherent contradiction between the two.

[0005] Therefore, designing a defense method that is independent of patches, requires no pre-training, and can efficiently detect and block adversarial patches even when the shape, size, and location of the patches are unknown, while taking into account both defense performance and time efficiency, has become a key breakthrough in improving the adversarial robustness of object detectors. Summary of the Invention

[0006] To address the technical problems of existing adversarial patch defense methods, such as strong patch dependency, high computational cost, and insufficient generalization ability, this application provides a zero-training adversarial patch defense method and system based on multi-domain quality difference perception. Based on the key observation that adversarial patch regions exhibit different manifold distribution characteristics in the latent space compared to natural images, this application separates clean samples from adversarial samples through manifold deviation detection, accurately locates adversarial patch positions using inverse generative difference perception, and effectively removes adversarial patches through generative inpainting. Thus, it achieves efficient defense against the shape, size, and location of unknown patches without any pre-training.

[0007] To achieve the above objectives, this application employs the following technical solution:

[0008] This application presents a zero-training adversarial patch defense method based on multi-domain quality difference awareness, comprising the following steps:

[0009] Step 1: For the input image Construct latent space priors to obtain clean samples For the obtained clean sample Block-based latent coding extraction is performed to obtain the latent coding of each image block. Based on the latent coding of each image block and the clean sample... Mahalanobis distance between Generate manifold deviation heatmaps and determine spatial masks for suspicious patch regions. And attack determination is based on manifold deviation heatmap;

[0010] Step 2: Based on the manifold deviation heatmap generated in Step 1, construct a region-weighted inverse optimization objective function. Based on the inverse optimization objective function, use a variational autoencoder... Iterative search in the latent space to find a generator Reconstructed Images and Clean Samples Optimal latent variable that minimizes differences Using the optimal latent variable Calculate pixel-level reconstruction difference maps for the corresponding reconstructed images, and output the final patch localization map using these pixel-level reconstruction difference maps. ;

[0011] Step 3: Analyze the final patch location map output in Step 2. Morphological processing is performed, and the resulting localization map is refined to obtain a refined localization mask. The location of the adversarial patch is located using the localization mask. Then, context-aware inpainting technology is used to accurately remove the adversarial patch within the current localization mask range, ensuring that the processed image can be correctly identified by the object detector.

[0012] A further improvement of this application is that, in step 1, for the input image... Constructing latent space priors refers to the process of establishing priors for the input image. Preprocessing is performed, and the latent codes of the clean samples obtained after preprocessing are extracted. The mean vector of the low-dimensional latent space distribution is then calculated based on the latent codes. Specifically, it includes the following steps:

[0013] Step 1.1.1: For the input image Preprocessing to obtain clean samples ,in, This indicates the total number of input images;

[0014] Step 1.1.2: Employ a pre-trained variational autoencoder Clean samples obtained from preprocessing The encoding process is as follows:

[0015] ,

[0016] in, For variational autoencoders The encoder, for Latent variables are collected through this encoding process, gathering all clean samples. latent coding ;

[0017] Step 1.1.3, Latent coding based on step 1.1.2 Calculate the mean vector of the low-dimensional latent space distribution. :

[0018] ,

[0019] in, This represents the image index.

[0020] A further improvement of this application is that, in step 1, the obtained clean sample... The specific steps for extracting block latent codes are as follows:

[0021] Step 1.2.1: Process the obtained clean sample Each image in the data is divided into Overlapping image patches ;

[0022] Step 1.2.2: Pass each overlapping image patch through a variational autoencoder. encoder in Mapping to the latent space yields the latent code for each image patch. :

[0023] ,

[0024] in, Indicates the index of the number of currently overlapping image patches. , Indicates the currently mapped overlapping image patch. This indicates the total number of overlapping image blocks.

[0025] A further improvement in this application is that in step 1, the latent coding of each image patch and clean samples are used... Mahalanobis distance between Generate manifold deviation heatmaps and determine spatial masks for suspicious patch regions. Specifically, it includes the following steps:

[0026] Step 1.3.1: Calculate the latent code for each image block. With clean samples Mahalanobis distance between :

[0027] ;

[0028] in, For transpose;

[0029] Step 1.3.2: Generate manifold deviation heatmap: (This step involves calculating the Mahalanobis distance.) Map back to overlapping image patches The corresponding spatial location yields the manifold deviation heatmap. :

[0030] ;

[0031] in, This indicates traversing all covered spatial locations. Image Patch Index ;

[0032] Step 1.3.3: Based on the manifold deviation heatmap Determine the spatial mask of the suspicious patch area :

[0033] ;

[0034] Step 1.3.4: Set the Mahalanobis distance threshold. Statistical manifold deviation from heatmap The distance exceeds the Mahalanobis distance threshold. area of ​​continuous region :

[0035] ;

[0036] If the area of ​​the continuous region Less than the preset area threshold If the image is not subjected to an adversarial patch attack, the normal detection result is output directly; otherwise, proceed to step 2 for reverse generation and localization.

[0037] A further improvement in this application is that step 2 specifically includes the following steps:

[0038] Step 2.1, Latent variable initialization: Initialize clean samples via encoder Obtain the initial latent variables ;

[0039] Step 2.2: Construct the inverse optimization objective function with region weighting:

[0040] ,

[0041] ,

[0042] ,

[0043] ,

[0044] ,

[0045] in, This represents the weighted pixel reconstruction loss. Indicates perceived loss. Indicates the prior loss in the latent space. For spatial location The weighting coefficients, This is the weight decay factor for the patch region. Represents a variational autoencoder No. Feature map of the layer For feature layer set, The first The number of channels, height, and width of the layer feature map. and To balance the hyperparameters of each loss term, For the input image height, For the input image width, For clean samples In spatial location The RGB three-channel pixel value vector at that location, For generator;

[0046] Step 2.3: Apply gradient descent to the inverse optimization objective function defined in Step 2.2. Perform iterative optimization, latent variables The update rules are as follows:

[0047]

[0048] in, For learning rate, For iteration rounds, Indicates the current gradient direction. and The first Wheel and the first The latent variables after each iteration are continuously iterated and optimized until convergence.

[0049] Step 2.4: After the iterative optimization converges, the optimal latent variables are obtained. Through variational autoencoder generator Generate the optimal reconstructed image Calculate pixel-level reconstruction error map :

[0050]

[0051] in, For clean samples In spatial location The RGB three-channel pixel value vector at that location, To reconstruct the image In spatial location RGB pixel value vector at the location, pixel-level reconstruction error map The manifold deviation information in the latent space is decoded into patch position signals in the pixel space, initially realizing the spatial localization of the patches and reconstructing the pixel-level error map. This effectively revealed the spatial location of the patch;

[0052] Step 2.5, Multi-scale reconstruction consistency enhancement: Repeat the pixel-level reconstruction error map of steps 2.1-2.3 at multiple downsampling scales. The generation process, let the scale set be... , in scale The reconstruction error map is obtained below. Reconstruction error maps at various scales Upsampled to the original resolution and then stacked, the enhanced reconstruction error map is obtained. :

[0053]

[0054] in, This represents the total number of sampling scales. This indicates a bilinear upsampling operation;

[0055] Step 2.6: Analyze the enhanced reconstruction difference map. Adaptive threshold segmentation is performed to obtain the final patch location map:

[0056]

[0057] Among them, threshold , and Reconstructing the difference diagram The mean and standard deviation, As an adjustment factor, output the final patch location map. This is used for fine-tuning of the mask in subsequent steps.

[0058] A further improvement in this application is that, in step 3, the final patch location map output in step 2 is... Morphological processing is performed, and the morphologically processed positioning map is refined to obtain a refined positioning mask. The specific steps include the following:

[0059] Step 3.1, Morphological Post-processing: Positioning the final patch map. For morphological processing, the opening operation is defined as:

[0060]

[0061] The closing operation is defined as:

[0062]

[0063] in, For erosion operators, This is the expansion operator. Indicates the input image. For predefined structural elements, the final patch location map is obtained by using an "open-close-open" operation sequence. Processing is performed to obtain the initial positioning map. :

[0064]

[0065] Step 3.2: Initial positioning image after morphological processing As the initial label, it is refined using a fully connected conditional random field, and the mask boundary is optimized using color, spatial, and edge information between image pixels:

[0066]

[0067] in, This represents the total energy function of a fully connected conditional random field. The initial positioning diagram is used to define the univariate potential. supply, For a binary potential, it is defined as:

[0068]

[0069] in, For pixels Spatial location coordinates, For pixels Spatial location coordinates, For pixels RGB color vector values, For pixels RGB color vector values, Penalty for inconsistent labeling , , , , The CRF hyperparameters are obtained by approximating the conditional random field (CRF) using a mean field. After the above processing steps, a refined localization mask is finally obtained. ;

[0070] Step 3.3, Context-Aware Generative Patch Repair: The refined localization mask obtained in Step 3.2 Guided by [the relevant authority], using pre-trained generators The refined positioning mask obtained in step 3.2 The corresponding regions are then semantically insulated and repaired. The repair strategy is as follows: keep the pixels outside the masked region unchanged, and search for the optimal repair latent variable in the latent space so that the image reconstructed by the generator is semantically and texturally consistent with the surrounding background in the masked region.

[0071] ,

[0072] in, This represents element-wise multiplication. This represents the weighting coefficient of the perceived loss. To mitigate the perceptual loss in mask boundary regions and ensure semantic coherence between the repaired content and the surrounding background, For the optimal repair latent variable in the latent space, The generator produces the following image:

[0073]

[0074] Step 3.4: Transfer the repaired image The input object detector performs normal detection and outputs the final detection result.

[0075] This application provides a zero-training adversarial patch defense system based on multi-domain quality difference perception. The system includes a manifold deviation detection module, an inverse generation localization module, and a generative instigation module. The manifold deviation detection module determines whether an image has been attacked and generates initial suspected patch regions. Guided by these initial suspected patch regions, the inverse generation localization module optimizes the latent space and utilizes pixel-level reconstruction of the difference map to output a final patch localization map. The generative repair module utilizes refined positioning masks. The patched areas are refined and repaired to obtain the repaired image. Finally The input object detector performs normal detection and outputs the final detection result.

[0076] A further improvement of this application is that the manifold deviation detection module includes:

[0077] Latent space prior building unit: Using variational autoencoders, we collect the latent codes of all clean samples, calculate the mean vector of the latent space distribution, and construct the statistical prior of the latent space prior natural image manifold.

[0078] Block-based latent coding extraction unit: Divides each image in the clean sample into overlapping image blocks, and each image block is mapped to a low-dimensional latent space by an encoder to obtain the latent coding of each image block;

[0079] Mahalanobis distance calculation unit: calculates the Mahalanobis distance between the latent code of each image patch and the clean sample, generates a manifold deviation heatmap, sets a Mahalanobis distance threshold, and determines the area of ​​continuous regions in the manifold deviation heatmap that exceed the Mahalanobis distance threshold as suspicious patch regions;

[0080] Attack determination unit: Calculates the area of ​​the suspected patch region. If the area is less than the preset area threshold, the image is determined not to have been attacked by the patch, and the normal detection result is directly output; otherwise, it enters the reverse generation and localization module.

[0081] The inverse positioning module includes:

[0082] Latent variable initialization unit: The clean sample is encoded to obtain the initial latent variables. At the same time, a spatial mask of the suspected patch region is generated based on the manifold deviation heatmap to guide the weight allocation in the subsequent optimization process.

[0083] Region-weighted inverse optimization unit: Constructs an inverse optimization objective function with region weighting, including weighted pixel reconstruction loss, perceptual loss and latent space prior loss;

[0084] Reconstruction difference map generation unit: After iterative optimization convergence, a pixel-level reconstruction error map is generated;

[0085] Multi-scale consistency enhancement unit: Repeat the inverse optimization and reconstruction difference map generation process at multiple downsampling scales, upsample the difference maps at each scale to the original resolution and then superimpose them to enhance the consistency of patch localization and suppress noise;

[0086] Adaptive threshold segmentation unit: Performs adaptive threshold segmentation on the enhanced reconstructed difference map to obtain the final patch localization map;

[0087] The generative repair module includes:

[0088] Morphological processing unit: Performs morphological processing of the final patch positioning map by sequentially performing opening, closing and opening operations.

[0089] Conditional Random Field Refinement Unit: The initial localization map after morphological processing is used as the initial label, and a fully connected conditional random field is used for refinement to generate a refined localization mask.

[0090] Context-aware inpainting unit: Utilizes a pre-trained generator to refine the localization mask. The corresponding region is semantically perceptually filled and repaired. The optimal repair latent variable is searched in the latent space to make the image reconstructed by the generator semantically and texturally consistent with the surrounding background in the mask region. Finally, the repaired image is input into the object detector for normal detection and the final detection result is output.

[0091] The beneficial effects of this application are:

[0092] This application does not rely on any pre-training for adversarial patches and does not require prior knowledge of the shape, size, and location of the patches. By utilizing the generative model's prior knowledge of natural image manifolds, it can generalize to any type of adversarial patch, fundamentally solving the problem of insufficient generalization ability in existing adversarial training methods and possessing a universal defense capability that is zero-training and patch-independent.

[0093] This application separates clean samples from adversarial samples through manifold deviation detection. For clean samples that have not been attacked, only a lightweight latent space Mahalanobis distance calculation is required to directly output the results, avoiding unnecessary inverse optimization and repair processes, and significantly reducing computational overhead.

[0094] This application employs a region-weighted latent space inverse optimization strategy. The generator reconstructs the image to be consistent with the input in natural regions while preserving differences in patch regions. The reconstructed difference map is used to accurately locate the patch positions. Multi-scale reconstruction consistency enhancement further suppresses texture noise interference, achieving high-precision localization of adversarial patches with natural appearances and demonstrating accurate inverse generation localization capabilities.

[0095] This application can be embedded into any existing object detector as a plug-and-play defense module without modifying the original detector architecture or retraining, and has good compatibility and easy deployment. Attached Figure Description

[0096] Figure 1 This is a schematic diagram of the overall process of this application.

[0097] Figure 2 This is a schematic diagram of the process for generating a manifold deviation heatmap and performing attack determination in this application.

[0098] Figure 3 This is a flowchart illustrating the reverse generation positioning process in this application.

[0099] Figure 4 This is a schematic diagram of the generative mask refinement and patch repair process in this application.

[0100] Figure 5 This is the average accuracy of the defense methods of this application and existing methods against various patch attacks.

[0101] Figure 6This is a diagram illustrating the defense success rate (DSR) of different defense methods against patch attacks at different locations.

[0102] Figure 7 This is a diagram comparing the defensive effectiveness and time efficiency of different defense methods. Detailed Implementation

[0103] The embodiments of this application will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details should not be used to limit this application. That is, in some embodiments of this application, these practical details are not necessary. In addition, for the sake of simplicity, some conventional structures and components will be shown in the drawings in a simple schematic manner.

[0104] This application is implemented using a zero-training adversarial patch defense system, which includes a manifold deviation detection module, an inverse generation localization module, and a generative insulation module. The manifold deviation detection module determines whether an image has been attacked and generates initial suspected patch regions. Guided by the initial suspected patch regions, the inverse generation localization module optimizes the latent space and uses pixel-level reconstruction of the difference map to output the final patch localization map. The generative repair module utilizes refined positioning masks. The patched areas are refined and repaired to obtain the repaired image. Finally The input object detector performs normal detection and outputs the final detection result.

[0105] The manifold deviation detection module includes:

[0106] Latent space prior building unit: Using variational autoencoders, we collect the latent codes of all clean samples, calculate the mean vector of the latent space distribution, and construct the statistical prior of the latent space prior natural image manifold.

[0107] Block-based latent coding extraction unit: Divides each image in the clean sample into overlapping image blocks, and each image block is mapped to a low-dimensional latent space by an encoder to obtain the latent coding of each image block;

[0108] Mahalanobis distance calculation unit: calculates the Mahalanobis distance between the latent code of each image patch and the clean sample, generates a manifold deviation heatmap, sets a Mahalanobis distance threshold, and determines the area of ​​continuous regions in the manifold deviation heatmap that exceed the Mahalanobis distance threshold as suspicious patch regions;

[0109] Attack determination unit: Calculates the area of ​​the suspected patch region. If the area is less than the preset area threshold, the image is determined not to have been attacked by the patch, and the normal detection result is directly output; otherwise, it enters the reverse generation and localization module.

[0110] The inverse positioning module includes:

[0111] Latent variable initialization unit: The clean sample is encoded to obtain the initial latent variables. At the same time, a spatial mask of the suspected patch region is generated based on the manifold deviation heatmap to guide the weight allocation in the subsequent optimization process.

[0112] Region-weighted inverse optimization unit: Constructs an inverse optimization objective function with region weighting, including weighted pixel reconstruction loss, perceptual loss and latent space prior loss;

[0113] Reconstruction difference map generation unit: After iterative optimization convergence, a pixel-level reconstruction error map is generated;

[0114] Multi-scale consistency enhancement unit: Repeat the inverse optimization and reconstruction difference map generation process at multiple downsampling scales, upsample the difference maps at each scale to the original resolution and then superimpose them to enhance the consistency of patch localization and suppress noise;

[0115] Adaptive threshold segmentation unit: Performs adaptive threshold segmentation on the enhanced reconstructed difference map to obtain the final patch localization map;

[0116] The generative repair module includes:

[0117] Morphological processing unit: Performs morphological processing of the final patch positioning map by sequentially performing opening, closing and opening operations.

[0118] Conditional Random Field Refinement Unit: The initial localization map after morphological processing is used as the initial label, and a fully connected conditional random field is used for refinement to generate a refined localization mask.

[0119] Context-aware inpainting unit: Utilizes a pre-trained generator to refine the localization mask. The corresponding region is semantically perceptually filled and repaired. The optimal repair latent variable is searched in the latent space to make the image reconstructed by the generator semantically and texturally consistent with the surrounding background in the mask region. Finally, the repaired image is input into the object detector for normal detection and the final detection result is output.

[0120] like Figure 1 As shown, this application presents a zero-training adversarial patch defense method based on multi-domain quality difference awareness, applicable to tasks such as autonomous driving and video surveillance. Taking video surveillance as an example, this application uses the INRIAPerson dataset, a commonly used dataset for video surveillance, to provide a specific defense implementation process, including the following steps:

[0121] Step 1: This application uses the INRIAPerson dataset from the video surveillance field as the input image data for this example. (The text then abruptly shifts to a different topic:) Construct latent space priors to obtain clean samples For the obtained clean sample Block-based latent coding extraction is performed to obtain the latent coding of each image block. Based on the latent coding of each image block and the clean sample... Mahalanobis distance between Generate manifold deviation heatmaps and determine spatial masks for suspicious patch regions. It also determines attacks based on manifold deviation heatmaps.

[0122] like Figure 2 As shown, this application specifically includes the following steps:

[0123] Step 1.1: Construct the input image for the INRIAPerson video surveillance dataset. And preprocessing to obtain clean samples ,in, This represents the total number of input images, as used in this example experiment. Set to 3000, input image height Set to 1920, width Set to 1080;

[0124] Step 1.2: Employ a pre-trained variational autoencoder Clean samples obtained from preprocessing The encoding process is as follows:

[0125] ,

[0126] in, For variational autoencoders The encoder, through the encoder Clean sample Mapped to the low-dimensional latent space of deep learning models, for Latent variables are collected through this encoding process, gathering all clean samples. latent coding Variational autoencoder Includes an encoder and a generator via encoder Clean sample Mapped to the low-dimensional latent space of the deep learning model, and reconstructed from the latent variables in the subsequent step 2 by the generator to obtain a reconstructed image that conforms to the distribution of natural images;

[0127] Step 1.3, Latent coding based on step 1.1.2 Calculate the mean vector of the low-dimensional latent space distribution. :

[0128] ,

[0129] in, Indicates the currently calculated number of... Image index, in this example The range of values ​​is .

[0130] Step 1.4: Process the obtained clean sample Each image in the data is divided into Overlapping image patches In this example The optimal value is 66;

[0131] Step 1.5: Transform each overlapping image patch through a variational autoencoder. encoder in Mapping to the latent space yields the latent code for each image patch. :

[0132] ,

[0133] in, Indicates the index of the number of currently overlapping image patches. , Indicates the currently mapped overlapping image patch. This represents the total number of overlapping image patch divisions; in this example... The value is .

[0134] Step 1.6: Calculate the latent code for each image block. With clean samples Mahalanobis distance between :

[0135] ,

[0136] Mahalanobis distance measures the degree to which the latent encoding deviates from the clean image manifold. The latent encoding of natural image patches follows the distribution of the training data, resulting in a smaller Mahalanobis distance. The adversarial patch, as an optimized non-natural pattern, deviates from the manifold in its latent encoding, resulting in a larger Mahalanobis distance. .

[0137] Step 1.7: Generate manifold deviation heatmap: (This step involves calculating the Mahalanobis distance.) Map back to overlapping image patches The corresponding spatial location yields the manifold deviation heatmap. :

[0138] ;

[0139] in, This indicates traversing all covered spatial locations. Image Patch Index For overlapping regions, the maximum Mahalanobis distance is used to ensure that no potential outliers are missed. In this example... The value is set to 16.5.

[0140] Step 1.8: Based on the manifold deviation heatmap Determine the spatial mask of the suspicious patch area :

[0141] ,

[0142] This mask is used to guide the allocation of region weights in subsequent optimization processes.

[0143] Step 1.9: Set the Mahalanobis distance threshold. Preferred ,in and The mean and standard deviation of the Mahalanobis distance on the clean validation set are given, respectively, and the statistical manifold deviation heatmap is also shown. The distance exceeds the Mahalanobis distance threshold. area of ​​continuous region :

[0144] ;

[0145] If the area of ​​the continuous region Less than the preset area threshold , If the area is preferably 1% of the total image area, the image is determined to be free from adversarial patch attacks, and a normal detection result is directly output; otherwise, proceed to step 2 for reverse generation and localization. Manifold deviation Mahalanobis distance heatmap It also serves as the initial guide for the reverse generation optimization in step 2.

[0146] Step 2: Based on the manifold deviation heatmap generated in Step 1, construct a region-weighted inverse optimization objective function. Based on the inverse optimization objective function, use a variational autoencoder... Iterative search in the latent space to find a generator Reconstructed Images and Clean Samples Optimal latent variable that minimizes differences Using the optimal latent variable Calculate pixel-level reconstruction difference maps for the corresponding reconstructed images, and output the final patch localization map using these pixel-level reconstruction difference maps. .like Figure 3As shown, step 2 specifically includes the following steps:

[0147] Step 2.1, Latent variable initialization: Initialize clean samples via encoder Obtain the initial latent variables This provides a starting point for optimization in step 2.2.

[0148] Step 2.2: Construct a region-weighted inverse optimization objective function and optimize the latent variables using gradient descent. This makes the generator Reconstructed images and clean samples The goal is to achieve the closest possible approximation in non-patched regions, while allowing for differences in patched regions. The inverse optimization objective function is defined as follows:

[0149] ,

[0150] ,

[0151] ,

[0152] ,

[0153] ,

[0154] in, This represents the weighted pixel reconstruction loss, assigning higher weights to non-patch areas and lower weights to suspected patch areas. The perceptual loss is represented by a pre-trained neural network extracting features and measuring the difference between the reconstructed image and the input image in the high-level semantic feature space. This represents the prior loss in the latent space, constraining latent variables to not deviate too far from the prior distribution, and preserving the visual realism of the generated image. Spatial location The weighting coefficients, This is the weight decay factor for the patch region, preferably 0.1, indicating that a relatively large reconstruction error in the patch region is allowed during the optimization process. Represents a variational autoencoder The Feature map of the layer For the selected feature layer set, The first The number of channels, height, and width of the layer feature map. and To balance the hyperparameters of each loss term, For the input image height, For the input image width, For clean samples In spatial location The RGB three-channel pixel value vector at that location, For generator; optimization target The essential function is to transform the spatial location information of the adversarial patch from the abstract signal of manifold deviation in latent space into an observable signal of reconstruction error in pixel space. This optimization objective provides direction for the optimization iteration in step 2.3.

[0155] Step 2.3: Iterative optimization solution. Gradient descent is used to optimize the inverse objective function defined in Step 2.2. Perform iterative optimization, latent variables The update rules are as follows:

[0156] ,

[0157] in, The learning rate is preferably 0.01; The number of iterations is preferably 15. Indicates the current gradient direction. and The first Wheel and First The latent variables after each iteration are continuously iterated and optimized until convergence; as iterations proceed, the latent variables... The image gradually moves in a direction that can reconstruct an image consistent with the input image in the non-patch region, while the patch region, due to its lower weight, retains the reconstruction differences.

[0158] Step 2.4: Generate pixel-level reconstructed difference maps. After iterative optimization converges, the optimal latent variables are obtained. Through variational autoencoder generator Generate the optimal reconstructed image Calculate pixel-level reconstruction error map :

[0159] ,

[0160] in, For clean samples In spatial location The RGB three-channel pixel value vector at that location, To reconstruct the image In spatial location RGB pixel value vector at the location, pixel-level reconstruction error map Decoding manifold deviation information in latent space into patch location signals in pixel space initially achieves spatial localization of patches, serving as a direct basis for subsequent patch localization and removal. However, the generation at a single scale... Issues such as localized texture noise interference, blurred patch boundaries, or discontinuous responses within patches may still exist, requiring further enhancement through multi-scale fusion. Since the optimization objective forces consistent reconstruction in non-patch areas while allowing larger deviations in patch areas, pixel-level reconstruction error maps... It exhibits a significant peak in the patched region and approaches zero in the natural image region. Pixel-level reconstruction error map. This effectively revealed the spatial location of the patch;

[0161] Step 2.5: Enhanced Consistency of Multi-Scale Reconstruction. To improve positioning accuracy and suppress noise, the pixel-level reconstruction error maps from steps 2.1-2.3 are repeated at multiple downsampling scales. The generation process, let the scale set be... , in scale The reconstruction error map is obtained below. Reconstruction error maps at various scales Upsampled to the original resolution and then stacked, the enhanced reconstruction error map is obtained. :

[0162] ,

[0163] in, The total number of sampling scales. This indicates a bilinear upsampling operation. Multi-scale stacking enhances the response consistency of the patch region at different resolutions, while suppressing texture noise interference at a single scale.

[0164] Step 2.6, Adaptive Threshold Segmentation: Reconstructing the difference map after enhancement. Adaptive threshold segmentation is performed to obtain the final patch location map. :

[0165] ,

[0166] Among them, threshold , and Reconstructing the difference diagram The mean and standard deviation, As an adjustment factor, output the final patch location map. This is used for fine-tuning of the mask in subsequent steps.

[0167] Step 3: Analyze the final patch location map output in Step 2. Morphological processing is performed, and the resulting localization map is refined to obtain a refined localization mask. This mask is used to pinpoint the location of adversarial patches. Then, context-aware inpainting techniques are used to precisely remove the adversarial patches within the currently located mask area, ensuring that the processed image can be correctly identified by the object detector. Figure 4 As shown, step 3 specifically includes the following steps:

[0168] Step 3.1, Morphological Post-processing: Positioning the final patch map. Morphological processing is performed to remove isolated noise points and fill holes in the localization map. The opening operation is defined as:

[0169] ,

[0170] The closing operation is defined as:

[0171] ,

[0172] in, For erosion operators, This is the expansion operator. Indicates the input image. For predefined structural elements, the final patch location map is obtained by using an "open-close-open" operation sequence. Processing is performed to obtain the initial positioning map. :

[0173] ,

[0174] Step 3.2, Conditional Random Field (CRF) Refinement. The initial localization map after morphological processing... As the initial label, it is refined using a fully connected conditional random field, and the mask boundary is optimized using color, spatial, and edge information between image pixels:

[0175] ,

[0176] in, This represents the total energy function of a fully connected conditional random field. The initial positioning diagram is used to define the univariate potential. supply, For a binary potential, it is defined as:

[0177] ,

[0178] in, For pixels Spatial location coordinates, For pixels Spatial location coordinates, For pixels RGB color vector values, For pixels RGB color vector values, Penalty for inconsistent labeling , , , , For CRF hyperparameters, the best option is... , , , , By approximating the conditional random field (CRF) with a mean field, and following the above processing steps, a refined localization mask is finally obtained. Through refined positioning masks This provides precise spatial guidance for the patch region in step 3.3, i.e. how to locate the adversarial patch. Guided by this mask, step 3.3 will use the generative model to perform semantically coherent pixel filling on the mask region. The two work together to complete the division of labor and how to fix the patch.

[0179] Step 3.3, Context-Aware Generative Patch Repair: The refined localization mask obtained in Step 3.2 Guided by [the relevant authority], using pre-trained generators The refined positioning mask obtained in step 3.2 The corresponding regions are then semantically insulated and repaired. The repair strategy is as follows: keep the pixels outside the masked region unchanged, and search for the optimal repair latent variable in the latent space so that the image reconstructed by the generator is semantically and texturally consistent with the surrounding background in the masked region.

[0180] ,

[0181] in, This represents element-wise multiplication. This represents the perceptual loss weighting coefficient, which is set to 0.1 in this embodiment. To mitigate the perceptual loss in mask boundary regions and ensure semantic coherence between the restored content and the surrounding background, For the optimal repair latent variable in the latent space, The generator produces the following image:

[0182] ,

[0183] Step 3.4: Transfer the repaired video surveillance images The input object detector performs normal detection and outputs the final detection result.

[0184] To verify this application, the defensive effectiveness of the method in this application is verified, specifically as follows:

[0185] For the test image, execute steps 1 to 3 sequentially, and output the detection result after defense processing;

[0186] Defense performance is evaluated using Defense Success Rate (DSR), which is calculated as follows:

[0187] ,

[0188] in, This indicates the number of times the object detector successfully defended against attacks. This indicates the total number of patch attacks.

[0189] like Figure 5 The three different adversarial patch generation methods shown—AdvPatch, TC-EGA, and T-SEA—attacked various object detection models, including YOLOv3, YOLOv5, and Faster R-CNN. Figure 5 (a) in the image is a detector based on Faster R-CNN. Figure 5 (b) in the example uses a detector based on YOLOv3. Figure 5 (c) in the diagram uses YOLOv5 as the base detector. A higher mean accuracy indicates better defensive performance of the model. Under the three attacks tested, our proposed method achieved the best defensive performance.

[0190] like Figure 6 As shown, to ensure the validity of the test results, this application conducted defense tests on the INRIAPerson test dataset, a commonly used dataset in the video surveillance field. A total of 200 adversarial patch attacks were performed, with patches placed in different locations including the center, top and bottom, and left and right. The defense success rate of each method was evaluated to verify the performance of the proposed method in the face of different patches and changes in their positions. Figure 6 (a) in the example uses the Faster RCNN detector as a benchmark. Figure 6 (b) in the example uses the YOLOv3 detector as a reference. Figure 6 (c) in the figure is based on the YOLOR detector. A higher defense success rate indicates better defense performance of the model. Under the attack conditions at different locations tested, the defense performance of our proposed method is optimal.

[0191] Figure 7This paper compares the defense effectiveness and time efficiency of different defense methods. Taking the YOLOv3 detector as an example, considering the advantages of the proposed method in detection performance and the very small time overhead on clean samples, the proposed method exhibits stronger overall performance advantages.

[0192] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A zero-training adversarial patch defense method based on multi-domain quality difference awareness, characterized in that: The zero-training adversarial patch defense method specifically includes the following steps: Step 1: For the input image Construct latent space priors to obtain clean samples For the obtained clean sample Block-based latent coding extraction is performed to obtain the latent coding of each image block. Based on the latent coding of each image block and the clean sample... Mahalanobis distance between Generate manifold deviation heatmaps and determine spatial masks for suspicious patch regions. And attack determination is based on manifold deviation heatmap; Step 2: Based on the manifold deviation heatmap generated in Step 1, construct a region-weighted inverse optimization objective function. Based on the inverse optimization objective function, use a variational autoencoder... Iterative search in the latent space to find a generator Reconstructed Images and Clean Samples Optimal latent variable that minimizes differences Using the optimal latent variable Calculate pixel-level reconstruction difference maps for the corresponding reconstructed images, and output the final patch localization map using these pixel-level reconstruction difference maps. Specifically, the steps include the following: Step 2.1, Latent variable initialization: Initialize clean samples via encoder Obtain the initial latent variables ; Step 2.2: Construct the inverse optimization objective function with region weighting. : ， ， ， ， ， in, This represents the weighted pixel reconstruction loss. Indicates perceived loss. Indicates the prior loss in the latent space. Spatial location The weighting coefficients, This is the weight decay factor for the patch region. Represents a variational autoencoder No. Feature map of the layer For feature layer set, The first The number of channels, height, and width of the layer feature map. and To balance the hyperparameters of each loss term, For the input image height, For the input image width, For clean samples In spatial location The RGB three-channel pixel value vector at that location, For variational autoencoders generator; Step 2.3: Apply gradient descent to the inverse optimization objective function defined in Step 2.

2. Perform iterative optimization, latent variables The update rules are as follows: ， in, For learning rate, For iteration rounds, Indicates the current gradient direction. and The first Wheel and the first The latent variables after each iteration are continuously iterated and optimized until convergence. Step 2.4: After the iterative optimization converges, the optimal latent variables are obtained. Through variational autoencoder generator Generate the optimal reconstructed image Calculate pixel-level reconstruction error map : ， in, For clean samples In spatial location The RGB three-channel pixel value vector at that location, To reconstruct the image In spatial location RGB pixel value vector at the location, pixel-level reconstruction error map The manifold deviation information in the latent space is decoded into patch position signals in the pixel space, initially realizing the spatial localization of the patches and reconstructing the pixel-level error map. This effectively revealed the spatial location of the patch; Step 2.5, Multi-scale reconstruction consistency enhancement: Repeat the pixel-level reconstruction error map of steps 2.1-2.3 at multiple downsampling scales. The generation process, let the scale set be... , in scale The reconstruction error map is obtained below. Reconstruction error maps at various scales Upsampled to the original resolution and then stacked, the enhanced reconstruction error map is obtained. : ， in, The total number of sampling scales. This indicates a bilinear upsampling operation; Step 2.6: Analyze the enhanced reconstruction difference map. Adaptive threshold segmentation is performed to obtain the final patch location map: ， Among them, threshold , and Reconstructing the difference diagram The mean and standard deviation, As an adjustment factor, output the final patch location map. Used for fine-tuning of the mask in subsequent steps; Step 3: Analyze the final patch location map output in Step 2. Morphological processing is performed, and the resulting localization map is refined to obtain a refined localization mask. The location of the adversarial patch is located using the localization mask. Then, context-aware inpainting technology is used to accurately remove the adversarial patch within the current localization mask range, ensuring that the processed image can be correctly identified by the object detector.

2. The zero-training adversarial patch defense method based on multi-domain quality difference awareness according to claim 1, characterized in that: In step 1, for the input image Constructing latent space priors refers to the process of establishing priors for the input image. Preprocessing was performed, and the clean sample obtained after preprocessing was extracted. The latent code is obtained, and the mean vector of the low-dimensional latent space distribution is calculated based on the latent code. Specifically, it includes the following steps: Step 1.1.1: For the input image Preprocessing to obtain clean samples ,in, This indicates the total number of input images; Step 1.1.2: Employ a pre-trained variational autoencoder Clean samples obtained from preprocessing The encoding process is as follows: ， in, For variational autoencoders The encoder, for Latent variables are collected through this encoding process, gathering all clean samples. latent coding ; Step 1.1.3, Latent coding based on step 1.1.2 Calculate the mean vector of the low-dimensional latent space distribution. : ， in, This represents the image index.

3. The zero-training adversarial patch defense method based on multi-domain quality difference awareness according to claim 1, characterized in that: In step 1, the obtained clean sample The specific steps for extracting block latent codes are as follows: Step 1.2.1: Process the obtained clean sample Each image in the data is divided into Overlapping image patches ; Step 1.2.2: Pass each overlapping image patch through a variational autoencoder. encoder in Mapping to the latent space yields the latent code for each image patch. : ， in, Indicates the index of the number of currently overlapping image patches. , This represents the overlapping image patch of the current mapping. This indicates the total number of overlapping image blocks.

4. The zero-training adversarial patch defense method based on multi-domain quality difference awareness according to claim 3, characterized in that: In step 1, the latent coding of each image patch and clean samples are used as a basis. Mahalanobis distance between Generate manifold deviation heatmaps and determine spatial masks for suspicious patch regions. The attack determination is based on the manifold deviation heatmap, specifically including the following steps: Step 1.3.1: Calculate the latent code for each image block. With clean samples Mahalanobis distance between : ； in, For transpose; Step 1.3.2: Generate manifold deviation heatmap: (This step involves calculating the Mahalanobis distance.) Map back to overlapping image patches The corresponding spatial location yields the manifold deviation heatmap. : ； in, This indicates traversing all covered spatial locations. Image Patch Index ; Step 1.3.3 Based on the manifold deviation heatmap Determine the spatial mask of the suspicious patch area : ； Step 1.3.4: Set the Mahalanobis distance threshold. Statistical manifold deviation from heatmap The distance exceeds the Mahalanobis distance threshold. area of ​​continuous region : ； If the area of ​​the continuous region Less than the preset area threshold If the image is not subjected to an adversarial patch attack, the normal detection result is output directly; otherwise, proceed to step 2 for reverse generation and localization.

5. The zero-training adversarial patch defense method based on multi-domain quality difference awareness according to claim 1, characterized in that: In step 3, the final patch location map output in step 2 is processed. Morphological processing is performed, and the morphologically processed positioning map is refined to obtain a refined positioning mask. The specific steps include the following: Step 3.1, Morphological Post-processing: Positioning the final patch map. For morphological processing, the opening operation is defined as: The closing operation is defined as: ,in, For erosion operators, This is the expansion operator. Indicates the input image. For predefined structuring elements, the final patch location map is obtained by performing an open-close-open operation sequence. Processing is performed to obtain the initial positioning map. : ， Step 3.2: Initial positioning image after morphological processing As the initial label, it is refined using a fully connected conditional random field, and the mask boundary is optimized using color, spatial, and edge information between image pixels: ， in, This represents the total energy function of a fully connected conditional random field. As a univariate potential, it is based on the initial positioning diagram. supply, For a binary potential, it is defined as: ， in, For pixels Spatial location coordinates, For pixels Spatial location coordinates, For pixels RGB color vector values, For pixels RGB color vector values, Penalty for inconsistent labeling , , , , The hyperparameters of the fully connected conditional random field are approximated by the mean field. After the above processing steps, a refined localization mask is finally obtained. ; Step 3.3, Context-Aware Generative Patch Repair: The refined localization mask obtained in Step 3.2 Guided by [the relevant authority], using pre-trained generators fine-grained positioning mask The corresponding regions are then semantically insulated and repaired. The repair strategy is as follows: keep the pixels outside the masked region unchanged, and search for the optimal repair latent variable in the latent space so that the image reconstructed by the generator is semantically and texturally consistent with the surrounding background in the masked region. ， in, This represents element-wise multiplication. This represents the weighting coefficient of the perceived loss. To mitigate the perceptual loss in mask boundary regions and ensure semantic coherence between the repaired content and the surrounding background, For the optimal repair latent variable in the latent space, The generator produces the following image: ， Step 3.4: Transfer the repaired image The input object detector performs normal detection and outputs the final detection result.

6. A zero-training adversarial patch defense system that implements the multi-domain quality difference awareness zero-training adversarial patch defense method according to any one of claims 1-5, characterized in that: The zero-training adversarial patch defense system includes a manifold deviation detection module, an inverse generation localization module, and a generative insulation module. The manifold deviation detection module determines whether an image has been attacked and generates initial suspected patch regions. Guided by the initial suspected patch regions, the inverse generation localization module optimizes the latent space and uses pixel-level reconstruction of the difference map to output the final patch localization map. The generative repair module utilizes refined positioning masks. The patched areas are refined and repaired to obtain the repaired image. Finally The input object detector performs normal detection and outputs the final detection result.

7. The zero-training adversarial patch defense system according to claim 6, characterized in that: The manifold deviation detection module includes: Latent space prior building unit: Using variational autoencoders, we collect the latent codes of all clean samples, calculate the mean vector of the latent space distribution, and construct the statistical prior of the latent space prior natural image manifold. Block-based latent coding extraction unit: Divides each image in the clean sample into overlapping image blocks, and each image block is mapped to a low-dimensional latent space by an encoder to obtain the latent coding of each image block; Mahalanobis distance calculation unit: calculates the Mahalanobis distance between the latent code of each image patch and the clean sample, generates a manifold deviation heatmap, sets a Mahalanobis distance threshold, and determines the area of ​​continuous regions in the manifold deviation heatmap that exceed the Mahalanobis distance threshold as suspicious patch regions; Attack determination unit: Calculates the area of ​​the suspicious patch region. If the area of ​​the suspicious patch region is less than the preset area threshold, it is determined that the image has not been attacked by the patch and the normal detection result is directly output; otherwise, it enters the reverse generation and localization module.

8. The zero-training adversarial patch defense system according to claim 6, characterized in that: The inverse positioning module includes: Latent variable initialization unit: The clean sample is encoded to obtain the initial latent variables. At the same time, a spatial mask of the suspected patch region is generated based on the manifold deviation heatmap to guide the weight allocation in the subsequent optimization process. Region-weighted inverse optimization unit: Constructs an inverse optimization objective function with region weighting, including weighted pixel reconstruction loss, perceptual loss and latent space prior loss; Reconstruction difference map generation unit: After iterative optimization convergence, a pixel-level reconstruction error map is generated; Multi-scale consistency enhancement unit: Repeat the inverse optimization and pixel-level reconstruction difference map generation process at multiple downsampling scales, upsample the difference maps at each scale to the original resolution and then superimpose them to enhance the consistency of patch localization and suppress noise; Adaptive threshold segmentation unit: Adaptive threshold segmentation is performed on the enhanced reconstructed difference map to obtain the final patch location map.

9. The zero-training adversarial patch defense system according to claim 6, characterized in that: The generative repair module includes: Morphological processing unit: Performs morphological processing of the final patch positioning map by sequentially performing opening, closing and opening operations. Conditional Random Field Refinement Unit: The initial localization map after morphological processing is used as the initial label, and a fully connected conditional random field is used for refinement to generate a refined localization mask. Context-aware inpainting unit: Utilizes a pre-trained generator to refine the localization mask. The corresponding region is semantically perceptually filled and repaired. The optimal repair latent variable is searched in the latent space to make the image reconstructed by the generator semantically and texturally consistent with the surrounding background in the mask region. Finally, the repaired image is input into the object detector for normal detection and the final detection result is output.

Images