Adversarial patch detection and positioning method based on local image block sequence modeling

Abstract

The invention provides an adversarial patch detection and positioning method based on local image block sequence modeling, and the method comprises the steps: carrying out the preprocessing of a to-be-detected image, and carrying out the division of a multi-scale sliding window, and obtaining a plurality of local image blocks in a spatial order; extracting multi-dimensional image features for each local image block, and splicing the multi-dimensional image features into an image block feature sequence according to a sliding sequence; inputting the image block feature sequence into a self-attention Transform model subjected to sequence smoothness constraint training, and outputting an abnormal score sequence of each image block; screening abnormal candidate regions according to a preset abnormal score threshold value and a mutation rate threshold value based on the abnormal score sequence and the adjacent block mutation rate; performing consistency verification and region merging on the abnormal candidate regions obtained under different scales to form an abnormal candidate region set; and selecting the first K image blocks with the highest confidence in the abnormal candidate region set, generating a closed patch positioning frame based on spatial connectivity analysis, and obtaining the overall position of the adversarial patch.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, specifically relating to an adversarial patch detection and localization method based on local image patch sequence modeling. Background Technology

[0002] With the widespread application of deep learning in tasks such as image recognition, object detection, and semantic segmentation, it has played a crucial role in key areas such as autonomous driving, security monitoring, facial recognition, and medical imaging. However, existing deep learning models still lack robustness to input perturbations, making them highly vulnerable to adversarial attacks. Attackers can effectively mislead models into outputting incorrect results by adding minute perturbations to the input image, leading to serious security risks. Among the various forms of adversarial attacks, adversarial patch attacks have attracted considerable attention due to their stealth and practical operability. Unlike traditional global perturbations, this type of attack induces the model to make incorrect judgments by attaching fake patches, such as small patterns, stickers, or films, to local areas of the image. For example, attackers can attach fake patches to traffic signs, causing the autonomous driving system to misinterpret "speed limit" as "stop," thereby triggering a serious traffic accident.

[0003] Existing detection and defense methods against patch attacks mainly fall into the following categories: Statistical feature-based methods: These methods traverse image regions through a sliding window and extract entropy values ​​to calculate anomaly scores. While simple to implement and highly interpretable, these methods rely heavily on manually generated features, are easily affected by complex image backgrounds, and lack generalization ability.

[0004] Image reconstruction-based methods utilize autoencoders or generative adversarial networks to reconstruct the input image and then locate patch regions through residual analysis. These methods can adapt to different image styles, but they are costly to train and their effectiveness depends on the distribution of the training data; once the patch morphology differs significantly from the training data, detection performance degrades.

[0005] Attention-based methods: These methods locate regions that may be affected by patches by analyzing the attention distribution or feature response anomalies in the intermediate layers of deep neural networks. While these methods can improve robustness to some extent, they are heavily reliant on the interpretation of the attention map and are easily affected by hyperparameter settings.

[0006] Although the above methods have improved patch detection capabilities to some extent, they still face many challenges: (1) most methods independently model local regions, lacking the utilization of the overall contextual semantics of the image; (2) heuristic methods rely on fixed thresholds and lack a unified optimization framework; (3) deep model-based methods lack interpretability and are difficult to accurately locate the patch center; (4) most methods have limited generalization ability for changes in patch position, shape, and color. Therefore, there is an urgent need for an adversarial patch detection and localization method that can combine local image patches with global contextual information and possesses high accuracy, strong generalization, and strong interpretability. Summary of the Invention

[0007] This invention proposes an adversarial patch detection and localization method based on local image patch sequence modeling, aiming to solve the technical problems of insufficient detection stability, limited semantic modeling ability, and poor adaptability to practical applications of existing patch detection technologies.

[0008] To address the aforementioned technical problems, this invention provides an adversarial patch detection and localization method based on local image patch sequence modeling, comprising the following steps: Step S1: Preprocess the image to be detected and divide it using a multi-scale sliding window to obtain several local image blocks with spatial order; Step S2: Extract multidimensional image features for each of the local image blocks and stitch them together in sliding order to form an image block feature sequence; Step S3: Input the image patch feature sequence into a self-attention Transformer model trained with sequence smoothness constraints, and output the anomaly score sequence for each image patch; Step S4: Based on the abnormal scoring sequence and the mutation rate of adjacent blocks, filter abnormal candidate regions according to the preset abnormal scoring threshold and mutation rate threshold; Step S5: Perform consistency verification and region merging on the abnormal candidate regions obtained at different scales to form a set of abnormal candidate regions; Step S6: Select the top K image patches with the highest confidence from the set of abnormal candidate regions, generate closed patch location boxes based on spatial connectivity analysis, and obtain the overall location of the adversarial patch.

[0009] Preferably, the preprocessing in step S1 includes: uniform size scaling, color space conversion, standardization, edge alignment, and size cropping; the multi-scale sliding window division includes: sliding windows of each scale across the entire image with a fixed step size to form local image blocks with spatial order.

[0010] Preferably, the multidimensional image features in step S2 include: grayscale mean, grayscale standard deviation, edge sharpness of the grayscale image, information entropy of the grayscale image, color richness index, and pixel standard deviation of three channels in the HSV color space, wherein the color richness index is obtained by combining the saturation mean and saturation standard deviation.

[0011] Preferably, the image patch feature sequence is spliced ​​in a sliding order from left to right and from top to bottom, and the sequence is used as input to the self-attention Transformer model after feature standardization to maintain the statistical stationarity of the sequence.

[0012] Preferably, the self-attention Transformer model includes a feature embedding layer, a self-attention encoding module, and a position-wise regression layer, wherein the position-wise regression layer is used to output the anomaly score sequence for each image patch.

[0013] Preferably, the self-attention Transformer model employs the following sequence smoothness constraint function. Apply constraints: ; In the formula, Indicates the length of the feature sequence of the image patch; Indicates the first Predicted anomaly scores for features of individual image patches.

[0014] Preferably, the self-attention Transformer model employs a weakly supervised training mechanism, using a scoring function based on the statistical difference between the whole image and local areas to generate pseudo-labels, and performs supervised training on the image patch feature sequence to obtain anomaly scoring sequences.

[0015] Preferably, the expression for the scoring function is: ; In the formula, This represents the score of the i-th image patch; Indicates the weighting coefficient; This represents the average gray value of the i-th image patch; This represents the average gray level of the entire image; This represents the standard deviation of the grayscale values ​​of the i-th image patch; K represents the standard deviation of the gray levels of the entire image; K represents the number of statistical features. The index value of the k-th image feature of the i-th image block.

[0016] Preferably, step S5 includes: fusing abnormal candidate regions collected by sliding windows at different scales, performing consistency verification by comparing the spatial overlap relationship of candidate regions at different scales, merging intersecting or adjacent candidate regions, and removing scattered, small-area or naturally highly complex regions to reduce false detections.

[0017] Preferably, the patch positioning box in step S6 is output in the form of a rectangle or a heatmap.

[0018] The beneficial effects of the present invention include at least the following: 1) Introducing local sequential modeling to improve detection stability: By dividing the image into a multi-scale sliding window, a set of local image patches is formed. A Transformer network based on a self-attention mechanism is then used to globally model the image patch sequence, enabling joint analysis of local features and contextual information. This method can effectively distinguish between complex regions with natural textures and real adversarial patch regions, thus significantly improving detection accuracy and robustness.

[0019] 2) Overcoming the problem of local isolation: Traditional methods typically score local regions independently, lacking the utilization of context and easily leading to misjudgment. This invention constructs an ordered sequence of image patches and performs overall modeling, enabling anomaly judgment to refer to the semantic consistency of the context, thereby improving the accuracy of patch localization and reducing misjudgment of normal complex regions.

[0020] 3) Reduced reliance on manual annotation: This invention proposes a weakly supervised training mechanism that generates pseudo-labels using a scoring function based on the statistical differences between the whole image and local areas, and combines gray-level mean difference, variance difference, and multi-dimensional features for supervised training. This method eliminates the need for manually labeled patch masks, significantly reducing data annotation costs, while enhancing the model's generalization ability in different scenarios.

[0021] 4) Enhanced anomaly detection sensitivity: During training, this invention introduces a sequence smoothness constraint function to ensure the smoothness of the score sequence in non-anomaly regions; during inference, it further captures anomalous fluctuations in local structures by calculating the mutation rate of anomaly scores. This method can more sensitively detect patch regions that disrupt semantic coherence, achieving high-precision detection and localization.

[0022] 5) Improve overall patch localization accuracy: After screening candidate regions, this invention integrates detection results at different scales to eliminate missed detections and false detections caused by mismatch in a single scale; and combines the spatial intersection and connectivity analysis of Top-K high-confidence regions to generate a closed patch localization box, thereby achieving accurate localization of the entire patch range and avoiding scattered detection results.

[0023] 6) Excellent scalability and application value: The method framework of this invention is not only applicable to adversarial patch detection, but can also be extended to various visual tasks such as face recognition, traffic sign recognition, and industrial quality inspection. By flexibly adjusting the sliding window scale, feature dimension, and network depth, this method can quickly adapt to different task requirements, combining theoretical innovation with engineering practicality.

[0024] In summary, this invention proposes a novel adversarial patch detection and localization method from the perspective of image region sequence structure modeling. This method overcomes the following problems in existing technologies by introducing multi-scale sliding windows, serialized feature modeling, global consistency analysis using self-attention mechanisms, and weakly supervised training: high false positive and false negative rates due to local detection and global semantic isolation; limited expressive power of manual heuristic scoring methods, making it difficult to adapt to diverse image scenarios; high dependence of model training on accurate patch masks, lacking universality and low-cost feasibility; fragmented detection results, making it difficult to accurately locate the overall spatial range of patches; and insufficient robustness under complex interference conditions, limiting application scenarios. The detection system proposed in this invention theoretically breaks through the limitations of traditional heuristic thresholding methods, achieving unified modeling of local and global image consistency; in application, it improves the robustness, flexibility, and adaptability of the detection system, possessing significant scientific research value and broad industrial application prospects. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0027] like Figure 1 As shown, this embodiment of the invention provides an adversarial patch detection and localization method based on local image patch sequence modeling, including the following steps: Step S1: Preprocess the image to be detected and divide it using a multi-scale sliding window to obtain several local image blocks with spatial order.

[0028] Specifically, the system receives the image to be detected, which can be an RGB color image, a grayscale image, or a multi-channel image. To ensure consistency and robustness in subsequent processing modules, the following preprocessing operations are performed on the input image: uniform size scaling, color space conversion (such as grayscale to HSV), standardization, edge alignment, and size cropping. Subsequently, a multi-scale sliding window operation is used to traverse the entire image. The window radius can be set to different scales such as 3, 5, and 7, and slides sequentially with a fixed step size. In this way, several spatially ordered local regions are extracted, forming a multi-scale image patch set. Multi-scale partitioning can simultaneously capture the fine-grained features of patches and large-scale anomaly features, avoiding missed detections at a single scale.

[0029] Step S2: Extract multidimensional image features for each local image patch and stitch them together in sliding order to form an image patch feature sequence.

[0030] Specifically, multidimensional image features are extracted, including: gray mean, gray standard deviation, edge sharpness (i.e., Laplacian operator variance), gray entropy, standard deviation of hue, saturation, and brightness of the color map, and color richness index, where the color richness index is obtained by combining the saturation mean and saturation standard deviation.

[0031] The aforementioned feature values ​​are standardized and concatenated into a feature vector of uniform dimension. Then, the feature vectors of all image patches are concatenated in a sliding order, from left to right and from top to bottom, to obtain an image patch feature sequence. This sequence reflects the statistical and semantic relationships between local regions of the image, forming a sequential structural modeling representation of the entire image. In natural images, the sequence should be smooth and continuous; if a patch attack exists, significant abrupt changes will occur in local regions.

[0032] Step S3: Input the image patch feature sequence into the self-attention Transformer model trained with sequence smoothness constraints, and output the anomaly score sequence for each image patch.

[0033] Specifically, the image patch feature sequence is input into a neural network model trained by a sequence smoothness constraint function. In this embodiment, the model uses a Transformer network based on a self-attention mechanism. The self-attention Transformer model includes a feature embedding layer, a self-attention encoding module, and a position-wise regression layer. The position-wise regression layer is used to output the anomaly score sequence for each image patch.

[0034] This model can learn the contextual relationships between image patches, capture abnormal patterns in the sequence, and output an anomaly score sequence for each image patch. During training, the self-attention Transformer model employs a weakly supervised training mechanism. It uses a scoring function based on the statistical difference between the entire image and local areas to generate pseudo-labels, and supervises the training of the image patch feature sequences to obtain the anomaly score sequence. Secondly, a sequence smoothness constraint is added to the loss function to ensure that scores in non-abnormal regions are stable and scores in abnormal regions are abrupt. The expression for the scoring function is as follows: ; In the formula, This represents the score of the i-th image patch; Indicates the weighting coefficient; This represents the average gray value of the i-th image patch; This represents the average gray level of the entire image; This represents the standard deviation of the grayscale values ​​of the i-th image patch; K represents the standard deviation of the gray levels of the entire image; K represents the number of statistical features. The index value of the k-th image feature of the i-th image block.

[0035] Taking the image features referenced in this embodiment as an example, their expression is: ; In the formula, This represents the Laplacian variability of an image patch, which is also an indicator of sharpness. Represents the grayscale entropy of an image patch; It represents color richness, and is a combination of the mean and standard deviation of saturation. , and It represents the pixel standard deviation of the three channels in the HSV color space.

[0036] In this embodiment, the weighting coefficient is set as follows: , , , , and .

[0037] To enhance the model's ability to learn the structure of image patch sequences, this embodiment of the invention incorporates a sequence smoothness constraint term into the loss function. Suppress unnecessary fluctuations in non-attack areas to ensure that anomalous mutations are highlighted: in, The length of the image patch sequence. For the first The mechanism calculates the predicted anomaly score for each image patch. This mechanism enables the model to automatically learn the smoothness of normal sequences and the abrupt changes in abnormal regions during training, thus allowing for more robust patch region identification during the inference phase.

[0038] Step S4: Based on the abnormal scoring sequence and the mutation rate of adjacent blocks, filter abnormal candidate regions according to the preset abnormal scoring threshold and mutation rate threshold.

[0039] Specifically, the abnormal score sequences are analyzed, and the mutation rate between adjacent scores is calculated. for: ; If a certain image block satisfy and If so, then that location is determined to be a suspected patch center. Here... and These are the thresholds for mutation magnitude and outliers, respectively. This represents the average score of the entire sequence.

[0040] Step S5: Perform consistency verification and region merging on the abnormal candidate regions obtained at different scales to form a set of abnormal candidate regions.

[0041] Specifically, to avoid false positives or false negatives caused by mismatch in a single-scale window, the detection results of candidate regions at different scales are fused. Through consistency verification and region merging, interference caused by scale inconsistency is eliminated, resulting in a set of candidate patch regions with enhanced robustness.

[0042] Step S6: Select the top K image patches with the highest confidence from the set of abnormal candidate regions, generate closed patch localization boxes based on spatial connectivity analysis, and obtain the overall location of the adversarial patch.

[0043] Specifically, from the candidate region set, the Top-K image patches with the highest confidence are selected and merged into a closed patch location box using spatial intersection or region connectivity analysis. The final output is the overall spatial location of the patch, which can be visualized by overlaying a rectangle or heatmap onto the original image. Simultaneously, the system can save the detection coordinates, anomaly score sequences, and scoring curves as JSON files or database records to support subsequent auditing and feedback.

[0044] The method of this invention supports the following deployment modes: In single-image offline processing mode, users can input images or an image catalog, and the system automatically completes the patch detection process, outputting the results for each image in the form of a visual graph and structured data. Suitable for model testing, image review, research validation, etc.

[0045] The real-time image stream inference mode allows the system to input image streams (such as video streams and camera data) and perform sliding window feature extraction and fast image sequence inference processing in real time, completing image block sequence analysis and anomaly localization within milliseconds. It is suitable for deployment in scenarios such as intelligent transportation, security cameras, and unmanned equipment.

[0046] The image recognition main model works collaboratively. This patch detection module can be embedded in an existing image recognition system as a pre-processing or post-processing module. It can shield patch areas before recognition or provide audit information after recognition, thereby improving the system's robustness and security protection capabilities.

[0047] Typical application scenarios are as follows: Traffic sign recognition system: Used to automatically identify road traffic signs that have been patched with attack patches, avoiding model misinterpretation of critical instructions such as speed limits and stops, and ensuring the safety of autonomous driving.

[0048] Facial recognition system: used to identify facial images of people wearing adversarial glasses, stickers, or other obstructions, improving system accuracy and preventing identity fraud.

[0049] Industrial vision inspection systems are used to identify abnormal patterns, defects, or stains on material surfaces and equipment structures, and can be applied to automated quality inspection lines. This implementation features flexible structure, simple deployment, and strong adaptability. It can effectively detect various forms of adversarial patches without relying on manual labels and a large number of training samples, and also possesses strong cross-scenario generalization capabilities. The entire system has a clear structure and efficient process, and can be widely used as an important technical module for image content security protection in practical engineering systems.

[0050] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0051] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for adversarial patch detection and localization based on local image patch sequence modeling, characterized in that: Includes the following steps: Step S1: Preprocess the image to be detected and divide it using a multi-scale sliding window to obtain several local image blocks with spatial order; Step S2: Extract multidimensional image features for each of the local image blocks and stitch them together in sliding order to form an image block feature sequence; Step S3: Input the image patch feature sequence into a self-attention Transformer model trained with sequence smoothness constraints, and output the anomaly score sequence for each image patch; Step S4: Based on the abnormal scoring sequence and the mutation rate of adjacent blocks, filter abnormal candidate regions according to the preset abnormal scoring threshold and mutation rate threshold; Step S5: Perform consistency verification and region merging on the abnormal candidate regions obtained at different scales to form a set of abnormal candidate regions; Step S6: Select the top K image patches with the highest confidence from the set of abnormal candidate regions, generate closed patch location boxes based on spatial connectivity analysis, and obtain the overall location of the adversarial patch.

2. The adversarial patch detection and localization method based on local image patch sequence modeling according to claim 1, characterized in that: The preprocessing in step S1 includes: uniform size scaling, color space conversion, normalization, edge alignment and size cropping; the multi-scale sliding window division includes: sliding windows of each scale across the entire image with a fixed step size to form local image blocks with spatial order.

3. The adversarial patch detection and localization method based on local image patch sequence modeling according to claim 2, characterized in that: The multidimensional image features mentioned in step S2 include: grayscale mean, grayscale standard deviation, edge sharpness of grayscale image, information entropy of grayscale image, color richness index, and pixel standard deviation of three channels in HSV color space, wherein the color richness index is obtained by combining the saturation mean and saturation standard deviation.

4. The adversarial patch detection and localization method based on local image patch sequence modeling according to claim 3, characterized in that: The image patch feature sequences are spliced ​​in a sliding order from left to right and from top to bottom. After feature standardization, the sequences are used as input to the self-attention Transformer model to maintain the statistical stationarity of the sequences.

5. The adversarial patch detection and localization method based on local image patch sequence modeling according to claim 1, characterized in that: The self-attention Transformer model includes a feature embedding layer, a self-attention encoding module, and a position-wise regression layer, which is used to output the anomaly score sequence for each image patch.

6. The adversarial patch detection and localization method based on local image patch sequence modeling according to claim 5, characterized in that: The self-attention Transformer model employs the following sequence smoothness constraint function. Apply constraints: ； In the formula, Indicates the length of the feature sequence of the image patch; Indicates the first Predicted anomaly scores for features of individual image patches.

7. The adversarial patch detection and localization method based on local image patch sequence modeling according to claim 6, characterized in that: The self-attention Transformer model employs a weakly supervised training mechanism, using a scoring function based on the statistical differences between the whole image and local areas to generate pseudo-labels, and performs supervised training on the image patch feature sequence to obtain anomaly scoring sequences.

8. The adversarial patch detection and localization method based on local image patch sequence modeling according to claim 7, characterized in that: The expression for the scoring function is: ； In the formula, This represents the score of the i-th image patch; Indicates the weighting coefficient; This represents the average gray value of the i-th image patch; This represents the average gray level of the entire image; This represents the standard deviation of the grayscale values ​​of the i-th image patch; K represents the standard deviation of the gray levels of the entire image; K represents the number of statistical features. The index value of the k-th image feature of the i-th image block.

9. The adversarial patch detection and localization method based on local image patch sequence modeling according to claim 1, characterized in that: Step S5 includes: fusing abnormal candidate regions collected by sliding windows at different scales, performing consistency verification by comparing the spatial overlap relationship of candidate regions at different scales, merging intersecting or adjacent candidate regions, and removing scattered, small-area or naturally highly complex regions to reduce false detections.

10. The adversarial patch detection and localization method based on local image patch sequence modeling according to claim 1, characterized in that: The patch positioning box described in step S6 is output in the form of a rectangle or a heatmap.

Images