Adversarial patch detection and localization method and system based on inter-frame similarity

Abstract

The application provides an inter-frame similarity-based adversarial patch detection and positioning method, which comprises the following steps: extracting a shallow feature map of a key video frame, selecting multiple candidate windows from the shallow feature map in a sliding window manner, and regarding any candidate window as an important window when the number of LISFs in the candidate window exceeds a screening threshold; using each important window as a mask to shield the video frame and performing image detection to obtain a mask detection result corresponding to each important window; performing a monopolist voting on all mask detection results to determine the position of the adversarial patch in the key video frame; and eliminating the interference of the adversarial patch on adjacent video frames of the key video frame according to inter-frame similarity. The application also provides an inter-frame similarity-based adversarial patch detection and positioning system and a data processing device for adversarial patch detection and positioning.

Description

Technical Field

[0001] This invention belongs to the field of neural network security, specifically relating to a method and system for detecting and locating adversarial patches in videos. Background Technology

[0002] Powered by deep neural network (DNN) technology, video object detection has achieved tremendous success and is beginning to drive existing industries such as autonomous driving, surveillance systems, drones, and robotics. For example, the market for video recognition-based autonomous driving is projected to jump to $77 billion by 2035 (representing 25% of the entire automotive market), attracting the attention of giants like Tesla, Audi, and Waymo.

[0003] While DNN-driven video object detection is gaining increasing importance, the inherent vulnerability of DNNs has become a significant problem for video object detection tasks, especially in security-related scenarios. DNN technology has proven to be highly susceptible to hostile attacks. Figures 1A-1D As shown, an attacker wearing a shirt with adversarial patches can fool a DNN-based human detector, making it undetectable. The attack remains successful even in more complex scenarios, such as walking, sitting, and movement. Such attacks pose significant potential dangers in surveillance and autonomous driving applications, making video detectors difficult to apply in life-threatening situations. Therefore, robust video target detection systems capable of resisting adversarial attacks are urgently needed and crucial.

[0004] After observing the vulnerabilities of neural network technology, researchers have proposed several defense systems for image classification tasks. However, their defense methods are difficult to apply to video object detection for the following reasons: 1) Their schemes target adversarial attacks, making them difficult to use in video object detection tasks. Adversarial attacks rely on the input image and have poor transferability. In real-world scenarios, attackers cannot obtain the attack scenario in advance, nor can they obtain adversarial samples for every frame of the video in real time. Therefore, adversarial attacks are not a practical attack model for video object detection tasks in the physical environment. 2) Existing methods rely on analyzing the depth features of each frame, failing to utilize the temporal and spatial similarity between adjacent frames in the video data. For example, previous studies required analyzing the complete activation path or activation map of each frame to detect abnormal inputs. However, the metrics used for these detections vary greatly across different inputs, making it impossible to accelerate them using consecutive video frames. 3) Previous schemes only detect inputs subjected to adversarial attacks, failing to guarantee the normal output of the system under attack; or the detection and recovery schemes are based on empirical observations and cannot resist adaptive attacks. Local Gradient Smoothing (LGS) assumes that pixel values ​​at the adversarial patch are not smooth, so it calculates local gradients and uses regions with large local gradients as adversarial patches. Digital Watermarking (DW) uses predicted labels to guide a neural network model to generate a saliency map, and then filters the saliency map to locate the adversarial patch. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes an adversarial patch detection and localization method based on inter-frame similarity, using optical flow in adjacent video frames. The method includes: acquiring a key video frame; extracting a shallow feature map of the key video frame; selecting multiple candidate windows from the shallow feature map using a sliding window approach; when the number of locally important shallow neurons in any candidate window exceeds a screening threshold, that candidate window is designated as an important window; using each important window as a mask to occlude the video frame and performing image detection to obtain the mask detection result corresponding to each important window; performing monopolistic voting on all mask detection results; when a mask detection result differs from other mask detection results, and all other mask detection results are the same, the important window corresponding to that mask detection result is the position of the adversarial patch in the key video frame; and eliminating the interference of the adversarial patch in adjacent video frames of the key video frame based on inter-frame similarity.

[0006] The adversarial patch detection and localization method of the present invention involves using the DIS optical flow algorithm to infer the position of the adversarial patch in the adjacent video frame based on the position of the adversarial patch in the key video frame, removing the adversarial patch from the adjacent video frame, and then performing image detection.

[0007] The adversarial patch detection and localization method of the present invention extracts the depth features of the key video frame after removing the adversarial patch, maps the depth features onto the adjacent video frame to obtain the depth features of the adjacent video frame, and performs image detection on the adjacent video frame using the depth features of the adjacent video frame.

[0008] The anti-patch detection and localization method of the present invention includes a sliding window whose size is the maximum size of the anti-patch predicted in the key video frame, and a sliding step of 1 pixel; when multiple important windows overlap, the important window in the center position among the multiple important windows is retained.

[0009] This invention also proposes an adversarial patch detection and localization system based on inter-frame similarity, comprising: a mask selection module for acquiring important windows as masks; wherein, a key video frame is acquired, a shallow feature map of the key video frame is extracted, and multiple candidate windows are selected from the shallow feature map using a sliding window method; when the number of locally important shallow neurons in any candidate window exceeds a screening threshold, the candidate window is designated as an important window; a result acquisition module for occluding the video frame with each important window as a mask and performing image detection to obtain the mask detection result corresponding to each important window; a detection and localization module for performing monopolist voting on all mask detection results; when a mask detection result differs from other mask detection results, and all other mask detection results are the same, the important window corresponding to the mask detection result is the position of the adversarial patch in the key video frame; and an interference cancellation module for eliminating the interference of the adversarial patch in adjacent video frames of the key video frame based on inter-frame similarity.

[0010] The adversarial patch detection and localization system of the present invention includes an interference cancellation module comprising: an adversarial patch mapping module, configured to infer the position of the adversarial patch in a video frame adjacent to the key video frame using the DIS optical flow algorithm based on the position of the adversarial patch in the key video frame, remove the adversarial patch from the adjacent video frame, and then perform image detection.

[0011] The anti-patch detection and localization system of the present invention includes an interference cancellation module comprising: a depth feature mapping module, used to extract the depth features of the key video frame after removing the anti-patch, map the depth features onto the adjacent video frame to obtain the depth features of the adjacent video frame, and perform image detection on the adjacent video frame using the depth features of the adjacent video frame.

[0012] The anti-patch detection and localization system of the present invention has the following features: the size of the sliding window is the maximum size of the anti-patch that is predicted to exist in the key video frame, and the sliding step of the sliding window is 1 pixel; when multiple important windows overlap, the important window in the center position among the multiple important windows is retained.

[0013] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that, when the computer-executable instructions are executed, the adversarial patch detection and localization method based on inter-frame similarity as described above is implemented.

[0014] The present invention also proposes a data processing apparatus, including a computer-readable storage medium as described above, wherein when the processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it performs adversarial patch detection and localization based on inter-frame similarity. Attached Figure Description

[0015] Figure 1A-1D This is a diagram illustrating the adv T-shirt attack.

[0016] Figure 2 This is a flowchart of the adversarial patch detection and localization method based on inter-frame similarity of the present invention.

[0017] Figure 3A , 3B This is a schematic diagram of the computing framework structure of the present invention.

[0018] Figure 4 This is a schematic diagram of the overall hardware architecture of the computing framework of the present invention.

[0019] Figure 5A , 5B This is a schematic diagram of the search and voting logic circuit structure of the present invention.

[0020] Figure 6 This is a schematic diagram of the computational flow of the normal image and occlusion image under intra-frame optimization according to the present invention.

[0021] Figure 7 This is a schematic diagram of the data flow for data reuse in this invention.

[0022] Figure 8 This is a schematic diagram of the data processing device of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0024] This invention targets adversarial patching attacks, which are more practical, feasible, and aggressive. These attacks generate specific patterns that, when placed on images or printed and then placed in the real world, retain a strong offensive effect, posing a significant threat to real-world deep learning systems. The inventors reduce redundant computation by analyzing the spatiotemporal similarity of adversarial patches between consecutive frames of a video. Furthermore, in analyzing the shallow features of normal and patched images, the inventors discovered that images with adversarial patching attacks have highly activated neurons concentrated in the patch region at the shallow layer. Therefore, the attack effect of the adversarial patch can be located and removed by detecting the concentrated distribution of high activation values. Further research revealed that this feature can effectively resist adaptive attacks. In addition, based on the characteristics of the proposed algorithm, the inventors proposed two defense frameworks and provided corresponding hardware support.

[0025] The purpose of this invention is to overcome the shortcomings of existing technologies in defending against adversarial patch attacks in video target detection systems. It proposes a method for locating and eliminating adversarial patches based on the detection of local shallow important neurons. Furthermore, it provides two defense frameworks for accuracy and performance, as well as hardware support for the algorithm, thus forming a complete and robust video target detection system. When applied to neural network-based video target detection systems, it can defend against real-world adversarial patch attacks, preventing attackers from evading target detection through adversarial patches.

[0026] The key technical points of this invention include:

[0027] 1. An adversarial patch localization method based on the detection of important shallow neurons: It can determine whether the input image of the system is attacked by adversarial patches, i.e., whether adversarial patches exist; if so, it can locate the position of the adversarial patch and eliminate its influence.

[0028] 2. An optical flow-based inter-frame optimization computation framework: Based on the similarity between video frames, optical flow is used to estimate the location of adversarial patches in adjacent frames or the depth features of adjacent frames after removing the influence of adversarial patches, replacing the more expensive parts of the adversarial patching algorithm. Two frameworks are proposed to address two user needs: one for high accuracy and one for high performance.

[0029] 3. Hardware acceleration design based on computational reuse and overall algorithm support: By utilizing the computational redundancy during multiple inferences of a single frame image in the localization algorithm, computational reuse is achieved, and hardware support is provided for other parts to realize a real-time robust video target detection system.

[0030] This invention designs a complete and robust video target detection system with hardware and software collaboration, including adversarial patch detection and localization algorithms, two computational frameworks utilizing inter-frame spatiotemporal redundancy, and a hardware system designed to provide effective support for the algorithms.

[0031] First, this invention proposes an adversarial patch detection and localization method based on locally important shallow neurons. The output of a video object detection system depends on the overall image information of each frame in the input video. Adversarial patch attacks require the use of very small regions to control the predicted output. Therefore, adversarial patches that control the output can be detected by occluding each small region in the image, and the influence of the attack can be removed by eliminating the adversarial patch region. However, occluding each sub-region of the image one by one is very time-consuming; for a 224×224 resolution image, this method requires 1446 seconds to complete the detection of one image. To reduce the detection overhead, this invention proposes a detection method based on locally important shallow neurons (LISF) to significantly reduce candidate detection regions and improve detection efficiency. Here, LISF, also known as locally important shallow feature, is obtained by summing the shallow feature map along the channel dimension to obtain a two-dimensional feature map. Each position in the two-dimensional feature map is considered a neuron, and LISF refers to the neuron corresponding to the feature map position with a maximum activation value (e.g., exceeding a threshold). This method is based on the following observations: 1) When locally important neurons are occluded, the results of normal images are more stable than those of attacked images. 2) When adversarial patches are printed and placed in a real scene, the positions of the adversarial patches in different frames of the corresponding video show obvious continuity. Therefore, the positions of adversarial patches in adjacent frames can be inferred using inter-frame similarity. In other words, the output of a normal image does not depend on certain highly concentrated important neurons in the shallow layers; while the local characteristics of the adversarial patch itself cause it to rely on localized important neurons to control the output; furthermore, inferring the positions of adversarial patches in adjacent frames using inter-frame similarity can reduce computational overhead.

[0032] Furthermore, while adversarial patch detection methods can effectively detect and locate adversarial patches in a single image, obtaining candidate windows requires multiple inferences at different occlusion locations, resulting in significant computational overhead. Simultaneously, processing video data frame-by-frame, although potentially effective in defense, fails to effectively utilize the similarity between video frames. Optical flow, referring to the relative displacement vector between each pixel in an image and its corresponding (most similar) pixels in neighboring images, can be used to describe spatial movement and mapping relationships between images, reducing computational resource consumption.

[0033] Based on the above description, this method proposes an adversarial patch detection and localization method based on LISF and optical flow inter-frame similarity calculation. Figure 2 This is a flowchart of the adversarial patch detection and localization method based on inter-frame similarity of the present invention. Figure 2 As shown, the anti-patch detection and location method of the present invention includes:

[0034] Step S1: Select key video frames and obtain shallow feature maps of the key video frames;

[0035] Step S2: Obtain multiple candidate windows from this shallow feature map. This invention uses a sliding window method, where the window size is the upper limit of the adversarial patch size, and the sliding window's step size is 1 pixel. Multiple candidate windows are obtained from the shallow feature map using the sliding window extraction method, and the number of LISFs within each candidate window is counted. When the number of important neurons in a candidate window exceeds the selection threshold, this candidate window is considered an important window where the adversarial patch may exist. When the number of LISFs in multiple candidate windows exceeds the selection threshold and these candidate windows overlap, only the candidate window located at the center of these overlapping windows is retained as an important window.

[0036] Step S3: After obtaining the above important windows, use them as masks to occlude the corresponding regions of the key video frame images and perform network inference to obtain the model output results after each important window occludes the key video frame images.

[0037] Step S4: Perform monopolistic voting on all outputs. Monopolistic voting here means first identifying the monopolist (a unique result), and then using this monopolist as the correct result. In this object detection framework, each result is a list, and a monopolist is an item in a result that is not present in any other result list (i.e., a "human" object "hidden" by the adversarial patch in the image). Assume there are k importance windows P1, P2, ..., P... k The output results correspond to L1, L2, ..., L k First, check if L is present. i If it differs from all other outputs, and the other outputs remain consistent, then P... i The output results are controlled, and the important window P is... i There is a countermeasure patch. Therefore, only by using P... i Only after occlusion can the network output the correct result. If there is only one candidate window, i.e., k=1, then this candidate window is designated as the important window P1. The network inferences to obtain the video frame image model output L1 with P1 as the mask. L1 is compared with L0, where L0 refers to the output of the original video frame, i.e., the image without mask occlusion. If L1≠L0, then P1 is the adversarial patch, and L1 is the correct output. When none of the above conditions exist among the candidate results, the image is considered a normal image, and the system performs majority voting on the candidate results to determine the final result.

[0038] Step S5: Utilize inter-frame similarity to infer the location of adversarial patches in video frames adjacent to the key video frame (non-key video frames) to remove adversarial patches from non-key video frames, or obtain the depth features of non-key video frames to avoid adversarial patch interference when performing image detection on non-key video frames. To reduce computational overhead, this invention only performs complete adversarial patch detection on key frames, while for non-key frames, it utilizes inter-frame spatiotemporal information, i.e., optical flow, to locate or eliminate the influence of adversarial patches. In this invention, optical flow is implemented using two methods: DIS (Dense Inverse Search) and SpyNet (Spatial Pyramid Network).

[0039] For accuracy-oriented detection schemes (AO, such as...) Figure 3A As shown, optical flow is used to map the detected adversarial patch locations in key video frames onto non-key frames. This image-level mapping can obtain accurate adversarial patch locations in non-key frames due to inter-frame similarity. Furthermore, by occluding the patches, rapid defense of non-key frames can be achieved.

[0040] For performance-oriented testing schemes (PO, such as...) Figure 3B As shown, the network model is first split into a high-computation feature extraction part (NN-prefix) at the front and a low-computation detection part (NN-suffix) at the back. Next, after detecting and removing adversarial patches for keyframes, clean depth features of the keyframes are obtained through the NN-prefix. Then, for non-keyframes, the optical flow between them and the keyframes is calculated, and the clean depth features of the keyframes are mapped onto the current non-keyframe space using this optical flow, resulting in clean depth features for the non-keyframes. Finally, these depth features are input into the low-computation NN-suffix part to obtain the output of the defended non-keyframes. Because the non-keyframe part eliminates the overhead of adversarial patch detection and avoids the high-computation feature extraction NN-prefix part, the PO execution framework can execute more efficiently.

[0041] This invention also proposes a hardware acceleration design. Although the Themis algorithm can be implemented using a purely software approach, this approach is inefficient for the following reasons: 1) In typical DNN accelerators, the candidate window search process required by Themis lacks computational parallelism with the normal image inference process, resulting in low efficiency in candidate window search. 2) During the voting phase, multiple candidate windows need to be occluded and inferred separately. In this process, due to the locality of adversarial patches, benign features unaffected by adversarial patches are repeatedly computed, introducing significant unnecessary performance overhead. Therefore, this invention designs a hardware architecture for Themis, implementing various logic circuits required by the algorithm and utilizing data and computation reuse to efficiently execute the defense algorithm. The top-level design of the hardware architecture is as follows: Figure 4 As shown, in addition to typical DNN accelerator components (including computational unit (PE) array, scalar function unit (SFU), global buffer, and control logic), Themis adds LISF search logic components for searching adversarial patch candidate regions, masked neuron buffer (MNB) for optimizing data and computation reuse, and voting logic circuitry for determining the final result.

[0042] Observing the significant amount of redundant computation in multiple intra-frame inferences, Themis proposed an intra-frame optimization scheme. For example... Figure 6 As shown, the occluded image and the normal image share most pixels, differing only around the occluded region. Therefore, by calculating the occluded region features of different layers separately and then stitching them back into the feature map of the normal image layer by layer, the final output of the occluded map can be obtained, thus eliminating the redundant calculation of the common region feature map. The detailed occluded region calculation process is as follows: Given an image (224×224 pixels) with an occluded region (50×50 pixels). The pixel values ​​of the occluded region are set to 0 to eliminate the influence of this region on the result. However, because the bias parameter of the convolutional neural network is not necessarily 0, and the convolution kernel is not necessarily 1 (when it is greater than 1, the pixels around the occluded region must also be considered), the occluded region in the intermediate layer feature map cannot be simply set to 0. Instead, the entire affected region from the previous layer is used as input for recalculation. For example, when calculating... Figure 6 When the C1 layer of the image is occluded, the complete region (occluded region + surrounding filled region) used for recalculation is 53×53, which is the occluded red region plus the filled purple region.

[0043] LISF search circuit logic. The LISF search circuit outputs the coordinates of the region where important neurons are clustered. This process includes the following three steps: 1) Obtain the binary image of important neurons in the first layer feature map. Because sorting is very time-consuming and incurs additional hardware overhead, an adaptive threshold method based on the maximum value is used here. Neurons with a value greater than a certain threshold coefficient multiplied by the maximum value in the feature map, β*featuremax, are considered important neurons, and the corresponding position in the binary image is set to 1; otherwise, it is 0. 2) Search for candidate windows. A sliding window is applied to the binary image, and important neurons within the window are counted. When the number of important neurons is greater than a threshold θ, the window is considered an important window. To reduce hardware overhead, such as... Figure 5A As shown, Themis uses incremental cumulative counting. To count the number of important neurons in window [i,j+1,i+s,j+1+s], the count of important neurons in column j+1 is simply subtracted from the count of important neurons in window [i,j,i+s,j+s], and the count of important neurons in column j+s+1 is added. 3) Delete important windows that overlap. If the overlap area of ​​two important windows is greater than 30%, they are considered as a single candidate window, and only one of them is retained.

[0044] MNB Caching. To effectively support the computational reuse of normal features and optimize data access for adversarial features, a Masked Neuron Caching (MNB) is proposed to cache and fill boundary regions for adversarial feature computation of candidate regions. This is practically feasible because the model network structure is known, and the filling region is determined after obtaining the coordinates of the candidate region. The data flow for data reuse is as follows: Figure 7 As shown, ① after analyzing the shallow feature map, candidate windows and their corresponding filled region coordinates are obtained. During normal image inference, the features of the filled boundary regions are stored in the MNB. ② Subsequently, during the inference of occluded images, all candidate regions are processed in batches in parallel. For each layer, the computational unit (PE) array reads the adversarial features of the candidate window regions from the global cache and the filled boundary region features from the MNB, combines the two as input, and calculates the adversarial features of the candidate window regions for the next layer. ③ After completing the calculation of this layer, the adversarial features are stored in the global cache. Because the filled boundary regions are very small compared to the complete activation map, each PE array's MNB is configured to be 8KB.

[0045] Voting logic component. Basic voting logic is as follows: Figure 5B As shown, Themis uses a comparator array to implement the output results L1, L2, ..., L of the occluded image. k A pairwise comparison. If there exists a result L... i If the image differs from all others, but the other results are the same, then the image is considered to have an anti-patch, and the repaired result is L. iOtherwise, it is considered a normal image, and a majority vote is taken.

[0046] The computational process is as follows: First, a new input image is received. The DNN accelerator executes Layer 1 and performs search logic on the output feature map of Layer 1 to obtain candidate windows and their corresponding fill boundary region coordinates. At this point, normal image inference is performed, and during inference, each layer stores all fill region features in the MNB cache according to the fill region coordinates of that layer. After inferring the original image, inference of occluded images begins. As described in the MNB cache section above, the PE array interacts with the global cache and MNB layer by layer, processing only windows and fill regions, and parallelizing the inference of all candidate occluded images to obtain the inference result for the occluded image. Then, voting logic is used to detect adversarial patches, and the final result is given. For non-keyframes, the optical flow with the previous keyframe is first calculated. For AO configurations, the adversarial patch position of the keyframe is then used to estimate the adversarial patch for the current frame, thus obtaining the occluded image, and normal inference is performed. For the PO configuration, after the adversarial patch detection is completed in the keyframe, the depth features of the NN-prefix part need to be calculated and saved in the global cache. When executing the current non-keyframe, the clean keyframe depth features are read from the global cache, and the optical flow is used to map them to obtain the current frame. Then, the NN-suffix is ​​executed to obtain the prediction result.

[0047] Figure 8 This is a schematic diagram of the data processing apparatus of the present invention. Figure 8 As shown, embodiments of the present invention also provide a computer-readable storage medium and a data processing apparatus. The computer-readable storage medium of the present invention stores computer-executable instructions. When these computer-executable instructions are executed by the processor of the data processing apparatus, the aforementioned adversarial patch detection and localization method based on inter-frame similarity is implemented. Those skilled in the art will understand that all or part of the steps in the above method can be implemented by a program instructing related hardware (e.g., processor, FPGA, ASIC, etc.), and the program can be stored in a readable storage medium, such as a read-only memory, a disk, or an optical disk. All or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiments can be implemented in hardware, for example, by an integrated circuit to implement its corresponding function, or it can be implemented as a software functional module, for example, by a processor executing a program / instruction stored in memory to implement its corresponding function. Embodiments of the present invention are not limited to any particular combination of hardware and software.

[0048] This invention uses the FLIC, LSP, MS COCO, and ImageNet datasets as benchmarks. For algorithm accuracy, detection success rate and mean average precision (mAP) are used as metrics; for hardware, running speed and energy consumption are used as metrics.

[0049] Regarding detection algorithms, the proposed algorithm offers better defense and stronger resistance to adaptive attacks. To demonstrate the difference before and after the attack, only images that could be correctly predicted by the detection model in the dataset were selected as test samples; therefore, the detection success rate of the datasets before the attack was 100%. After the attack, the detection success rates of the four datasets dropped to 13.0%, 9.6%, 29.5%, and 14.0%, respectively. After using the proposed detection algorithm for defense, the detection success rates increased to 93.5%, 93.9%, 90.6%, and 85.8%. Furthermore, under adaptive attacks (where the attacker understands the defender's defense methods and adjusts accordingly), the detection success rate still reached 76.5%. Compared to previous techniques, the effectiveness of local gradient smoothing and digital watermarking methods in defending against FLIC and MS COCO was tested. After using local gradient smoothing for defense, the detection success rates increased to 71.9% and 70.8%, but decreased to 13.4% and 27.0% under adaptive attacks. Digital watermarking methods are not suitable for this task because the target detection system outputs no labels to guide the generation of the required saliency map.

[0050] The optical flow-based intra-frame optimization computation framework supports multiple configurations, maintaining accuracy requirements under different defense configurations. Using the Adv T-shirt dataset as a benchmark, detection success rate and mAP are used as metrics, and two optical flow computation methods are selected: the DIS optical flow method based on traditional image processing methods and the SpyNet optical flow method based on deep networks. The accuracy-oriented framework AO includes the following configurations: AO-ND (no defense), AO-Full (defense for each frame of the video individually), AO-Dis (using the DIS optical flow method to accelerate defense), and AO-Spynet (using the SpyNet optical flow method to accelerate defense). This invention improves the detection success rate from only 4.4% under the attacked AO-ND configuration to 98.1%, 95.1%, and 95.1% under the AO-Full, AO-Dis, and AO-Spynet configurations, respectively; and improves the mAP metric from only 3% under the AO-ND configuration to 68%, 66%, and 66% under the AO-Full, AO-Dis, and AO-Spynet configurations, respectively. The performance-oriented framework PO includes the following configurations: PO-ND provides no defense, PO-Dis uses the DIS optical flow method to accelerate defense, and PO-Spynet uses the SpyNet method to accelerate defense. This invention improves the detection success rate from only 4.8% under the attacked PO-ND configuration to 93.8% and 94.1% under the PO-Dis and PO-Spynet configurations, respectively; and improves the mAP from only 5% under the PO-ND configuration to 53% and 53% under the PO-Dis and PO-Spynet configurations, respectively.

[0051] With hardware acceleration, this invention can run in real time under both frameworks. This section uses four networks—YOLOv2, YOLOv3, RetinaNet, and Faster R-CNN—as benchmarks. Detection algorithms incur computational overhead, and AO-Full achieves a 3.3x speedup compared to AO-ND. By applying hardware acceleration to these four networks, AO-Spynet incurs only 26.5%, 16.7%, 15.6%, and 21% speedup compared to AO-ND; PO-Spynet incurs only 28.2%, 20.3%, 14.5%, and 26.5% speedup compared to PO-ND. Furthermore, compared to implementations on general-purpose CPUs, the basic hardware of this invention achieves a speedup of 16.3x to 60.3x; compared to GPU Titan V, it achieves a speedup of 9.47x to 29.7x. Adding intra-frame computation reuse optimization can further achieve a speedup of 1.20x to 1.75x. Through experiments, this invention can run at 36 and 59 frames per second under the AO and PO frameworks respectively, which can meet the real-time requirements.

[0052] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. A method for detecting and locating adversarial patches based on inter-frame similarity, characterized in that, include: Obtain key video frames, extract shallow feature maps of key video frames, sum the shallow feature maps according to the channel dimension to obtain two-dimensional feature maps, take each position in the two-dimensional feature map as a neuron, and take neurons whose activation values ​​exceed the selected threshold as local shallow important neurons. Multiple candidate windows are selected from the shallow feature map using a sliding window method. When the number of important shallow neurons in any candidate window exceeds the screening threshold, the candidate window is selected as the important window. The video frame is occluded using each important window as a mask, and image detection is performed to obtain the mask detection result for each important window. A monopolist vote is performed on all mask detection results. If a mask detection result is different from other mask detection results and all other mask detection results are the same, then the important window corresponding to the mask detection result is the position of the adversarial patch in the key video frame. The interference of the adversarial patch with the adjacent video frames of the critical video frame is eliminated based on inter-frame similarity.

2. The anti-patch detection and location method as described in claim 1, characterized in that, Based on the position of the adversarial patch in the key video frame, the DIS optical flow algorithm is used to infer the position of the adversarial patch in the adjacent video frames. After removing the adversarial patch from the adjacent video frames, image detection is performed.

3. The anti-patch detection and location method as described in claim 1, characterized in that, Extract the depth features from the key video frame after removing the adversarial patch, map the depth features onto the adjacent video frame to obtain the depth features of the adjacent video frame, and perform image detection on the adjacent video frame using the depth features of the adjacent video frame.

4. The anti-patch detection and location method as described in claim 1, characterized in that, The sliding window is sized to the maximum size of the adversarial patch that is predicted to exist in the key video frame, and the sliding step of the window is 1 pixel; when multiple important windows overlap, the important window in the center position among the multiple important windows is retained.

5. A system for detecting and locating adversarial patches based on inter-frame similarity, characterized in that, include: The mask selection module is used to obtain important windows for use as masks. Specifically, it acquires key video frames, extracts shallow feature maps from these frames, sums the shallow feature maps along the channel dimension to obtain a two-dimensional feature map, uses each position in this two-dimensional feature map as a neuron, and identifies neurons with activation values ​​exceeding a selection threshold as locally important shallow neurons. Multiple candidate windows are selected from this shallow feature map using a sliding window method. When the number of locally important shallow neurons in any candidate window exceeds a selection threshold, that candidate window is designated as an important window. The result acquisition module is used to mask the video frame with each important window as a mask and perform image detection to obtain the mask detection result corresponding to each important window. The detection and localization module is used to perform monopolistic voting on all mask detection results. When a mask detection result is different from other mask detection results, and all other mask detection results are the same, the important window corresponding to the mask detection result is the position of the adversarial patch in the key video frame. The interference cancellation module is used to eliminate the interference of the adversarial patch in adjacent video frames of the critical video frame based on inter-frame similarity.

6. The anti-patch detection and location system as described in claim 5, characterized in that, The interference cancellation module includes: The adversarial patch mapping module is used to infer the position of the adversarial patch in the adjacent video frames based on the position of the adversarial patch in the key video frame using the DIS optical flow algorithm, and then remove the adversarial patch from the adjacent video frames before performing image detection.

7. The anti-patch detection and location system as described in claim 5, characterized in that, The interference cancellation module includes: The deep feature mapping module is used to extract the depth features of the key video frame after removing the adversarial patch, map the depth features onto the adjacent video frame to obtain the depth features of the adjacent video frame, and perform image detection on the adjacent video frame using the depth features of the adjacent video frame.

8. The anti-patch detection and location system as described in claim 5, characterized in that, The sliding window is sized to the maximum size of the adversarial patch that is predicted to exist in the key video frame, and the sliding step of the window is 1 pixel; when multiple important windows overlap, the important window in the center position among the multiple important windows is retained.

9. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, the adversarial patch detection and localization method based on inter-frame similarity as described in any one of claims 1 to 4 is implemented.

10. A data processing apparatus comprising a computer-readable storage medium as claimed in claim 9, wherein when a processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it performs adversarial patch detection and localization based on inter-frame similarity.

Images