An adversarial sample defense system and method based on image masking
By processing the input image using an image mask-based defense system, generating multiple image copies and performing weighted voting, the vulnerability of the target detector to adversarial examples is solved, achieving effective resistance to adversarial attacks and improving the robustness of the system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Chinese People's Liberation Army Cyberspace Force Information Engineering University
- Filing Date
- 2022-12-08
- Publication Date
- 2026-05-26
Smart Images
Figure CN116416453B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of adversarial example defense for target detection, and more particularly to an adversarial example defense system and method based on image masking. Background Technology
[0002] With the rapid development of deep learning technology, computing power, and the volume of sample data, deep learning-based target detection and image recognition technologies have penetrated into various fields of society, such as autonomous driving, biomedicine, mobile payment, and security monitoring. However, deep learning recognition models rely on statistical learning based on large-scale, high-quality training data, which can only provide limited generalization ability and robustness guarantees. They exhibit vulnerability when faced with carefully designed inputs containing subtle perturbations (Sun Hao, Chen Jin, Lei Lin, et al. A review of adversarial robustness techniques for deep convolutional neural network image recognition models [J]. Journal of Radar. 2021, 10(04).), and are easily deceived by imperceptible subtle perturbations, causing model inference errors and seriously affecting the realization of the original design functions of the system. Existing defense methods (Yu Zhengfei, Yan Qiao, Zhou Yun. A review of adversarial machine learning research for cyberspace defense [J]. Acta Automatica Sinica. 2022, 48(07):) often need to be combined with specific target detectors, requiring modifications to existing target detectors and lacking universality. Therefore, there is an urgent need for a more general adversarial example defense method that is compatible with existing target detectors and improves the robustness and security of target detection systems. Summary of the Invention
[0003] This invention addresses the vulnerability of existing deep learning-based target detectors to carefully designed adversarial examples by proposing an image mask-based adversarial example defense system and method. This method requires no modification to the target detector, only processing of the input image, and is compatible with existing target detectors. It enhances the target detection system's ability to resist adversarial example attacks, comprehensively improving the system's robustness and security.
[0004] To achieve the above objectives, the present invention adopts the following technical solution:
[0005] This invention proposes an adversarial sample defense system based on image masking, including a target detection and localization module, a key target extraction module, a key target image cropping module, a target mask processing module, an image mask processing module, an image classification module, and a weighted voting module;
[0006] The target detection and localization module is implemented by a target detection algorithm based on a convolutional neural network, which is used to infer the target information contained in the input image and to detect and localize multiple targets in the input image.
[0007] The key target extraction module filters key target information in the input image based on the set of key target categories of interest and the set target existence probability threshold.
[0008] The key target image cropping module extracts key target images based on key target information in the input image.
[0009] The target mask processing module generates an image mask containing the image processing area based on the key target information of the input image, performs adversarial sample perturbation elimination processing on the input image based on the image mask, and generates a copy of the input image.
[0010] The image mask processing module generates an image mask containing the image processing area based on the key target image, and performs adversarial sample perturbation elimination processing on the key target image based on the image mask, and generates a copy of the key target image;
[0011] The image classification module is implemented by an image classification algorithm based on a convolutional neural network, which is used to classify the input key target image and infer the category of the target contained in the input target image;
[0012] The weighted voting module is used to summarize the classification results of key targets, perform weighted statistics on the classification results, output the classification results, and issue alarms when there are potential adversarial sample targets.
[0013] Furthermore, the target detection and localization module is specifically used for:
[0014] Implement input image The reasoning of the target information contained therein predicts the targets contained in the input image and generates a target information tensor O; where h, w, and c represent the height, width, and number of RGB channels of the input image, respectively.
[0015] The target information tensor O is shown in equation (1), containing n targets. Each row of the tensor represents the relevant prediction information of one target:
[0016]
[0017] Where x and y represent the horizontal and vertical coordinates of the target center point on the input image, respectively; h and w represent the height and width of the target, respectively; t∈T represents the target category; and p∈(0,1) represents the probability that the target belongs to category t.
[0018] Furthermore, the key target extraction module is specifically used for:
[0019] Based on the target information tensor O of the input image, filter out the target information t. i ∈T key And p i>p th The goal is to generate a tensor O of key target information. key There are m key objectives in total, as shown in equation (2):
[0020]
[0021] Where T key p represents the set of key target categories of interest. th This represents the probability threshold for adopting the target.
[0022] Furthermore, the key target image cropping module is specifically used for:
[0023] Based on the key target information tensor O of the input image key Traversing tensor O key The image generates a key target image by cropping out the area containing the key target based on the x-coordinate and y-coordinate of the target center point, as well as the height h and width w.
[0024] Furthermore, the target mask processing module is specifically used for:
[0025] Determine the image mask implementation area: based on the key target information tensor O of the input image. key Traversing tensor O key The target information is processed in each row, and the mask implementation area of each key target in the input image is determined based on the x-coordinate and y-coordinate of the target center point, as well as the height h and width w.
[0026] Image mask generation: Divide each mask implementation region of the input image into a k×s grid, where the height and width of each square are respectively... Randomly select z squares from k×s squares as the final image processing implementation area, and generate an image mask containing the image processing area;
[0027] Image processing: Based on the image mask, adversarial sample perturbation elimination processing is performed on the image grid of the image processing implementation area. The perturbation elimination processing can be performed by setting the pixel value of the processing area to 0, taking the average value, or setting it to a set value, and generating the corresponding input image copy.
[0028] Repeat the above process to generate two different image masks, and finally generate two different copies of the input image; the original input image and the two different copies of the input image constitute the input image copy set of the input image.
[0029] Furthermore, the image mask processing module is specifically used for:
[0030] Determine the image mask implementation area: The entire image range of each key target image generated by the key target image cropping module is determined as the input image mask implementation area;
[0031] Image mask generation: The mask application region of the key target image is divided into a k×s grid, where the height and width of each square are respectively... Randomly select z squares from k×s squares as the final image processing implementation area, and generate an image mask containing the image processing area;
[0032] Image processing: Based on the image mask, adversarial sample perturbation elimination processing is performed on the image grid of the image processing implementation area. The perturbation elimination processing can be carried out by setting the pixel value of the processing area to 0, taking the average value, or setting it to a set value, and generating corresponding key target image copies.
[0033] Repeat the above process to generate two different image masks for each key target image, and finally generate two different key target image copies; the original key target image and the two different key target image copies constitute the key target image copy set for that key target.
[0034] Furthermore, the image classification module is specifically used for:
[0035] Achieve input of key target images The classification is used to deduce the category of the target contained in the input target image; where h, w, and c represent the height, width, and number of RGB channels of the key target image, respectively.
[0036] Furthermore, the weighted voting module is specifically used for:
[0037] Summary of Key Target Classification Results: This summary includes the key target classification information generated by the target detection and localization module for the input image replica set, as well as the target classification information generated by the image classification module for the key target image replica set.
[0038] Statistical analysis of key target classification results: The weights of the key target classification results generated by the target detection and localization module for the input image copy set are set as α1, α2, and α3, respectively; the weights of the key target classification information results generated by the image classification module for the key target image copy set are set as β1, β2, and β3, respectively; for each key target in the input image, the classification results generated by the target detection and localization module and the image classification module are weighted, statistically analyzed, and sorted.
[0039] Key target classification result output: For each key target in the input image, select the classification result with the largest weight in the classification results as the final classification result of the key target and output it; if there are multiple classification results for a key target in the input image, record the relevant input image and classification result information, and output the alarm information of potential adversarial example target.
[0040] A second aspect of this invention proposes an adversarial example defense method based on image masks, comprising:
[0041] Step 1: Input the input image containing potential adversarial example targets into the target detection and localization module to detect and localize the targets, and generate the target information tensor of the input image;
[0042] Step 2: The key target extraction module extracts the key targets in the input image based on the target information tensor of the input image and generates a key target information tensor.
[0043] Step 3: The key target image cropping module crops the input image based on the key target information tensor of the input image to generate a key target image;
[0044] Step 4: The target mask processing module applies an image mask to the region where the key target is located in the input image based on the key target information tensor of the input image, generating multiple copies of the input image to form an input image copy set;
[0045] Step 5: The image masking module applies an image mask to the extracted key target image to generate multiple key target image copies, forming a key target image copy set;
[0046] Step 6: Input the generated set of input image copies into the target detection and localization module to detect targets again, generating key target classification information and localization information;
[0047] Step 7: Input the generated set of key target image copies into the image classification module to classify the targets and generate key target classification information;
[0048] Step 8: Based on the key target classification information generated in Steps 6 and 7, the weighted voting module generates key target classification results through weighted voting and issues warnings for potential adversarial sample targets.
[0049] Further, step 1 includes:
[0050] The target detection and localization module processes the input image. The reasoning based on the target information contained in the input image predicts the targets contained in the input image and generates a target information tensor O;
[0051] The input image can be represented by a tensor, that is, the input image Where h, w, and c represent the height, width, and number of RGB channels of the input image, respectively;
[0052] The target information tensor O is shown in equation (1), containing n targets. Each row of the tensor represents the relevant prediction information of one target:
[0053]
[0054] Where x and y represent the horizontal and vertical coordinates of the target center point on the input image, respectively; h and w represent the height and width of the target, respectively; t∈T represents the target category; and p∈(0,1) represents the probability that the target belongs to category t.
[0055] Further, step 2 includes:
[0056] The set of key target categories of interest is represented by T. key The probability threshold for adopting a target is represented by p. th This means that only when p i >p th The target is confirmed to exist in the image;
[0057] The key target extraction module filters out target information t based on the target information tensor O of the input image. i ∈T key And p i >p th The goal is to generate a tensor O of key target information. key There are m key objectives in total, as shown in equation (2):
[0058]
[0059] Further, step 3 includes:
[0060] The key target image cropping module uses the key target information tensor O of the input image. key Traversing tensor O key The system extracts target information from each row and, based on the horizontal and vertical coordinates (x and y) of the target center point, as well as the height and width (h and w), crops out the area containing the key target to generate a key target image.
[0061] Further, step 4 includes:
[0062] Step 4.1: Determine the image mask implementation area; the target mask processing module uses the key target information tensor O of the input image. key Traversing tensor O key The target information is processed in each row, and the mask implementation area of each key target in the input image is determined based on the horizontal and vertical coordinates x and y of the target center point, as well as the height and width h and w.
[0063] Step 4.2: Image Mask Generation; Divide each mask implementation region of the input image into a k×s grid, where the height and width of each square are respectively... Randomly select z squares from k×s squares as the final image processing implementation area, and generate an image mask containing the image processing area;
[0064] Step 4.3: Image Processing; Based on the image mask, adversarial sample perturbation removal processing is performed on the image grid of the image processing area. The perturbation removal processing can be performed by setting the pixel values in the processing area to 0, taking the average value, or setting them to a set value; After completing the image adversarial sample perturbation removal processing, the corresponding input image copy can be generated.
[0065] Step 4.4: Repeat steps 4.2 and 4.3 to generate two different image masks and finally generate two different input image copies; the original input image and the two different input image copies constitute the input image copy set of the input image.
[0066] Further, step 5 includes:
[0067] Step 5.1: Determine the image mask implementation area; The entire image range of each key target image generated in Step 3 is determined as the input image mask implementation area;
[0068] Step 5.2: Image Mask Generation; Divide the mask application region of the key target image into a k×s grid, with the height and width of each square being respectively... Randomly select z squares from k×s squares as the final image processing implementation area, and generate an image mask containing the image processing area;
[0069] Step 5.3: Image processing; Based on the image mask, adversarial sample perturbation removal processing is performed on the image grid of the image processing area. The perturbation removal processing can be performed by setting the pixel value in the processing area to 0, taking the average value, or setting it to a set value; After completing the image adversarial sample perturbation removal processing, the corresponding key target image copy can be generated.
[0070] Step 5.4: Repeat steps 5.2 and 5.3 to generate two different image masks for each key target image, and finally generate two different key target image copies; the original key target image and the two different key target image copies constitute the key target image copy set of the key target.
[0071] Further, step 7 includes:
[0072] The image classification module classifies the input key target image and infers the category of the target contained in the input target image;
[0073] The key target images in the key target image replica set can be represented by tensors, i.e., target images. Where h, w, and c represent the height, width, and number of RGB channels of the key target image, respectively.
[0074] Further, step 8 includes:
[0075] Step 8.1: Summarize the classification results of key targets; Summarize the classification information of key targets generated in Step 6 for the input image replica set, and the target classification information generated in Step 7 for the key target image replica set;
[0076] Step 8.2: Statistics of Key Target Classification Results; Set the weights α1, α2, and α3 for the key target classification results generated by the target detection and localization module for the input image copy set (original input image and two different input image copies); set the weights β1, β2, and β3 for the key target classification information results generated by the image classification module for the key target image copy set (original key target image and two different key target image copies); for each key target in the input image, perform weighted statistics and sorting of the classification results generated by the target detection and localization module and the image classification module.
[0077] Step 8.3: Output the classification results of key targets; For each key target in the input image, select the classification result with the largest weight in the classification results as the final classification result of the key target and output it; If there are multiple classification results for a certain key target in the input image, record the relevant input image and classification result information, and output the alarm information of potential adversarial sample targets.
[0078] Compared with the prior art, the present invention has the following beneficial effects:
[0079] Deep learning-based object detectors are vulnerable to carefully designed inputs containing subtle perturbations, easily being fooled by imperceptible subtle disturbances, leading to model inference errors. Existing adversarial example defense methods often require integration with specific object detectors and modifications, lacking general applicability. This invention, however, requires no modification to the object detector, only processing of the input image. It is widely compatible with existing object detectors, possesses good versatility, and effectively enhances the object detection system's resistance to adversarial example attacks. Attached Figure Description
[0080] Figure 1This is a schematic diagram of the architecture of an adversarial sample defense system based on image mask according to an embodiment of the present invention;
[0081] Figure 2 This is an image illustration of an adversarial sample defense method based on image masking according to an embodiment of the present invention;
[0082] Figure 3 This is a basic flowchart of an adversarial sample defense method based on image masking, according to an embodiment of the present invention. Detailed Implementation
[0083] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:
[0084] like Figure 1 As shown, an adversarial sample defense system based on image masking includes a target detection and localization module, a key target extraction module, a key target image cropping module, a target mask processing module, an image mask processing module, an image classification module, and a weighted voting module.
[0085] The target detection and localization module is mainly implemented by a target detection algorithm based on a convolutional neural network, which can infer the target information contained in the input image and realize the detection and localization of multiple targets in the input image;
[0086] The key target extraction module filters key target information in the input image based on the set of key target categories of interest and the set target existence probability threshold.
[0087] The key target image cropping module extracts key target images based on key target information in the input image.
[0088] The target mask processing module generates an image mask containing the image processing area based on the key target information of the input image, and performs adversarial sample perturbation elimination processing on the input image based on the image mask, and generates a copy of the input image;
[0089] The image mask processing module generates an image mask containing the image processing area based on the key target image, and performs adversarial sample perturbation elimination processing on the key target image based on the image mask, and generates a copy of the key target image;
[0090] The image classification module is mainly implemented by an image classification algorithm based on a convolutional neural network, which can classify the input key target image and infer the category of the target contained in the input target image;
[0091] The weighted voting module is used to summarize the classification results of key targets, perform weighted statistics on the classification results, output the classification results, and issue alarms when there are potential adversarial sample targets.
[0092] Furthermore, the target detection and localization module is specifically used for:
[0093] Implement input image The reasoning of the target information contained therein predicts the targets contained in the input image and generates a target information tensor O; where h, w, and c represent the height, width, and number of RGB channels of the input image, respectively.
[0094] The target information tensor O is shown in equation (1), containing n targets. Each row of the tensor represents the relevant prediction information of one target:
[0095]
[0096] Where x and y represent the horizontal and vertical coordinates of the target center point on the input image, respectively; h and w represent the height and width of the target, respectively; t∈T represents the target category; and p∈(0,1) represents the probability that the target belongs to category t.
[0097] Furthermore, the key target extraction module is specifically used for:
[0098] Based on the target information tensor O of the input image, filter out the target information t. i ∈T key And p i >p th The goal is to generate a tensor O of key target information. key There are m key objectives in total, as shown in equation (2):
[0099]
[0100] Where T key p represents the set of key target categories of interest. th This represents the probability threshold for adopting the target.
[0101] Furthermore, the key target image cropping module is specifically used for:
[0102] Based on the key target information tensor O of the input image key Traversing tensor O key The image generates a key target image by cropping out the area containing the key target based on the x-coordinate and y-coordinate of the target center point, as well as the height h and width w.
[0103] Furthermore, the target mask processing module is specifically used for:
[0104] Determine the image mask implementation area: based on the key target information tensor O of the input image. key Traversing tensor O keyThe target information is processed in each row, and the mask implementation area of each key target in the input image is determined based on the x-coordinate and y-coordinate of the target center point, as well as the height h and width w.
[0105] Image mask generation: Divide each mask implementation region of the input image into a k×s grid, where the height and width of each square are respectively... Randomly select z squares from k×s squares as the final image processing implementation area, and generate an image mask containing the image processing area;
[0106] Image processing: Based on the image mask, adversarial sample perturbation elimination processing is performed on the image grid of the image processing implementation area. The perturbation elimination processing can be performed by setting the pixel value of the processing area to 0, taking the average value, or setting it to a set value, and generating the corresponding input image copy.
[0107] Repeat the above process to generate two different image masks, and finally generate two different copies of the input image; the original input image and the two different copies of the input image constitute the input image copy set of the input image.
[0108] Furthermore, the image mask processing module is specifically used for:
[0109] Determine the image mask implementation area: The entire image range of each key target image generated by the key target image cropping module is determined as the input image mask implementation area;
[0110] Image mask generation: The mask application region of the key target image is divided into a k×s grid, where the height and width of each square are respectively... Randomly select z squares from k×s squares as the final image processing implementation area, and generate an image mask containing the image processing area;
[0111] Image processing: Based on the image mask, adversarial sample perturbation elimination processing is performed on the image grid of the image processing implementation area. The perturbation elimination processing can be carried out by setting the pixel value of the processing area to 0, taking the average value, or setting it to a set value, and generating corresponding key target image copies.
[0112] Repeat the above process to generate two different image masks for each key target image, and finally generate two different key target image copies; the original key target image and the two different key target image copies constitute the key target image copy set for that key target.
[0113] Furthermore, the image classification module is specifically used for:
[0114] Achieve input of key target images The classification is used to deduce the category of the target contained in the input target image; where h, w, and c represent the height, width, and number of RGB channels of the key target image, respectively.
[0115] Furthermore, the weighted voting module is specifically used for:
[0116] Summary of Key Target Classification Results: This summary includes the key target classification information generated by the target detection and localization module for the input image replica set, as well as the target classification information generated by the image classification module for the key target image replica set.
[0117] Statistical analysis of key target classification results: The weights of the key target classification results generated by the target detection and localization module for the input image copy set are set as α1, α2, and α3, respectively; the weights of the key target classification information results generated by the image classification module for the key target image copy set are set as β1, β2, and β3, respectively; for each key target in the input image, the classification results generated by the target detection and localization module and the image classification module are weighted, statistically analyzed, and sorted.
[0118] Key target classification result output: For each key target in the input image, select the classification result with the largest weight in the classification results as the final classification result of the key target and output it; if there are multiple classification results for a key target in the input image, record the relevant input image and classification result information, and output the alarm information of potential adversarial example target.
[0119] Based on the above embodiments, the present invention also proposes an adversarial example defense method based on image masks, as illustrated in the schematic diagram of the relevant images. Figure 2 As shown, the main flow of this method is as follows: Figure 3 As shown, it includes:
[0120] Step 1: Input the input image containing potential adversarial example targets into the target detection and localization module to detect and localize the targets, and generate the target information tensor of the input image;
[0121] Step 2: The key target extraction module extracts the key targets in the input image based on the target information tensor of the input image and generates a key target information tensor.
[0122] Step 3: The key target image cropping module crops the input image based on the key target information tensor of the input image to generate a key target image;
[0123] Step 4: The target mask processing module applies an image mask to the region where the key target is located in the input image based on the key target information tensor of the input image, generating multiple copies of the input image to form an input image copy set;
[0124] Step 5: The image masking module applies an image mask to the extracted key target image to generate multiple key target image copies, forming a key target image copy set;
[0125] Step 6: Input the generated set of input image copies into the target detection and localization module to detect targets again, generating key target classification information and localization information;
[0126] Step 7: Input the generated set of key target image copies into the image classification module to classify the targets and generate key target classification information;
[0127] Step 8: Based on the key target classification information generated in Steps 6 and 7, the weighted voting module generates key target classification results through weighted voting and issues warnings for potential adversarial sample targets.
[0128] Further, step 1 includes:
[0129] The input image can be represented by a tensor, that is, the input image Where h, w, and c represent the height, width, and number of RGB channels of the input image, respectively;
[0130] The target information tensor O is shown in equation (1), containing n targets. Each row of the tensor represents the relevant prediction information of one target:
[0131]
[0132] Where x and y represent the horizontal and vertical coordinates of the target center point on the input image, respectively; h and w represent the height and width of the target, respectively; t∈T represents the target category; and p∈(0,1) represents the probability that the target belongs to category t.
[0133] The object detection and localization module is mainly implemented by an object detection algorithm based on convolutional neural networks, which can detect and locate input images. The reasoning based on the target information contained in the input image predicts the targets contained in the input image and generates a target information tensor O.
[0134] Further, step 2 includes:
[0135] The set of key target categories of interest is represented by T. key The probability threshold for adopting a target is represented by p. th This means that only when p i >p th The target is confirmed to exist in the image;
[0136] The key target extraction module filters out target information t based on the target information tensor O of the input image. i ∈T key And pi >p th The goal is to generate a tensor O of key target information. key There are m key objectives in total, as shown in equation (2):
[0137]
[0138] Further, step 3 includes:
[0139] The key target image cropping module uses the key target information tensor O of the input image. key Traversing tensor O key The system extracts target information from each row and, based on the horizontal and vertical coordinates (x and y) of the target center point, as well as the height and width (h and w), crops out the area containing the key target to generate a key target image.
[0140] Further, step 4 includes:
[0141] Step 4.1: Determine the image mask implementation area. The target mask processing module uses the key target information tensor O of the input image... key Traversing tensor O key The target information is processed in each row, and the mask implementation area of each key target in the input image is determined based on the horizontal and vertical coordinates (x, y) of the target center point, as well as the height and width (h, w).
[0142] Step 4.2: Image Mask Generation. Divide each mask implementation region of the input image into a k×s grid, where the height and width of each square are respectively... Randomly select z squares from k×s squares as the final image processing area, and generate an image mask containing the image processing area.
[0143] Step 4.3: Image Processing. Based on the image mask, adversarial sample perturbation removal processing is performed on the image grid of the image processing area. Perturbation removal processing can be performed by setting the pixel values in the processing area to 0, taking the average value, or setting them to a set value. After completing the image adversarial sample perturbation removal processing, the corresponding input image copy can be generated.
[0144] Step 4.4: Repeat steps 4.2 and 4.3 to generate two different image masks and finally generate two different input image copies; the original input image and the two different input image copies constitute the input image copy set of the input image.
[0145] Further, step 5 includes:
[0146] Step 5.1: Determine the image mask implementation area. The entire image range of each key target image generated in Step 3 is determined as the input image mask implementation area.
[0147] Step 5.2: Image Mask Generation. The mask application region of the key target image is divided into a k×s grid, where the height and width of each square are respectively... Randomly select z squares from k×s squares as the final image processing area, and generate an image mask containing the image processing area.
[0148] Step 5.3: Image Processing. Based on the image mask, adversarial sample perturbation removal processing is performed on the image grid of the image processing area. Perturbation removal processing can be performed by setting the pixel values in the processing area to 0, taking the average value, or setting them to a set value. After completing the image adversarial sample perturbation removal processing, the corresponding key target image copy can be generated.
[0149] Step 5.4: Repeat steps 5.2 and 5.3 to generate two different image masks for each key target image, and finally generate two different key target image copies; the original key target image and the two different key target image copies constitute the key target image copy set of the key target.
[0150] Further, step 7 includes:
[0151] The key target images in the key target image replica set can be represented by tensors, i.e., target images. Where h, w, and c represent the height, width, and number of RGB channels of the key target image, respectively;
[0152] The image classification module is mainly implemented by an image classification algorithm based on convolutional neural networks. It can classify the input key target image and infer the category of the target contained in the input target image.
[0153] Further, step 8 includes:
[0154] Step 8.1: Summarize the classification results of key targets. Summarize the classification information of key targets generated in Step 6 for the input image replica set, and the target classification information generated in Step 7 for the key target image replica set.
[0155] Step 8.2: Statistics of Key Target Classification Results. The weights α1, α2, and α3 are set for the key target classification results generated by the target detection and localization module for the input image copy set (the original input image and two different input image copies); the weights β1, β2, and β3 are set for the key target classification information results generated by the image classification module for the key target image copy set (the original key target image and two different key target image copies); for each key target in the input image, the classification results generated by the target detection and localization module and the image classification module are weighted, statistically analyzed, and sorted.
[0156] Step 8.3: Output of classification results for key targets. For each key target in the input image, select the classification result with the largest weight from the classification results as the final classification result for that key target and output it; if there are multiple classification results for a certain key target in the input image, record the relevant input image and classification result information, and output the alarm information of potential adversarial example targets.
[0157] In summary, this invention addresses the vulnerability of existing deep learning-based target detectors to carefully designed adversarial examples by proposing an image mask-based adversarial example defense system and method. This method requires no modification to the target detector, only processing of the input image. It is widely compatible with existing target detectors, possesses good versatility, effectively enhances the target detection system's ability to resist adversarial example attacks, and comprehensively improves the system's robustness and security.
[0158] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. An adversarial sample defense system based on image masking, characterized in that, It includes a target detection and localization module, a key target extraction module, a key target image cropping module, a target mask processing module, an image mask processing module, an image classification module, and a weighted voting module; The target detection and localization module is implemented by a target detection algorithm based on a convolutional neural network, which is used to infer the target information contained in the input image and to detect and localize multiple targets in the input image. The key target extraction module filters key target information in the input image based on the set of key target categories of interest and the set target existence probability threshold. The key target image cropping module extracts key target images based on key target information in the input image. The target mask processing module generates an image mask containing the image processing area based on the key target information of the input image, performs adversarial sample perturbation elimination processing on the input image based on the image mask, and generates a copy of the input image. The target detection and localization module is further used to perform target detection and localization on the input image copy, and generate key target classification information; The image mask processing module generates an image mask containing the image processing area based on the key target image, and performs adversarial sample perturbation elimination processing on the key target image based on the image mask, and generates a copy of the key target image; The image classification module is implemented by an image classification algorithm based on a convolutional neural network, which is used to classify the key target image copies and generate key target classification information; The weighted voting module is used to summarize the classification information generated by the target detection and localization module and the classification information generated by the image classification module, and to complete the weighted statistics of the classification results of key targets, output the classification results, and issue alarms when there are potential adversarial sample targets.
2. The adversarial sample defense system based on image mask according to claim 1, characterized in that, The target detection and localization module is specifically used for: Implement input image The reasoning based on the target information contained therein predicts the targets contained in the input image and generates a target information tensor O; where These represent the height, width, and number of RGB channels of the input image, respectively. The target information tensor O is shown in equation (1), containing n targets. Each row of the tensor represents the relevant prediction information of one target: （1） Where x and y represent the horizontal and vertical coordinates of the target's center point on the input image, respectively, and h and w represent the target's height and width, respectively. Indicates the category of the target. This represents the probability that the target belongs to category t.
3. The adversarial sample defense system based on image mask according to claim 2, characterized in that, The key target extraction module is specifically used for: Based on the target information tensor O of the input image, filter out the target information. The goal is to generate a tensor O of key target information. key There are m key objectives in total, as shown in equation (2): （2） in This represents the set of key target categories of interest. This represents the probability threshold for adopting the target.
4. The adversarial sample defense system based on image mask according to claim 3, characterized in that, The key target image cropping module is specifically used for: Based on the key target information tensor O of the input image key Traversing tensor O key The image generates a key target image by cropping out the area containing the key target based on the x-coordinate and y-coordinate of the target center point, as well as the height h and width w.
5. The adversarial sample defense system based on image mask according to claim 1, characterized in that, The target mask processing module is specifically used for: Determine the image mask implementation area: based on the key target information tensor O of the input image. key Traversing tensor O key The target information is processed in each row, and the mask implementation area of each key target in the input image is determined based on the x-coordinate and y-coordinate of the target center point, as well as the height h and width w. Image mask generation: The input image is segmented into regions for each mask application. The grid has a height and width of 100 squares. ;from Randomly select z squares from the grid as the final image processing implementation area, and generate an image mask containing the image processing area; Image processing: Based on image masking, adversarial sample perturbation elimination is performed on the image grid of the image processing area to generate a corresponding copy of the input image; Repeat the above process to generate two different image masks, and finally generate two different copies of the input image; the original input image and the two different copies of the input image constitute the input image copy set of the input image.
6. The adversarial sample defense system based on image mask according to claim 1, characterized in that, The image mask processing module is specifically used for: Determine the image mask implementation area: The entire image range of each key target image generated by the key target image cropping module is determined as the input image mask implementation area; Image mask generation: The mask application region of the key target image is segmented into... The grid has a height and width of 100 squares. ;from Randomly select z squares from the grid as the final image processing implementation area, and generate an image mask containing the image processing area; Image processing: Based on image masking, adversarial sample perturbation elimination processing is performed on the image grid of the image processing implementation area to generate corresponding key target image copies; Repeat the above process to generate two different image masks for each key target image, and finally generate two different copies of the key target image; The original key target image and two different key target image copies constitute the key target image copy set for that key target.
7. The adversarial sample defense system based on image mask according to claim 1, characterized in that, The image classification module is specifically used for: Achieve input of key target image The classification is used to infer the category of the target contained in the input target image; where These represent the height, width, and number of RGB channels of the key target image, respectively.
8. The adversarial sample defense system based on image mask according to claim 1, characterized in that, The weighted voting module is specifically used for: Summary of Key Target Classification Results: This summary includes the key target classification information generated by the target detection and localization module for the input image replica set, as well as the target classification information generated by the image classification module for the key target image replica set. Key target classification result statistics: The weights of the key target classification results generated by the target detection and localization module for the input image replica set are set as follows: ; The weights of the key target classification information results generated by the image classification module for the key target image replica set are set as follows: For each key target in the input image, the classification results generated by the target detection and localization module and the image classification module are weighted, statistically analyzed, and sorted. Key target classification results output: For each key target in the input image, the classification result with the largest weight in the classification results is selected as the final classification result for that key target and output. If a key target in the input image has multiple classification results, the relevant input image and classification result information are recorded, and an alarm message indicating the presence of a potential adversarial example target is output.
9. A method for image mask-based adversarial example defense based on an image mask-based adversarial example defense system according to any one of claims 1-8, characterized in that, include: Step 1: Input the input image containing potential adversarial example targets into the target detection and localization module to detect and localize the targets, and generate the target information tensor of the input image; Step 2: The key target extraction module extracts the key targets in the input image based on the target information tensor of the input image and generates a key target information tensor. Step 3: The key target image cropping module crops the input image based on the key target information tensor of the input image to generate a key target image; Step 4: The target mask processing module applies an image mask to the region where the key target is located in the input image based on the key target information tensor of the input image, generating multiple copies of the input image to form an input image copy set; Step 5: The image masking module applies an image mask to the extracted key target image to generate multiple key target image copies, forming a key target image copy set; Step 6: Input the generated set of input image copies into the target detection and localization module to detect targets again, generating key target classification information and localization information; Step 7: Input the generated set of key target image copies into the image classification module to classify the targets and generate key target classification information; Step 8: Based on the key target classification information generated in Steps 6 and 7, the weighted voting module generates key target classification results through weighted voting and issues warnings for potential adversarial sample targets.