Method, system, medium and electronic device for improving the security of perception algorithm confrontation
By building adversarial samples of white box and black box attack scenarios, improving the sample quality of the perceptual algorithm and adjusting the algorithm parameters, the problem of vulnerability of the perceptual algorithm against adversarial sample attacks is solved, and its security and robustness are improved.
Patent Information
- Application Number
- CN202411002175.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing perception algorithms are vulnerable to sample attacks, resulting in wrong judgments and decisions, affecting their security and reliability.
By constructing adversarial samples in white box attack scenarios and black box attack scenarios, different attack types are simulated, sample quality is improved, and algorithm parameters and structures are adjusted by identifying accuracy, the model's resistance is enhanced.
It improves the reliability and robustness of the perception algorithm, enhances the security of adversariality, prevents attack type learning, and improves the model's ability to resist attacks.
Smart Images

Figure CN118520930B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine learning, and in particular to a method, system, medium and electronic device for improving the anti-security of perception algorithms. Background Art
[0002] With the widespread application of perception algorithms in various fields, their security issues have become increasingly prominent. Adversarial samples can deceive intelligent perception algorithms and lead to wrong judgments and decisions. Studies have shown that some intelligent perception algorithms, including perception algorithms that perform well in many aspects (such as convolutional neural networks and recurrent neural networks), are obviously vulnerable to adversarial sample attacks that are strategically modified by attackers.
[0003] Therefore, developing a method to improve the adversarial security of perception algorithms is crucial to ensure the reliability and robustness of perception algorithms. Summary of the invention
[0004] The embodiments of the present application provide a method, system, medium and electronic device for improving the security of perception algorithms against attacks. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.
[0005] In a first aspect, an embodiment of the present application provides a method for improving the security of a perception algorithm against attacks, the method comprising:
[0006] Collect multiple categories of original images suitable for the intelligent perception algorithm to be tested for data augmentation processing to obtain an augmented data set;
[0007] Based on the augmented dataset, construct adversarial samples for white-box attack scenarios and black-box attack scenarios;
[0008] Based on the preset analysis indicators, the data set quality analysis of the adversarial sample is performed to obtain the data quality analysis results of the adversarial sample;
[0009] Adjust the adversarial sample according to the data quality analysis results, input the adversarial sample into the intelligent perception algorithm to be tested, and output the recognition result of the adversarial sample;
[0010] Based on the recognition results of the adversarial samples, the recognition accuracy of the intelligent perception algorithm to be tested is calculated, and when the recognition accuracy is less than a preset threshold, the algorithm parameters and structure of the intelligent perception algorithm to be tested are adjusted, and the steps of inputting the adversarial samples into the intelligent perception algorithm to be tested are continued until the recognition accuracy is greater than or equal to the preset threshold.
[0011] Optionally, construct adversarial samples for white-box attack scenarios based on the augmented dataset, including:
[0012] From the augmented dataset, select target images that can be correctly classified by the intelligent perception algorithm to be tested;
[0013] Acquire algorithm parameters of the intelligent perception algorithm to be tested, and generate gradient information of the intelligent perception algorithm to be tested according to the acquired algorithm parameters;
[0014] By using the algorithm and gradient information used in the white-box attack scenario, a perturbation of a preset size and a preset gradient direction is added to each target image to generate samples in the white-box attack scenario corresponding to each target image;
[0015] Input the samples of the white-box attack scenario corresponding to each target image into the intelligent perception algorithm to be tested, and output the image type corresponding to the samples of the white-box attack scenario corresponding to each target image;
[0016] When the image type corresponding to the sample in the white-box attack scenario corresponding to each target image is different from the image type of each target image, the sample in the white-box attack scenario corresponding to each target image is used as the adversarial sample in the white-box attack scenario.
[0017] Optionally, construct adversarial samples for black-box attack scenarios based on the augmented dataset, including:
[0018] Obtain a target perception algorithm whose similarity in structure and performance with the intelligent perception algorithm to be tested is greater than a preset threshold;
[0019] Generate initial adversarial samples on the target perception algorithm by using algorithms used in white-box attack scenarios and augmenting datasets;
[0020] Input the initial adversarial sample into the intelligent perception algorithm to be tested to determine the deception rate of the initial adversarial sample;
[0021] When the deception rate of the initial adversarial sample is less than a preset threshold, the estimated gradient of the intelligent perception algorithm to be tested is queried through the algorithm used in the black box attack scenario;
[0022] According to the estimated gradient, the initial adversarial sample is updated to obtain the adversarial sample in the black-box attack scenario.
[0023] Optionally, the preset analysis indicators include qualitative evaluation indicators and quantitative evaluation indicators; the quantitative evaluation indicators include the number of first samples and the first sample distribution of different categories, the deception ratio of the first adversarial samples, and the success ratio of the first adversarial samples under different attack intensities;
[0024] Based on the preset analysis indicators, the data set quality analysis of the adversarial sample is performed to obtain the data quality analysis results of the adversarial sample, including:
[0025] According to the qualitative evaluation indicators, the quality of the data set of adversarial samples is analyzed to obtain the qualitative evaluation results of the adversarial samples;
[0026] Calculate the number and distribution of samples of different categories of adversarial samples;
[0027] Input the adversarial sample into the pre-trained machine learning model and output the adversarial sample deception ratio corresponding to the adversarial sample; the pre-trained machine learning model is a mathematical model used for classifier accuracy reduction test and can calculate the deception ratio based on the test results;
[0028] By pre-setting multiple perturbation sizes and adversarial samples, the success ratio of adversarial samples under different attack intensities is determined;
[0029] Compare the number of samples and sample distribution of different categories of adversarial samples, the deception ratio of adversarial samples, and the success ratio under different attack intensities with the number of first samples and first sample distribution of different categories, the deception ratio of first adversarial samples, and the success ratio of first adversarial samples under different attack intensities to obtain quantitative evaluation results of adversarial samples;
[0030] The qualitative and quantitative evaluation results are used as the data quality analysis results of adversarial samples.
[0031] Optionally, the qualitative evaluation indicators include authenticity indicators, effectiveness indicators, diversity indicators, concealment indicators, and transferability indicators;
[0032] According to the qualitative evaluation indicators, the quality of the adversarial sample dataset is analyzed to obtain the qualitative evaluation results of the adversarial sample, including:
[0033] Send adversarial examples to multiple expert scoring clients;
[0034] Receive the scores of authenticity index, effectiveness index, diversity index, concealment index and transferability index from each expert rating client;
[0035] The scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index of each expert scoring client feedback are weighted and averaged to obtain the final scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index;
[0036] The final scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index are used as the qualitative evaluation results of the adversarial samples.
[0037] Optionally, calculate the number and distribution of samples of different categories of adversarial samples, including:
[0038] Iterate over each image in the adversarial example;
[0039] Based on the type of each traversed image, count the number of samples in each preset category;
[0040] Storing a mapping relationship between each preset type and the number of samples of each preset category;
[0041] Calculate the ratio of the number of samples in each preset category to the total number of adversarial samples to obtain the sample ratio of each preset category;
[0042] Based on the mapping relationship and the sample ratio of each preset category, the sample distribution of different categories of adversarial samples is drawn.
[0043] Optionally, the method further includes:
[0044] When the recognition accuracy is greater than or equal to a preset threshold, the intelligent perception algorithm to be tested is deployed to obtain a pre-trained image recognition model;
[0045] When receiving an image to be recognized, inputting the image to be recognized into a pre-trained image recognition model;
[0046] Output the recognition result corresponding to the image to be recognized.
[0047] In a second aspect, an embodiment of the present application provides a system for improving the security of perception algorithms against attacks, the system comprising:
[0048] A data augmentation processing module is used to collect multiple types of original images suitable for the intelligent perception algorithm to be tested for data augmentation processing to obtain an augmented data set;
[0049] The adversarial sample construction module is used to construct adversarial samples in white-box attack scenarios and black-box attack scenarios based on the augmented dataset;
[0050] A data quality analysis module is used to perform data set quality analysis on adversarial samples based on preset analysis indicators to obtain data quality analysis results of adversarial samples;
[0051] The recognition result output module is used to adjust the adversarial sample according to the data quality analysis results, input the adversarial sample into the intelligent perception algorithm to be tested, and output the recognition result of the adversarial sample;
[0052] The algorithm adjustment module is used to calculate the recognition accuracy of the intelligent perception algorithm to be tested based on the recognition results of the adversarial samples, and when the recognition accuracy is less than a preset threshold, adjust the algorithm parameters and structure of the intelligent perception algorithm to be tested, and continue to execute the steps of inputting the adversarial samples into the intelligent perception algorithm to be tested until the recognition accuracy is greater than or equal to the preset threshold.
[0053] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.
[0054] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.
[0055] The technical solution provided by the embodiments of the present application may have the following beneficial effects:
[0056] In the embodiments of the present application, on the one hand, by constructing adversarial samples under white-box attack scenarios and black-box attack scenarios as input data of the perception algorithm, the white-box attack scenarios and black-box attack scenarios can simulate attacks from different angles, so that the constructed adversarial samples contain rich attack types, thereby improving the sample quality, and then improving the reliability and robustness of the perception algorithm; on the other hand, by using the constructed adversarial samples to input the intelligent perception algorithm to be tested, the model can be prevented from learning the attack type, and at the same time, the recognition readiness rate is used to adjust the algorithm parameters and structure of the intelligent perception algorithm to be tested, so that the model's ability to resist attacks is continuously improved, thereby improving the adversarial security of the perception algorithm.
[0057] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0059] Figure 1 It is a flowchart of a method for improving the security of perception algorithms provided by an embodiment of the present application;
[0060] Figure 2 This is a schematic diagram of a category method of data augmentation provided in an embodiment of the present application;
[0061] Figure 3 It is a structural diagram of a system for improving the security of perception algorithms provided by an embodiment of the present application;
[0062] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0063] The following description and the accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them.
[0064] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0065] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of systems and methods consistent with some aspects of the present application as detailed in the attached claims.
[0066] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.
[0067] The present application provides a method, system, medium and electronic device for improving the adversarial security of perception algorithms to solve the problems existing in the above-mentioned related technical issues. In the technical solution provided by the present application, on the one hand, by constructing adversarial samples under white-box attack scenarios and black-box attack scenarios as input data of the perception algorithm, the white-box attack scenarios and black-box attack scenarios can simulate attacks from different angles, so that the constructed adversarial samples contain rich attack types, thereby improving the sample quality, and then improving the reliability and robustness of the perception algorithm; on the other hand, using the constructed adversarial samples to input the intelligent perception algorithm to be tested can make the model not learn the attack type, and at the same time use the recognition readiness rate to adjust the algorithm parameters and structure of the intelligent perception algorithm to be tested, so that the model's ability to resist attacks is continuously improved, thereby improving the adversarial security of the perception algorithm. The following uses an exemplary embodiment to explain in detail.
[0068] The following will be combined with the attached Figure 1 -Attached Figure 2 , a method for improving the security of perception algorithms against attacks provided by an embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a system for improving the security of perception algorithms against attacks based on a von Neumann system. The computer program can be integrated into an application or run as an independent tool application.
[0069] See also Figure 1 , which is a flowchart of a method for improving the security of perception algorithms in an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:
[0070] S101, collecting multiple categories of original images suitable for the intelligent perception algorithm to be tested for data augmentation processing to obtain an augmented data set;
[0071] Among them, perception algorithm refers to an algorithm that simulates human perception ability, collects data through sensors, cameras, microphones and other devices, and uses machine learning, deep learning and other technologies to analyze and process. This algorithm has a wide range of applications in different application scenarios. The intelligent perception algorithms to be tested are neural networks that need to improve adversarial security, such as convolutional neural networks (CNN), deep residual networks (ResNet), and YOLO (You Only Look Once). Convolutional neural network (CNN): used for tasks such as image classification, object detection and image segmentation. Deep residual network (ResNet): a deep learning model used to solve the problem of difficult deep network training. YOLO (You Only Look Once): a real-time object detection algorithm used to identify and locate objects in images.
[0072] Data augmentation is a technique used in machine learning to increase data diversity and improve the generalization ability of models. For image data, data augmentation is particularly effective because it can simulate changes in the real world, thereby helping the model learn more robust features. Data augmentation categories include flipping, translation, rotation, scaling, color change, and noise. Figure 2 shown.
[0073] In the embodiment of the present application, firstly, multiple types of original images suitable for the intelligent perception algorithm to be tested are collected, and then the multiple types of original images are processed by using the example Figure 2 The augmented data set is obtained by processing it in the augmentation method shown.
[0074] In one possible implementation, a suitable augmentation technology is selected according to task requirements and data characteristics, and parameters are determined for the selected augmentation technology, such as flipping direction, rotation angle, scaling ratio, etc. During the data loading or preprocessing stage, data augmentation processing is performed on multiple types of original images suitable for the intelligent perception algorithm to be tested.
[0075] S102, constructing adversarial samples in white-box attack scenarios and black-box attack scenarios based on the augmented data set;
[0076] Among them, the black-box attack scenario and the white-box attack scenario describe the attacker's understanding of and access rights to the algorithm.
[0077] In the white-box attack scenario, it is assumed that the attacker can obtain all the information of the perception algorithm to be attacked, including the algorithm architecture and algorithm parameters of the perception algorithm, the training method used to train the perception algorithm, the data set, the training hyperparameters, etc. The algorithms in the white-box attack scenario include the GSM algorithm, the PGD algorithm, the DeepFool algorithm, the JSMA algorithm, etc.
[0078] In a black-box attack scenario, the attacker knows nothing about the internal information of the target model and can only infer the behavior of the model through the input-output relationship of the model. Algorithms in black-box attack scenarios include the ZOO algorithm and the AutoZOOM algorithm.
[0079] In an embodiment of the present application, in the process of constructing adversarial samples in a white-box attack scenario based on an augmented data set, the process includes: selecting a target image that can be correctly classified by the intelligent perception algorithm to be tested from the augmented data set; obtaining the algorithm parameters of the intelligent perception algorithm to be tested, and generating gradient information of the intelligent perception algorithm to be tested based on the obtained algorithm parameters; adding a disturbance of a preset size and a preset gradient direction to each target image through the algorithm and gradient information used in the white-box attack scenario, and generating samples in the white-box attack scenario corresponding to each target image; inputting the samples in the white-box attack scenario corresponding to each target image into the intelligent perception algorithm to be tested, and outputting the image type corresponding to the samples in the white-box attack scenario corresponding to each target image; when the image type corresponding to the samples in the white-box attack scenario corresponding to each target image is different from the image type of each target image, the samples in the white-box attack scenario corresponding to each target image are used as adversarial samples in the white-box attack scenario.
[0080] In an embodiment of the present application, in the process of constructing an adversarial sample in a black-box attack scenario based on an augmented data set, the process includes: obtaining a target perception algorithm whose similarity in structure and performance with the intelligent perception algorithm to be tested is greater than a preset threshold; generating an initial adversarial sample on the target perception algorithm through the algorithm used in the white-box attack scenario and the augmented data set; inputting the initial adversarial sample into the intelligent perception algorithm to be tested to determine the deception rate of the initial adversarial sample; when the deception rate of the initial adversarial sample is less than the preset threshold, querying the estimated gradient of the intelligent perception algorithm to be tested through the algorithm used in the black-box attack scenario; based on the estimated gradient, updating the initial adversarial sample to obtain the adversarial sample in the black-box attack scenario.
[0081] S103, performing a data set quality analysis on the adversarial sample based on preset analysis indicators to obtain a data quality analysis result of the adversarial sample;
[0082] Among them, the preset analysis indicators include qualitative evaluation indicators and quantitative evaluation indicators; the quantitative evaluation indicators include the number of first samples and the first sample distribution of different categories, the deception ratio of the first adversarial samples, and the success ratio of the first adversarial samples under different attack intensities.
[0083] In an embodiment of the present application, in the process of performing data set quality analysis on adversarial samples based on preset analysis indicators to obtain data quality analysis results of adversarial samples, the process includes: performing data set quality analysis on adversarial samples according to qualitative evaluation indicators to obtain qualitative evaluation results of adversarial samples; calculating the number of samples and sample distribution of different categories of adversarial samples; inputting the adversarial samples into a pre-trained machine learning model, and outputting the adversarial sample deception ratio corresponding to the adversarial samples; the pre-trained machine learning model is a mathematical model used for classifier accuracy reduction test, and the deception ratio can be calculated based on the test results; determining the success ratio of adversarial samples under different attack intensities through multiple pre-set perturbation sizes and adversarial samples; comparing the number of samples and sample distribution of different categories of adversarial samples, the deception ratio of adversarial samples, and the success ratio under different attack intensities with the number of first samples and the first sample distribution of different categories, the deception ratio of first adversarial samples, and the success ratio of first adversarial samples under different attack intensities to obtain quantitative evaluation results of adversarial samples; and using the qualitative evaluation results and the quantitative evaluation results as the data quality analysis results of the adversarial samples.
[0084] Among them, qualitative evaluation indicators include authenticity indicators, effectiveness indicators, diversity indicators, hidden indicators and transferability indicators.
[0085] In an embodiment of the present application, a data set quality analysis is performed on the adversarial sample according to the qualitative evaluation index. In the process of obtaining the qualitative evaluation result of the adversarial sample, the adversarial sample is sent to multiple expert scoring clients; the scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index fed back from each expert scoring client are received; the scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index fed back by each expert scoring client are weighted and averaged to obtain the final scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index; the final scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index are used as the qualitative evaluation result of the adversarial sample.
[0086] In an embodiment of the present application, the process of calculating the number of samples and sample distribution of different categories of adversarial samples includes: traversing each image in the adversarial sample; counting the number of samples of each preset category based on the type carried by each traversed image; storing the mapping relationship between each preset type and the number of samples of each preset category; calculating the ratio of the number of samples of each preset category to the total number of samples of the adversarial sample, and obtaining the sample ratio of each preset category; based on the mapping relationship and the sample ratio of each preset category, drawing the sample distribution of different categories of the adversarial sample.
[0087] S104, adjusting the adversarial sample according to the data quality analysis result, inputting the adversarial sample into the intelligent perception algorithm to be tested, and outputting the recognition result of the adversarial sample;
[0088] Among them, the process of adjusting adversarial samples based on the results of data quality analysis involves optimizing existing adversarial sample datasets to improve their effectiveness and practicality in security testing.
[0089] In some embodiments, in the process of adjusting the adversarial samples according to the data quality analysis results, the existing adversarial sample data set is thoroughly reviewed, and samples that do not meet the requirements, such as those that are not realistic enough or cannot effectively deceive the model, are identified based on the data quality analysis results and removed from the adversarial samples. Alternatively, the perturbation parameters in the FGSM algorithm can be adjusted to generate more concealed adversarial samples.
[0090] S105, based on the recognition result of the adversarial sample, calculate the recognition accuracy of the intelligent perception algorithm to be tested, and when the recognition accuracy is less than a preset threshold, adjust the algorithm parameters and structure of the intelligent perception algorithm to be tested, and continue to execute the steps of inputting the adversarial sample into the intelligent perception algorithm to be tested until the recognition accuracy is greater than or equal to the preset threshold.
[0091] In some embodiments, after obtaining the recognition results of the adversarial samples, the number of adversarial samples that are correctly identified can be counted based on the recognition results of the adversarial samples, and the ratio between the number of adversarial samples that are correctly identified and the total number of adversarial samples can be calculated, and the ratio is used as the recognition accuracy of the intelligent perception algorithm to be tested.
[0092] In some embodiments, when the recognition accuracy is greater than or equal to a preset threshold, the intelligent perception algorithm to be tested is deployed to obtain a pre-trained image recognition model; when an image to be recognized is received, the image to be recognized is input into the pre-trained image recognition model; and the recognition result corresponding to the image to be recognized is output.
[0093] In the technical solution provided by the present application, on the one hand, by constructing adversarial samples under white-box attack scenarios and black-box attack scenarios as input data of the perception algorithm, the white-box attack scenarios and black-box attack scenarios can simulate attacks from different angles, so that the constructed adversarial samples contain rich attack types, thereby improving the sample quality, and then improving the reliability and robustness of the perception algorithm; on the other hand, by using the constructed adversarial samples to input the intelligent perception algorithm to be tested, the model can be prevented from learning the attack type, and at the same time, the recognition readiness rate is used to adjust the algorithm parameters and structure of the intelligent perception algorithm to be tested, so that the model's ability to resist attacks is continuously improved, thereby improving the adversarial security of the perception algorithm.
[0094] The following is a system embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the system embodiment of the present application, please refer to the method embodiment of the present application.
[0095] See also Figure 3 , which shows a schematic diagram of the structure of a system for improving the security of perception algorithms against adversarial attacks provided by an exemplary embodiment of the present application. The system for improving the security of perception algorithms against adversarial attacks can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The system 1 includes a data augmentation processing module 10, an adversarial sample construction module 20, a data quality analysis module 30, a recognition result output module 40, and an algorithm adjustment module 50.
[0096] The data augmentation processing module 10 is used to collect multiple types of original images suitable for the intelligent perception algorithm to be tested for data augmentation processing to obtain an augmented data set;
[0097] An adversarial sample construction module 20, used to construct adversarial samples in white-box attack scenarios and black-box attack scenarios based on the augmented data set;
[0098] A data quality analysis module 30 is used to perform a data set quality analysis on the adversarial sample based on preset analysis indicators to obtain a data quality analysis result of the adversarial sample;
[0099] The recognition result output module 40 is used to adjust the adversarial sample according to the data quality analysis result, input the adversarial sample into the intelligent perception algorithm to be tested, and output the recognition result of the adversarial sample;
[0100] The algorithm adjustment module 50 is used to calculate the recognition accuracy of the intelligent perception algorithm to be tested based on the recognition results of the adversarial samples, and when the recognition accuracy is less than a preset threshold, adjust the algorithm parameters and structure of the intelligent perception algorithm to be tested, and continue to execute the steps of inputting the adversarial samples into the intelligent perception algorithm to be tested until the recognition accuracy is greater than or equal to the preset threshold.
[0101] It should be noted that the system for improving the security of perception algorithms against attacks provided in the above embodiments only uses the division of the above functional modules as an example when executing the method for improving the security of perception algorithms against attacks. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system for improving the security of perception algorithms against attacks provided in the above embodiments and the method for improving the security of perception algorithms against attacks belong to the same concept. The implementation process is detailed in the method embodiment, which will not be repeated here.
[0102] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0103] In the technical solution provided by the present application, on the one hand, by constructing adversarial samples under white-box attack scenarios and black-box attack scenarios as input data of the perception algorithm, the white-box attack scenarios and black-box attack scenarios can simulate attacks from different angles, so that the constructed adversarial samples contain rich attack types, thereby improving the sample quality, and then improving the reliability and robustness of the perception algorithm; on the other hand, by using the constructed adversarial samples to input the intelligent perception algorithm to be tested, the model can be prevented from learning the attack type, and at the same time, the recognition readiness rate is used to adjust the algorithm parameters and structure of the intelligent perception algorithm to be tested, so that the model's ability to resist attacks is continuously improved, thereby improving the adversarial security of the perception algorithm.
[0104] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the methods for improving the anti-security of perception algorithms provided by the above-mentioned various method embodiments.
[0105] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the methods for improving the security of perception algorithms in the above-mentioned various method embodiments.
[0106] See also Figure 4 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .
[0107] The communication bus 1002 is used to realize the connection and communication between these components.
[0108] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.
[0109] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).
[0110] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts of the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.
[0111] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage system located away from the aforementioned processor 1001. As Figure 4 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program for improving the security of the perception algorithm.
[0112] exist Figure 4 In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call the application program for improving the security of the perception algorithm against the security stored in the memory 1005, and specifically perform the following operations:
[0113] Collect multiple categories of original images suitable for the intelligent perception algorithm to be tested for data augmentation processing to obtain an augmented data set;
[0114] Based on the augmented dataset, construct adversarial samples for white-box attack scenarios and black-box attack scenarios;
[0115] Based on the preset analysis indicators, the data set quality analysis of the adversarial sample is performed to obtain the data quality analysis results of the adversarial sample;
[0116] Adjust the adversarial sample according to the data quality analysis results, input the adversarial sample into the intelligent perception algorithm to be tested, and output the recognition result of the adversarial sample;
[0117] Based on the recognition results of the adversarial samples, the recognition accuracy of the intelligent perception algorithm to be tested is calculated, and when the recognition accuracy is less than a preset threshold, the algorithm parameters and structure of the intelligent perception algorithm to be tested are adjusted, and the steps of inputting the adversarial samples into the intelligent perception algorithm to be tested are continued until the recognition accuracy is greater than or equal to the preset threshold.
[0118] In one embodiment, when the processor 1001 constructs an adversarial sample in a white-box attack scenario according to the augmented data set, the processor 1001 specifically performs the following operations:
[0119] From the augmented dataset, select target images that can be correctly classified by the intelligent perception algorithm to be tested;
[0120] Acquire algorithm parameters of the intelligent perception algorithm to be tested, and generate gradient information of the intelligent perception algorithm to be tested according to the acquired algorithm parameters;
[0121] By using the algorithm and gradient information used in the white-box attack scenario, a perturbation of a preset size and a preset gradient direction is added to each target image to generate samples in the white-box attack scenario corresponding to each target image;
[0122] Input the samples of the white-box attack scenario corresponding to each target image into the intelligent perception algorithm to be tested, and output the image type corresponding to the samples of the white-box attack scenario corresponding to each target image;
[0123] When the image type corresponding to the sample in the white-box attack scenario corresponding to each target image is different from the image type of each target image, the sample in the white-box attack scenario corresponding to each target image is used as the adversarial sample in the white-box attack scenario.
[0124] In one embodiment, when the processor 1001 constructs an adversarial sample in a black box attack scenario according to the augmented data set, the processor 1001 specifically performs the following operations:
[0125] Obtain a target perception algorithm whose similarity in structure and performance with the intelligent perception algorithm to be tested is greater than a preset threshold;
[0126] Generate initial adversarial samples on the target perception algorithm by using algorithms used in white-box attack scenarios and augmenting datasets;
[0127] Input the initial adversarial sample into the intelligent perception algorithm to be tested to determine the deception rate of the initial adversarial sample;
[0128] When the deception rate of the initial adversarial sample is less than a preset threshold, the estimated gradient of the intelligent perception algorithm to be tested is queried through the algorithm used in the black box attack scenario;
[0129] According to the estimated gradient, the initial adversarial sample is updated to obtain the adversarial sample in the black-box attack scenario.
[0130] In one embodiment, when the processor 1001 performs a data set quality analysis on the adversarial sample based on a preset analysis indicator and obtains a data quality analysis result of the adversarial sample, the processor 1001 specifically performs the following operations:
[0131] According to the qualitative evaluation indicators, the quality of the data set of adversarial samples is analyzed to obtain the qualitative evaluation results of the adversarial samples;
[0132] Calculate the number and distribution of samples of different categories of adversarial samples;
[0133] Input the adversarial sample into the pre-trained machine learning model and output the adversarial sample deception ratio corresponding to the adversarial sample; the pre-trained machine learning model is a mathematical model used for classifier accuracy reduction test and can calculate the deception ratio based on the test results;
[0134] By pre-setting multiple perturbation sizes and adversarial samples, the success ratio of adversarial samples under different attack intensities is determined;
[0135] Compare the number of samples and sample distribution of different categories of adversarial samples, the deception ratio of adversarial samples, and the success ratio under different attack intensities with the number of first samples and first sample distribution of different categories, the deception ratio of first adversarial samples, and the success ratio of first adversarial samples under different attack intensities to obtain quantitative evaluation results of adversarial samples;
[0136] The qualitative and quantitative evaluation results are used as the data quality analysis results of adversarial samples.
[0137] In one embodiment, when the processor 1001 performs a data set quality analysis on the adversarial sample according to the qualitative evaluation index to obtain a qualitative evaluation result of the adversarial sample, the processor 1001 specifically performs the following operations:
[0138] Send adversarial examples to multiple expert scoring clients;
[0139] Receive the scores of authenticity index, effectiveness index, diversity index, concealment index and transferability index from each expert rating client;
[0140] The scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index of each expert scoring client feedback are weighted and averaged to obtain the final scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index;
[0141] The final scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index are used as the qualitative evaluation results of the adversarial samples.
[0142] In one embodiment, when calculating the number and distribution of samples of different categories of adversarial samples, the processor 1001 specifically performs the following operations:
[0143] Iterate over each image in the adversarial example;
[0144] Based on the type of each traversed image, count the number of samples in each preset category;
[0145] Storing a mapping relationship between each preset type and the number of samples of each preset category;
[0146] Calculate the ratio of the number of samples in each preset category to the total number of adversarial samples to obtain the sample ratio of each preset category;
[0147] Based on the mapping relationship and the sample ratio of each preset category, the sample distribution of different categories of adversarial samples is drawn.
[0148] In one embodiment, the processor 1001 further performs the following operations:
[0149] When the recognition accuracy is greater than or equal to a preset threshold, the intelligent perception algorithm to be tested is deployed to obtain a pre-trained image recognition model;
[0150] When receiving an image to be recognized, inputting the image to be recognized into a pre-trained image recognition model;
[0151] Output the recognition result corresponding to the image to be recognized.
[0152] In the technical solution provided by the present application, on the one hand, by constructing adversarial samples under white-box attack scenarios and black-box attack scenarios as input data of the perception algorithm, the white-box attack scenarios and black-box attack scenarios can simulate attacks from different angles, so that the constructed adversarial samples contain rich attack types, thereby improving the sample quality, and then improving the reliability and robustness of the perception algorithm; on the other hand, by using the constructed adversarial samples to input the intelligent perception algorithm to be tested, the model can be prevented from learning the attack type, and at the same time, the recognition readiness rate is used to adjust the algorithm parameters and structure of the intelligent perception algorithm to be tested, so that the model's ability to resist attacks is continuously improved, thereby improving the adversarial security of the perception algorithm.
[0153] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program for improving the security of the perception algorithm against attacks can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium of the program for improving the security of the perception algorithm against attacks can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.
[0154] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. A method for improving the security of perception algorithms, characterized in that: The method comprises: Collect multiple categories of original images suitable for the intelligent perception algorithm to be tested for data augmentation processing to obtain an augmented data set; Constructing adversarial samples in white-box attack scenarios and black-box attack scenarios based on the augmented data set; Based on the preset analysis indicators, the data set quality analysis is performed on the adversarial sample to obtain the data quality analysis result of the adversarial sample; wherein, The preset analysis indicators include qualitative evaluation indicators and quantitative evaluation indicators; the quantitative evaluation indicators include the number of first samples and the first sample distribution of different categories, the deception ratio of the first adversarial samples, and the success ratio of the first adversarial samples under different attack intensities; The step of performing a data set quality analysis on the adversarial sample based on a preset analysis indicator to obtain a data quality analysis result of the adversarial sample includes: According to the qualitative evaluation index, performing a data set quality analysis on the adversarial sample to obtain a qualitative evaluation result of the adversarial sample; Calculate the number and distribution of samples of different categories of the adversarial samples; Input the adversarial sample into a pre-trained machine learning model, and output the adversarial sample deception ratio corresponding to the adversarial sample; the pre-trained machine learning model is a mathematical model used for classifier accuracy reduction test, and the deception ratio can be calculated based on the test results; Determine the success ratio of the adversarial sample under different attack intensities by using a plurality of pre-set perturbation sizes and the adversarial sample; Compare the number of samples of different categories and sample distribution, the deception ratio of the adversarial samples, and the success ratio under different attack intensities of the adversarial samples with the number of first samples of different categories and the first sample distribution, the deception ratio of the first adversarial samples, and the success ratio of the first adversarial samples under different attack intensities to obtain a quantitative evaluation result of the adversarial samples; Using the qualitative evaluation result and the quantitative evaluation result as the data quality analysis result of the adversarial sample; Adjusting the adversarial sample according to the data quality analysis result, inputting the adversarial sample into the intelligent perception algorithm to be tested, and outputting the recognition result of the adversarial sample; Based on the recognition result of the adversarial sample, the recognition accuracy of the intelligent perception algorithm to be tested is calculated, and when the recognition accuracy is less than a preset threshold, the algorithm parameters and structure of the intelligent perception algorithm to be tested are adjusted, and the step of inputting the adversarial sample into the intelligent perception algorithm to be tested is continued until the recognition accuracy is greater than or equal to the preset threshold.
2. The method according to claim 1, characterized in that The step of constructing an adversarial sample in a white-box attack scenario according to the augmented data set includes: Selecting, from the augmented data set, a target image that can be correctly classified by the intelligent perception algorithm to be tested; Acquire algorithm parameters of the intelligent perception algorithm to be tested, and generate gradient information of the intelligent perception algorithm to be tested according to the acquired algorithm parameters; By using the algorithm used in the white-box attack scenario and the gradient information, a disturbance of a preset size and a preset gradient direction is added to each target image to generate a sample in the white-box attack scenario corresponding to each target image; Input the samples in the white-box attack scenario corresponding to each target image into the intelligent perception algorithm to be tested, and output the image type corresponding to the samples in the white-box attack scenario corresponding to each target image; When the image type corresponding to the sample in the white-box attack scenario corresponding to each target image is different from the image type of each target image, the sample in the white-box attack scenario corresponding to each target image is used as an adversarial sample in the white-box attack scenario.
3. The method according to claim 1, characterized in that The step of constructing an adversarial sample in a black box attack scenario according to the augmented data set includes: Acquire a target perception algorithm whose similarity in structure and performance to the intelligent perception algorithm to be tested is greater than a preset threshold; Generating an initial adversarial sample on the target perception algorithm by using an algorithm used in a white-box attack scenario and the augmented data set; Inputting the initial adversarial sample into the intelligent perception algorithm to be tested to determine the deception rate of the initial adversarial sample; When the deception rate of the initial adversarial sample is less than a preset threshold, querying the estimated gradient of the intelligent perception algorithm to be tested by using an algorithm used in a black box attack scenario; According to the estimated gradient, the initial adversarial sample is updated to obtain an adversarial sample in a black box attack scenario.
4. The method according to claim 1, characterized in that The qualitative evaluation indicators include authenticity indicators, effectiveness indicators, diversity indicators, concealment indicators and transferability indicators; The step of performing a data set quality analysis on the adversarial sample according to the qualitative evaluation index to obtain a qualitative evaluation result of the adversarial sample includes: Sending the adversarial sample to multiple expert scoring clients; Receive the scores of authenticity index, effectiveness index, diversity index, concealment index and transferability index from each expert rating client; The scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index fed back by each expert scoring client are weighted and averaged to obtain the final scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index; The final scores corresponding to the authenticity index, effectiveness index, diversity index, concealment index and transferability index are used as qualitative evaluation results of the adversarial sample.
5. The method according to claim 1, characterized in that The calculating the number of samples and sample distribution of different categories of the adversarial samples includes: Traversing each image in the adversarial example; Based on the type of each traversed image, count the number of samples in each preset category; Storing a mapping relationship between each preset type and the number of samples of each preset category; Calculate the ratio of the number of samples of each preset category to the total number of samples of the adversarial samples to obtain the sample ratio of each preset category; Based on the mapping relationship and the sample ratio of each preset category, the sample distribution of different categories of the adversarial samples is drawn.
6. The method according to claim 1, characterized in that The method further comprises: When the recognition accuracy is greater than or equal to a preset threshold, the intelligent perception algorithm to be tested is deployed to obtain a pre-trained image recognition model; When receiving an image to be recognized, inputting the image to be recognized into a pre-trained image recognition model; Output the recognition result corresponding to the image to be recognized.
7. A system for improving the security of perception algorithms against attacks, characterized in that: The system comprises: A data augmentation processing module is used to collect multiple types of original images suitable for the intelligent perception algorithm to be tested for data augmentation processing to obtain an augmented data set; An adversarial sample construction module, used to construct adversarial samples in white-box attack scenarios and black-box attack scenarios based on the augmented data set; A data quality analysis module is used to perform a data set quality analysis on the adversarial sample based on preset analysis indicators to obtain a data quality analysis result of the adversarial sample; wherein, The preset analysis indicators include qualitative evaluation indicators and quantitative evaluation indicators; the quantitative evaluation indicators include the number of first samples and the first sample distribution of different categories, the deception ratio of the first adversarial samples, and the success ratio of the first adversarial samples under different attack intensities; The data quality analysis module is specifically used for: According to the qualitative evaluation index, performing a data set quality analysis on the adversarial sample to obtain a qualitative evaluation result of the adversarial sample; Calculate the number and distribution of samples of different categories of the adversarial samples; Input the adversarial sample into a pre-trained machine learning model, and output the adversarial sample deception ratio corresponding to the adversarial sample; the pre-trained machine learning model is a mathematical model used for classifier accuracy reduction test, and the deception ratio can be calculated based on the test results; Determine the success ratio of the adversarial sample under different attack intensities by using a plurality of pre-set perturbation sizes and the adversarial sample; Compare the number of samples of different categories and sample distribution, the deception ratio of the adversarial samples, and the success ratio under different attack intensities of the adversarial samples with the number of first samples of different categories and the first sample distribution, the deception ratio of the first adversarial samples, and the success ratio of the first adversarial samples under different attack intensities to obtain a quantitative evaluation result of the adversarial samples; Using the qualitative evaluation result and the quantitative evaluation result as the data quality analysis result of the adversarial sample; A recognition result output module, used for adjusting the adversarial sample according to the data quality analysis result, inputting the adversarial sample into the intelligent perception algorithm to be tested, and outputting the recognition result of the adversarial sample; An algorithm adjustment module is used to calculate the recognition accuracy of the intelligent perception algorithm to be tested based on the recognition result of the adversarial sample, and when the recognition accuracy is less than a preset threshold, adjust the algorithm parameters and structure of the intelligent perception algorithm to be tested, and continue to execute the step of inputting the adversarial sample into the intelligent perception algorithm to be tested until the recognition accuracy is greater than or equal to the preset threshold.
8. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 6.
9. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method according to any one of claims 1 to 6.
