Safety critical intelligent software confrontation sample generation and robustness test system

Abstract

The invention discloses an adversarial sample generation and robustness test system for safety-critical intelligent software. The system comprises a model and data set import module, a model adaptation module, an adversarial sample generation module, a test execution module and a robustness evaluation module, wherein the model and data set importing module is used for importing a tested model and data; the model adaptation module analyzes the structure and parameters of the input model; the adversarial sample generation module integrates various adversarial sample generation methods, and a user can select one or more methods according to specific requirements to generate different types of adversarial samples; and the test execution module is responsible for flow configuration, test execution and result display, and provides a visual interface and an evaluation assembly line configuration unit. According to the method, the reliability and the safety of the intelligent software in various safety-critical applications are improved, the risk caused by confronting sample attacks is reduced, and important support is provided for constructing a safer intelligent system.

Description

TECHNICAL FIELD

[0001] The present application relates to a safety-critical intelligent software adversarial sample generation and robustness testing system, belonging to the technical field of software testing. BACKGROUND

[0002] With the rapid development of artificial intelligence (AI) technology, intelligent software has been widely applied in various fields, especially in safety-critical fields such as medical treatment, autonomous driving, and financial security. However, the safety and robustness of intelligent software become urgent problems when it faces complex and variable real-world environments. The robustness of intelligent software, especially the defense capability against adversarial sample attacks, is directly related to its reliability and safety in practical applications. Therefore, it is particularly important to develop a general safety-critical intelligent software adversarial sample generation system.

[0003] Adversarial examples are input data that can cause intelligent models to make errors in inference by applying subtle perturbations to the original data. Adversarial sample attacks can not only deceive intelligent models, but also may lead to serious security problems. For example, a small error in an image recognition model may cause a spacecraft to misjudge the space environment, affecting the successful execution of the task, or even endangering safety. In the field of autonomous driving, attackers may cause autonomous vehicles to misidentify traffic signs by making minor modifications, leading to traffic accidents. Therefore, it is of great practical significance to research and develop a system that can generate various adversarial samples to evaluate and improve the robustness of intelligent software.

[0004] Currently, adversarial sample generation methods mainly focus on the following directions:

[0005] 1) Gradient-based methods: use model gradient information to quickly generate adversarial samples, such as FGSM and PGD attacks.

[0006] 2) Optimization-based methods: generate adversarial samples by optimizing specific objective functions, such as C&W attacks.

[0007] 3) Generation model-based methods: use techniques such as generative adversarial networks (GAN) to create more natural adversarial samples.

[0008] 4) Black-box attack methods: generate adversarial samples without knowing the internal structure of the model, common black-box methods include query-based ZOO, AutoZOO, NES, Bandits, and decision-based Boundary Attack.

[0009] These methods show different effects in different application scenarios and model structures. However, most of these methods are designed for specific models and lack generality.

[0010] Existing intelligent software testing tools mainly include:

[0011] 1) Open source framework: such as Google's Cleverhans and IBM's Adversarial Robustness Toolbox (ART), which provides a variety of adversarial sample generation algorithms, but mainly for professional researchers, lacking a user-friendly interface; After evaluation, the user needs to manually summarize and analyze the robustness-related data; Cannot give defense suggestions to improve model security.

[0012] 2) Commercial platform: AI security testing platforms developed by some network security companies, such as advertorch developed by BorealisAI, a Canadian bank, mainly for financial, security and other fields, which is difficult to meet the robustness testing needs of intelligent software of various task types.

[0013] 3) Academic research platform: Some universities and research institutions develop adversarial sample evaluation systems, such as the adversarial attack and defense platform ARES developed by Tsinghua University for computer vision tasks, but are usually limited to specific task types and specific frameworks.

[0014] The common shortcomings of these existing tools mainly include the following aspects: First, there is a lack of comprehensive support for diversified tasks of security-critical intelligent software, making it difficult to cover multiple application scenarios such as image processing, speech recognition, and natural language processing. It is difficult to adapt to the multiple formats and special deployment environments of security-critical intelligent software, such as embedded systems and real-time operating systems, and lacks universality. The robustness evaluation dimension is often single, making it difficult to comprehensively evaluate the robustness characteristics of intelligent software such as accuracy, robustness, and attack success rate, and unable to give defense suggestions. In addition, the automation level of these tools is generally insufficient, which is difficult to apply under the strict time and resource constraints of security-critical tasks. SUMMARY

[0015] The technical problem solved by the present application is to overcome the shortcomings of the prior art and provide a security-critical intelligent software adversarial sample generation and robustness testing system suitable for multiple format models and integrating multiple adversarial sample generation methods to provide comprehensive robustness evaluation for intelligent software.

[0016] The technical solution of the present application is: a security-critical intelligent software adversarial sample generation and robustness testing system, comprising: a model and dataset import module, a model adaptation module, an adversarial sample generation module, a robustness evaluation module, and a test execution module; wherein:

[0017] The model and dataset import module provides a data receiving interface, so that a user imports a to-be-tested intelligent software model and a required test dataset from outside and stores them; allows the user to perform preprocessing operations on the test dataset; and provides data calling functions to the model adaptation module and the adversarial sample generation module;

[0018] The model adaptation module accepts a model loading instruction from the test execution module, obtains an intelligent model format for adapting a corresponding task type from the model loading instruction, calls the to-be-tested intelligent software model from the model and dataset import module, parses and identifies the structure and parameters of the to-be-tested intelligent software model, and sends the to-be-tested intelligent software model and the structure and parameters thereof to the adversarial sample generation module and the test execution module;

[0019] The adversarial sample generation module integrates multiple adversarial sample generation methods, receives the to-be-tested intelligent software model and the structure and parameters thereof sent by the model adaptation module, and calls a test dataset from the model and dataset import module; receives an adversarial sample generation instruction sent by the test execution module, selects an adversarial sample generation method specified in the adversarial sample generation instruction, generates an adversarial sample in combination with the test dataset, the to-be-tested intelligent software model and the structure and parameters thereof, and sends the adversarial sample to the test execution module;

[0020] The test execution module is responsible for process configuration, test execution and result display, provides a visual interface and an evaluation pipeline configuration unit; based on the visual interface, the test execution module provides a process configuration interface for a user, so that the user configures a test task process through the process configuration interface, sends a model loading instruction to the model adaptation module and an adversarial sample generation instruction to the adversarial sample generation module based on the task process; after receiving the to-be-tested intelligent software model and the corresponding adversarial sample, the test execution module inputs them to the evaluation pipeline configuration unit, selects a test link and detailed parameters of each link through the evaluation pipeline configuration unit, the evaluation pipeline configuration unit executes an adversarial test process, the test execution module displays the adversarial test progress and the test result in real time through the visual interface, saves the test result in a storage space, and sends the test result to the robustness evaluation module; the test execution module receives a robustness evaluation result returned by the robustness evaluation module and displays the robustness evaluation result.

[0021] The robustness evaluation module evaluates the to-be-tested intelligent software model after receiving the test result of the test execution module, evaluates the robustness of the to-be-tested intelligent software model when facing an adversarial sample, generates a robustness evaluation result, and sends the robustness evaluation result to the test execution module.

[0022] Preferably, the preprocessing operations in the model and dataset import module include data cleaning, data regularization and data augmentation on the imported dataset;

[0023] The data augmentation for the image data includes rotation, scaling, flipping, color distortion, and splicing.

[0024] Preferably, the task types include image processing models, text classification models, and speech recognition models.

[0025] The intelligent models of different formats are adapted, including Pytorch, Tensorflow, ONNX, and NCCN.

[0026] Preferably, different task types and intelligent model formats correspond to different adversarial sample generation methods, and the specific units are as follows:

[0027] For image classification tasks, the intelligent models in Pytorch and Tensorflow formats are supported, and white-box adversarial attack methods, black-box adversarial attack methods, and physical world adversarial attack methods are selected.

[0028] For object detection tasks, intelligent models in Pytorch, Tensorflow, and NCNN formats are supported, and white-box adversarial attack methods and black-box adversarial attack methods are selected; the NCNN format can also select a black-box adversarial attack method based on migration and a black-box adversarial attack method based on query.

[0029] For semantic segmentation tasks, the system supports intelligent models in ONNX and NCNN formats; the ONNX format selects a black-box adversarial attack method based on migration, a black-box adversarial attack method based on query, and an adversarial attack method based on a conversion model; the NCNN format selects a black-box adversarial attack method based on migration.

[0030] For speech recognition tasks, the system supports intelligent models in Pytorch format, and selects a PGD speech adversarial attack method.

[0031] For text classification tasks, the system supports intelligent models in Pytorch format, and selects a TextFooler text adversarial attack method to mislead the prediction of the model by changing the words in the input text.

[0032] Preferably, the robustness evaluation results generated by the robustness evaluation module include overall scores, main indicators, and defense strategy suggestions.

[0033] Preferably, the overall scores are weighted averages of each evaluation link, and the weights are determined according to different application scenarios.

[0034] Preferably, the main indicators in the robustness evaluation results include adversarial attack success rates, average attack times, average perturbation distances, average classification accuracies, average intersection over union (IoU) values, and perturbation curve graphs.

[0035] Adversarial attack success rate: indicates the proportion of samples in the adversarial test samples that successfully cause the model to produce incorrect output or misclassification among the total number of attack samples; the higher the adversarial attack success rate indicates that the model is more vulnerable to attack; otherwise, it can better resist adversarial attacks;

[0036] Average attack times Reflects the degree of disturbance when the model is successfully attacked, the smaller the average attack times, the easier the model is attacked, and the worse the robustness, specifically

[0037]

[0038] Refers to the average value of attack times after M test pictures are selected to perform adversarial attacks, wherein n i is the attack times, indicating the minimum attack times used to obtain an adversarial test sample that can interfere with the model after a limited number of disturbance attacks on the ith picture;

[0039] Average disturbance distance Indicates the average value of the minimum disturbance distance between each picture and its own adversarial test sample after M test pictures are selected to perform adversarial attacks; the smaller the average disturbance distance, the smaller the distance between the original picture and the adversarial test sample picture; specifically:

[0040]

[0041] Wherein:

[0042]

[0043] (x, y) represents a certain pixel and its surrounding area, μ x , μ y is the mean of the pixel points; σ xy is the pixel covariance, σ x , σ y is the pixel standard deviation, C1, C2 is a hyperparameter;

[0044] Average classification accuracy mAP: indicates the proportion of correct identification of the model on the entire test set; the higher the average classification accuracy, the more stable the model in identifying targets; specifically:

[0045]

[0046] Wherein, TP represents the number of positive samples correctly detected as positive samples, FP represents the number of negative samples incorrectly identified as positive samples; FN represents the number of positive samples incorrectly identified as negative samples, then is the recall rate of the model, is the precision rate of the model;

[0047] Mean Intersection over Union (MIoU): The lower the average intersection over union, the worse the model's robustness to perturbation attacks; specifically:

[0048]

[0049] where IoU is the intersection over union, n represents the number of classes, Intersection() refers to the intersection region of a certain class, and Union() refers to the union region of a certain class; label i represents the pixel label set of the i-th class.

[0050] Preferably, the defense strategy suggestion indicates the target class that needs attention and suggests the defense method to be used according to the specific results of the adversarial attack, including adversarial training, input transformation, adversarial detection, defensive distillation, and random smoothing; specifically:

[0051] Adversarial training: using adversarial samples and normal samples together for model training to improve the robustness of the model to adversarial samples;

[0052] Input transformation: pre-processing the input data to offset the effects of adversarial perturbations; methods include random cropping, translation, rotation, and color jittering;

[0053] Adversarial detection: training an auxiliary model or adding a detection mechanism to identify adversarial samples and take defensive measures when adversarial samples are detected;

[0054] Defensive distillation: training a teacher model and using the soft labels of the model to train a student model;

[0055] Random smoothing: adding random noise to the input to make the model insensitive to perturbations.

[0056] Compared with the prior art, the present application has the following advantages:

[0057] (1) The system of the present application can support intelligent models of multiple task types, including image classification, object detection, semantic segmentation, speech recognition, and natural language processing; therefore, the system can evaluate the robustness of intelligent software in different application fields;

[0058] (2) The system of the present application supports intelligent models in multiple common formats for security-critical intelligent software, including Pytorch, Tensorflow, Onnx, and Nccn, which are suitable for different types of intelligent software;

[0059] (3) The system of the present application integrates multiple mainstream adversarial sample generation methods, such as FGSM, PGD, and C&W attacks, and can also extend new adversarial sample generation techniques according to specific needs, providing comprehensive robustness evaluation for intelligent software;

[0060] (4)The system of the application can automatically generate a detailed robustness evaluation report containing various indicators, helping users understand the performance of intelligent software when facing different adversarial sample attacks; these diversity indicators can be used to guide the optimization and improvement of security-related intelligent software, improving the robustness and security in practical applications. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The figure is a schematic diagram of the system architecture of the application;

[0062] Figure 2 The figure is a schematic diagram of the adversarial method of the application; DETAILED DESCRIPTION

[0063] In order to better evaluate the security of intelligent software, there is an urgent need for a general system that can be applied to various format models and integrates various adversarial sample generation methods. Such a tool can not only help developers understand the potential weaknesses of security-related intelligent software in depth, but also provide strong support for the quality assurance of security-related intelligent systems.

[0064] The application provides a security-related intelligent software adversarial sample generation and robustness testing system, which includes five functional modules: model and dataset import module, model adaptation module, adversarial sample generation module, robustness evaluation module, and test execution module.

[0065] 1) The model and dataset import module is used to import the model and data to be tested, obtain the model and dataset to be processed through a preset method, and can manage and configure the test dataset, supporting functions such as import, quantity selection, and data enhancement of the test dataset. Users can diversify the configuration of the test dataset through this module to meet different testing needs.

[0066] 2) The model adaptation module is used to adapt intelligent models of different task types, including image processing models, text classification models, and speech recognition models; and adapt intelligent models of different formats, including Pytorch, Tensorflow, ONNX, and NCCN. This module can parse the structure and parameters of the input model to ensure that the adversarial sample generation method can be correctly applied to various models.

[0067] 3) The adversarial sample generation module integrates various adversarial sample generation methods, including white-box adversarial attacks, black-box adversarial attacks, physical world adversarial attacks, and adversarial sample generation methods designed for specific format models (Onnx, Ncnn) and specific task models (such as speech and text), including but not limited to. Users can select one or more methods according to specific needs to generate different types of adversarial samples.

[0068] 4) The test execution module is used for automatic execution of the adversarial sample generation process. According to the test link selected by the user and the detailed parameters of each link, the preset pipeline is converted into a command executed in sequence, the automatic test process is started, and the test start success or failure message and real-time progress are fed back in real time; the execution result of the test link is obtained, and all test results are saved in the storage space.

[0069] 5) The robustness evaluation module is used for evaluating the generated adversarial samples and measuring the robustness of intelligent software when facing adversarial samples. The module includes various evaluation indexes such as accuracy, perturbation distance, attack success rate, etc., to help users comprehensively understand the robustness performance of intelligent software. At the same time, the robustness problem of the intelligent software under test is pointed out and defense suggestions are given.

[0070] The following further describes a safety-critical intelligent software adversarial sample generation and robustness test system provided by an embodiment of the application in combination with the accompanying drawings.

[0071] The system provided by the embodiment of the application is composed as shown in Figure 1

[0072] (1) Model and dataset import module

[0073] The model dataset import and preprocessing module is a basic module of the entire adversarial sample generation tool. The module is responsible for importing the intelligent software model to be tested and its training dataset from an external data source, and performing necessary data preprocessing operations. The preprocessing operations include data cleaning, data normalization and data augmentation, etc. Data cleaning ensures the integrity and consistency of the dataset; data normalization is used to adjust the range and unit of the data; the user can also select the required data augmentation method to enhance the diversity and balance of the dataset. The data set augmentation methods that can be used in this tool are shown in Table 1;

[0074] Table 1 Data set augmentation methods contained in the application

[0075]

[0076]

[0077] (2) Model adaptation module

[0078] The model adaptation module is used to adapt intelligent models of different task types, including image processing models, text classification models, speech recognition models, etc.; and adapt intelligent models of different formats, including Pytorch, Tensorflow, ONNX, and NCCN four formats. The module can parse the structure and parameters of the input model to ensure that the adversarial sample generation method can be correctly applied to various models.

[0079] ​(3) The adversarial sample generation module

[0080] The existing robustness evaluation tool is often based on a fixed attack strategy (a fixed set of attack methods, or a fixed number of iterations and perturbation size of the adversarial sample) when evaluating the intelligent model, and the evaluation result is greatly affected by the specific method and parameter, and the safety of the model cannot be fully evaluated. The adversarial sample generation module in the application can execute a plurality of different attack strategy subtasks according to the model and data information to be processed, wherein each attack strategy can correspond to a set of attack parameters, and the user can select different perturbation sizes, iteration numbers and attack method parameters for combination, thereby constructing a plurality of different adversarial samples, so as to more comprehensively evaluate the safety of the model.

[0081] As shown in Figure 2 The system framework proposed by the application is divided into image classification tasks, target detection tasks, semantic segmentation tasks, speech recognition tasks and text classification tasks according to different tasks. For each task, the system can adapt to a plurality of mainstream machine learning frameworks, such as Pytorch, Tensorflow, NCNN and ONNX, and each framework supports a plurality of adversarial attack methods.

[0082] Firstly, for the image classification task, the system proposed by the application supports intelligent models in Pytorch and Tensorflow formats. In these two formats, the system can perform different types of adversarial attacks, including white-box adversarial attacks, black-box adversarial attacks and physical world adversarial attacks. Among them, the white-box adversarial attack assumes that the attacker has complete knowledge of the model, while the black-box adversarial attack assumes that the attacker knows nothing about the model, and the physical world adversarial attack is used to simulate the attack situation in the actual physical environment.

[0083] Secondly, in terms of target detection tasks, the application supports intelligent models in Pytorch, Tensorflow and NCNN formats. Similar to image classification, the system supports white-box and black-box adversarial attacks in these three formats. In addition, the NCNN format also includes migration-based black-box adversarial attacks and query-based black-box adversarial attacks, further enhancing the flexibility and diversity of adversarial sample generation.

[0084] For semantic segmentation tasks, the system supports intelligent models in ONNX and NCNN formats. The ONNX format supports migration-based black-box adversarial attacks, query-based black-box adversarial attacks and conversion model-based adversarial attacks. In the NCNN format, the system also supports migration-based black-box adversarial attacks.

[0085] In the speech recognition task, the system supports intelligent models in Pytorch format and can generate PGD speech adversarial samples. PGD (Projected Gradient Descent) is a commonly used optimization algorithm widely used to generate adversarial samples.

[0086] Finally, in the text classification task, the system supports intelligent models in Pytorch format and uses the TextFooler text adversarial attack method. TextFooler is a popular text adversarial attack algorithm that misleads the model's prediction by changing the words in the input text.

[0087] Specifically, the adversarial sample generation method contained in the present application is shown in Table 2.

[0088] Table 2 Adversarial sample generation method contained in the embodiment of the present application

[0089]

[0090]

[0091]

[0092] (4) Test automation execution module

[0093] In the test automation execution module, the system converts the pre-set pipeline into commands executed in sequence according to the test steps selected by the user in the evaluation pipeline configuration module unit and the detailed parameters of each step, starts the automation test process, and feeds back the test start success or failure message and real-time progress in real time; obtains the execution result of the test step, and saves all test results in the storage space.

[0094] (5) Robustness evaluation module

[0095] The robustness evaluation module generates a final evaluation report according to the user's adversarial sample method configuration and the evaluation results saved during the test execution process. Specifically, the robustness evaluation report contains the overall score, main indicators, and defense strategy suggestions. The overall score is the weighted average of each evaluation step, and the weight depends on different application scenarios. In this system, the following indicators are used to evaluate the robustness of intelligent models:

[0096] 1) Adversarial attack success rate

[0097] Adversarial Attack Success Rate (ASR): indicates the proportion of attackers successfully deceiving the model, i.e. the proportion of samples in the adversarial test samples that successfully cause the model to produce an incorrect output or misclassification among the total number of attack samples. A higher ASR indicates that the model is vulnerable to attack, and the model is relatively fragile. On the contrary, a lower ASR indicates that the model has higher robustness and can better resist adversarial attacks.

[0098] 2) Average number of attacks

[0099] Attack number n i : refers to the minimum number of attacks used to obtain an adversarial test sample that can interfere with the neural network after a limited number of perturbation attacks on the i th picture.

[0100] Average number of attacks refers to the average value of the number of attacks after selecting M test pictures to perform adversarial attacks, i.e.

[0101]

[0102] The average number of attacks reflects the degree of perturbation when the model is successfully attacked. The smaller the average number of attacks, the easier the model is attacked, and further indicates that the model has poorer robustness.

[0103] 3) Average perturbation distance

[0104] Average perturbation distance refers to the average value of the minimum perturbation distance between each picture and its adversarial test sample after selecting M test pictures to perform adversarial attacks; the average perturbation distance is an index reflecting the degree of perturbation of the test picture when the model is successfully attacked. The smaller the average perturbation distance, the smaller the distance between the original picture and the adversarial test sample picture;

[0105] In the system, the perturbation distance is measured by the structural similarity index (SSIM), and the specific formula is as follows:

[0106]

[0107] wherein μ x , μ y is the mean of the pixel points; σ xy is the pixel covariance, which is used to measure the overall error of the pixel points, σ x , σ y is the pixel standard deviation, and C1, C2 are hyperparameters, which are used to avoid instability when the value of the formula approaches 0;

[0108] To reflect the degree of disturbance, we use 1-SSIM(x, y) as the perturbation distance value of each picture, that is, the closer the SSIM is to 1, the smaller the image difference and the smaller the perturbation distance. Finally, by calculating the average perturbation distance of all successfully attacked samples, the average perturbation distance is obtained

[0109]

[0110] 4) Average classification accuracy

[0111] Mean Average Precision (mAP): represents the proportion of correct identification by the model on the entire test set. A higher average classification accuracy means that the model can accurately identify the target in most cases and performs more stably. By conducting adversarial attacks on the model to be tested, the change in average classification accuracy during this process can be measured, and the vulnerability of the target recognition model to perturbation attacks can be evaluated. Average classification accuracy can be calculated by recall and precision, and the calculation formula is as follows:

[0112]

[0113] Where TP represents the number of positive samples correctly detected as positive samples, FP represents the number of negative samples incorrectly identified as positive samples, and FN represents the number of positive samples incorrectly identified as negative samples, then is the recall of the model, is the precision of the model.

[0114] 5) Average Intersection over Union

[0115] Intersection over Union (IoU): also known as Jaccard similarity coefficient, is a measure of the overlap between two sets. In the semantic segmentation task, IoU is used to measure the matching degree between each pixel predicted by the model and the true label.

[0116] Mean Intersection over Union (mIoU): is the average value of IoU for each class in the entire test set, reflecting the classification accuracy of the model for different classes. By conducting adversarial attacks on the model to be tested, the change in average Intersection over Union between the original result and the result after the attack can be measured, and the lower the average Intersection over Union, the worse the robustness of the model to perturbation attacks

[0117] The formula is as follows, where n represents the number of classes, Intersection() represents the intersection area of a certain class, and Union() represents the union area of a certain class:

[0118]

[0119] label i represents a pixel label set of the i-th class.

[0120] 6) Disturbance curve

[0121] In one embodiment of the present embodiment, the attack strategy includes the disturbance size of the adversarial sample constructed according to the strategy and / or the number of iterations of the adversarial sample according to the strategy, so that the corresponding disturbance curve can be obtained, including: drawing a curve graph with the disturbance size or the number of iterations of the generated adversarial sample as the x-axis and the recognition accuracy of the corresponding model as the y-axis two-dimensional coordinates; or drawing a curve graph with the disturbance size or the number of iterations of the generated adversarial sample as the x-axis and the recognition accuracy of the corresponding model as the y-axis two-dimensional coordinates; if there are multiple categories of attack parameters in the attack strategy, the same number of two-dimensional coordinate curve graphs can be drawn according to the number of categories, and then the final evaluation result is calculated according to the weight of each category.

[0122] When facing the threat of adversarial samples, intelligent models can use various defense methods to improve their robustness and security. The defense strategy suggestion points out the target categories that need to be paid attention to and the defense methods that are recommended to be used according to the specific results of the adversarial attack. The recommended defense methods mainly include:

[0123] 1) Adversarial training.

[0124] Adversarial training improves the robustness of the model to adversarial samples by using adversarial samples together with normal samples for model training. The model is constantly exposed to adversarial samples during training, learning to recognize and resist these attacks.

[0125] 2) Input transformation.

[0126] Input transformation counteracts the effects of adversarial perturbations by preprocessing input data. Common transformation methods include random cropping, translation, rotation, color jittering, etc.

[0127] 3) Adversarial detection.

[0128] Adversarial detection methods identify adversarial samples by training an auxiliary model or adding a detection mechanism, and take defensive measures (such as rejecting classification or further processing) when adversarial samples are detected.

[0129] 4) Defensive distillation.

[0130] Defensive distillation trains a model with higher robustness through distillation technology. First, a teacher model is trained, and then a student model is trained using the soft labels of the model to make it more robust to adversarial perturbations.

[0131] 5) Random Smoothing.

[0132] The random smoothing method improves robustness by adding random noise to the input, making the model insensitive to small perturbations.

[0133] Embodiment:

[0134] The following takes the intelligent software in a certain intelligent car control module as an example to illustrate the test process of the system. The function of the software is to take a photo of a certain target object, perform semantic segmentation on the obtained image, and guide the movement decision of the intelligent car according to the difference between the position of each part after segmentation in the whole image and the preset reference position. The composition of the intelligent software includes a test dataset T, a learning code L (including a learning code and an inference code), and an intelligent model M.

[0135] (1) The user newly creates an adversarial sample generation project, specifies and uploads the test dataset T, learning code, inference code L, and intelligent model M of the project in the model and dataset import module.

[0136] (2) In the model adaptation module, the user can select the model task type (semantic segmentation) and format (ONNX). The model adaptation module will analyze the structure and parameters of the input model to ensure that the task and model format correspond accurately, and the adversarial sample generation method can run correctly on the model.

[0137] (3) In the adversarial sample generation module, the system generates corresponding adversarial samples according to the user-selected adversarial sample generation method and perturbation parameters, and saves them in the specified storage location. According to the model characteristics analysis, the user can select appropriate attack parameters, such as: attack strength (attack times per perturbation) = 10000; perturbation pixel value (VisitConstant) = 50; backtracking value (Backtracking Constant) = 25. Run the adversarial sample generation algorithm.

[0138] (4) The test execution module coordinates and manages the workflow of the aforementioned modules, converts the preset pipeline into commands executed in sequence according to the user-selected test links and detailed parameters of each link in the evaluation pipeline configuration unit, and starts the automated test process. During the test execution process, the test execution module will monitor the running state of each module in real time to ensure the smooth progress of the entire test process. Finally, the module saves the original test results for the robustness evaluation module to perform summary calculation.

[0139] (5) In the robustness evaluation module, the specific meaning of each evaluation index in the ONNX format semantic segmentation task is as shown in Table 3

[0140] Table 3 Robustness test indexes of ONNX format semantic segmentation model

[0141]

[0142] The robustness evaluation module finally generates a complete test result report according to the evaluation results of the adversarial samples and the original test data set, shows the comparison between the original picture and the adversarial sample picture and the performance of the model under the adversarial attack, and provides detailed robustness analysis results and improvement suggestions to help users further optimize the intelligent software model.

[0143] In view of the problems of limited model adaptation, insufficient method types and lack of robustness evaluation of existing adversarial sample generation tools, the present application provides a general security-critical intelligent software adversarial sample generation system to improve the security and robustness of intelligent software in practical application. Specifically, the system supports intelligent models of multiple formats and multiple task types, and for different task types, multiple adversarial sample generation methods can be selected, so that it has wide applicability and flexibility. Through the generated adversarial samples, the performance of intelligent software in the face of different attack means can be comprehensively evaluated, and potential security vulnerabilities and weak links can be identified. In addition, the system can also automatically generate a robustness evaluation report to provide targeted optimization suggestions for developers and guide them to improve the defense mechanism of intelligent software. The present application not only improves the reliability and security of intelligent software in various security-critical applications, reduces the risk brought by adversarial sample attacks, and provides important support for building a safer intelligent system.

[0144] The contents not described in detail in the specification of the present application belong to the prior art known to those skilled in the art.

Claims

1. A security-critical intelligent software adversarial sample generation and robustness testing system, characterized in that: Comprise: Model and dataset import module, model adaptation module, adversarial sample generation module, robustness evaluation module, test execution module; wherein: The model and dataset import module provides a data receiving interface, allowing users to import the intelligent software model to be tested and the test dataset required from the outside and store them; allows users to perform preprocessing operations on the test dataset; at the same time, it provides data calling functions to the model adaptation module and the adversarial sample generation module; The model adaptation module accepts the model loading instruction of the test execution module, obtains the intelligent model format used to adapt to the corresponding task type from the model loading instruction, calls the intelligent software model to be tested from the model and dataset import module, parses and identifies the structure and parameters of the intelligent software model to be tested, and sends the intelligent software model to be tested and its structure and parameters to the adversarial sample generation module and the test execution module; The adversarial sample generation module integrates multiple adversarial sample generation methods, receives the intelligent software model to be tested and its structure and parameters sent by the model adaptation module, and calls the test dataset from the model and dataset import module; receives the adversarial sample generation instruction sent by the test execution module, selects the adversarial sample generation method specified in the adversarial sample generation instruction, generates adversarial samples based on the test dataset, the intelligent software model to be tested and its structure and parameters, and sends them to the test execution module; The test execution module is responsible for process configuration, test execution and result display, and provides a visual interface and an evaluation pipeline configuration unit; based on the visual interface, the process configuration interface is provided to the user, so that the user can configure the test task process through the process configuration interface, and based on the task process, the model loading instruction is sent to the model adaptation module, and the adversarial sample generation instruction is sent to the adversarial sample generation module; after receiving the intelligent software model to be tested and the corresponding adversarial samples, they are input to the evaluation pipeline configuration unit, the test links and detailed parameters of each link are selected through the evaluation pipeline configuration unit, the evaluation pipeline configuration unit executes the adversarial test process, and the adversarial test progress and test results are displayed in real time through the visual interface. The test results are saved in the storage space and sent to the robustness evaluation module; the robustness evaluation result returned by the robustness evaluation module is received and displayed; The robustness evaluation module receives the test results of the test execution module and evaluates the intelligent software model to be tested, evaluates the robustness of the intelligent software model to be tested when facing adversarial samples, generates a robustness evaluation result, and sends it to the test execution module. 2.The system of claim 1, wherein: The preprocessing operation in the model and dataset import module includes data cleaning, data normalization and data augmentation on the imported dataset; Wherein the data augmentation for image data includes rotation, scaling, flipping, color distortion and splicing; the user selects the required data augmentation method to enhance the diversity and balance of the dataset.

3. The safety-critical intelligent software adversarial sample generation and robustness test system of claim 1, wherein: The task type includes image processing model, text classification model and speech recognition model; Adapt intelligent models of different formats, including: Pytorch, Tensorflow, ONNX, NCCN.

4. The system of claim 1, wherein: Different task types and intelligent model formats correspond to different adversarial sample generation methods, with specific units: For image classification tasks, support Pytorch and Tensorflow format intelligent models, select white-box adversarial attack method, black-box adversarial attack method and physical world adversarial attack method; For object detection tasks, support Pytorch, Tensorflow and NCNN format intelligent models, select white-box adversarial attack method, black-box adversarial attack method; Among them, NCNN format can also select black-box adversarial attack method based on migration and black-box adversarial attack method based on query; For semantic segmentation tasks, the system supports ONNX and NCNN format intelligent models; ONNX format selects black-box adversarial attack method based on migration, black-box adversarial attack method based on query and adversarial attack method based on conversion model; NCNN format selects black-box adversarial attack method based on migration; For speech recognition tasks, the system supports Pytorch format intelligent models, and selects PGD speech adversarial attack method; For text classification tasks, the system supports Pytorch format intelligent models, and selects TextFooler text adversarial attack method to mislead the model's prediction by changing the words in the input text.

5. The system of claim 1, wherein: The robustness evaluation results generated by the robustness evaluation module include: overall score, main indicators and defense strategy suggestions.

6. The system of claim 5, wherein: The overall score is the weighted average of each evaluation link, and the weight is determined according to different application scenarios.

7. The system of claim 5, wherein: The main indicators in the robustness evaluation results include: adversarial attack success rate, average attack times, average perturbation distance, average classification accuracy, average intersection over union, and perturbation curve graph; Specifically: Adversarial attack success rate: indicates the proportion of samples that successfully cause the model to produce incorrect output or misclassification among the total number of attack samples; The higher the adversarial attack success rate, the more vulnerable the model is to attack; Conversely, it can better resist adversarial attacks; average attack times The smaller the average attack times, the more easily the model is attacked and the worse the robustness when the disturbance degree of the reflection model is successfully attacked. denotes the average number of attacks after M images are selected to perform the adversarial attack, where n i is the number of attacks, indicating the minimum number of attacks used to obtain an adversarial test sample capable of interfering with the model after a limited number of perturbation attacks on the i-th image. Average perturbation distance The average perturbation distance represents the average value of the minimum perturbation distance between each picture and the adversarial test sample after selecting M pictures to perform adversarial attacks. The smaller the average perturbation distance, the smaller the distance between the original picture and the adversarial test sample picture. Specifically: Among them: (x, y) represents a certain pixel and its surrounding area, μ x , μ y is the mean of the pixel; σ xy is the pixel covariance, σ x , σ y is the pixel standard deviation, C1, C2 are hyperparameters; Average classification accuracy mAP: indicates the proportion of correct identification of the model on the entire test set; The higher the average classification accuracy, the more stable the model's performance in identifying targets; Specifically: Wherein, TP represents the number of positive samples that are correctly detected as positive samples, FP represents the number of negative samples that are incorrectly identified as positive samples; FN represents the number of positive samples that are incorrectly identified as negative samples, then is the recall rate of the model, is the precision rate of the model; Average intersection over union MIoU: The lower the average intersection over union, the worse the model's robustness to perturbation attacks; Specifically: wherein, IoU is the intersection over union, n represents the number of classes, Intersection() represents the intersection region of a certain class, Union() represents the union region of a certain class; label i represents the pixel label set of the i-th class.

8. The system of claim 5, wherein: Defense strategy suggestions point out the target categories that need attention and suggest the use of defense methods, including: adversarial training, input transformation, adversarial detection, defensive distillation, and random smoothing; Specifically: Adversarial training: use adversarial samples and normal samples together for model training, thereby improving the model's robustness to adversarial samples; Input transformation: preprocess the input data to offset the effects of adversarial perturbations; Methods include random cropping, translation, rotation, and color jitter; Adversarial detection: train an auxiliary model or add a detection mechanism to identify adversarial samples and take defensive measures when adversarial samples are detected; Defensive distillation: Train a teacher model, and use the soft labels of this model to train a student model; Random smoothing: Add random noise to the input, so that the model is not sensitive to perturbations.

Images