Model security assessment method, equipment, storage medium and device

Abstract

The present invention discloses a model security assessment method, equipment, storage medium and device. The present invention performs recognition performance test on the model to be tested to obtain recognition performance test results; performs robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain robust performance test results; performs backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain backdoor risk test results; performs comprehensive security analysis on the model to be tested according to the performance test results, the robust performance test results and the model backdoor risk test results. Compared with the prior art, which has not yet established a unified deep learning model security assessment specification and has a single assessment dimension, resulting in poor availability, ease of use and other aspects of the model security performance assessment method, the present invention realizes multi-dimensional evaluation of model security performance, which can comprehensively guarantee the model's security performance and improve the model's security performance.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model security assessment method, equipment, storage medium and device. Background Art

[0002] With the improvement of computing power and the development of big data, artificial intelligence technology, especially deep learning theory and methods, has made significant breakthroughs and has been widely used in many fields such as computer vision, speech recognition, and natural language processing. While artificial intelligence promotes the development of various fields of social economy from digitalization and informatization to intelligence, it also faces serious security threats. At present, artificial intelligence algorithms represented by deep neural networks have unresolved technical limitations such as weak robustness and unexplainability, which has led to the emergence of new security attack methods such as adversarial sample attacks and model backdoor attacks that undermine the confidentiality, integrity, and availability of artificial intelligence algorithms and data.

[0003] Some companies and universities at home and abroad have conducted research on artificial intelligence security assessment technology, and have developed a series of counterattack and defense, backdoor attack and defense methods and related integrated tools, which can achieve relatively good results in certain scenarios.

[0004] However, at this stage, various tools are mainly based on algorithm integration, and the evaluation technology mainly uses the success rate of adversarial sample attacks as a single evaluation indicator. There are also some non-universal indicators that can only be applied to small models and specific models. A unified deep learning model security assessment specification has not yet been established, so the usability, ease of use, and universality are relatively poor.

[0005] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of the present invention is to provide a model security assessment method, equipment, storage medium and device, aiming to solve the technical problem that the prior art has not yet established a unified deep learning model security assessment specification, resulting in poor availability and ease of use of model security performance assessment methods.

[0007] To achieve the above object, the present invention provides a model security assessment method, which comprises the following steps:

[0008] Perform recognition performance test on the model to be tested and obtain recognition performance test results;

[0009] Performing a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result;

[0010] Performing a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result;

[0011] A comprehensive security analysis is performed on the model to be tested based on the performance test results, the robust performance test results and the model backdoor risk test results.

[0012] Optionally, the step of performing a recognition performance test on the model to be tested and obtaining a recognition performance test result includes:

[0013] Perform data augmentation on the preset initial test set to construct a generalized test set;

[0014] Performing a recognition performance test on the model to be tested based on the preset initial test set to obtain a first ACC indicator;

[0015] Performing a recognition performance test on the model to be tested based on the generalized test set to obtain a second ACC indicator;

[0016] A recognition performance test result is determined according to the first ACC indicator and the second ACC indicator.

[0017] Optionally, the step of performing a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result includes:

[0018] Acquire simulated access conditions of the model to be tested, wherein the simulated access conditions include three conditions: completely acquiring model parameters and output information, acquiring complete output information but unable to access model parameters, and acquiring output category information but unable to access model parameters;

[0019] A target model is selected from the preset white box model or the preset black box model according to the simulated access situation to perform a robust performance test on the model to be tested, so as to obtain a robust performance test result.

[0020] Optionally, the step of selecting a target model from the preset white box model or the preset black box model according to the simulated access situation to perform a robust performance test on the model to be tested and obtaining a robust performance test result includes:

[0021] When the simulated access situation is to obtain complete output information but unable to access model parameters, selecting the preset white box model as the target model;

[0022] Generate an adversarial sample according to the preset white box model and the initial test set;

[0023] A robust performance test is performed on the model to be tested according to the adversarial sample to obtain a robust performance test result corresponding to a preset multi-dimensional indicator.

[0024] Optionally, the step of selecting a target model from the preset white box model or the preset black box model according to the simulated access situation to perform a robust performance test on the model to be tested and obtaining a robust performance test result includes:

[0025] When the simulated access situation is to completely obtain model parameters and output information, selecting the preset black box model as the target model;

[0026] Generate an adversarial sample according to the preset black box model and the initial test set;

[0027] A robust performance test is performed on the model to be tested according to the adversarial sample to obtain a robust performance test result corresponding to a preset multi-dimensional indicator.

[0028] Optionally, the step of selecting a target model from the preset white box model or the preset black box model according to the simulated access situation to perform a robust performance test on the model to be tested and obtaining a robust performance test result includes:

[0029] When the simulated access situation is to obtain output category information but cannot access model parameters, selecting the preset black box model as the target model;

[0030] Generate an adversarial sample according to the preset black box model and the initial test set;

[0031] The model to be tested is subjected to a robust performance test according to the adversarial sample to obtain robust performance test results corresponding to the attack success rate dimension, the maximum disturbance limit dimension, the model query number dimension, and the attack consumption dimension.

[0032] Optionally, the step of performing a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result includes:

[0033] Obtaining the access status of the model to be tested, wherein the access status includes two states: only obtaining the weight parameter information of the output layer of the model to be tested and accessing the model to be tested but only obtaining the output category information;

[0034] A test scenario is determined according to the access status, and a target model is selected from the preset white box model or the preset black box model according to the test scenario to perform a backdoor risk test on the model to be tested to obtain a backdoor risk test result.

[0035] Optionally, the step of determining a test scenario according to the access status, selecting a target model from the preset white box model and the preset black box model according to the test scenario to perform a backdoor risk test on the model to be tested, and obtaining a backdoor risk test result includes:

[0036] When the access state is to obtain only the weight parameter information of the output layer of the model to be detected, determining that the test scenario is a white box scenario, and using the preset white box model as the target model;

[0037] A backdoor risk test is performed on the model to be tested according to the preset white box model and the preset TPR model evaluation method to obtain a backdoor risk test result.

[0038] Optionally, the step of performing a backdoor risk test on the model to be tested according to the preset white box model and the preset TPR model evaluation method to obtain a backdoor risk test result includes:

[0039] Solve the constraints of the model to be tested according to the preset white box model and the preset TPR model evaluation method to obtain the weight distribution information of the model output layer;

[0040] It is determined whether the model to be tested has a backdoor risk based on the weight distribution information and a preset anomaly detection function, and a backdoor risk test result is obtained.

[0041] Optionally, the step of determining a test scenario according to the access status, selecting a target model from the preset white box model and the preset black box model according to the test scenario to perform a backdoor risk test on the model to be tested, and obtaining a backdoor risk test result includes:

[0042] When the access state is accessing the model to be tested but only obtaining output category information, determining that the test scenario is a black box scenario, and using the preset black box model as a target model;

[0043] A backdoor risk test is performed on the model to be tested according to the preset black box model and the preset AEVA backdoor attack method to obtain a backdoor risk test result.

[0044] Optionally, after the step of performing a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result, the step further includes:

[0045] When the backdoor risk test result indicates that there is a backdoor risk, a target attack category corresponding to the backdoor risk is determined.

[0046] Optionally, after the step of performing a comprehensive security analysis on the model to be tested according to the performance test result, the robust performance test result and the model backdoor risk test result, the step further includes:

[0047] Obtaining an overall safety performance score corresponding to the model to be tested;

[0048] An evaluation report is generated based on the overall security performance score and the various dimensional indicators included in the performance test results, the robust performance test results, and the model backdoor risk test results.

[0049] In addition, to achieve the above-mentioned purpose, the present invention also proposes a model security assessment device, which includes a memory, a processor, and a model security assessment program stored in the memory and executable on the processor, and the model security assessment program is configured to implement the steps of model security assessment as described above.

[0050] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a model security assessment program is stored, and when the model security assessment program is executed by a processor, the steps of the model security assessment method described above are implemented.

[0051] In addition, to achieve the above-mentioned purpose, the present invention also proposes a model security assessment device, which includes:

[0052] The recognition performance evaluation module is used to perform recognition performance test on the model to be tested and obtain recognition performance test results;

[0053] A robust performance evaluation module, used to perform a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result;

[0054] A backdoor risk assessment module, used to perform a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result;

[0055] The security comprehensive analysis module is used to perform a security comprehensive analysis on the model to be tested based on the performance test results, the robust performance test results and the model backdoor risk test results.

[0056] The present invention performs recognition performance test on the model to be tested to obtain recognition performance test results; performs robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain robust performance test results; performs backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain backdoor risk test results; and performs a comprehensive security analysis on the model to be tested according to the performance test results, the robust performance test results and the model backdoor risk test results. Compared with the prior art, which has not yet established a unified deep learning model security assessment specification and has a single assessment dimension, resulting in poor availability, ease of use and other aspects of the model security performance assessment method, the present invention realizes multi-dimensional evaluation of model security performance, which can comprehensively guarantee the model's security performance and improve the model's security performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a structural diagram of a model security assessment device for a hardware operating environment involved in an embodiment of the present invention;

[0058] Figure 2 It is a flowchart of the first embodiment of the model security assessment method of the present invention;

[0059] Figure 3 A schematic diagram of the model security assessment process of the first embodiment of the model security assessment method of the present invention;

[0060] Figure 4 It is a flow chart of the second embodiment of the model security assessment method of the present invention;

[0061] Figure 5 This is a schematic diagram of multi-dimensional evaluation of a model in the second embodiment of the model security assessment method of the present invention;

[0062] Figure 6 It is a flowchart of the third embodiment of the model security assessment method of the present invention;

[0063] Figure 7 It is a structural block diagram of the first embodiment of the model safety assessment device of the present invention.

[0064] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0065] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0066] Reference Figure 1 , Figure 1 It is a schematic diagram of the structure of a model security assessment device for a hardware operating environment involved in an embodiment of the present invention.

[0067] like Figure 1As shown, the model security assessment device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The wired interface of the user interface 1003 may be a USB interface in the present invention. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable memory (Non-volatile Memory, NVM), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0068] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the model safety assessment device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0069] like Figure 1 As shown, the memory 1005 identified as a computer storage medium may include an operating system, a network communication module, a user interface module, and a model security assessment program.

[0070] exist Figure 1 In the model security assessment device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the user device; the model security assessment device calls the model security assessment program stored in the memory 1005 through the processor 1001, and executes the model security assessment method provided by the embodiment of the present invention.

[0071] Based on the above hardware structure, an embodiment of the model security assessment method of the present invention is proposed.

[0072] Reference Figure 2 , Figure 2 The figure is a flow chart of the first embodiment of the model security assessment method of the present invention, and the first embodiment of the model security assessment method of the present invention is proposed.

[0073] In this embodiment, the model security assessment method includes the following steps:

[0074] Step S10: Perform a recognition performance test on the model to be tested to obtain a recognition performance test result.

[0075] It should be noted that the execution subject in this embodiment can be a device with a model security assessment system, such as: a server, a smart phone, a notebook, a computer, and a tablet, etc., and can also be other model security assessment devices that can achieve the same or similar functions. This embodiment does not limit this. Here, the above-mentioned computer will be used as an example to illustrate this embodiment and the following embodiments. The model security assessment method proposed in this scheme is a deep learning model security assessment scheme that integrates security assessment and robust enhancement. In view of the model security assessment results, this scheme will also analyze and explain according to different indicators, and give protection enhancement suggestions, including active defense modules and passive defense modules. Among them, the active defense module mainly optimizes and enhances the model itself through methods such as adversarial training; the passive defense module mainly processes and detects the model input samples through methods such as image transformation or local features, without changing the model itself. This scheme can be based on the ResNet50 model trained with the Cifar10 data set as an example to illustrate each embodiment.

[0076] In order to solve the problems of lack of unified evaluation standards, single evaluation dimension and limited evaluation indicators in the current deep learning model security evaluation, this scheme proposes a multi-dimensional and comprehensive deep learning model security evaluation technology and evaluation system, which belongs to the field of artificial intelligence security. The main idea of ​​this scheme is to combine multi-dimensional performance and security risks, and comprehensively consider the security performance of deep learning models, including (1) the recognition ability of deep learning models for clean samples with the same distribution, (2) the recognition ability of deep learning models for generalized clean samples, (3) the recognition ability or attack success rate of deep learning models generated by different principles in various scenarios such as white box and black box, and the maximum perturbation amount and other evaluation indicators, (4) whether the deep learning model faces the security risk of backdoor attacks.

[0077] The model security assessment system proposed in this solution is a deep learning model security assessment solution that integrates security assessment and robustness enhancement. Based on the model security assessment results, this system will also analyze and explain according to different indicators, and give protection enhancement suggestions, including active defense modules and passive defense modules. The active defense module mainly optimizes and enhances the model itself through methods such as adversarial training; the passive defense module mainly processes and detects the model input samples through methods such as image transformation or local features, without changing the model itself.

[0078] It is understandable that the model to be tested can be a model of various forms and scenarios that require security performance testing, and the model can be a model selected by the user. The recognition performance test can refer to a test that mainly evaluates the generalization of the recognition ability of the model to be tested, and the recognition performance test result can be a model recognition performance score obtained by testing a test set constructed based on multiple transformation methods such as image transformation, pixel transformation, natural scene transformation, style transfer network, etc.

[0079] It should be understood that the model security assessment system in this solution supports the uploading and processing of CV field datasets. Users can upload a set of clean test sets as the initial test datasets according to the task type of the model to be tested. In addition, the system has built-in multiple commonly used datasets, including Cifar10, ImageNet, etc. Users can select the corresponding test set according to the task type of the model to be tested. This test set is used as the preset initial test set for model evaluation testing.

[0080] In a specific implementation, the recognition performance of the model to be tested can be tested by performing a test set constructed by performing multiple transformation methods such as image transformation, pixel transformation, natural scene transformation, style transfer network, etc. on a preset initial test set to obtain a model recognition performance score.

[0081] Furthermore, the step S10 also includes: performing data enhancement on a preset initial test set to construct a generalized test set; performing a recognition performance test on the model to be tested based on the preset initial test set to obtain a first ACC indicator; performing a recognition performance test on the model to be tested based on the generalized test set to obtain a second ACC indicator; and determining a recognition performance test result based on the first ACC indicator and the second ACC indicator.

[0082] It should be noted that the preset initial test set can be a set of clean test sets uploaded by the user as the initial test data set D0 according to the task type of the model to be tested, and based on the initial test set D0, data enhancement is performed to construct a generalized test set, wherein the construction method includes multiple transformation methods such as image transformation, pixel transformation, natural scene transformation, style transfer network, etc. The generalized test set may include a test set generated based on multiple transformation methods such as image transformation, pixel transformation, natural scene transformation, style transfer network, etc. For example: calling the image transformation method to construct a set of generalized test sets D1 that have been flipped and scaled; calling the pixel transformation method to construct a set of generalized test sets D2 with various noises added; calling the natural scene transformation method to construct a set of generalized test sets D3 with natural noise added; calling the style transfer network to construct a set of generalized test sets D4 with artistic style.

[0083] It is understandable that the first ACC index refers to the recognition performance test of the model to be tested on the preset initial test set, and the recognition performance of the model on the preset initial test set is obtained; the second ACC index is based on the generalization test set to test the recognition performance of the model to be tested, and the recognition performance of the model on the generalization test set is obtained. The ACC index of the model to be tested on the initial test set and the generalization test set is calculated, and the model recognition performance score is obtained by weighted summation.

[0084] In the specific implementation, the evaluation and analysis of the recognition performance of deep learning models will simulate various normal / abnormal input situations that the model may face in non-attack scenarios. Therefore, in addition to analyzing the model's recognition ability for identically distributed data, it will also construct possible generalized test data to comprehensively evaluate the generalization of the deep learning model's recognition ability. The test data set consists of two parts: the initial test data set and the generalized test data set. Among them, the initial test data is uploaded by the user or selected from the designated public training set; the generalized test data set will be constructed using image transformation methods (such as flipping, rotating, cropping, scaling, and translation), pixel transformation (such as adding salt and pepper noise, Gaussian noise, random noise, etc.), natural scene transformation (such as adding natural noise such as rain, fog, and snow), style transfer network (such as oil painting, sketching, and other artistic styles) and other types of transformation methods.

[0085]

[0086] In the evaluation of model recognition performance, the ACC performance of the model on multiple test data sets is mainly used as the evaluation indicator. The higher the ACC on each test data set, the better the recognition performance of the model for normal data and generalized data. The calculation formula for the evaluation score of model recognition performance is as follows: Score acc =α0*ACC D0 +α1*ACC D1 +α2*ACC D2 +α3*ACC D3 +α4*

[0087] ACC D4 ;

[0088] Among them, ACC D0 represents the recognition performance of the model on the initial test set D0, ACC D1 represents the recognition performance of the model on the generalization test set D1, ACC D2 represents the recognition performance of the model on the generalization test set D2, ACC D3 represents the recognition performance of the model on the generalization test set D3, ACC D4 represents the recognition performance of the model on the generalization test set D4, and α0, α1, α2, α3, and α4 represent the corresponding weight coefficients respectively.

[0089] Step S20: performing a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result.

[0090] It should be noted that the preset white-box model can be a pre-set model for testing model performance. The model can be a user uploading the network structure and weight parameters in the form of a compressed package. This mode can achieve more efficient and comprehensive evaluation and analysis. The preset black-box model can be a pre-packaged API / SDK, i.e. a black-box model. This mode is closer to the actual attack and defense scenario and can more accurately reflect the security performance of the model to be tested when facing real attacks.

[0091] It is understandable that when performing robust performance evaluation on the model to be tested, the model to be tested is taken as the attacked model, and the initial test set D0 is taken as the original clean sample. The white box attack or black box attack algorithm corresponding to the preset white box model or the preset black box model is used to generate corresponding adversarial samples, and then multi-dimensional indicator information is obtained by attacking the model to be tested based on the adversarial samples, and then the robust performance score is determined based on the multi-dimensional indicator information.

[0092] Step S30: performing a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result.

[0093] It should be noted that the backdoor risk test is performed on the model to be tested based on the preset white box model or the preset black box model to determine whether there is a backdoor risk and obtain a judgment result. The backdoor risk test result is determined according to the judgment result. The judgment result includes two results: the existence of a backdoor risk and the non-existence of a backdoor risk.

[0094] Furthermore, after step S30, the method further includes: when the backdoor risk test result indicates that there is a backdoor risk, determining a target attack category corresponding to the backdoor risk.

[0095] It should be noted that when the backdoor risk test result is that there is a backdoor risk, it is necessary to determine the target attack category of the backdoor risk in order to determine its indicator result. During the testing phase, when facing benign samples, the model backdoor will not be activated, and the model can perform normally on benign samples; when facing poisoned samples with triggers, the hidden backdoor in the model will be activated, and the model will classify the sample into the category specified in advance by the attacker.

[0096] It can be understood that benign samples are clean data samples that have not been poisoned by attackers, the image content has not been maliciously modified by anyone, and the image label is consistent with the image content; poisoned samples are data samples that have been inserted with triggers. In the training phase, poisoned samples are injected into the training data set to embed backdoors in the model; in the testing phase, any poisoned samples that have been inserted with triggers can activate the model backdoor and are eventually classified into the target category; source category: the original category of the poisoned image; also called the base category; target category: the category that the attacker hopes the poisoned pattern will be identified as. The sample category can be used as an evaluation indicator for backdoor risk, such as false, and the attack success rate (ASR) can also be used as an evaluation indicator for backdoor risk.

[0097] Step S40: Perform a comprehensive security analysis on the model to be tested according to the performance test results, the robust performance test results and the model backdoor risk test results.

[0098] It should be noted that, according to the performance test results, the robust performance test results and the model backdoor risk test results, an overall security comprehensive analysis is performed on the model to be tested to obtain the overall model evaluation result, and the security performance analysis result of the model to be tested is determined according to the overall evaluation result, the performance test results, the robust performance test results and the model backdoor risk test results. To facilitate model improvement, corresponding evaluation information can be generated according to the security performance analysis results.

[0099] In the specific implementation, comprehensive evaluation and analysis of the model's safety performance is performed: In this embodiment, a multi-dimensional and comprehensive safety performance evaluation system for deep learning models is proposed. For the model to be tested, its corresponding performance and score are obtained by comprehensively considering the evaluation tests in the above three dimensions, and then the overall score of the safety performance of the model to be tested is calculated according to the following formula.

[0100] Score=w1*Score acc +w2*Score adv +w3*Score back ;

[0101] Among them, Score acc Represents the recognition performance score of the model, Score aav Represents the robust performance score of the model, Score backrepresents the backdoor risk score of the model, and w1, w2, and w3 represent the weight coefficients of the three scores respectively. In actual application scenarios, the recognition performance of the model is generally high, and the probability of the model having a backdoor risk is relatively small. However, the risk of adversarial attacks is a security threat that every model will face. Therefore, the weight coefficient w2 accounts for the largest proportion in the model security score calculation formula of this solution. Finally, based on the model evaluation and analysis results, this system will give corresponding protection enhancement suggestions and provide corresponding active and passive defense methods to fully guarantee the security performance of the deep learning model.

[0102] Furthermore, in order to improve the safety performance of the model, corresponding evaluation suggestion information can be generated after obtaining the safety performance analysis results. After step S40, it also includes: obtaining the overall safety performance score corresponding to the model to be tested; generating an evaluation report based on the overall safety performance score and the various dimensional indicators contained in the performance test results, the robust performance test results and the model backdoor risk test results.

[0103] It should be noted that an evaluation report is generated based on the overall security performance score included in the overall evaluation results of the model obtained according to the above-mentioned overall comprehensive security analysis, as well as the various dimensional indicators included in the performance test results, the robust performance test results and the model backdoor risk test results.

[0104] It is understandable that the process of generating the evaluation report can be the following steps: (a) giving the overall security score of the model to be tested and the specific analysis of each indicator in the three evaluation modules; (b) protection enhancement suggestions: for the model security evaluation analysis in (a), the system will give protection enhancement suggestions for the model to be tested from the two perspectives of active defense and passive defense. If the user adopts the suggestions to perform model robustness enhancement, the system will continue to perform steps c) to e), otherwise the evaluation report will be output; (c) model robustness enhancement: the system can implement robust enhancement measures such as robust adversarial training in active defense and abnormal sample detection in passive defense on the model to be tested, and the robust model after protection enhancement can be downloaded and used by users; (d) robust model evaluation: the enhanced robust model is used as the model to be tested for performance test results, robustness test and backdoor risk test, and a security score and specific analysis are given; (e) comparative analysis of the two evaluations: the evaluation indicators of the initial model to be tested and the robust model are compared, and the comparison results, the evaluation analysis of the model to be tested and the evaluation analysis of the robust model are merged into the evaluation report, and the evaluation report is output.

[0105] In the specific implementation, to further illustrate the implementation process of this solution, you can refer to Figure 3The model security evaluation process diagram shown is, in which the above method is implemented by integrating the model upload module to be tested, the test set upload module, the model recognition performance evaluation module, the model robust performance evaluation module and the model backdoor risk assessment module, and the evaluation report is output based on the evaluation report generation module.

[0106] This embodiment obtains recognition performance test results by performing recognition performance test on the model to be tested; performs robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain robust performance test results; performs backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain backdoor risk test results; and performs a comprehensive security analysis on the model to be tested based on the performance test results, the robust performance test results and the model backdoor risk test results. Compared with the prior art, which has not yet established a unified deep learning model security assessment specification and has a single assessment dimension, resulting in poor availability, ease of use and other aspects of the model security performance assessment method, this embodiment implements multi-dimensional assessment of model security performance, which can comprehensively guarantee the model's security performance and improve the model's security performance.

[0107] Reference Figure 4 , Figure 4 This is a flow chart of the second embodiment of the model security assessment method of the present invention. Figure 2 The first embodiment shown provides a second embodiment of the model security assessment method of the present invention.

[0108] In this embodiment, the step S20 further includes:

[0109] Step S201: obtaining the simulated access status of the model to be tested.

[0110] It should be noted that the simulated access situations include three situations: completely obtaining model parameters and output information, obtaining complete output information but unable to access model parameters, and obtaining output category information but unable to access model parameters.

[0111] It is understandable that the evaluation and analysis of the robust performance of deep learning models will simulate different types and strengths of adversarial sample attacks that the model may face in the attack scenario, and evaluate the model's ability to resist adversarial sample attacks. In this embodiment, the robust performance evaluation can be divided into (1) white box scenarios, which can fully obtain model parameters and output information; (2) black box scenarios, which cannot access model parameters but can obtain complete output information, namely soft label status; (3) black box scenarios, which cannot access model parameters and can only obtain output category information, namely hard label status.

[0112] Step S202: selecting a target model from the preset white box model or the preset black box model according to the simulated access situation, performing a robust performance test on the model to be tested, and obtaining a robust performance test result.

[0113] It should be noted that the white-box scenario is tested with a preset white-box model, and the black-box scenario is tested with a preset black-box model. When evaluating the robust performance of the model to be tested, the model to be tested can be used as the attacked model, and the initial test set D0 can be used as the original clean sample. The white-box attack or black-box attack algorithm can be used to generate the corresponding adversarial samples; and the calculation steps are combined:

[0114] 1. Calculate the success rate ASR when adversarial samples attack the model to be tested;

[0115] 2. Calculate the maximum perturbation limit Epsilon added to the adversarial sample when the attack success rate reaches the threshold;

[0116] 3. Calculate the time consumed by adversarial sample generation when the attack success rate reaches the threshold;

[0117] 4. Calculate the number of attack iterations (Iters) or the number of queries to the model (Queries) when the attack success rate reaches the threshold;

[0118] 5. Calculate the change in the model's confidence in predicting the true labels of clean samples and adversarial samples;

[0119] 6. Calculate the robust performance score of the above indicator values ​​1 to 5, and output the robust performance score as the robust performance test result.

[0120] In the specific implementation, in order to further illustrate the difference between this solution and the existing technology, the following examples are given: Through the evaluation scheme of this solution, more than 10 common models with different scale data sets such as Cifar and ImageNet, 5 mainstream network structures, different network depths, and more than 10 adversarial training models constructed with no less than 5 active defense strategies are evaluated and analyzed. The security performance of different models facing new risks such as adversarial attacks can be evaluated and analyzed, and corresponding protection suggestions can be given. Taking the ResNet50 model trained with the CIFAR10 data set and the Inception_ResNet_V2 model trained with the ImageNet data set as examples, the various indicators measured by this system in multiple dimensions and the security scores of the models are shown in Tables 1 and 2:

[0121] Table 1 Evaluation results of ResNet50 in white box scenario

[0122]

[0123]

[0124] Table 2 Evaluation results of Inception_ResNet_V2 in white box scenario

[0125]

[0126]

[0127] Comparing Table 1 and Table 2, it can be seen from the results of the recognition performance evaluation that the recognition performance of the two models on the generalization test set has declined, and the decline of ResNet50 trained on CIFAR10 is more obvious; from the results of the robust performance evaluation, it can be seen that when facing the risk of white-box adversarial attacks, the robust performance of ResNet50 and Inception_ResNet_V2 faces huge challenges, which will seriously threaten the security of the model and reduce the security score of the model.

[0128] Through the above content and combined Figure 5 As shown in the diagram of multi-dimensional evaluation of the model, the core technical point of the security evaluation system for deep learning models proposed in this scheme is to consider the multi-dimensional security characteristics of deep learning models, including the model's recognition ability on normal samples, generalized samples, and adversarial samples, its resistance to adversarial samples of different categories and strengths, whether there is a backdoor risk, etc., to comprehensively evaluate the robustness and security performance of the model.

[0129] Furthermore, the step S202 also includes: when the simulated access situation is to obtain complete output information but cannot access model parameters, selecting the preset white box model as the target model; generating adversarial samples based on the preset white box model and the initial test set; performing a robust performance test on the model to be tested based on the adversarial samples, and obtaining a robust performance test result corresponding to a preset multi-dimensional indicator.

[0130] It should be noted that for scenario (1), when the simulated access situation is to obtain complete output information but cannot access model parameters, the preset white box model is selected as the target model; a comprehensive and in-depth robust performance test will be conducted, combining different types of adversarial attack methods with different attack intensities such as gradient-based and optimization-based, combined with multi-dimensional indicators such as attack success rate, maximum perturbation limit, and number of model queries, to comprehensively evaluate the model robust performance. Among them:

[0131] Adversarial sample attack success rate: Use the adversarial attack algorithm to generate adversarial samples, and calculate the proportion of samples in the adversarial samples that are incorrectly identified by the test model to the total number of samples. The attack success rate is most directly related to the adversarial risk faced by the model. The higher the index, the higher the model risk. In order to directly reflect the security performance of the model in the face of adversarial attack risks, this index accounts for the highest proportion in the model robustness score;

[0132] Maximum perturbation limit of adversarial samples: Generate adversarial samples under different norm limits using methods such as gradient attack and optimization attack. By continuously increasing the perturbation limit, the success rate of adversarial sample attacks gradually approaches the threshold T1. When T1 is reached, the attack stops. Calculate the maximum perturbation added to the sample when the attack success rate reaches T1. Take the weighted sum of the maximum perturbations of different attack methods under different norm limits to get the final maximum perturbation limit. The maximum perturbation directly affects the visual effect of the adversarial sample. The easier it is for the human eye to distinguish the adversarial sample, the lower the adversarial risk faced by the model. This can also indirectly reflect the adversarial attack risk faced by the model. Therefore, the proportion of this indicator in the model robustness score is second only to the adversarial sample attack success rate.

[0133] Attack iteration number: Generate adversarial samples using an attack method based on gradient iteration. When the adversarial sample successfully attacks the model to be tested, the iteration process is stopped, and the average number of iterations in the process of generating adversarial samples is calculated. The more iterations there are, the greater the resource consumption during the attack process, and the higher the resource cost the attacker has to pay. This indicator can reflect the adversarial attack risk faced by the model under different attack costs;

[0134] Attack time consumption: Generate adversarial samples using an attack method based on gradient iteration. When the adversarial sample successfully attacks the model to be tested, the iteration process is stopped, and the average time consumption required in the process of generating adversarial samples is calculated. The greater the time consumption, the higher the time cost for the attacker, which can indirectly reflect the robustness of the model in resisting adversarial attacks. Time consumption is not only related to the robustness of the model, but also has a certain correlation with the network depth of the model. In order to minimize the impact of network depth on model security evaluation, this indicator accounts for the smallest proportion in the model robustness score;

[0135] Change in true label confidence: Use the adversarial attack algorithm to generate adversarial samples, and calculate the difference in the model's prediction confidence for the true label on the adversarial sample and the clean sample. The greater the change in the true label confidence, the greater the impact of the adversarial sample on the model's prediction ability, and the higher the security risk of the model when facing adversarial attacks;

[0136] It is understandable that in scenario (1), the parameters and output information of the model can be fully obtained. For this reason, this solution adopts two types of adversarial attack methods, gradient-based and optimization-based, to generate diverse adversarial samples, so as to calculate the model robustness evaluation score in a more comprehensive and fine-grained manner. Among them, the gradient-based attack method can quickly generate adversarial samples of various strengths, and the optimization-based attack method can consider using more subtle perturbations as much as possible while completing the adversarial attack, which can more accurately infer the robust lower bound of the model to a certain extent.

[0137] In the specific implementation, the calculation formula of the model robustness test score of scenario (1) is expressed as follows:

[0138] Score adv =λ1*ASR+λ2*Epsilon+λ3*Iters+λ4*Time+λ5*Conf;

[0139] Among them, ASR represents the success rate of adversarial sample attack, Epsilon represents the maximum perturbation limit of adversarial samples, Iters represents the number of attack iterations, Time represents the attack time consumption, Conf represents the confidence change of the true label of the sample before and after the attack, and λ1, λ2, λ3, λ4, and λ5 represent their respective weight coefficients.

[0140] Furthermore, the step S202 also includes: when the simulated access situation is to fully obtain model parameters and output information, selecting the preset black box model as the target model; generating adversarial samples according to the preset black box model and the initial test set; performing a robust performance test on the model to be tested according to the adversarial samples, and obtaining a robust performance test result corresponding to a preset multi-dimensional indicator.

[0141] It should be noted that for scenario (2), when the simulated access situation is to fully obtain the model parameters and output information, the preset black box model is selected as the target model; this embodiment will use a score-based adversarial attack method, combined with multi-dimensional indicators such as attack success rate, maximum perturbation limit, and model query times, to comprehensively evaluate the model robustness performance. Among them:

[0142] Adversarial sample attack success rate: Using the score-based adversarial attack method, the model gradient is estimated with the model's prediction probability and adversarial samples are generated. The proportion of samples in the adversarial samples that are incorrectly identified by the test model to the total number of samples is calculated. The higher the attack success rate, the higher the security risk faced by the model;

[0143] Maximum perturbation limit for adversarial samples: Using a score-based adversarial attack method, the model gradient is estimated with the model's predicted probability and adversarial samples are generated. The perturbation limit is continuously increased so that the success rate of adversarial sample attacks gradually approaches the threshold T2 until T2 is reached, at which point the attack stops. The maximum perturbation added to the sample when the attack success rate reaches T2 is calculated. If the success rate still does not reach T2 when the perturbation amount is added to M, the attack success rate under the perturbation amount limit of M is used as the basis for evaluation. The larger the maximum perturbation amount, the lower the adversarial risk faced by the model;

[0144] Model query times: Calculates the average number of model queries to estimate model gradients during adversarial sample generation. The more times the model is queried, the greater the resource consumption during the attack, and the higher the resource cost to the attacker.

[0145] Attack time consumption: Calculate the average time consumption required for model gradient estimation in the process of generating adversarial samples. The greater the time consumption, the higher the time cost the attacker has to pay, and the more robust the model is;

[0146] Change in true label confidence: Calculate the difference in the model's prediction confidence for the true label on the adversarial sample and the clean sample. The greater the change in the true label confidence, the more accurate the model gradient estimation during the attack, and the greater the impact of the adversarial sample on the model's prediction ability;

[0147] It is understandable that in scenario (2), only the soft label of the model output can be obtained. Therefore, this embodiment adopts a score-based adversarial attack method to estimate the model gradient to generate adversarial samples, thereby calculating various evaluation indicators. The model robustness evaluation score calculation formula in this scenario is expressed as follows: Score adv =λ1*ASR+λ2*Epsilon+λ3*Queries+λ4*Time+λ5*Conf;

[0148] Among them, ASR represents the success rate of adversarial sample attack, Epsilon represents the maximum perturbation limit of adversarial samples, Queries represents the number of model queries, Time represents the attack time consumption, Conf represents the confidence change of the true label of the sample before and after the attack, and λ1, λ2, λ3, λ4, and λ5 represent their respective weight coefficients.

[0149] Furthermore, the step S202 also includes: when the simulated access situation is to obtain output category information but cannot access model parameters, selecting the preset black box model as the target model; generating adversarial samples according to the preset black box model and the initial test set; performing a robust performance test on the model to be tested according to the adversarial samples, and obtaining robust performance test results corresponding to the attack success rate dimension, the maximum perturbation limit dimension, the model query number dimension, and the attack consumption dimension.

[0150] It should be noted that for scenario (3), when the simulated access situation is to obtain output category information but cannot access model parameters, the preset black box model is selected as the target model; this embodiment will use the adversarial attack method based on decision boundaries, combined with multi-dimensional indicators such as attack success rate, maximum perturbation limit, and model query times, to comprehensively evaluate the model robustness performance. Among them:

[0151] Adversarial sample attack success rate: Using the decision boundary-based adversarial attack method, the model's predicted label is used to estimate the model's decision boundary and generate adversarial samples. The proportion of samples in the adversarial samples that are incorrectly identified by the model to be tested to the total number of samples is calculated. The higher the attack success rate, the more accurate the estimation of the model's decision boundary, and the higher the security risk faced by the model;

[0152] Maximum perturbation limit for adversarial samples: Calculate the maximum perturbation added to the sample when the success rate of adversarial sample attacks reaches the threshold T3. The larger the maximum perturbation, the lower the adversarial risk faced by the model.

[0153] Model query times: Calculate the average number of queries to the model in order to estimate the decision boundary of the model during the process of generating adversarial samples. The more times the model is queried, the more complex its decision boundary is, the more difficult it is for the adversarial sample to attack successfully, and the more robust the model is;

[0154] Attack time consumption: Calculate the average time spent estimating the model decision boundary in the process of generating adversarial samples. The greater the time consumption, the higher the time cost the attacker has to pay, and the more robust the model is;

[0155] It is understandable that in scenario (3), only the hard label output by the model can be obtained. For this reason, this solution adopts an adversarial attack method based on the decision boundary. By continuously accessing the model prediction value and gradually approaching its decision boundary to generate adversarial samples, various evaluation indicators are calculated. Since it is impossible to obtain the model's prediction probability for the sample, it is impossible to evaluate the change in the confidence of the true label in this scenario.

[0156] In the specific implementation, this scenario is the most suitable for the actual anti-attack scenario, so the model robustness score calculation formula is as follows:

[0157] Score adv =λ1*ASR+λ2*Epsilon+λ3*Queries+λ4*Time;

[0158] Among them, ASR represents the success rate of adversarial sample attacks, Epsilon represents the maximum perturbation limit of adversarial samples, Queries represents the number of model queries, Time represents the attack time consumption, and λ1, λ2, λ3, and λ4 represent their respective weight coefficients.

[0159] This embodiment obtains recognition performance test results by performing recognition performance test on the model to be tested; obtains simulated access conditions of the model to be tested, and the simulated access conditions include three conditions: fully obtaining model parameters and output information, obtaining complete output information but unable to access model parameters, and obtaining output category information but unable to access model parameters; selects a target model from the preset white box model or the preset black box model according to the simulated access conditions to perform robust performance test on the model to be tested to obtain robust performance test results; performs backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain backdoor risk test results; performs a comprehensive security analysis on the model to be tested according to the performance test results, the robust performance test results, and the model backdoor risk test results. Compared with the prior art, which has not yet established a unified deep learning model security assessment specification, the single evaluation dimension leads to poor availability, ease of use, and other aspects of the model security performance assessment method. This embodiment realizes multi-dimensional evaluation of model security performance, which can comprehensively guarantee the model's security performance and improve the model's security performance.

[0160] Reference Figure 6 , Figure 6 This is a flow chart of the third embodiment of the model security assessment method of the present invention. Figure 2 The first embodiment shown provides a second embodiment of the model security assessment method of the present invention.

[0161] In this embodiment, step S30 includes:

[0162] Step S301: Obtain the access status of the model to be tested.

[0163] It should be noted that the access status includes two states: only obtaining the output layer weight parameter information of the model to be tested and accessing the model to be tested but only obtaining the output category information.

[0164] Step S302: determining a test scenario according to the access status, selecting a target model from the preset white box model or the preset black box model according to the test scenario, performing a backdoor risk test on the model to be tested, and obtaining a backdoor risk test result.

[0165] It is understandable that for the deep learning model backdoor risk assessment and analysis, the model backdoor risk assessment in this embodiment will be divided into white box scenarios and black box scenarios according to the access status of the model to be tested. Then, the model to be tested is analyzed by the model backdoor detection method to determine whether there is a backdoor risk in the model to be tested. If there is a backdoor risk, the target attack category is given.

[0166] Furthermore, the step S302 also includes: when the access state is to only obtain the weight parameter information of the output layer of the model to be tested, determining that the test scenario is a white box scenario, and taking the preset white box model as the target model; performing a backdoor risk test on the model to be tested according to the preset white box model and the preset TPR model evaluation method to obtain a backdoor risk test result.

[0167] It should be noted that the preset TPR model evaluation method refers to a pre-set method for statistically analyzing the attack hit rate. Specifically, the model to be tested can be tested for backdoor risk using the preset white box model and the preset TPR model evaluation method, and the attack category and number in the backdoor risk test results can be statistically analyzed to determine whether the model to be tested has a backdoor risk. If so, the attack category is output.

[0168] Furthermore, the step of performing a backdoor risk test on the model to be tested according to the preset white box model and the preset TPR model evaluation method to obtain a backdoor risk test result includes: performing constraint solving on the model to be tested according to the preset white box model and the preset TPR model evaluation method to obtain weight distribution information of the model output layer; judging whether the model to be tested has a backdoor risk according to the weight distribution information and a preset anomaly detection function to obtain a backdoor risk test result.

[0169] In the specific implementation, for the white-box scenario: when only the weight parameter information of the output layer of the model to be tested is obtained, the TPR method will be used to construct the input data of the model output layer by using the generative model based on mutual information combined with the kurtosis loss function, and then the attack category logits vector kurtosis value is greater than other categories and the attacked category ||mask*pattern|| p Based on the two assumptions that the norm is as small as possible, constraints are solved to obtain the mask and pattern vectors corresponding to the weight distribution of potential triggers in the model output layer, and combined with the anomaly detection function based on the Kurtosis-Norm score to output whether the model to be detected has a backdoor risk. If there is a backdoor, the attacked category is output.

[0170] Furthermore, the step S302 also includes: when the access state is to access the model to be tested but only obtain output category information, determining that the test scenario is a black box scenario, and taking the preset black box model as the target model; performing a backdoor risk test on the model to be tested according to the preset black box model and the preset AEVA backdoor attack method to obtain a backdoor risk test result.

[0171] It should be noted that when the test scenario is a black box scenario, when the model to be tested can be accessed but only the output category information can be obtained, the AEVA backdoor attack method will be used. The adversarial samples are calculated through the Monte Carlo algorithm, and the global adversarial peak is obtained based on the adversarial singularity phenomenon analysis. Then, the Median Absolute Deviation (MAD) is used to find outliers to determine the existence of backdoor risks and their target labels.

[0172] This embodiment performs a recognition performance test on the model to be tested to obtain a recognition performance test result; obtains a simulated access situation of the model to be tested, and the simulated access situation includes three situations: completely obtaining model parameters and output information, obtaining complete output information but unable to access model parameters, and obtaining output category information but unable to access model parameters; selects a target model from the preset white box model or the preset black box model according to the simulated access situation to perform a robust performance test on the model to be tested to obtain a robust performance test result; obtains the access status of the model to be tested, and the access status includes only obtaining the output layer weight parameter information of the model to be tested and accessing the model to be tested but only obtaining the output category information. two states; determining a test scenario according to the access status, selecting a target model from the preset white box model or the preset black box model according to the test scenario to perform a backdoor risk test on the model to be tested, and obtaining a backdoor risk test result; performing a comprehensive security analysis on the model to be tested according to the performance test results, the robust performance test results, and the model backdoor risk test results. Compared with the prior art, a unified deep learning model security assessment specification has not yet been established, and the evaluation dimension is single, resulting in poor availability, ease of use, and other aspects of the model security performance assessment method. This embodiment realizes a multi-dimensional assessment of the model security performance, which can comprehensively guarantee the model's security performance and improve the model's security performance.

[0173] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a model security assessment program is stored, and when the model security assessment program is executed by a processor, the steps of the model security assessment method described above are implemented.

[0174] Reference Figure 7 , Figure 7 It is a structural block diagram of the first embodiment of the model safety assessment device of the present invention.

[0175] like Figure 7 As shown, the model security assessment device proposed in the embodiment of the present invention includes:

[0176] The recognition performance evaluation module 10 is used to perform a recognition performance test on the model to be tested and obtain a recognition performance test result;

[0177] A robust performance evaluation module 20, configured to perform a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result;

[0178] A backdoor risk assessment module 30 is used to perform a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result;

[0179] The security comprehensive analysis module 40 is used to perform a security comprehensive analysis on the model to be tested according to the performance test results, the robust performance test results and the model backdoor risk test results.

[0180] This embodiment obtains recognition performance test results by performing recognition performance test on the model to be tested; performs robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain robust performance test results; performs backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain backdoor risk test results; and performs a comprehensive security analysis on the model to be tested based on the performance test results, the robust performance test results and the model backdoor risk test results. Compared with the prior art, which has not yet established a unified deep learning model security assessment specification and has a single assessment dimension, resulting in poor availability, ease of use and other aspects of the model security performance assessment method, this embodiment implements multi-dimensional assessment of model security performance, which can comprehensively guarantee the model's security performance and improve the model's security performance.

[0181] Other embodiments or specific implementations of the model security assessment device of the present invention can refer to the above-mentioned method embodiments, which will not be described in detail here.

[0182] The present invention provides A1, a model security assessment method, the model security assessment method comprising the following steps:

[0183] Perform recognition performance test on the model to be tested and obtain recognition performance test results;

[0184] Performing a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result;

[0185] Performing a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result;

[0186] A comprehensive security analysis is performed on the model to be tested based on the performance test results, the robust performance test results and the model backdoor risk test results.

[0187] A2. In the model security assessment method described in A1, the step of performing a recognition performance test on the model to be tested and obtaining a recognition performance test result comprises:

[0188] Perform data augmentation on the preset initial test set to construct a generalized test set;

[0189] Performing a recognition performance test on the model to be tested based on the preset initial test set to obtain a first ACC indicator;

[0190] Performing a recognition performance test on the model to be tested based on the generalized test set to obtain a second ACC indicator;

[0191] A recognition performance test result is determined according to the first ACC indicator and the second ACC indicator.

[0192] A3. In the model security assessment method described in A2, the step of performing a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result comprises:

[0193] Acquire simulated access conditions of the model to be tested, wherein the simulated access conditions include three conditions: completely acquiring model parameters and output information, acquiring complete output information but unable to access model parameters, and acquiring output category information but unable to access model parameters;

[0194] A target model is selected from the preset white box model or the preset black box model according to the simulated access situation to perform a robust performance test on the model to be tested, so as to obtain a robust performance test result.

[0195] A4. The model security assessment method as described in A3, wherein the step of selecting a target model from the preset white box model or the preset black box model according to the simulated access situation to perform a robust performance test on the model to be tested, and obtaining a robust performance test result comprises:

[0196] When the simulated access situation is to obtain complete output information but unable to access model parameters, selecting the preset white box model as the target model;

[0197] Generate an adversarial sample according to the preset white box model and the initial test set;

[0198] A robust performance test is performed on the model to be tested according to the adversarial sample to obtain a robust performance test result corresponding to a preset multi-dimensional indicator.

[0199] A5. The model security assessment method as described in A3, wherein the step of selecting a target model from the preset white box model or the preset black box model according to the simulated access situation to perform a robust performance test on the model to be tested and obtaining a robust performance test result comprises:

[0200] When the simulated access situation is to completely obtain model parameters and output information, selecting the preset black box model as the target model;

[0201] Generate an adversarial sample according to the preset black box model and the initial test set;

[0202] A robust performance test is performed on the model to be tested according to the adversarial sample to obtain a robust performance test result corresponding to a preset multi-dimensional indicator.

[0203] A6. The model security assessment method as described in A3, wherein the step of selecting a target model from the preset white box model or the preset black box model according to the simulated access situation to perform a robust performance test on the model to be tested and obtaining a robust performance test result comprises:

[0204] When the simulated access situation is to obtain output category information but cannot access model parameters, selecting the preset black box model as the target model;

[0205] Generate an adversarial sample according to the preset black box model and the initial test set;

[0206] The model to be tested is subjected to a robust performance test according to the adversarial sample to obtain robust performance test results corresponding to the attack success rate dimension, the maximum disturbance limit dimension, the model query number dimension, and the attack consumption dimension.

[0207] A7. In the model security assessment method described in A1, the step of performing a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result comprises:

[0208] Obtaining the access status of the model to be tested, wherein the access status includes two states: only obtaining the weight parameter information of the output layer of the model to be tested and accessing the model to be tested but only obtaining the output category information;

[0209] A test scenario is determined according to the access status, and a target model is selected from the preset white box model or the preset black box model according to the test scenario to perform a backdoor risk test on the model to be tested to obtain a backdoor risk test result.

[0210] A8. The model security assessment method as described in A7, wherein the step of determining a test scenario according to the access status, selecting a target model from the preset white box model and the preset black box model according to the test scenario to perform a backdoor risk test on the model to be tested, and obtaining a backdoor risk test result comprises:

[0211] When the access state is to obtain only the weight parameter information of the output layer of the model to be detected, determining that the test scenario is a white box scenario, and using the preset white box model as the target model;

[0212] A backdoor risk test is performed on the model to be tested according to the preset white box model and the preset TPR model evaluation method to obtain a backdoor risk test result.

[0213] A9. The model security assessment method as described in A8, wherein the step of performing a backdoor risk test on the model to be tested according to the preset white box model and the preset TPR model assessment method to obtain a backdoor risk test result comprises:

[0214] Solve the constraints of the model to be tested according to the preset white box model and the preset TPR model evaluation method to obtain the weight distribution information of the model output layer;

[0215] It is determined whether the model to be tested has a backdoor risk based on the weight distribution information and a preset anomaly detection function, and a backdoor risk test result is obtained.

[0216] A10. The model security assessment method as described in A7, wherein the step of determining a test scenario according to the access status, selecting a target model from the preset white box model and the preset black box model according to the test scenario to perform a backdoor risk test on the model to be tested, and obtaining a backdoor risk test result comprises:

[0217] When the access state is accessing the model to be tested but only obtaining output category information, determining that the test scenario is a black box scenario, and using the preset black box model as a target model;

[0218] A backdoor risk test is performed on the model to be tested according to the preset black box model and the preset AEVA backdoor attack method to obtain a backdoor risk test result.

[0219] A11. The model security assessment method as described in A7, wherein after the step of performing a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model and obtaining the backdoor risk test result, the method further comprises:

[0220] When the backdoor risk test result indicates that there is a backdoor risk, a target attack category corresponding to the backdoor risk is determined.

[0221] A12. The model security assessment method as described in A1, after the step of performing a comprehensive security analysis on the model to be tested according to the performance test results, the robust performance test results and the model backdoor risk test results, further comprises:

[0222] Obtaining an overall safety performance score corresponding to the model to be tested;

[0223] An evaluation report is generated based on the overall security performance score and the various dimensional indicators included in the performance test results, the robust performance test results, and the model backdoor risk test results.

[0224] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0225] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as names.

[0226] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0227] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A model security assessment method, characterized in that: The model security assessment method comprises the following steps: Perform recognition performance test on the model to be tested and obtain recognition performance test results; The step of performing a recognition performance test on the model to be tested and obtaining a recognition performance test result includes: performing data enhancement on a preset initial test set to construct a generalized test set; performing a recognition performance test on the model to be tested based on the preset initial test set to obtain a first ACC indicator; performing a recognition performance test on the model to be tested based on the generalized test set to obtain a second ACC indicator; determining the recognition performance test result according to the first ACC indicator and the second ACC indicator, wherein the model security assessment method supports uploading and processing of CV field data sets, and the preset initial test set is a clean test set uploaded by the user according to the task type of the model to be tested; The generalization test set includes calling image transformation to construct a set of generalization test sets D1 after flipping and scaling; calling pixel transformation to construct a set of generalization test sets D2 with various noises added; calling natural scene transformation to construct a set of generalization test sets D3 with natural noise added; calling style transfer network to construct a set of generalization test sets D4 with artistic style; In the evaluation of model recognition performance, the ACC performance of the model to be tested on multiple groups of test data is used as an evaluation indicator. The higher the ACC on each group of test data, the better the recognition performance of the model for normal data and generalized data. The evaluation score calculation formula of the recognition performance of the model to be tested is expressed as follows: Score acc =α0*ACC D0 +α1*ACC D1 +α2*ACC D2 +α3*ACC D3 +α4*ACC D4 Among them, Score acc Indicates the recognition performance score of the model to be tested, ACC D0 represents the recognition performance of the model to be tested on the preset initial test set D0, ACC D1 represents the recognition performance of the model to be tested on the generalized test set D1, ACC D2 represents the recognition performance of the model to be tested on the generalized test set D2, ACC D3 represents the recognition performance of the model to be tested on the generalized test set D3, ACC D4 represents the recognition performance of the model to be tested on the generalized test set D4, α0, α1, α2, α3, α4, respectively represent the corresponding weight coefficients; The overall score of the safety performance of the model to be tested is: Score=w1*Score acc +w2*Score adv +w3*Score back ; Among them, Score is the overall score of the safety performance of the model to be tested. adv Indicates the robust performance score of the model to be tested, Score back represents the backdoor risk score of the model to be tested, w1, w2, and w3 represent the weight coefficients of the three scores respectively. The risk of counterattack is a security threat that every model will face, so the weight coefficient w2 accounts for the largest proportion in the security score calculation formula of the model to be tested; Performing a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result; The step of performing a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result includes: Selecting a target model from a preset white box model or a preset black box model based on the simulated access situation of the model to be tested to perform a robust performance test on the model to be tested, and obtaining a robust performance test result, wherein the simulated access situation includes completely obtaining model parameters and output information, obtaining complete output information but failing to access model parameters, and obtaining output category information but failing to access model parameters; Performing a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result; The step of performing a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result includes: Obtaining the access status of the model to be tested; When the access state is to obtain only the weight parameter information of the output layer of the model to be tested, determining that the test scenario is a white box scenario, and taking the preset white box model as the target model; performing a backdoor risk test on the model to be tested according to the preset white box model and the preset TPR model evaluation method, and obtaining a backdoor risk test result; Or, when the access state is to access the model to be tested but only obtain output category information, determine that the test scenario is a black box scenario, and use the preset black box model as the target model; perform a backdoor risk test on the model to be tested according to the preset black box model and the preset AEVA backdoor attack method to obtain a backdoor risk test result; A comprehensive security analysis is performed on the model to be tested based on the performance test results, the robust performance test results and the model backdoor risk test results.

2. The model security assessment method according to claim 1, characterized in that: After the step of performing a comprehensive security analysis on the model to be tested according to the performance test result, the robust performance test result and the model backdoor risk test result, the method further includes: Obtaining an overall safety performance score corresponding to the model to be tested; Generate an evaluation report based on the overall security performance score and various dimensional indicators included in the performance test results, the robust performance test results, and the model backdoor risk test results, wherein the evaluation report includes protection enhancement suggestions for the model to be tested corresponding to various angles of active defense and passive defense; If the user adopts the protection enhancement suggestion, robust adversarial training in active defense and abnormal sample detection in passive defense are performed on the model to be tested; The step of performing a robust performance test on the model to be tested and obtaining a robust performance test result includes: taking the model to be tested as the attacked model and the initial test set as the original clean sample, and using a white-box attack or black-box attack algorithm to generate a corresponding adversarial sample; and determining a robust performance score of an indicator value corresponding to the following calculation steps, and using the robust performance score as a robust performance test result: calculating the success rate ASR when the adversarial sample attacks the model to be tested; calculating the maximum perturbation limit Epsilon added to the adversarial sample when the attack success rate reaches a threshold; calculating the time consumption Time used to generate the adversarial sample when the attack success rate reaches the threshold; calculating the number of attack iterations Iters or the number of queries to the model Queries when the attack success rate reaches the threshold; and calculating the change Conf in the model's prediction confidence of the true labels of clean samples and adversarial samples.

3. The model security assessment method according to claim 2, characterized in that: The step of performing a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result includes: When the simulated access situation is to obtain complete output information but cannot access model parameters, the preset white box model is selected as the target model, and the model robustness performance of the model to be tested is comprehensively evaluated by combining at least one indicator based on gradient, optimization of different types and different attack intensities, attack success rate, maximum perturbation limit, and model query times. Among them, the gradient-based attack method can quickly generate adversarial samples of various intensities. The model robustness evaluation score calculation formula is as follows: Score adv =λ1*ASR+λ2*Epsilon+λ3*Iters+λ4* Time+λ5*Conf; Among them, ASR represents the success rate of adversarial sample attack, Epsilon represents the maximum perturbation limit of adversarial samples, Iters represents the number of attack iterations, Time represents the attack time consumption, Conf represents the confidence change of the true label of the sample before and after the attack, and λ1, λ2, λ3, λ4, and λ5 represent their respective weight coefficients.

4. The model security assessment method according to claim 2, characterized in that: The step of performing a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result includes: When the simulated access situation is to fully obtain the model parameters and output information, the preset black box model is selected as the target model, and the score-based adversarial attack method is used to comprehensively evaluate the model robustness performance of the model to be tested, combining the attack success rate, the maximum perturbation limit, and at least one indicator of the number of model queries. At this time, only the soft label of the model output can be obtained, so the score-based adversarial attack method is used to estimate the model gradient to generate adversarial samples, thereby calculating various evaluation indicators. The model robustness evaluation score calculation formula in this scenario is as follows: Score adv =λ1*ASR+λ2*Epsilon+λ6*Queries+λ4* Time+λ5*Conf; Among them, ASR represents the success rate of adversarial sample attack, Epsilon represents the maximum perturbation limit of adversarial samples, Queries represents the number of model queries, Time represents the attack time consumption, Conf represents the confidence change of the true label of the sample before and after the attack, and λ1, λ2, λ6, λ4, and λ5 represent their respective weight coefficients.

5. The model security assessment method according to claim 2, characterized in that: The step of performing a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result includes: When the simulated access situation is to obtain output category information but cannot access model parameters, the preset black box model is selected as the target model, and the model robustness performance of the model to be tested is comprehensively evaluated by using the adversarial attack method based on the decision boundary, combined with the attack success rate, the maximum perturbation limit, and at least one indicator of the number of model queries. In this scenario, only the hard label of the model output can be obtained. Therefore, the adversarial attack method based on the decision boundary is adopted. By continuously accessing the model prediction value, the decision boundary is gradually approached to generate adversarial samples, thereby calculating various evaluation indicators. Since the model's prediction probability for the sample cannot be obtained, the change in the true label confidence cannot be evaluated in this scenario. This scenario is the most suitable for the actual adversarial attack scenario. The model robustness evaluation score calculation formula is as follows: Score adv =λ1*ASR+λ2*Epsilon+λ6*Queries+λ4* Time; Among them, ASR represents the success rate of adversarial sample attacks, Epsilon represents the maximum perturbation limit of adversarial samples, Queries represents the number of model queries, Time represents the attack time consumption, and λ1, λ2, λ6, and λ4 represent their respective weight coefficients.

6. The model security assessment method according to claim 2, characterized in that: The step of selecting a target model from a preset white box model or a preset black box model based on the simulated access situation of the model to be tested to perform a robust performance test on the model to be tested and obtaining a robust performance test result includes: When the simulated access situation is to obtain complete output information but unable to access model parameters, selecting the preset white box model as the target model; Generate an adversarial sample according to the preset white box model and the initial test set; A robust performance test is performed on the model to be tested according to the adversarial sample to obtain a robust performance test result corresponding to a preset multi-dimensional indicator.

7. The model security assessment method according to claim 6, characterized in that: The step of selecting a target model from a preset white box model or a preset black box model based on the simulated access situation of the model to be tested to perform a robust performance test on the model to be tested and obtaining a robust performance test result includes: When the simulated access situation is to completely obtain model parameters and output information, selecting the preset black box model as the target model; Generate an adversarial sample according to the preset black box model and the initial test set; A robust performance test is performed on the model to be tested according to the adversarial sample to obtain a robust performance test result corresponding to a preset multi-dimensional indicator.

8. The model security assessment method according to claim 7, characterized in that: The step of selecting a target model from a preset white box model or a preset black box model based on the simulated access situation of the model to be tested to perform a robust performance test on the model to be tested and obtaining a robust performance test result includes: When the simulated access situation is to obtain output category information but cannot access model parameters, selecting the preset black box model as the target model; Generate an adversarial sample according to the preset black box model and the initial test set; The model to be tested is subjected to a robust performance test according to the adversarial sample to obtain robust performance test results corresponding to the attack success rate dimension, the maximum disturbance limit dimension, the model query number dimension, and the attack consumption dimension.

9. The model security assessment method according to claim 1, characterized in that: The step of performing a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result includes: Obtaining the access status of the model to be tested, wherein the access status includes two states: only obtaining the weight parameter information of the output layer of the model to be tested and accessing the model to be tested but only obtaining the output category information; A test scenario is determined according to the access status, and a target model is selected from the preset white box model or the preset black box model according to the test scenario to perform a backdoor risk test on the model to be tested to obtain a backdoor risk test result.

10. A model security assessment device, characterized in that: The model security assessment device comprises: a memory, a processor, and a model security assessment program stored in the memory and executable on the processor, wherein the model security assessment program implements the model security assessment method according to any one of claims 1 to 9 when executed by the processor.

11. A storage medium, characterized in that: The storage medium stores a model security assessment program, which, when executed by a processor, implements the model security assessment method according to any one of claims 1 to 9.

12. A model security assessment device, characterized in that: The model security assessment device is applied to the model security assessment method according to any one of claims 1 to 9, and the model security assessment includes: The recognition performance evaluation module is used to perform recognition performance test on the model to be tested and obtain recognition performance test results; A robust performance evaluation module, used to perform a robust performance test on the model to be tested based on a preset white box model or a preset black box model to obtain a robust performance test result; A backdoor risk assessment module, used to perform a backdoor risk test on the model to be tested based on the preset white box model or the preset black box model to obtain a backdoor risk test result; The security comprehensive analysis module is used to perform a security comprehensive analysis on the model to be tested based on the performance test results, the robust performance test results and the model backdoor risk test results.

Images