AI model safety protection and confrontation verification method and system

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an AI model safety protection and confrontation verification method and system. The method comprises the following steps: uploading a to-be-detected AI model and a test sample; performing data management on the AI model to be detected and the test sample; outputting a detection instruction; and based on the detection instruction, generating an adversarial sample according to the test sample, and performing security detection on the to-be-detected AI model according to the adversarial sample to obtain a detection result. Therefore, the safety of the model can be guaranteed, the rapid evaluation capability is provided for the research and development of the safety model, and the safety construction of the model is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an AI model security protection and adversarial verification method and system. Background Art

[0002] The rapid development of large AI models has also led to many security issues, especially in image recognition models that use algorithmic tools to achieve recognition errors and recognition deceptions, and in large language models that have security vulnerabilities and language cloning.

[0003] In open source model training, training based on different sample data can achieve preliminary model recognition effects, but it is impossible to solve the model's security vulnerabilities through training. In order to solve this problem, it is necessary to use adversarial models to complete the model's security verification, but this method has technical difficulties in practical applications and will increase learning costs and R&D time costs. In addition, new attack methods such as adversarial samples and model backdoors will cause artificial intelligence models to execute incorrect instructions, which may eventually cause serious security consequences. Summary of the invention

[0004] The present invention provides an AI model security protection and adversarial verification method and system, which can solve the problems in the prior art.

[0005] The present invention provides an AI model security protection and adversarial verification method, wherein the method comprises:

[0006] Upload the AI ​​model and test samples to be tested;

[0007] Perform data management on AI models and test samples to be tested;

[0008] Output detection instructions;

[0009] Based on the detection instructions, adversarial samples are generated according to the test samples, and security detection is performed on the AI ​​model to be detected based on the adversarial samples to obtain the detection results.

[0010] Preferably, data management of the AI ​​model to be tested and the test sample includes: data storage of model data of the AI ​​model to be tested and test data of the test sample.

[0011] Preferably, generating an adversarial sample according to a test sample includes:

[0012] Add noise or perturbation algorithms to test samples to generate adversarial samples.

[0013] Preferably, security detection is performed on the AI ​​model to be detected based on the adversarial sample, and the detection results obtained include:

[0014] S100, prepare the service to be tested;

[0015] S102, determine whether to use a custom data set for evaluation, if yes, go to S104, otherwise go to S106;

[0016] S104, create a data set, and go to S106;

[0017] S106, configure the evaluation plan;

[0018] S108, perform security testing on the AI ​​model to be tested based on the evaluation plan and adversarial samples;

[0019] S110, obtaining the detection result.

[0020] Preferably, the method further comprises: outputting the detection result.

[0021] The present invention also provides an AI model security protection and adversarial verification system, wherein the system comprises:

[0022] The user end is used to upload the AI ​​model and test samples to be tested;

[0023] Model business platform, used to manage data of AI models and test samples to be tested and output test instructions;

[0024] The model security detection platform is used to generate adversarial samples based on test samples based on detection instructions, and perform security detection on the AI ​​model to be detected based on the adversarial samples to obtain detection results.

[0025] Preferably, data management of the AI ​​model to be tested and the test sample includes: data storage of model data of the AI ​​model to be tested and test data of the test sample.

[0026] Preferably, generating an adversarial sample according to a test sample includes:

[0027] Add noise or perturbation algorithms to test samples to generate adversarial samples.

[0028] Preferably, security detection is performed on the AI ​​model to be detected based on the adversarial sample, and the detection results obtained include:

[0029] S100, prepare the service to be tested;

[0030] S102, determine whether to use a custom data set for evaluation, if yes, go to S104, otherwise go to S106;

[0031] S104, create a data set, and go to S106;

[0032] S106, configure the evaluation plan;

[0033] S108, perform security testing on the AI ​​model to be tested based on the evaluation plan and adversarial samples;

[0034] S110, obtaining the detection result.

[0035] Preferably, the model safety detection platform is also used to output detection results.

[0036] Through the above technical solution, the model and test sample data can be uploaded, the business platform completes data management, and the test platform generates adversarial samples based on the test sample data, determines the adversarial algorithm, and then performs security testing on the model to generate test results. In this way, the model security can be guaranteed, and rapid evaluation capabilities can be provided for the development of security models to ensure the construction of model security. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A flowchart of an AI model security protection and adversarial verification method according to an embodiment of the present invention is shown;

[0039] Figure 2 An exemplary flow chart of security detection in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0042] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0043] Figure 1 A flowchart of an AI model security protection and adversarial verification method according to an embodiment of the present invention is shown.

[0044] like Figure 1 As shown, an embodiment of the present invention provides an AI model security protection and adversarial verification method, wherein the method includes:

[0045] Upload the AI ​​model and test samples to be tested (test samples);

[0046] Perform data management on AI models and test samples to be tested;

[0047] Output detection instructions, wherein the instructions include detection function settings and model detection settings;

[0048] Based on the detection instructions, adversarial samples are generated according to the test samples, and security detection is performed on the AI ​​model to be detected based on the adversarial samples to obtain the detection results.

[0049] Among them, the AI ​​model can be an AI big model.

[0050] Through the above technical solution, the model and test sample data can be uploaded, the business platform completes data management, and the test platform generates adversarial samples based on the test sample data, determines the adversarial algorithm, and then performs security testing on the model to generate test results. In this way, the model security can be guaranteed, and rapid evaluation capabilities can be provided for the development of security models to ensure the construction of model security.

[0051] According to an embodiment of the present invention, data management of the AI ​​model to be detected and the test sample includes: data storing model data of the AI ​​model to be detected and test data of the test sample.

[0052] Before data storage is performed, data preprocessing and data filtering may be performed on the data to ensure data security.

[0053] According to an embodiment of the present invention, generating an adversarial sample according to a test sample includes:

[0054] Add noise or perturbation algorithms to test samples to generate adversarial samples.

[0055] According to an embodiment of the present invention, security detection is performed on an AI model to be detected based on adversarial samples, and the detection results obtained include:

[0056] S100, prepare the service to be tested;

[0057] S102, determine whether to use a custom data set for evaluation, if yes, go to S104, otherwise go to S106;

[0058] S104, create a data set, and go to S106;

[0059] S106, configure the evaluation plan;

[0060] S108, perform security testing on the AI ​​model to be tested according to the evaluation plan and adversarial samples (submit the evaluation task);

[0061] S110, obtain the test result (obtain the test report).

[0062] Among them, in the case of uncertainty of the type of security threat corresponding to the evaluation, the security risk type can be determined before S106. If the security risk type is adversarial sample attack, then go to S106; if the security risk type is model backdoor attack, then directly perform the security detection corresponding to the model backdoor attack (submit the evaluation task), and then obtain the evaluation result, such as Figure 2 shown.

[0063] That is, when the security threat is an adversarial sample attack, the platform supports multiple algorithms to generate adversarial samples for testing. During the test, the model effect changes under different algorithms, iteration times, and disturbance size attacks can be combined to obtain the model's security score. When the security threat is a model backdoor attack, the platform can search and restore the final result of the backdoor trigger for each category of the model, and determine whether the model has been implanted with a backdoor based on the discrete degree of the restored result. In this way, the model's security against adversarial sample attacks can be effectively evaluated and whether the model has been implanted with a backdoor can be detected.

[0064] More specifically, in the step of preparing the service to be tested, before performing the evaluation, you can first prepare the service to be evaluated and the data used for the evaluation. The present invention supports the use of the system's built-in data sets for evaluation. If the user needs a customized data set for evaluation, you can also create a customized data set for evaluation as required. In the subsequent evaluation link, the security of the model can be tested by submitting an evaluation task. When the evaluation security threat is an adversarial sample attack, the present invention supports the configuration of key parameters such as the evaluation method and the size of the disturbance that directly affect the success rate of the adversarial sample attack. After submitting the evaluation task, the platform will automatically evaluate the security of the AI ​​model and output an evaluation report after the evaluation is completed. When the evaluation security threat is a model backdoor attack, you can directly submit the service to be evaluated and select the corresponding data set to obtain the evaluation report after the evaluation is completed.

[0065] According to an embodiment of the present invention, the method further includes: outputting the detection result.

[0066] This makes it convenient for users to view the test report.

[0067] The AI ​​model security protection and adversarial verification method described in the present invention is described below with reference to examples.

[0068] In this example, the AI ​​model takes the image recognition model as an example: upload the trained image classification model, upload the test sample (for example, a cat), add noise or perturbation algorithm (for example, a hash function) that is invisible to the human eye to the uploaded test sample, and use the processed test sample to verify the image recognition model. The success rate is returned. If the recognition is a cat, it indicates that the recognition result is correct and the model detection passes. If the success rate is less than 90% or other animals are recognized, it indicates that the recognition result is incorrect and the model test fails.

[0069] An embodiment of the present invention further provides an AI model security protection and adversarial verification system, wherein the system includes:

[0070] The user end is used to upload the AI ​​model and test samples to be tested;

[0071] Model business platform (e.g., large model business platform), used to manage data of AI models and test samples to be tested and output test instructions;

[0072] A model security detection platform (e.g., a large model security detection platform) is used to generate adversarial samples based on test samples based on detection instructions, and perform security detection on the AI ​​model to be detected based on the adversarial samples to obtain detection results.

[0073] Through the above technical solution, the model and test sample data can be uploaded, the business platform completes data management, and the test platform generates adversarial samples based on the test sample data, determines the adversarial algorithm, and then performs security testing on the model to generate test results. In this way, the model security can be guaranteed, and rapid evaluation capabilities can be provided for the development of security models to ensure the construction of model security.

[0074] According to an embodiment of the present invention, data management of the AI ​​model to be detected and the test sample includes: data storing model data of the AI ​​model to be detected and test data of the test sample.

[0075] According to an embodiment of the present invention, generating an adversarial sample according to a test sample includes:

[0076] Add noise or perturbation algorithms to test samples to generate adversarial samples.

[0077] According to an embodiment of the present invention, security detection is performed on an AI model to be detected based on adversarial samples, and the detection results obtained include:

[0078] S100, prepare the service to be tested;

[0079] S102, determine whether to use a custom data set for evaluation, if yes, go to S104, otherwise go to S106;

[0080] S104, create a data set, and go to S106;

[0081] S106, configure the evaluation plan;

[0082] S108, perform security testing on the AI ​​model to be tested based on the evaluation plan and adversarial samples;

[0083] S110, obtaining the detection result.

[0084] According to an embodiment of the present invention, the model safety detection platform is also used to output detection results.

[0085] The above system and Figure 1 The method described above corresponds to the specific example. Figure 1 The description of the method will not be repeated here.

[0086] The AI ​​model security protection and adversarial verification method and system described in the present invention solve the problem of evaluating the security capabilities of large models in network security applications: 1. By analyzing the data set for large model training, preprocessing the data, and filtering the data, it is ensured that the data used to train the large model is safe and meets the requirements. 2. After the large model training is completed, the arranged test data is subjected to repeated verification attacks to autonomously discover security vulnerabilities, thereby optimizing the training network model and ensuring model security. 3. The platform uses basic model training samples (test samples) and adds a perturbation algorithm to process sample data to perform sample security verification and testing to achieve security adversarial verification of the model. The above processing is used to ensure the security of large models, and to provide rapid evaluation capabilities for the development of secure large models to ensure the safe construction of large models.

[0087] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the devices or elements referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention. The directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0088] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0089] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An AI model security protection and adversarial verification method, characterized in that: The method includes: Upload the AI ​​model and test samples to be tested; Perform data management on AI models and test samples to be tested; Output detection instructions; Based on the detection instructions, adversarial samples are generated according to the test samples, and security detection is performed on the AI ​​model to be detected based on the adversarial samples to obtain the detection results.

2. The method according to claim 1, characterized in that Data management of the AI ​​model to be tested and the test samples includes: data storage of model data of the AI ​​model to be tested and test data of the test samples.

3. The method according to claim 2, characterized in that Generating adversarial samples based on test samples includes: Add noise or perturbation algorithms to test samples to generate adversarial samples.

4. The method according to claim 3, characterized in that The AI ​​model to be tested is tested for security based on adversarial samples, and the test results include: S100, prepare the service to be tested; S102, determine whether to use a custom data set for evaluation, if yes, go to S104, otherwise go to S106; S104, create a data set, go to S106; S106, configure the evaluation plan; S108, perform security testing on the AI ​​model to be tested based on the evaluation plan and adversarial samples; S110, obtaining the detection result.

5. The method according to any one of claims 1 to 4, characterized in that The method also includes: outputting the detection result.

6. An AI model security protection and adversarial verification system, characterized in that: The system includes: The user end is used to upload the AI ​​model and test samples to be tested; Model business platform, used to manage data of AI models and test samples to be tested and output test instructions; The model security detection platform is used to generate adversarial samples based on test samples based on detection instructions, and perform security detection on the AI ​​model to be detected based on the adversarial samples to obtain detection results.

7. The system according to claim 6, characterized in that Data management of the AI ​​model to be tested and the test samples includes: data storage of model data of the AI ​​model to be tested and test data of the test samples.

8. The system according to claim 7, characterized in that Generating adversarial samples based on test samples includes: Add noise or perturbation algorithms to test samples to generate adversarial samples.

9. The system according to claim 8, characterized in that The AI ​​model to be tested is tested for security based on adversarial samples, and the test results include: S100, prepare the service to be tested; S102, determine whether to use a custom data set for evaluation, if yes, go to S104, otherwise go to S106; S104, create a data set, go to S106; S106, configure the evaluation plan; S108, perform security testing on the AI ​​model to be tested based on the evaluation plan and adversarial samples; S110, obtaining the detection result.

10. The system according to any one of claims 6 to 9, characterized in that: The model safety testing platform is also used to output testing results.

Images