Method and device for evaluating security risks of large models based on training scenarios

Abstract

The application relates to the technical field of artificial intelligence, in particular to a large model security risk evaluation method and device based on a training scene, wherein the method comprises the following steps: fine-tuning training is performed based on training scene information of a target large model to obtain a fine-tuning model; an attack instruction is used to attack the target large model to obtain an attacked model and a poisoned sample; the attacked model is subjected to defense processing based on a defense instruction and the poisoned sample, and the attacked model after the defense processing is attacked according to the attack instruction to obtain a poisoned model; a security risk score is generated by using the fine-tuning model and the poisoned model based on a target evaluation index, and the security risk level of the target large model is evaluated according to the security risk score. According to the application, the security risk score of the large model can be obtained by attacking and defense training of the large model, the security risk of the model can be directly measured, the optimization and improvement of the security defense level of the large model can be further realized, and the model is more stable and reliable.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large-scale model security risk assessment method and device based on training scenarios. Background Art

[0002] Large-scale language models have characteristics such as large amounts of training data, large number of model parameters, open source model parameter structure, and strong generalization ability of shallow feature representation. These characteristics also make the losses caused by large models more serious when facing security attacks. Therefore, security risk assessment of large models is particularly important.

[0003] Among the related technologies, the most representative security attack methods at this stage include adversarial attacks, backdoor attacks, and poisoning attacks. These can be used to evaluate the performance of the model after it is subjected to security attacks in various scenarios based on the security risks that the model may face.

[0004] However, related technologies can only evaluate smaller-scale models and are not applicable to large-scale model application scenarios. There is a lack of security risk assessment methods for large-scale language models, and the attack scenarios supported by risk assessment are relatively simple. It is impossible to evaluate the user's local models, data, attack methods, and defense methods. It is difficult to obtain model security assessment comparisons of different frameworks through unified risk assessment standards, which affects the protection level of large models and needs to be solved urgently. Summary of the Invention

[0005] This application provides a large-scale model security risk assessment method and device based on training scenarios to solve the problems that related technologies can only realize the assessment of small-scale models and are not applicable to large-scale model application scenarios. There is a lack of security risk assessment methods for large-scale language models, and the attack scenarios supported by risk assessment are relatively single. It is impossible to evaluate the user's local models, data, attack methods and defense methods. It is difficult to obtain model security assessment comparisons of different frameworks through unified risk assessment standards, which affects the protection level of large models.

[0006] The first aspect of the present application provides a large model security risk assessment method based on a training scenario, comprising the following steps: obtaining training scenario information, attack instructions, and defense instructions of a target large model; fine-tuning the target large model based on the training scenario information to obtain a fine-tuning model; attacking the target large model based on the attack instructions to obtain an attacked model and a poisoned sample; performing defense processing on the attacked model based on the defense instructions and the poisoned sample, and attacking the attacked model after defense processing according to the attack instructions to obtain a poisoned model; based on a target evaluation index, generating a security risk score using the fine-tuning model and the poisoned model, and evaluating the security risk level of the target large model according to the security risk score.

[0007] Optionally, in one embodiment of the present application, the fine-tuning training of the target large model based on the training scenario information includes: confirming the downstream task training set and the current training scenario of the target large model based on the training scenario information; and iteratively training the target large model in the current training scenario according to the downstream task training set until the target large model reaches a preset number of iterations.

[0008] Optionally, in one embodiment of the present application, the security risk score is generated based on the target evaluation index using the fine-tuning model and the poisoning model, including: testing the fine-tuning model and the poisoning model using a downstream task test set to obtain the indicator score corresponding to each indicator in the target evaluation index; and calculating the security risk score based on the weight value of each indicator and the indicator score.

[0009] Optionally, in one embodiment of the present application, the target evaluation indicators include downstream task performance indicators, anti-attack effect indicators and defense effect indicators.

[0010] The second aspect of the present application provides a large model security risk assessment device based on a training scenario, including: an acquisition module for acquiring training scenario information, attack instructions and defense instructions of a target large model; a training module for fine-tuning the target large model based on the training scenario information to obtain a fine-tuning model; an attack module for attacking the target large model based on the attack instruction to obtain an attacked model and a poisoned sample; a defense module for performing defense processing on the attacked model based on the defense instruction and the poisoned sample, and attacking the attacked model after defense processing according to the attack instruction to obtain a poisoned model; an evaluation module for generating a security risk score based on a target evaluation indicator using the fine-tuning model and the poisoned model, and evaluating the security risk level of the target large model according to the security risk score.

[0011] Optionally, in one embodiment of the present application, the training module includes: a confirmation unit for confirming the downstream task training set and the current training scene of the target large model based on the training scene information; an iteration unit for iteratively training the target large model in the current training scene according to the downstream task training set until the target large model reaches a preset number of iterations.

[0012] Optionally, in one embodiment of the present application, the evaluation module includes: a testing unit, used to test the fine-tuning model and the poisoning model using a downstream task test set to obtain an indicator score corresponding to each indicator in the target evaluation indicator; and a calculation unit, used to calculate the security risk score based on the weight value of each indicator and the indicator score.

[0013] Optionally, in one embodiment of the present application, the target evaluation indicators include downstream task performance indicators, anti-attack effect indicators and defense effect indicators.

[0014] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the large-model security risk assessment method based on training scenarios as described in the above embodiment.

[0015] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned large-model security risk assessment method based on training scenarios.

[0016] The fifth aspect of the present application provides a computer program, which, when executed, implements the above-mentioned large-model security risk assessment method based on training scenarios.

[0017] The embodiment of the present application can obtain the security risk score of the large model by attacking and defending the large model, so as to intuitively measure the security risk of the model, thereby further optimizing and improving the security defense level of the large model and making the model more stable and reliable. This solves the problems that the relevant technology can only realize the evaluation of small-scale models and cannot be applied to large-scale model application scenarios. There is a lack of security risk assessment methods for large-scale language models, and the attack scenarios supported by risk assessment are relatively single. It is impossible to evaluate the user's local model, data, attack method and defense method. It is difficult to obtain model security assessment comparisons of different frameworks through unified risk assessment standards, which affects the protection level of large models.

[0018] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A flowchart of a large model security risk assessment method based on a training scenario according to an embodiment of the present application;

[0021] Figure 2 This is a flowchart of a large model security risk assessment based on adversarial attack and adversarial training methods according to an embodiment of the present application;

[0022] Figure 3The following is an algorithm flow of a security risk assessment method according to an embodiment of the present application;

[0023] Figure 4 This is a flowchart of a large model security risk assessment based on a backdoor attack and anomaly detection method according to an embodiment of the present application;

[0024] Figure 5 This is a flowchart of a large model security risk assessment based on poisoning attacks and a robust aggregation method according to an embodiment of the present application;

[0025] Figure 6 Schematic diagram of the structure of a large-scale model security risk assessment device based on a training scenario according to an embodiment of the present application;

[0026] Figure 7 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0028] The following describes the large-scale model security risk assessment method and device based on the training scenario of the embodiment of the present application with reference to the accompanying drawings. The related technologies mentioned in the above background technology can only realize the assessment of small-scale models and cannot be applied to large-scale model application scenarios. There is a lack of security risk assessment methods for large-scale language models, and the attack scenarios supported by risk assessment are relatively single. It is impossible to evaluate the user's local model, data, attack methods and defense methods. It is difficult to obtain model security assessment comparisons of different frameworks through a unified risk assessment standard, which affects the protection level of the large model. The present application provides a large-scale model security risk assessment method based on a training scenario. In this method, the security risk score of the large model can be obtained by performing attack and defense training on the large model to intuitively measure the security risk of the model, thereby further optimizing and improving the security defense level of the large model and making the model more stable and reliable. This solves the problem that related technologies can only realize the evaluation of small-scale models and are not applicable to large-scale model application scenarios. There is a lack of security risk assessment methods for large-scale language models. The attack scenarios supported by risk assessment are relatively single, and it is impossible to evaluate the user's local models, data, attack methods, and defense methods. It is difficult to obtain model security assessment comparisons of different frameworks through unified risk assessment standards, which affects the protection level of large models.

[0029] Specifically, Figure 1A flowchart of a large-scale model security risk assessment method based on a training scenario provided in an embodiment of the present application.

[0030] like Figure 1 As shown, the large model security risk assessment method based on training scenarios includes the following steps:

[0031] In step S101, the training scenario information, attack instructions and defense instructions of the target large model are obtained.

[0032] It is understood that in the embodiments of the present application, the target large model can be a model for which security risk assessment is to be performed, such as a large-scale language model or a neural network model. Attack instructions can include adversarial attack methods, backdoor attack methods, poisoning attack methods, etc., while defense instructions can include adversarial training methods, anomaly detection methods, and robust detection methods.

[0033] Among them, users can build a customized configuration file according to their own task needs. The file should contain the basic information used in this evaluation corresponding to the training scenario information, such as the name and storage location of the selected large model, the storage information of the selected data set, the training parameters of the selected downstream task, and the output path of the evaluation results; as well as the attack and defense information corresponding to the attack instructions and defense instructions, such as the file location and attack parameters of the specific anti-attack method selected, and the file location and defense parameters of the selected defense method.

[0034] Further, the resources corresponding to the training scene information, attack instructions and defense instructions are initially read and preprocessed. The pre-trained large model file corresponding to the target large model required for evaluation can be downloaded and read according to the configuration information of the user, supporting the user to use a local model or a model introduced through a transformers library file, and setting the parameters corresponding to the model after reading according to the configuration information of the user; the required dataset file is downloaded according to the configuration information of the user, supporting the user to use a local dataset or a dataset introduced through a datasets library file, and pre-processing the dataset file according to the read model, eliminating the data that cannot be used, and saving the remaining data in a format that can be read by the model; the required downstream task training file is set according to the configuration information of the user, mainly setting the basic training information such as the number of training times nums, the number of training rounds epochs, the batch sample size batch_size, the training optimizer, the learning rate η, etc.; the required attack and defense method file is set according to the configuration information of the user, mainly performing a preliminary encapsulation of the attack method and the defense method selected by the user, encapsulating all of them into a standard attacker class file, and at the same time, similar to the downstream task training file, the basic attack and defense information such as the number of attack and defense times nums, the number of attack and defense rounds epochs, the random number seed seed, the learning rate η, etc. are also set, generating the downstream task training file and the attack and defense file required in the following embodiments.

[0035] In step S102, the target large model is fine-tuned based on the training scene information to obtain a fine-tuned model.

[0036] It can be understood that in the embodiments of the present application, the model can be fine-tuned according to the training task configured in the model in the training scene information, and the trained model is obtained as a fine-tuned model for comparison in the following step of model security risk evaluation.

[0037] Optionally, in an embodiment of the present application, fine-tuning the target large model based on the training scene information comprises: confirming the downstream task training set of the target large model and the current training scene based on the training scene information; and iteratively training the target large model in the current training scene according to the downstream task training set until the target large model reaches a preset number of iterations.

[0038] It should be noted that the preset number of iterations can be set by those skilled in the art according to the actual situation, which is not limited here.

[0039] In actual execution, the current training scene can include training parameters set for target large model fine-tuning training, such as basic training information such as training times nums, number of training rounds epochs each time, batch sample size batch_size, training optimizer optimizer, learning rate η, and the like, downstream task training set corresponding downstream task training file reading preprocessed model and dataset and the like resources, and the training data is iterated for epochs according to the target model.

[0040] Specifically, in the nth iteration, the sample data X of the downstream task training set is input into the target large model Once forward propagation is performed, the loss function of the model is calculated Where θ n is the current model parameter value. After obtaining the loss function, the model parameters are updated using back propagation and gradient descent method, and the specific formula is Where θ n+1 is the next round of model parameter value, is the gradient, which represents the fastest rising point of the current position of the function. Finally, a fine-tuned model that performs well in the corresponding downstream task is obtained.

[0041] In step S103, the target large model is attacked based on the attack instruction to obtain the attacked model and the poisoned sample.

[0042] It can be understood that in the embodiments of the present application, based on the attack instruction set by the user, the large model is attacked according to the attack file that has been configured, and an attacked model after attack is returned, and the corresponding poisoned sample is obtained according to the attack result, wherein the poisoned sample can be the poisoned training data set generated under the attack method.

[0043] For example, taking the TextFooler adversarial attack algorithm selected by the user as an example. First, the importance of the input text sentence X={x1,…,x n} is analyzed, wherein x i is the i-th token in the sentence, Y represents the true value label corresponding to X, is the error label, and F Y (X) is the score when the model outputs X as Y class, is the importance score of token x i , and the score difference between each token of X is used as the basis for importance score calculation, and the specific calculation formula is:

[0044]

[0045] After obtaining the importance of all tokens, they are sorted in descending order, and then tokens are replaced one by one until the output of the target model F changes or the candidate set for replacement is exhausted. Specifically, the strategy of synonym replacement is adopted during replacement. The corresponding synonym set is calculated for each token through the cosine distance, and the tokenx is used as the i Take the example, and select the top-N synonyms to form a candidate word set, and select the synonym x′ from it i TokenX i Replace the input sample from X to X′, calculate the semantic similarity between X and X′, and replace the x′ whose similarity exceeds the threshold i Delete it from the candidate set, and then select the next synonym in the candidate set, and repeat this process until the output result of the target model F changes or the candidate set is exhausted.

[0046] Finally, if the output result of the target model F changes, the adversarial attack is considered successful, and the generated adversarial sample is saved to obtain the poisoned sample. If the output result of the target model F does not change after exhausting the candidate set, the attack is considered to have failed. In the end, the attacked model is obtained under any result.

[0047] In step S104, defense processing is performed on the attacked model based on the defense instruction and the poisoned sample, and the attacked model after the defense processing is attacked according to the attack instruction to obtain the poisoned model.

[0048] It can be understood that in the embodiments of the present application, a defense method can be applied to the attacked model according to the defense file that has been set up, and the attacked model can be subjected to corresponding defense training and other processing based on the user's defense instructions and poisoned samples, and then attacked again according to the attack instructions, and the poisoned model after attack and defense is returned.

[0049] For example, if the defense instruction is adversarial training defense, the specific adversarial training defense method can be determined based on the user's configuration file, and the corresponding adversarial training defense can be performed on the attacked model. The adversarial sample is used as the poisoned sample to update the downstream task training set to obtain the adversarial training set. The attacked model is then subjected to adversarial training based on the adversarial training set and the defense instruction.

[0050] Specifically, based on the adversarial examples obtained in the above steps, the adversarial examples and their corresponding true value labels are combined into a new set of training data. For example, the original training data (X, Y) is used, where X is the original input text and Y is the corresponding true value label. X′ is the adversarial example generated based on it. X′ and Y are combined to form the new training data (X′, Y). All newly generated training data is added to the original training dataset. The model is then subjected to additional adversarial training using fine-tuning training to obtain the target model after adversarial training. After obtaining the target model after adversarial training, a new round of adversarial attacks is performed on the model according to the attack instructions to obtain the poisoned model.

[0051] In step S105, based on the target evaluation index, a security risk score is generated using the fine-tuning model and the poisoning model, and the security risk level of the target large model is evaluated according to the security risk score.

[0052] It can be understood that in the embodiments of the present application, the security risk of the original large model can be evaluated based on the obtained fine-tuning model and poisoning model, starting from multiple aspects corresponding to the target evaluation indicators, and the final security risk score can be output.

[0053] Among them, such as Figure 2 As shown in the figure, it is a flow chart of large model security risk assessment based on adversarial attack and adversarial training method according to the embodiment of the present application, which can be based on Figure 2 The security risk assessment process shown is based on the adversarial attack and adversarial training methods described in the above embodiments to obtain a fine-tuned model and a poisoned model, and further calculates the security risk score. According to the large model security risk score Score, the large model risk can be divided into 4 levels using the set default threshold: safe (Score < 0.2), relatively safe (0.2 ≤ Score < 0.5), risky (0.5 ≤ Score < 0.8), and serious risky (0.8 ≤ Score).

[0054] For example Figure 3 , is the algorithm flow of the security risk assessment method according to the embodiment of the present application, according to Figure 3The security risk assessment method shown can intuitively measure the risk of the model based on the model's security risk score, and use different categories of model security scores to evaluate the model's training from multiple perspectives, so as to further improve and optimize the target model according to the guidance of the security score, thereby improving the overall security of the target model and becoming a key reference privacy that can be selected at different times. In addition, this application supports users to provide highly liberalized assessment services. Users can freely choose to adjust the assessment parameters and strategies, and can also customize the large models, data sets, downstream training tasks, attack methods, defense methods, and evaluation indicators used in the assessment, thereby realizing the user's personalized security assessment needs, allowing security risk assessment of models, especially large models, in almost all scenarios, meeting the diverse needs of different users and application scenarios, and providing a key reference for users when selecting and training models.

[0055] Optionally, in one embodiment of the present application, a security risk score is generated based on the target evaluation index using a fine-tuning model and a poisoning model, including: testing the fine-tuning model and the poisoning model using a downstream task test set to obtain an indicator score corresponding to each indicator in the target evaluation index; and calculating the security risk score based on the weight value and indicator score of each indicator.

[0056] Optionally, in one embodiment of the present application, the target evaluation indicators include downstream task performance indicators, anti-attack effect indicators and defense effect indicators.

[0057] During the actual execution process, the security risk of the original large model can be evaluated from three aspects: downstream task performance, attack effect of attack method, and defense effect of defense method, and the final security risk score can be returned.

[0058] Specifically, the fine-tuned model and the poisoned model are evaluated at the downstream task performance level. The accuracy on the downstream task test set is used as the evaluation indicator. The fine-tuned model and the poisoned model are tested on the corresponding test set respectively. The security risk score obtained by the evaluation at this level is recorded as: S task The poisoning model is evaluated at the attack effect level, and the attack success rate and other evaluation indicators are used to test the poisoning model. The security score obtained from this level of evaluation is recorded as: S attack The poisoned model is evaluated at the defense effectiveness level. The performance of the defense method on the model is tested using the comparison of the success rate of the attack method before and after the defense, and the comparison of the accuracy of the model on the downstream task test set before and after the defense method as evaluation indicators. The security score obtained from this level of evaluation is recorded as: S defenseThe security scores of the above three levels are weighted and the final weighted calculation result Score is used as the overall security risk score of the large model output by the evaluation method, that is: Score = w task ·S task +w attack ·S attack +w defense ·S defense , where w i is the weight score corresponding to level i.

[0059] The working content of the embodiments of this application is described in detail below with specific examples.

[0060] For example, Figure 4 The flow chart of the large model security risk assessment based on the backdoor attack and anomaly detection method according to the embodiment of the present application is shown. Among them, the attack instruction is backdoor attack, and the defense instruction is anomaly detection. Then, when the large model is fine-tuned and trained based on the above steps to obtain a fine-tuned model, the specific backdoor attack method is determined according to the user's configuration file, and the corresponding attack is performed on the fine-tuned model to construct a backdoor training set used for the backdoor attack. The attack method first selects the target backdoor class t of the attack and the backdoor trigger trigger to be used, and then generates a backdoor training set based on the original training data set of the target model and the backdoor trigger, that is, the backdoor data set. Where X is a clean training sample, X′ is a backdoor data sample generated by X. It corresponds to the generation operation used by the backdoor attack method. Its main purpose is to embed the backdoor trigger into the original text data X. After obtaining the backdoor training set, the target model is trained on the backdoor according to the user's configuration information. The loss function of the training process is expanded from the model main task loss to the model main task loss plus the backdoor task loss, that is: L attack =L task (X,Y)+L backdoor (X′,t), where L attack , L task , L backdoor They represent the total loss function during the backdoor attack training process, the loss function of the model on the main task, and the loss function of the model on the backdoor task respectively. The corresponding backdoor attack training is completed by expanding the loss function. Finally, a backdoor model embedded with a backdoor and the generated backdoor samples are obtained. The backdoor model embedded with a backdoor and the backdoor training set are used as poisoned samples.

[0061] Furthermore, a specific anomaly detection method is determined based on the user's configuration file, and corresponding anomaly detection is performed on the backdoor model and backdoor samples. According to the anomaly detection selected by the user, it can be divided into two categories: data detection and model detection. For example, if the user chooses to use the ONION detection algorithm for defense, the defense method uses a pre-trained model such as GPT-2 for assistance. First, the test sample of the input target model is scanned, and abnormal words such as "cf" in the test sample are detected and removed. Specifically, each token in the test sample is traversed, and the degree of change in semantic perplexity after removing the current token is calculated to determine whether the current token contains a backdoor trigger. After completing the anomaly detection of the data or model, a new round of backdoor attack is performed on the model to obtain a poisoned model after the backdoor attack and anomaly detection defense. Finally, based on the process of the above embodiment, the security risk scores obtained under the backdoor attack method and the anomaly detection method are calculated to match the corresponding security risk level.

[0062] For example, Figure 5 The flowchart of the large model security risk assessment based on poisoning attack and robust aggregation method according to the embodiment of the present application is shown. Among them, the attack instruction is poisoning attack, and the defense instruction is robust aggregation defense. Then, when the large model is fine-tuned and trained based on the above steps to obtain a fine-tuned model, the specific poisoning attack method is determined according to the user's configuration file, and the fine-tuned model is attacked accordingly; a poisoned training set used for poisoning attack is constructed, and the attack file poisons the data according to the specific attack method. For example, if the user chooses to use label flipping attack, the corresponding original training data set is first obtained according to the data poisoning ratio set by the user, and the data labels in the training set are randomly flipped, such as converting the clean sample (X, Y) into a poisoned sample. in is a random label that is not equal to Y; after obtaining the poisoned training set, the target model is poisoned according to the user's configuration information. After a certain number of rounds of attack, the generated poisoned samples and the poisoned attacked model will be returned.

[0063] Further, a specific robust aggregation method is determined according to the user's profile, and a corresponding robust aggregation operation is performed on the poisoned training set; and the data distribution rule of the clean sample is obtained according to the clean sample set. First, the defense method will cluster the clean samples of each class according to the known clean sample set, so as to obtain the data distribution rule of the clean sample; after obtaining the possible data distribution rule of the clean sample through the unsupervised idea, the test sample is aggregated according to the output of the attacked model and the distribution of the corresponding clean sample, the cosine distance between the test sample and the corresponding clustering center is calculated, if it exceeds the manually set threshold ∈, it is considered that it is a poisoned sample, and it is removed, otherwise it is considered to be a clean sample, and it is added to the clean sample set, and the new data distribution generated after aggregation is updated; after completing the robust aggregation of the data, a new round of poisoning attack is performed on the model, and a poisoned model after poisoning attack and robust aggregation defense is obtained. Finally, the security risk score obtained by the poisoning attack method and the robust aggregation defense method based on the above embodiment process is calculated to match the corresponding security risk level.

[0064] Based on the above adversarial sample attack method, backdoor attack method, poisoning attack method, adversarial training method, anomaly detection method and robust detection method, the security risk of the large model itself can be comprehensively and carefully evaluated, and the corresponding robustness improvement suggestion is given while the model security risk is evaluated, so as to enhance various models, especially the robustness of large model technology to various attacks and abnormal inputs, so that it is more stable and reliable in actual application.

[0065] The large model security risk evaluation method based on training scenarios according to the embodiments of the present application can obtain the security risk score of the large model by attacking and defending the large model, so as to intuitively measure the security risk of the model, so as to further optimize and improve the security defense level of the large model, and make the model more stable and reliable. Therefore, the related art can only evaluate small-scale models, cannot be applied to large model application scenarios, lacks a security risk evaluation method for large-scale language models, the attack scene supported by the risk evaluation is relatively single, cannot evaluate the user's local model, data, attack method and defense method, it is difficult to obtain the model security evaluation comparison of different frameworks through a unified risk evaluation standard, and the protection level of the large model is affected.

[0066] Secondly, the large model security risk evaluation device based on training scenarios according to the embodiments of the present application is described with reference to the accompanying drawings.

[0067] Figure 6It is a structural diagram of a large-scale model security risk assessment device based on a training scenario in an embodiment of the present application.

[0068] like Figure 6 As shown, the large-scale model security risk assessment device 10 based on training scenarios includes: an acquisition module 100, a training module 200, an attack module 300, a defense module 400 and an evaluation module 500.

[0069] Among them, the acquisition module 100 is used to obtain the training scene information, attack instructions and defense instructions of the target large model.

[0070] The training module 200 is used to fine-tune the target large model based on the training scene information to obtain a fine-tuned model.

[0071] The attack module 300 is used to attack the target large model based on the attack instruction to obtain the attacked model and the poisoned sample.

[0072] The defense module 400 is used to perform defense processing on the attacked model based on the defense instruction and the poisoned sample, and attack the attacked model after the defense processing according to the attack instruction to obtain the poisoned model.

[0073] The evaluation module 500 is used to generate a security risk score based on the target evaluation index using the fine-tuning model and the poisoning model, and evaluate the security risk level of the target large model according to the security risk score.

[0074] Optionally, in one embodiment of the present application, the training module 200 includes: a confirmation unit and an iteration unit.

[0075] Among them, the confirmation unit is used to confirm the downstream task training set and current training scene of the target large model based on the training scene information.

[0076] The iteration unit is used to iteratively train the target large model in the current training scenario according to the downstream task training set until the target large model reaches a preset number of iterations.

[0077] Optionally, in one embodiment of the present application, the evaluation module 500 includes: a calculation unit and a testing unit.

[0078] Among them, the testing unit is used to test the fine-tuning model and the poisoning model using the downstream task test set to obtain the indicator score corresponding to each indicator in the target evaluation indicator.

[0079] The calculation unit is used to calculate the security risk score based on the weight value of each indicator and the indicator score.

[0080] Optionally, in one embodiment of the present application, the target evaluation indicators include downstream task performance indicators, anti-attack effect indicators and defense effect indicators.

[0081] It should be noted that the above explanation of the embodiment of the large-model security risk assessment method based on training scenarios is also applicable to the large-model security risk assessment device based on training scenarios in this embodiment, and will not be repeated here.

[0082] According to the large-scale model security risk assessment device based on training scenarios proposed in the embodiment of the present application, the security risk score of the large model can be obtained by attacking and defending the large model, so as to intuitively measure the security risk of the model, thereby further optimizing and improving the security defense level of the large model and making the model more stable and reliable. This solves the problems that the relevant technology can only evaluate small-scale models and cannot be applied to large-scale model application scenarios. There is a lack of security risk assessment methods for large-scale language models, and the attack scenarios supported by risk assessment are relatively simple. It is impossible to evaluate the user's local model, data, attack method and defense method. It is difficult to obtain model security assessment comparisons of different frameworks through unified risk assessment standards, which affects the protection level of large models.

[0083] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0084] A memory 701 , a processor 702 , and a computer program stored in the memory 701 and executable on the processor 702 .

[0085] When the processor 702 executes the program, the large model security risk assessment method based on the training scenario provided in the above embodiment is implemented.

[0086] Furthermore, the electronic device further includes:

[0087] The communication interface 703 is used for communication between the memory 701 and the processor 702 .

[0088] The memory 701 is used to store computer programs that can be run on the processor 702 .

[0089] The memory 701 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0090] If the memory 701, processor 702, and communication interface 703 are implemented independently, the communication interface 703, memory 701, and processor 702 can be interconnected via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0091] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0092] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0093] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the above-mentioned large-model security risk assessment method based on training scenarios.

[0094] This embodiment also provides a computer program, which, when executed, implements the above-mentioned large-model security risk assessment method based on training scenarios.

[0095] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0097] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0098] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0099] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0100] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0101] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0102] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A large-scale model security risk assessment method based on training scenarios, characterized in that: The following steps are involved: Obtain training scenario information, attack instructions, and defense instructions for the target large model; Based on the user-defined downstream task type, the target large model, the training scenario information, the attack instructions, and the defense instructions are uniformly encapsulated and preprocessed to build an extensible generalized execution process. The generalized execution process includes: Performing fine-tuning training on the target large model based on the training scenario information to obtain a fine-tuning model; Attacking the target large model based on the attack instruction to obtain an attacked model and a poisoned sample; Performing defense processing on the attacked model based on the defense instruction and the poisoned sample, and attacking the attacked model after the defense processing according to the attack instruction to obtain a poisoned model; Based on the target evaluation index, a security risk score is generated using the fine-tuning model and the poisoning model, and the security risk level of the target large model is evaluated according to the security risk score; The target evaluation indicators include downstream task performance indicators, attack method attack effect indicators, and defense method defense effect indicators.

2. The method according to claim 1, characterized in that The fine-tuning training of the target large model based on the training scenario information includes: Determine the downstream task training set and the current training scene of the target large model based on the training scene information; According to the downstream task training set, the target large model is iteratively trained in the current training scenario until the target large model reaches a preset number of iterations.

3. The method according to claim 1, characterized in that The step of generating a security risk score based on the target evaluation indicator and utilizing the fine-tuning model and the poisoning model includes: Using the downstream task test set to test the fine-tuning model and the poisoning model, and obtain the indicator score corresponding to each indicator in the target evaluation indicator; The security risk score is calculated based on the weight value of each indicator and the indicator score.

4. A large-scale model security risk assessment device based on training scenarios, characterized in that: include: The acquisition module is used to obtain the training scene information, attack instructions and defense instructions of the target large model; According to the user-defined downstream task type, the target large model, the training scenario information, the attack instructions and the defense instructions are uniformly encapsulated and preprocessed; A training module, configured to perform fine-tuning training on the target large model based on the training scenario information to obtain a fine-tuned model; An attack module, configured to attack the target large model based on the attack instruction to obtain an attacked model and a poisoned sample; a defense module, configured to perform defense processing on the attacked model based on the defense instruction and the poisoned sample, and attack the attacked model after the defense processing according to the attack instruction to obtain a poisoned model; An evaluation module, configured to generate a security risk score based on a target evaluation indicator using the fine-tuning model and the poisoning model, and evaluate the security risk level of the target large model according to the security risk score; The target evaluation indicators include downstream task performance indicators, attack method attack effect indicators, and defense method defense effect indicators.

5. The device according to claim 4, characterized in that The training module includes: A confirmation unit, configured to confirm a downstream task training set and a current training scene of the target large model based on the training scene information; An iteration unit is used to iteratively train the target large model in the current training scenario according to the downstream task training set until the target large model reaches a preset number of iterations.

6. The device according to claim 4, characterized in that The evaluation module includes: A testing unit, configured to test the fine-tuning model and the poisoning model using a downstream task test set to obtain an indicator score corresponding to each indicator in the target evaluation indicator; A calculation unit is used to calculate the security risk score based on the weight value of each indicator and the indicator score.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large-model security risk assessment method based on a training scenario as described in any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the large-model security risk assessment method based on training scenarios as described in any one of claims 1-3.

9. A computer program product comprising a computer program, characterized in that The computer program is executed to implement the large-model security risk assessment method based on training scenarios as described in any one of claims 1-3.

Images