Large language model security assessment method and device and electronic equipment

Abstract

The embodiment of the invention is suitable for the technical field of artificial intelligence models, and provides a large language model safety evaluation method and device and electronic equipment, and the method comprises the steps: obtaining safety evaluation information obtained by carrying out the evaluation of a plurality of evaluation samples through a preset safety evaluation model, and obtaining an uncertainty value corresponding to the safety evaluation information; the evaluation sample comprises input information and output information generated by the to-be-evaluated large model for each piece of input information; determining a dynamic threshold according to the sampling sample set; aiming at the evaluation sample of which the uncertainty value is greater than a dynamic threshold value, determining an expert label aiming at the evaluation sample as safety evaluation information of the evaluation sample; and according to each evaluation sample and the corresponding safety evaluation information, generating a safety evaluation result of the to-be-evaluated large model. According to the embodiment of the invention, the method can achieve the efficient safety evaluation of the to-be-evaluated large model, and guarantees the accuracy and reliability of the safety evaluation result of the to-be-evaluated large model.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application belongs to the technical field of artificial intelligence models, and particularly relates to a large language model security evaluation method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of generative artificial intelligence technology, artificial intelligence large models have shown great capabilities in the field of natural language processing. The security and privacy protection of artificial intelligence large models have increasingly attracted attention. Artificial intelligence large language model security evaluation faces unique challenges such as diversified risk categories, large data size, and real-time requirements.

[0003] Current security evaluation methods for artificial intelligence large models mostly rely on manual judgment, which is costly and inefficient. Although some research attempts to use artificial intelligence models for preliminary evaluation, these methods lack statistical error rate guarantees, and the reliability of the evaluation results cannot meet actual needs.

[0004] Therefore, there is an urgent need for an efficient, accurate, and cost-controllable large language model security evaluation method. SUMMARY

[0005] Therefore, the embodiment of the present application provides a large language model security evaluation method and device and electronic equipment to efficiently complete the security evaluation of the to-be-evaluated large model and ensure the accuracy and reliability of the security evaluation results of the to-be-evaluated large model.

[0006] The first aspect of the embodiment of the present application provides a large language model security evaluation method, comprising: obtaining security evaluation information obtained by a preset security evaluation model for a plurality of evaluation samples, and an uncertainty value corresponding to the security evaluation information; the evaluation sample includes input information and output information generated by a to-be-evaluated large model for each input information; determining a dynamic threshold according to a sampling sample set; the sampling sample set includes a plurality of evaluation samples, and the uncertainty value and expert labels corresponding to the evaluation samples; for the evaluation sample whose uncertainty value is greater than the dynamic threshold, determining that the expert label for the evaluation sample is the security evaluation information of the evaluation sample; generating a security evaluation result of the to-be-evaluated large model according to each evaluation sample and the corresponding security evaluation information.

[0007] In some implementations of the first aspect, the security evaluation result includes an evaluation error rate and an evaluation confidence of the evaluation error rate, and the method further comprises: If the evaluation error rate is greater than the preset overall error threshold or the evaluation confidence is less than the preset confidence threshold, the number of evaluation samples in the sampling sample set is increased, and the step of determining the dynamic threshold according to the sampling sample set is returned.

[0008] In some implementations of the first aspect, the determining the dynamic threshold according to the sampling sample set and the expert labels comprises: sampling from the plurality of evaluation samples to obtain a plurality of sampling samples; determining expert labels corresponding to each input information in the sampling samples; calculating an error upper limit value according to the sampling samples and the expert labels; determining a dynamic threshold according to the error upper limit value.

[0009] In some implementations of the first aspect, the calculating the error upper limit value according to the sampling sample set and the expert labels comprises: determining target samples from the sampling samples; the output information of the target samples is different from the expert labels and the uncertainty value is higher than an uncertainty threshold; calculating a weighted loss value for each target sample; determining a weighted loss mean value and a weighted loss standard deviation value based on the weighted loss values; determining a product value of a normal distribution quantile corresponding to a preset confidence threshold and the weighted loss standard deviation value; determining a sum of the weighted loss mean value and the product value as an error upper limit value corresponding to the uncertainty threshold.

[0010] In some implementations of the first aspect, the determining the dynamic threshold according to the error upper limit value comprises: determining a plurality of risk categories; the following steps are performed for each risk category: determining a category error threshold; the category error threshold is used to represent the proportion of output information errors of the same category; determining a dynamic threshold; the dynamic threshold is the minimum one of the uncertainty thresholds corresponding to each error upper limit value greater than the category error threshold.

[0011] In some implementations of the first aspect, if the risk category is a high-risk category, the determining the category error threshold comprises: determining an error coefficient; the error coefficient is a positive number less than 1; determining a product of the error coefficient and the overall error threshold as the category error threshold of the high-risk category.

[0012] In some implementations of the first aspect, if the risk category is a low-risk category, the determining the category error threshold comprises: determining, based on the sampling samples, a category weight corresponding to each different risk category; determining a sum of the category weights of each low-risk category as a low-risk category weight; determining a sum of the category weights of each high-risk category as a high-risk category weight; determining, based on the low-risk category weight, the high-risk category weight, the error coefficient, and the overall error threshold, a category error threshold of the low-risk category; wherein a product of the category error threshold of the low-risk category and the low-risk category weight is not greater than a sum of a product of a category error threshold of the high-risk category and the high-risk category weight.

[0013] In some implementations of the first aspect, the determining, based on the sampling samples, a category weight corresponding to each different risk category comprises: determining a risk severity score value corresponding to each risk category and a sampling sample proportion; the sampling sample proportion is a proportion of a number of sampling samples corresponding to the same risk category and a total amount of sampling samples; determining, according to the risk severity score value, the sampling sample proportion, a first adjustment coefficient corresponding to the risk severity score value, and a second adjustment coefficient corresponding to the sampling sample proportion, a category weight.

[0014] In some implementations of the first aspect, before the determining, for the evaluation sample whose uncertainty value is greater than the dynamic threshold, that the expert label for the evaluation sample is the safety evaluation information of the evaluation sample, the method further comprises: determining a risk category of the evaluation sample; determining an evaluation sample group in which the evaluation sample is located and a true error rate corresponding to the evaluation sample group; the evaluation sample group corresponds to the same risk category and the same uncertainty value interval; the true error rate is determined based on the sampling sample set; calculating an uncertainty mean value of the evaluation sample group; when a difference between the uncertainty mean value and the true error rate is greater than a preset deviation tolerance value, determining, based on the true error rate, the uncertainty value of the evaluation sample.

[0015] A second aspect of the embodiments of the present application provides a large language model safety evaluation device, comprising: The model evaluation module is configured to obtain safety evaluation information obtained by a preset safety evaluation model by evaluating a plurality of evaluation samples, and an uncertainty value corresponding to the safety evaluation information; the evaluation samples include input information and output information generated by the large model to be evaluated for each input information; The dynamic threshold determination module is configured to determine a dynamic threshold according to a sampling sample set; the sampling sample set includes a plurality of evaluation samples, uncertainty values corresponding to the evaluation samples, and expert labels; The expert label replacement module is configured to, for the evaluation sample with the uncertainty value greater than the dynamic threshold, determine that the expert label for the evaluation sample is the safety evaluation information of the evaluation sample. The safety evaluation result generation module is configured to generate a safety evaluation result of the large model to be evaluated according to each evaluation sample and the corresponding safety evaluation information.

[0016] The third aspect of the embodiments of the present application provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the electronic device implements the large language model safety evaluation method as described in the first aspect.

[0017] The fourth aspect of the embodiments of the present application provides a computer program product, including a computer program, and when the computer program is executed, the large language model safety evaluation method as described in the first aspect is executed.

[0018] The fifth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the large language model safety evaluation method as described in the first aspect is implemented.

[0019] The embodiments of the present application have the following beneficial effects: By acquiring safety evaluation information obtained by a preset safety evaluation model for a plurality of evaluation samples, and an uncertainty value corresponding to the safety evaluation information; the evaluation samples include input information and output information generated by a large model to be evaluated for each input information; a dynamic threshold is determined according to a sampling sample set; the sampling sample set includes a plurality of evaluation samples, and the uncertainty values and expert labels corresponding to the evaluation samples; for the evaluation samples with the uncertainty values greater than the dynamic threshold, the expert label for the evaluation samples is determined as the safety evaluation information of the evaluation samples; according to each evaluation sample and the corresponding safety evaluation information, a safety evaluation result of the large model to be evaluated is generated, so as to obtain the safety evaluation information obtained by the safety evaluation model automatically evaluating the output information generated by the large model to be evaluated, and the uncertainty of the safety evaluation information generated by the safety evaluation model is quantified to obtain the uncertainty value. By comparing the dynamic threshold with the uncertainty value, and expert review of the evaluation samples with the uncertainty values greater than the dynamic threshold, the expert label is used as the safety evaluation information of the evaluation samples with the uncertainty values greater than the dynamic threshold, expert review of the high-uncertainty evaluation samples is realized, so as to ensure the accuracy and reliability of the final safety evaluation result. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is a schematic diagram of a large language model safety evaluation method provided by an embodiment of the present application; Figure 2 is a flow framework schematic diagram of a large language model safety evaluation method provided by an embodiment of the present application; Figure 3 is a schematic diagram of a large language model safety evaluation device provided by an embodiment of the present application; Figure 4 is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application, but it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0023] It should be understood that the term "comprises" as used in the specification and the appended claims indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0024] It should also be understood that the term "and / or" as used in the specification and the appended claims, means any one or more of the associated listed items, as well as all possible combinations of the items.

[0025] As used in the specification and the appended claims, the term "if' can be interpreted as meaning "when" or "once" or "in response to a determination" or "in response to a detection" depending on the context. Similarly, the phrase "if determined" or "if detected [the described condition or event]" can be interpreted to mean "once determined" or "in response to a determination" or "once detected [the described condition or event]" or "in response to a detection [the described condition or event]" depending on the context.

[0026] In addition, in the description of the specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0027] In the present specification, the phrase "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, the appearances of the phrase "in one embodiment", "in some embodiments", "in other some embodiments", "in yet some embodiments" or the like in various places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically specified. The terms "comprise", "include", "have" and their conjugates mean "including but not limited to", unless otherwise specifically specified.

[0028] The technical solutions of the application are described below through specific embodiments.

[0029] Referring to Figure 1 , a schematic diagram of a large language model security evaluation method provided by an embodiment of the application is shown, which can specifically include the following steps: At step 101, safety evaluation information obtained by a preset safety evaluation model for evaluating a plurality of evaluation samples is acquired, and an uncertainty value corresponding to the safety evaluation information is acquired; the evaluation samples include input information and output information generated by a to-be-evaluated large model for each input information; A generative artificial intelligence large language model that needs to be evaluated, i.e., a to-be-evaluated large model, and a plurality of input information that needs to be processed by the to-be-evaluated large model are predetermined.

[0030] The input information can be input to the to-be-evaluated large model, and the to-be-evaluated large model generates corresponding output information for each input information, and the evaluation samples are obtained based on the mutually corresponding input information and output information.

[0031] The artificial intelligence model used for safety evaluation of the evaluation samples is predetermined as a safety evaluation model, and the safety evaluation model is guided to perform safety evaluation on the evaluation samples to obtain corresponding safety evaluation information and an uncertainty value. One evaluation sample corresponds to one safety evaluation information, and the safety evaluation information is used to represent whether the output information generated by the to-be-evaluated large model for the input information is compliant. The safety evaluation information includes safety and insecurity. That is, the safety evaluation information of one evaluation sample is safety or insecurity. If the safety evaluation information is safety, it means that the safety evaluation model identifies that the output information of the evaluation sample is compliant. If the safety evaluation information is insecurity, it means that the safety evaluation model identifies that the output information of the evaluation sample is not compliant.

[0032] In a specific implementation, the input information of each evaluation sample can be marked as a question, and the output information can be marked as an answer. The safety evaluation model is guided to output a safety evaluation value for the output of each evaluation sample and a credibility of the safety evaluation value, and the uncertainty value is obtained based on the credibility.

[0033] For example, the safety evaluation model is guided to output a safety evaluation value of 0 or 1, and the credibility range is an integer from 0 to 10. If the safety evaluation value is 1, it means that the safety evaluation information is safety. If the safety evaluation value is 0, it means that the safety evaluation information is insecurity. If the credibility is 0, it means that the safety evaluation value is completely credible. If the credibility is 10, it means that the safety evaluation value is completely not credible. The credibility is normalized to the interval [0, 1] as an uncertainty score (u) The higher the uncertainty score is, the greater the uncertainty of the safety evaluation information of the safety evaluation model is.

[0034] At step 102, a dynamic threshold is determined according to a sampling sample set; the sampling sample set includes a plurality of evaluation samples, uncertainty values corresponding to the evaluation samples, and expert labels; The expert label is the true safety information determined by a designated person for each evaluation sample in the sampling sample test set.

[0035] By evaluating the difference between the output information and the true safety information in the sample, the error rate of all evaluation samples can be estimated, and the estimated error rate is used to determine the dynamic threshold, so that the evaluation samples with an uncertainty value greater than the dynamic threshold are manually confirmed, and the error rate of all evaluation samples is not necessarily determined.

[0036] In step 103, for the evaluation samples with an uncertainty value greater than the dynamic threshold, the expert label for the evaluation sample is determined as the safety evaluation information of the evaluation sample. For evaluation samples with an uncertainty value greater than the dynamic threshold, manual confirmation can be performed, specifically, the expert label of the evaluation sample can be received by the designated personnel, and the expert label is safety or insecurity. The safety evaluation information obtained by the safety evaluation model is replaced by the expert label to complete the expert review of the evaluation samples with an uncertainty value greater than the dynamic threshold, and the true safety evaluation information is obtained.

[0037] By automatically selecting evaluation samples that need expert review according to the dynamic threshold, the workload of manual processing is significantly reduced, and the efficiency of safety evaluation is improved.

[0038] In an example, only 10% of the evaluation samples need expert review, and the remaining 90% of the evaluation samples use the safety evaluation information output by the safety evaluation model as the final result. Compared with the way of manually determining the safety evaluation information, about 90% of the time and labor cost is saved, and the efficiency of safety evaluation of the to-be-evaluated large model is greatly improved.

[0039] In step 104, the safety evaluation result of the to-be-evaluated large model is generated according to each evaluation sample and the corresponding safety evaluation information.

[0040] The evaluation samples and the safety evaluation information corresponding to the evaluation samples can be counted to obtain the safety evaluation result of the to-be-evaluated large model. The safety evaluation result can include the proportion of evaluation samples with safety evaluation information as safety, to determine the safety of the to-be-evaluated large model for generation processing. The safety evaluation result can also include the evaluation error rate and the evaluation confidence of the evaluation error rate, and the evaluation error rate and the evaluation confidence can represent the performance of the to-be-evaluated model for generation processing.

[0041] Since the above steps compare the uncertainty value of the evaluation sample with the dynamic threshold, and the dynamic threshold is based on the estimated error rate of all evaluation samples, the dynamic threshold can be adjusted by the size of the expected value, and the evaluation samples with an uncertainty value greater than the dynamic threshold are reviewed by experts, so that the evaluation error rate of the safety evaluation result is less than the expected value, thereby ensuring the accuracy of the safety evaluation result.

[0042] In the embodiments of the present application, in addition to the expert label requiring manual processing, the determination of the safety evaluation information, the generation of the uncertainty value, the comparison of the uncertainty value with the dynamic threshold value, and the safety evaluation result in the above process can all be automatically processed without full manual processing, which can quickly respond to large model updates, complete the safety evaluation of the to-be-tested large model in a short time, break through the efficiency bottleneck of pure manual evaluation, and meet the iteration timeliness requirements.

[0043] In the embodiments of the present application, the safety evaluation information obtained by the preset safety evaluation model for evaluating a plurality of evaluation samples and the uncertainty value corresponding to the safety evaluation information are obtained; the evaluation sample includes input information and output information generated by the to-be-evaluated large model for each input information; a dynamic threshold value is determined according to a sampling sample set; the sampling sample set includes a plurality of evaluation samples, and the uncertainty value and the expert label corresponding to the evaluation sample; for the evaluation sample with the uncertainty value greater than the dynamic threshold value, the expert label for the evaluation sample is determined as the safety evaluation information of the evaluation sample; and a safety evaluation result of the to-be-evaluated large model is generated according to each evaluation sample and the corresponding safety evaluation information, so that the safety evaluation information obtained by the safety evaluation model automatically evaluating the output information generated by the to-be-evaluated large model is obtained, and the uncertainty of the safety evaluation information generated by the safety evaluation model is quantified to obtain the uncertainty value. By comparing the dynamic threshold value with the uncertainty value, and expert review of the evaluation sample with the uncertainty value greater than the dynamic threshold value, the expert label is used as the safety evaluation information of the evaluation sample with the uncertainty value greater than the dynamic threshold value, expert review of the high-uncertainty evaluation sample is realized, so as to ensure the accuracy and reliability of the final safety evaluation result.

[0044] In some implementation manners of the embodiments of the present application, the safety evaluation result includes an evaluation error rate and an evaluation confidence of the evaluation error rate, and the method further includes: In the case that the evaluation error rate is greater than a preset overall error threshold value or the evaluation confidence is less than a preset confidence threshold value, the number of evaluation samples in the sampling sample set is increased, and the step of determining the dynamic threshold value according to the sampling sample set is returned.

[0045] The overall error threshold value can be determined in advance before step 102. Since the higher the uncertainty value, the greater the possibility of difference between the output information and the true safety information, for the same to-be-tested sample, the proportion of evaluation samples requiring expert review can be adjusted by adjusting the size of the dynamic threshold value. The smaller the dynamic threshold value, the greater the proportion of evaluation samples requiring expert review, and the lower the evaluation error rate of the corresponding safety evaluation result.

[0046] As an example, the preset confidence threshold value is (1-α), and the overall error rate can be calculated as wherein the safety assessment information output by the safety assessment model, is an expert label. The confidence upper limit CI of the overall error rate is determined.

[0047]

[0048] If the CI is not greater than the overall error threshold, it is determined that the evaluation confidence is not less than the preset confidence threshold; if the CI is greater than the overall error threshold, it is determined that the evaluation confidence is less than the preset confidence threshold. Wherein z is a normal distribution z value.

[0049] In the case where the evaluation error rate is greater than the preset overall error threshold or the evaluation confidence is less than the preset confidence threshold, the number of evaluation samples in the sampling sample set is increased, and the dynamic threshold determined according to the sampling sample set is returned to improve the evaluation error rate and the evaluation confidence of the safety assessment result.

[0050] Referring to Figure 2 , a flow framework schematic diagram of another large language model safety assessment method provided by the embodiments of the present application is shown.

[0051] In some implementation modes of the embodiments of the present application, the determination of the dynamic threshold according to the sampling sample set comprises: sampling a plurality of evaluation samples to obtain a plurality of sampling samples; determining the expert label corresponding to each input information in the sampling sample; calculating the error upper limit value according to the sampling sample and the expert label; and determining the dynamic threshold according to the error upper limit value.

[0052] Randomly extract evaluation samples covering various risk dimensions as sampling samples, perform expert review on the sampling samples to obtain expert labels of the sampling samples, and obtain a sampling sample set based on the sampling samples and the expert labels , m is the number of sampling samples. is a sampling sample, is the safety assessment information output by the safety assessment model for the sampling sample, is a true artificial label, is the uncertainty value output by the safety assessment model for the sampling sample.

[0053] Wherein, for different expert labels of the same sampling sample, the expert label with the highest proportion is taken as the true artificial label. For example: 10000 evaluation samples are sampled, 500 are extracted as sampling samples, and each sampling sample is cross-labeled by 3 designated units, and the majority result is taken as the expert label.

[0054] Based on the sampled samples and the expert labels corresponding to each sampled sample, an upper error limit is calculated, and a dynamic threshold is obtained based on the upper error limit, so that the error rate of the security assessment result in step 104 is not greater than the upper error limit.

[0055] In some implementations of the embodiments of the present application, the upper error limit value is calculated based on the sampling samples and the expert labels, including: determining a target sample from the sampling samples; the output information of the target sample is different from the expert label and the uncertainty value is higher than the uncertainty critical value; calculating a weighted loss value for each target sample; determining a weighted loss mean and a weighted loss standard deviation value based on the weighted loss value; determining the product value of the normal distribution quantile corresponding to a preset confidence threshold and the weighted loss standard deviation value; and determining that the sum of the weighted loss mean and the product value is the upper error limit value corresponding to the uncertainty critical value.

[0056] A weighted loss formula can be constructed to calculate the weighted loss value of each sample .

[0057] ,in The uncertainty critical value is used as a constraint condition to screen out sampling samples whose uncertainty value is not greater than the uncertainty critical value and calculate the weighted loss value.

[0058] in, is a binary variable indicating whether sample i is drawn ( Indicates being selected. 0 means not selected). is the sampling probability of sample i. is an indicator function if and only if ≠ When the indicator function takes the value of 1, otherwise the indicator function takes the value of 0.

[0059] pass and The security assessment model of the sample set is used to implement weighted loss of the sampled samples whose security assessment information output is different from the actual manual label, and the total error of all evaluation samples is evaluated.

[0060] Furthermore, the calculation formula for the upper limit of the construction error is: .

[0061] Among them, the weighted loss mean , weighted loss standard deviation , is the confidence threshold quantile of the standard normal distribution.

[0062] In some implementations of the embodiments of the present application, the determining the dynamic threshold according to the error upper limit value comprises: determining a plurality of risk categories; and performing the following steps for each risk category: determining a category error threshold, the category error threshold being used to represent the proportion of output information errors of the same category; and determining a dynamic threshold, the dynamic threshold being the minimum one of the uncertainty critical values corresponding to each error upper limit value greater than the category error threshold.

[0063] In the embodiments of the present application, different risk categories are divided, and the different risk categories are further divided into a high-risk category and a low-risk category, and a high-risk category set and a low-risk category set are obtained accordingly.

[0064] By setting the category error threshold corresponding to each risk category, the dynamic threshold is determined by the formula: , c is used to identify the risk category, and the minimum one of the uncertainty critical values corresponding to each error upper limit value greater than the category error threshold is determined as the target uncertainty critical value by the above formula, and the target uncertainty critical value is taken as the dynamic threshold of the risk category.

[0065] In some implementations of the embodiments of the present application, if the risk category is a high-risk category, the determining the category error threshold comprises: determining an error coefficient, the error coefficient being a positive number less than 1; and determining the product of the error coefficient and the overall error threshold as the category error threshold of the high-risk category.

[0066] In a specific implementation, a constraint condition of the overall error threshold is set: . is the category weight of the risk category c, and satisfies , and the definition of ,

[0067] In order to ensure that the error threshold of the high-risk category is low, a positive number less than 1 is determined as the error coefficient, and the product of the error coefficient and the overall error threshold is taken as the category error threshold of the high-risk category .

[0068] For example, the error coefficient is 0.6, and the overall error threshold is , then .

[0069] For the overall error threshold ϵ, the overall error of all sampling samples is , wherein .

[0070] wherein, for , , H is used to identify the high-risk category set, , L is used to identify a low-risk category set.

[0071] In some implementations of the embodiments of the present application, if the risk category is a low-risk category, the determining of the category error threshold comprises: determining, based on the sampling sample, a category weight corresponding to each different risk category; determining a sum of the category weights of each low-risk category as a low-risk category weight; determining a sum of the category weights of each high-risk category as a high-risk category weight; determining, based on the low-risk category weight, the high-risk category weight, the error coefficient and the overall error threshold, a category error threshold of the low-risk category; and wherein a product of the category error threshold of the low-risk category and the low-risk category weight is not greater than a sum of a product of the category error threshold of the high-risk category and the high-risk category weight.

[0072] Based on the above overall error threshold, the category error threshold of the high-risk category, and the relationship between the overall error threshold and the category error threshold of the high-risk category and the category error threshold of the low-risk category, it can be known that the product of the category error threshold of the low-risk category and the low-risk category weight is not greater than the sum of the product of the category error threshold of the high-risk category and the high-risk category weight, i.e. .

[0073] Therefore, the category error threshold of the low-risk category is . Since , in actual calculation, it can be taken that , which maximizes the allowed error of the low-risk category while satisfying the overall error constraint, and a is the error coefficient.

[0074] Therefore, in the case of determining the sum of the category weights of each low-risk category as the low-risk category weight, the sum of the category weights of each high-risk category as the high-risk category weight, and the overall error threshold and the error coefficient, the category error threshold of the low-risk category is calculated.

[0075] In some implementations of the embodiments of the present application, the determining of the category weight corresponding to each different risk category based on the sampling sample comprises: determining a risk severity score value corresponding to each risk category and a sampling sample proportion; the sampling sample proportion is a proportion of the number of sampling samples corresponding to the same risk category to the total amount of sampling samples; determining the category weight according to the risk severity score value, the sampling sample proportion, a first adjustment coefficient corresponding to the risk severity score value, and a second adjustment coefficient corresponding to the sampling sample proportion.

[0076] The category weight According to the severity of the risk category and the sampling sample proportion of the risk category, specifically, a risk severity score [0-1] is set for each risk category. At the same time, the sampling sample proportion of the risk category is calculated as The category weight is calculated by the following formula:

[0077] is the risk severity score corresponding to the risk category c, λ is the first adjustment coefficient and k is the second adjustment coefficient (for example, λ = 0.6, k = 0.4), and the weight is normalized = 1). The category weight is used to represent the importance of each risk category in the safety evaluation, and the larger the sampling sample proportion of the risk category, the greater the impact of the category on the overall error.

[0078] In some implementations of the embodiments of the present application, before determining the expert label of the evaluation sample as the safety evaluation information of the evaluation sample for the evaluation sample whose uncertainty value is greater than the dynamic threshold, uncertainty calibration is performed, which includes: determining the risk category of the evaluation sample; determining the evaluation sample group to which the evaluation sample belongs and the true error rate corresponding to the evaluation sample group; the evaluation sample group corresponds to the same risk category and the same uncertainty value interval; the true error rate is determined based on the sampling sample set; calculating the uncertainty mean of the evaluation sample group; when the difference between the uncertainty mean and the true error rate is greater than a preset deviation tolerance, determining the uncertainty value of the evaluation sample based on the true error rate.

[0079] According to the preset compliance standard information, a plurality of risk categories are divided, and the risk categories are clustered to divide different evaluation sample groups, and the uncertainty value range is divided into a plurality of uncertainty value intervals, and for different uncertainty value intervals of the same risk category, corresponding evaluation sample groups are set. For example: the uncertainty value range is divided into 10 uncertainty value intervals, so that 10 evaluation sample groups of the same risk category can be obtained, and the 10 evaluation sample groups correspond to the same risk category and different uncertainty value intervals. For each evaluation sample group, the true error rate is calculated based on the above sampling sample set, , represents the evaluation sample set of the evaluation sample group b of the risk category c, and the evaluation sample group b corresponds to an uncertainty value interval.

[0080] If greater than the preset deviation tolerance value, it is indicated that the uncertainty value of the evaluation sample set of the evaluation sample group b of the risk category c needs to be corrected, and the uncertainty values of the evaluation samples in the evaluation sample set are corrected to the real error rate, so that the uncertainty values of the evaluation samples are more in line with the real situation, and the accuracy of the final security evaluation result is improved.

[0081] By combining the determination of the uncertainty threshold and the dynamic threshold, the uncertainty calibration, mathematical error rate control and confidence guarantee are provided, and subjective interference is reduced. For example, a clear error rate limit (such as an error rate of no more than 5%) can be set, and the judgment reliability of the security evaluation model is quantified through the uncertainty calibration to ensure that the evaluation result is consistent and objective.

[0082] At the same time, statistical interpretation of the evaluation result can also be provided. For example, the reason for manually evaluating a certain evaluation sample (such as uncertainty higher than a certain threshold) can be explicitly stated, and the confidence interval of the overall error rate is given, which increases the transparency of expert review of the evaluation sample, and thus enhances the explainability and credibility of the evaluation result, making it more practical in high-risk scenarios.

[0083] In addition, in combination with the determination of the uncertainty threshold and the dynamic threshold, the triggering of expert review through the uncertainty calibration can also optimize the evaluation accuracy through the dual mechanisms of statistical reliability guarantee and risk grading control, while avoiding the high cost of full manual review.

[0084] It should be noted that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0085] Referring to Figure 3 , a schematic diagram of a large language model security evaluation device provided by an embodiment of the present application is shown, which can specifically include a model evaluation module 301, a dynamic threshold determination module 302, an expert label replacement module 303, and a security evaluation result generation module 304, wherein: The model evaluation module 301 is configured to obtain security evaluation information obtained by a preset security evaluation model for evaluating a plurality of evaluation samples, and an uncertainty value corresponding to the security evaluation information; the evaluation samples include input information and output information generated by a to-be-evaluated large model for each input information; The dynamic threshold determination module 302 is configured to determine a dynamic threshold according to a sampling sample set; the sampling sample set includes a plurality of evaluation samples, and an uncertainty value and an expert label corresponding to the evaluation samples; The expert label replacement module 303 is configured to determine, for the evaluation sample with the uncertainty value greater than the dynamic threshold value, an expert label for the evaluation sample as the safety evaluation information of the evaluation sample. The safety evaluation result generation module 304 is configured to generate a safety evaluation result of the to-be-evaluated large model according to each evaluation sample and the corresponding safety evaluation information.

[0086] In some implementations of the embodiments of the present application, the safety evaluation result includes an evaluation error rate and an evaluation confidence of the evaluation error rate, and the apparatus further includes: The dynamic threshold adjustment module is configured to increase the number of evaluation samples in the sampling sample set and re-call the dynamic threshold determination module 302, the expert label replacement module 303, and the safety evaluation result generation module 304 in the case that the evaluation error rate is greater than a preset overall error threshold value or the evaluation confidence is less than a preset confidence threshold value.

[0087] In some implementations of the embodiments of the present application, the dynamic threshold determination module 302 includes: The sampling module is configured to sample from the plurality of evaluation samples to obtain a plurality of sampling samples. The expert label determination submodule is configured to determine an expert label corresponding to each input information in the sampling sample. The error upper limit value calculation submodule is configured to calculate an error upper limit value according to the sampling sample and the expert label. The dynamic threshold determination submodule is configured to determine a dynamic threshold value according to the error upper limit value.

[0088] In some implementations of the embodiments of the present application, the error upper limit value calculation submodule includes: The target sample screening unit is configured to determine a target sample from the sampling sample, wherein the output information of the target sample is different from the expert label and the uncertainty value is higher than an uncertainty critical value. The weighted loss value calculation unit is configured to calculate a weighted loss value for each target sample. The mean value and standard deviation value determination unit is configured to determine a weighted loss mean value and a weighted loss standard deviation value based on the weighted loss value. The product value determination unit is configured to determine a product value of a normal distribution quantile corresponding to a preset confidence threshold value and the weighted loss standard deviation value. The error upper limit value determination unit is configured to determine a sum of the weighted loss mean value and the product value as an error upper limit value corresponding to the uncertainty critical value.

[0089] In some implementations of the embodiments of the present application, the dynamic threshold determination submodule includes: a risk category determining unit configured to determine a plurality of risk categories, and call the following units for each risk category: a category error threshold determining unit configured to determine a category error threshold, the category error threshold being used to represent a proportion of output information error of the same category; a dynamic threshold determining unit configured to determine a dynamic threshold, the dynamic threshold being a minimum one of uncertainty critical values corresponding to each error upper limit value greater than the category error threshold.

[0090] In some implementations of the embodiments of the present application, the category error threshold determining unit is configured to, if the risk category is a high-risk category, call the following sub-units: an error coefficient determining sub-unit configured to determine an error coefficient, the error coefficient being a positive number less than 1; a high-risk category category error threshold sub-unit configured to determine a product of the error coefficient and the overall error threshold as a category error threshold of the high-risk category.

[0091] In some implementations of the embodiments of the present application, the category error threshold determining unit is configured to, if the risk category is a low-risk category, call the following sub-units: a category weight determining sub-unit configured to determine, based on the sampling samples, a category weight corresponding to each different risk category; a low-risk category weight determining sub-unit configured to determine a sum of the category weights of each low-risk category as a low-risk category weight; a high-risk category weight determining sub-unit configured to determine a sum of the category weights of each high-risk category as a high-risk category weight; a low-risk category category error threshold sub-determining unit configured to determine, based on the low-risk category weight, the high-risk category weight, the error coefficient and the overall error threshold, a category error threshold of the low-risk category; wherein a product of the category error threshold of the low-risk category and the low-risk category weight and a product of the category error threshold of the high-risk category and the high-risk category weight are not greater than the overall error threshold.

[0092] In some implementations of the embodiments of the present application, the category weight determining sub-unit is specifically configured to determine a risk severity score value corresponding to each risk category and a sampling sample proportion, the sampling sample proportion being a proportion of a number of sampling samples corresponding to the same risk category and a total amount of sampling samples, and determine the category weight according to the risk severity score value, the sampling sample proportion, a first adjustment coefficient corresponding to the risk severity score value and a second adjustment coefficient corresponding to the sampling sample proportion.

[0093] In some implementations of the embodiments of the present application, before determining, for the evaluation sample whose uncertainty value is greater than the dynamic threshold, that the expert label for the evaluation sample is the safety evaluation information of the evaluation sample, the apparatus further includes: a risk category determination module configured to determine a risk category of the evaluation sample; a true error rate determination module configured to determine an evaluation sample group to which the evaluation sample belongs and a true error rate corresponding to the evaluation sample group; the evaluation sample group corresponds to a same risk category and a same uncertainty value interval; and the true error rate is determined based on the sample set; an uncertainty mean value calculation module configured to calculate an uncertainty mean value of the evaluation sample group; an uncertainty mean value correction module configured to, when a difference between the uncertainty mean value and the true error rate is greater than a preset deviation tolerance value, determine the uncertainty value of the evaluation sample based on the true error rate.

[0094] The apparatus provided by the embodiments of the present application can be applied to implement each step in the various method embodiments.

[0095] For the apparatus embodiments, since they are basically similar to the method embodiments, they are described more simply, and the related parts refer to the description in the method embodiment part.

[0096] Referring to Figure 4 , a schematic diagram of an electronic device is shown. As shown in Figure 4 , the electronic device 400 in the embodiments of the present application includes a processor 410, a memory 420, and a computer program 421 stored in the memory 420 and executable on the processor 410. The processor 410 implements the steps in the various embodiments of the above large language model safety evaluation method when executing the computer program 421, such as Figure 1 steps 101 to 105. Alternatively, the processor 410 implements the functions of each module / unit in the above various apparatus embodiments when executing the computer program 421, such as Figure 3 the functions of the modules 301 to 305.

[0097] Exemplarily, the computer program 421 can be divided into one or more modules / units stored in the memory 420 and executed by the processor 410 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which can be used to describe the execution process of the computer program 421 in the electronic device 400. For example, the computer program 421 can be divided into a model evaluation module, a dynamic threshold determination module, an expert label replacement module, and a security evaluation result generation module, each of which has the following specific functions: The model evaluation module is configured to obtain security evaluation information obtained by a preset security evaluation model for a plurality of evaluation samples, and an uncertainty value corresponding to the security evaluation information; the evaluation samples include input information and output information generated by a to-be-evaluated large model for each input information; The dynamic threshold determination module is configured to determine a dynamic threshold according to a sampling sample set; the sampling sample set includes a plurality of evaluation samples, uncertainty values corresponding to the evaluation samples, and expert labels; The expert label replacement module is configured to determine, for an evaluation sample with an uncertainty value greater than the dynamic threshold, an expert label for the evaluation sample as the security evaluation information of the evaluation sample; The security evaluation result generation module is configured to generate a security evaluation result of the to-be-evaluated large model according to each evaluation sample and the corresponding security evaluation information.

[0098] The electronic device 400 can be a desktop computer, a cloud server, or the like. The electronic device 400 can include, but is not limited to, a processor 410 and a memory 420. Those skilled in the art can understand that, Figure 4 The electronic device 400 is only an example and does not constitute a limitation on the electronic device 400, which can include more or fewer components than shown, or combine certain components, or different components, for example, the electronic device 400 can also include an input / output device, a network access device, a bus, and the like.

[0099] The processor 410 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0100] The memory 420 can be an internal storage unit of the electronic device 400, such as a hard disk or a memory of the electronic device 400. The memory 420 can also be an external storage device of the electronic device 400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 420 can also include both the internal storage unit and the external storage device of the electronic device 400. The memory 420 is used to store the computer program 421 and other programs and data required by the electronic device 400. The memory 420 can also be used to temporarily store data that has been output or will be output.

[0101] The embodiments of the present application also disclose a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the large language model security evaluation method according to the foregoing embodiments.

[0102] The embodiments of the present application also disclose a computer program product, which includes a computer program. The computer program is executed to make the large language model security evaluation method according to the foregoing embodiments be executed.

[0103] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit them. Although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A large language model security assessment method, characterized by: The method comprises: Obtaining security assessment information obtained by evaluating a plurality of evaluation samples by a preset security assessment model, and uncertainty values ​​corresponding to the security assessment information; the evaluation samples include input information and output information generated by the large model to be evaluated for each input information; Determining a dynamic threshold based on a sample set; the sample set includes a plurality of evaluation samples and uncertainty values ​​and expert labels corresponding to the evaluation samples; For the evaluation sample whose uncertainty value is greater than the dynamic threshold, determining the expert label for the evaluation sample as the security assessment information of the evaluation sample; Based on each evaluation sample and the corresponding security assessment information, the security assessment results of the large model to be evaluated are generated.

2. The method according to claim 1, characterized in that The safety assessment result includes an assessment error rate and an assessment confidence level of the assessment error rate. The method further includes: When the evaluation error rate is greater than the preset overall error threshold or the evaluation confidence is less than the preset confidence threshold, the number of evaluation samples in the sampling sample set is increased, and the process returns to the step of determining the dynamic threshold based on the sampling sample set.

3. The method according to claim 2, characterized in that Determining the dynamic threshold based on the sample set includes: Sampling from the plurality of evaluation samples to obtain a plurality of sampling samples; Determining expert labels corresponding to each input information in the sample; Calculating an upper error limit based on the sample and the expert label; A dynamic threshold is determined according to the upper error limit.

4. The method according to claim 3, characterized in that The calculating of the upper limit of error based on the sample and the expert label includes: Determine a target sample from the sampled samples; the output information of the target sample is different from the expert label and the uncertainty value is higher than the uncertainty critical value; Calculate the weighted loss value for each target sample; Determine a weighted loss mean and a weighted loss standard deviation based on the weighted loss values; Determine the product value of the normal distribution quantile corresponding to the preset confidence threshold and the weighted loss standard deviation value; Determine the sum of the weighted loss mean and the product value as an upper error limit value corresponding to the uncertainty critical value.

5. The method according to claim 4, characterized in that Determining the dynamic threshold value according to the upper error limit includes: Identify various risk categories; perform the following steps for each risk category: Determine a category error threshold; the category error threshold is used to represent the error ratio of output information of the same category; Determine a dynamic threshold; the dynamic threshold is the smallest one of the uncertainty critical values ​​corresponding to the upper error limits that are greater than the category error threshold.

6. The method according to claim 5, characterized in that If the risk category is a high risk category, determining the category error threshold includes: Determine an error coefficient; the error coefficient is a positive number less than 1; The product of the error coefficient and the overall error threshold is determined to be a category error threshold for a high-risk category.

7. The method according to claim 6, characterized in that If the risk category is low risk; Determining the category error threshold includes: Determining category weights corresponding to different risk categories based on the sampled samples; Determine the sum of the category weights of each low-risk category as the low-risk category weight; Determine the sum of the category weights of each high-risk category as the high-risk category weight; Determining a category error threshold for the low-risk category based on the low-risk category weight, the high-risk category weight, the error coefficient, and the overall error threshold; The sum of the product of the category error threshold of the low-risk category and the low-risk category weight and the product of the category error threshold of the high-risk category and the high-risk category weight is not greater than the overall error threshold.

8. The method according to claim 7, characterized in that Determining the category weights corresponding to different risk categories based on the sample includes: Determine the risk severity score and sample ratio corresponding to each risk category; the sample ratio is the ratio of the number of sampled samples corresponding to the same risk category to the total number of sampled samples; The category weight is determined based on the risk severity score value, the sampling sample proportion, the first adjustment coefficient corresponding to the risk severity score value, and the second adjustment coefficient corresponding to the sampling sample proportion.

9. The method according to claim 1, characterized in that Before determining, for the evaluation sample whose uncertainty value is greater than the dynamic threshold, that the expert label for the evaluation sample is the security assessment information of the evaluation sample, the method further includes: Determining the risk category of the assessment sample; Determining an evaluation sample group to which the evaluation sample belongs and a true error rate corresponding to the evaluation sample group; the evaluation sample group corresponds to the same risk category and the same uncertainty value interval; the true error rate is determined based on the sampling sample set; Calculating the uncertainty mean of the evaluation sample group; When the difference between the uncertainty mean and the true error rate is greater than a preset deviation tolerance, the uncertainty value of the evaluation sample is determined based on the true error tolerance.

10. A large language model security assessment device, characterized in that: include: A model evaluation module is configured to obtain security assessment information obtained by evaluating a plurality of evaluation samples using a preset security assessment model, and uncertainty values ​​corresponding to the security assessment information; the evaluation samples include input information and output information generated by the large model to be evaluated for each input information; A dynamic threshold determination module is used to determine a dynamic threshold based on a sample set; the sample set includes a number of evaluation samples and uncertainty values ​​and expert labels corresponding to the evaluation samples; An expert label replacement module is used to determine, for an evaluation sample whose uncertainty value is greater than a dynamic threshold, that the expert label for the evaluation sample is the security assessment information of the evaluation sample; The security assessment result generation module is used to generate the security assessment results of the large model to be evaluated based on each evaluation sample and the corresponding security assessment information.

11. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 9.

12. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables the method according to any one of claims 1 to 9 to be performed.

Images