Large Language Model Security Assessment Methods, Devices and Electronic Equipment

Abstract

This application applies to the field of artificial intelligence model technology, providing a method, apparatus, and electronic device for security assessment of large language models. The method includes: acquiring security assessment information obtained by a preset security assessment model evaluating multiple test samples, and the uncertainty value corresponding to the security assessment information; the test samples include input information and output information generated by the large model to be evaluated for each input information; determining a dynamic threshold based on a sample set; for test samples with uncertainty values ​​greater than the dynamic threshold, determining expert labels for the test samples as the security assessment information of the test samples; and generating a security assessment result for the large model to be evaluated based on each test sample and its corresponding security assessment information. This application embodiment can efficiently complete the security assessment of the large model to be evaluated and ensure the accuracy and reliability of the security assessment results.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application belongs to the technical field of artificial intelligence models, and particularly relates to a large language model security evaluation method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of generative artificial intelligence technology, artificial intelligence large models have shown great capabilities in the field of natural language processing. The security and privacy protection of artificial intelligence large models have increasingly attracted attention. Artificial intelligence large language model security evaluation faces unique challenges such as diversified risk categories, large data size, and real-time requirements.

[0003] Current security evaluation methods for artificial intelligence large models mostly rely on manual judgment, which is costly and inefficient. Although some research attempts to use artificial intelligence models for preliminary evaluation, these methods lack statistical error rate guarantees, and the reliability of the evaluation results is difficult to meet actual needs.

[0004] Therefore, there is an urgent need for an efficient, accurate, and cost-controllable large language model security evaluation method. SUMMARY

[0005] Therefore, the embodiment of the present application provides a large language model security evaluation method and device and electronic equipment to efficiently complete the security evaluation of the to-be-evaluated large model and ensure the accuracy and reliability of the security evaluation results of the to-be-evaluated large model.

[0006] The first aspect of the embodiment of the present application provides a large language model security evaluation method, comprising:

[0007] obtaining security evaluation information obtained by a preset security evaluation model for a plurality of evaluation samples, and an uncertainty value corresponding to the security evaluation information; the evaluation sample includes input information and output information generated by a to-be-evaluated large model for each input information;

[0008] determining a dynamic threshold according to a sampling sample set; the sampling sample set includes a plurality of evaluation samples, and the uncertainty values and expert labels corresponding to the evaluation samples;

[0009] for the evaluation sample whose uncertainty value is greater than the dynamic threshold, determining that the expert label for the evaluation sample is the security evaluation information of the evaluation sample;

[0010] generating a security evaluation result of the to-be-evaluated large model according to each evaluation sample and the corresponding security evaluation information.

[0011] In some implementations of the first aspect, the security evaluation result includes an evaluation error rate and an evaluation confidence of the evaluation error rate, and the method further comprises:

[0012] In a case where the evaluation error rate is greater than a preset overall error threshold or the evaluation confidence is less than a preset confidence threshold, the number of evaluation samples in the sampling sample set is increased, and the step of determining the dynamic threshold according to the sampling sample set is returned to.

[0013] In some implementations of the first aspect, the determining the dynamic threshold according to the sampling sample set and the expert labels comprises:

[0014] sampling from the plurality of evaluation samples to obtain a plurality of sampling samples;

[0015] determining expert labels corresponding to each input information in the sampling samples;

[0016] calculating an error upper limit value according to the sampling samples and the expert labels;

[0017] determining a dynamic threshold according to the error upper limit value.

[0018] In some implementations of the first aspect, the calculating the error upper limit value according to the sampling samples and the expert labels comprises:

[0019] determining target samples from the sampling samples; the output information of the target samples is different from the expert labels and the uncertainty value is higher than an uncertainty threshold;

[0020] calculating a weighted loss value for each target sample;

[0021] determining a weighted loss mean value and a weighted loss standard deviation value based on the weighted loss value;

[0022] determining a product value of a normal distribution quantile corresponding to a preset confidence threshold and the weighted loss standard deviation value;

[0023] determining a sum of the weighted loss mean value and the product value as an error upper limit value corresponding to the uncertainty threshold.

[0024] In some implementations of the first aspect, the determining the dynamic threshold according to the error upper limit value comprises:

[0025] determining a plurality of risk categories; the following steps are performed for each risk category:

[0026] determining a category error threshold; the category error threshold is used to represent a proportion of output information errors of the same category;

[0027] determining a dynamic threshold; the dynamic threshold is a minimum one of uncertainty thresholds corresponding to each error upper limit value greater than the category error threshold.

[0028] In some implementations of the first aspect, if the risk category is a high risk category, the determining the category error threshold comprises:

[0029] determining an error coefficient, the error coefficient being a positive number less than 1;

[0030] determining a product of the error coefficient and the overall error threshold as the category error threshold for the high risk category.

[0031] In some implementations of the first aspect, if the risk category is a low risk category, the determining the category error threshold comprises:

[0032] determining a category weight corresponding to each different risk category based on the sampling samples;

[0033] determining a sum of the category weights of each low risk category as a low risk category weight;

[0034] determining a sum of the category weights of each high risk category as a high risk category weight;

[0035] determining a category error threshold for the low risk category based on the low risk category weight, the high risk category weight, the error coefficient and the overall error threshold;

[0036] wherein a product of the category error threshold for the low risk category and the low risk category weight is not greater than a sum of a product of the category error threshold for the high risk category and the high risk category weight.

[0037] In some implementations of the first aspect, the determining the category weight corresponding to each different risk category based on the sampling samples comprises:

[0038] determining a risk severity score value corresponding to each risk category and a sampling sample proportion, the sampling sample proportion being a proportion of a number of sampling samples corresponding to the same risk category and a total amount of sampling samples;

[0039] determining the category weight according to the risk severity score value, the sampling sample proportion, a first adjustment coefficient corresponding to the risk severity score value, and a second adjustment coefficient corresponding to the sampling sample proportion.

[0040] In some implementations of the first aspect, before the determining the expert label for the evaluation sample as the safety evaluation information of the evaluation sample for the evaluation sample with the uncertainty value greater than the dynamic threshold, the method further comprises:

[0041] determining a risk category of the evaluation sample;

[0042] The evaluation sample group to which the evaluation sample belongs and the true error rate corresponding to the evaluation sample group are determined; the evaluation sample group corresponds to the same risk category and the same uncertainty value range; the true error rate is determined based on the sampling sample set;

[0043] Calculate the mean uncertainty of the evaluation sample group;

[0044] When the difference between the mean uncertainty and the true error rate is greater than a preset deviation tolerance, the uncertainty value of the evaluation sample is determined based on the true error tolerance.

[0045] A second aspect of this application provides a large language model security evaluation apparatus, comprising:

[0046] The model evaluation module is used to obtain security evaluation information obtained by evaluating multiple evaluation samples by a preset security evaluation model, as well as the uncertainty value corresponding to the security evaluation information; the evaluation samples include input information and output information generated by the large model to be evaluated for each input information;

[0047] A dynamic threshold determination module is used to determine a dynamic threshold based on a sample set; the sample set includes several evaluation samples and the uncertainty values ​​and expert labels corresponding to the evaluation samples;

[0048] The expert label replacement module is used to determine the expert label for the evaluation sample as the security assessment information of the evaluation sample for the evaluation sample whose uncertainty value is greater than the dynamic threshold.

[0049] The security assessment result generation module is used to generate security assessment results for the large model to be evaluated based on each evaluation sample and the corresponding security assessment information.

[0050] A third aspect of this application provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the large language model security assessment method as described in the first aspect above.

[0051] A fourth aspect of this application provides a computer program product, including a computer program that, when run, causes the large language model security assessment method described in the first aspect above to be executed.

[0052] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the large language model security assessment method as described in the first aspect above.

[0053] The embodiments of this application have the following beneficial effects:

[0054] By acquiring security assessment information obtained from evaluating multiple test samples using a preset security assessment model, and the corresponding uncertainty values, the following steps are taken: The test samples include input information and output information generated by the large model to be evaluated for each input; a dynamic threshold is determined based on a sample set; the sample set includes several test samples, their corresponding uncertainty values, and expert labels; for test samples with uncertainty values ​​greater than the dynamic threshold, the expert labels for those samples are determined as the security assessment information; based on each test sample and its corresponding security assessment information, a security assessment result for the large model to be evaluated is generated. This allows the security assessment model to automatically evaluate the output information generated by the large model to be evaluated, obtaining security assessment information, and quantifying the uncertainty of the security assessment information generated by the model to obtain an uncertainty value. By setting a dynamic threshold and comparing it with the uncertainty value, and by conducting expert review on test samples with uncertainty values ​​greater than the dynamic threshold, and using expert labels as the security assessment information for test samples with uncertainty values ​​greater than the dynamic threshold, expert review of high-uncertainty test samples is achieved to ensure the accuracy and reliability of the final security assessment result. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of a large language model security assessment method provided in an embodiment of this application;

[0057] Figure 2 This is a schematic diagram of the process framework of a large language model security assessment method provided in an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of a large language model security assessment device provided in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0061] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0062] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0063] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0064] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0065] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0066] The technical solution of this application will be described below through specific embodiments.

[0067] Reference Figure 1The diagram illustrates a method for security assessment of large language models provided in an embodiment of this application, which may specifically include the following steps:

[0068] Step 101: Obtain security assessment information obtained by evaluating multiple test samples using a preset security assessment model, and the uncertainty value corresponding to the security assessment information; the test samples include input information and output information generated by the large model to be evaluated for each input information;

[0069] The generative AI large language model to be evaluated is determined in advance, i.e. the large model to be evaluated, and multiple input information that the large model to be evaluated needs to process.

[0070] Input information can be input into the large model to be evaluated, and the large model to be evaluated can generate corresponding output information for each input information. Evaluation samples are obtained based on the corresponding input and output information.

[0071] An AI model for security assessment of the test samples is pre-defined as the security assessment model. This model is then guided to perform security assessments on the test samples, yielding corresponding security assessment information and uncertainty values. Each test sample corresponds to one security assessment information, which characterizes whether the output information generated by the large model under assessment in response to the input information is compliant. Security assessment information includes both "safe" and "insecure." Specifically, a test sample's security assessment information is either "safe" or "insecure." If the information is "safe," it indicates that the security assessment model recognizes the output information of the test sample as compliant; if it is "insecure," it indicates that the output information of the test sample is not compliant.

[0072] In practical implementation, the input information of each evaluation sample can be labeled as a question and the output information can be labeled as an answer. This guides the security assessment model to output a security assessment value and the credibility of the security assessment value for each evaluation sample, and the uncertainty value is obtained based on the credibility.

[0073] For example: the security assessment model outputs a security assessment value of 0 or 1, with a confidence level ranging from 0 to 10. A security assessment value of 1 indicates that the security assessment information is secure, while a security assessment value of 0 indicates that the security assessment information is insecure. A confidence level of 0 indicates that the security assessment value is completely reliable, while a confidence level of 10 indicates that the security assessment value is completely unreliable. The confidence level normalized value range [0,1] is used as the uncertainty score. The higher the uncertainty score, the greater the uncertainty of the security assessment information in the security assessment model.

[0074] Step 102: Determine the dynamic threshold based on the sample set; the sample set includes several evaluation samples and the uncertainty values ​​and expert labels corresponding to the evaluation samples.

[0075] Expert tags are the real security information determined by designated personnel for each evaluation sample in the sampled test set.

[0076] By comparing the output information in the evaluation samples with the actual security information, the error rate in all evaluation samples can be estimated. The estimated error rate determines a dynamic threshold, which allows for manual verification of evaluation samples with uncertainty values ​​greater than the dynamic threshold, thus ensuring that the error rate of all evaluation samples is not necessarily a fixed value.

[0077] Step 103: For the evaluation sample whose uncertainty value is greater than the dynamic threshold, determine the expert label for the evaluation sample as the security assessment information of the evaluation sample;

[0078] For evaluation samples with uncertainty values ​​exceeding the dynamic threshold, manual verification is performed. Specifically, this involves receiving expert labels from designated personnel, indicating whether the sample is safe or unsafe. These expert labels replace the security assessment information obtained from the security assessment model, thus completing the expert review of evaluation samples with uncertainty values ​​exceeding the dynamic threshold and obtaining accurate security assessment information.

[0079] By automatically selecting evaluation samples that require expert review based on dynamic thresholds, the workload of manual processing is significantly reduced, and the efficiency of security assessment is improved.

[0080] In one example, only 10% of the evaluation samples require expert review, while the remaining 90% are evaluated using security assessment information output by the security assessment model. Compared to manually determining all security assessment information, this saves approximately 90% of time and manpower costs, significantly improving the efficiency of security assessments of large-scale models.

[0081] Step 104: Based on each evaluation sample and the corresponding security assessment information, generate the security assessment results of the large model to be evaluated.

[0082] The security assessment results of the large model under evaluation can be obtained by statistically analyzing the evaluation samples and their corresponding security assessment information. These results can include the percentage of evaluation samples with secure security assessment information, thus determining the security of the large model's generation and processing. The results can also include the evaluation error rate and its confidence level; the evaluation error rate and confidence level characterize the performance of the model's generation and processing.

[0083] Since the above steps compare the uncertainty value of the evaluation sample with the dynamic threshold, which is based on the estimated error rate of all evaluation samples, the dynamic threshold can be adjusted by the magnitude of the expected value. Furthermore, by having the evaluation samples with uncertainty values ​​greater than the dynamic threshold reviewed by experts, the evaluation error rate of the safety assessment results can be made less than the expected value, thereby ensuring the accuracy of the safety assessment results.

[0084] In this embodiment, except for expert tags which require manual processing, the determination of security assessment information, the generation of uncertainty values, the comparison of uncertainty values ​​with dynamic thresholds, and the security assessment results in the above process can all be automated without full manual processing. This allows for rapid response to large model updates, and the security assessment of the large model under test can be completed in a short time, breaking through the efficiency bottleneck of purely manual assessment and meeting the requirements of iteration timeliness.

[0085] In this embodiment, security assessment information obtained by a preset security assessment model evaluating multiple test samples, and the uncertainty value corresponding to the security assessment information, are acquired. The test samples include input information and output information generated by the large model to be evaluated for each input. A dynamic threshold is determined based on a sampling sample set. The sampling sample set includes several test samples, their corresponding uncertainty values, and expert labels. For test samples with uncertainty values ​​greater than the dynamic threshold, the expert labels for those samples are determined as the security assessment information. Based on each test sample and its corresponding security assessment information, a security assessment result for the large model to be evaluated is generated. This allows the security assessment model to automatically evaluate the output information generated by the large model to be evaluated, obtaining security assessment information, and quantifying the uncertainty of the security assessment information generated by the model to obtain an uncertainty value. By setting a dynamic threshold and comparing it with the uncertainty value, and by conducting expert review on test samples with uncertainty values ​​greater than the dynamic threshold, and using expert labels as the security assessment information for test samples with uncertainty values ​​greater than the dynamic threshold, expert review of high-uncertainty test samples is achieved to ensure the accuracy and reliability of the final security assessment result.

[0086] In some implementations of this application, the security assessment result includes an assessment error rate and an assessment confidence level of the assessment error rate; the method further includes:

[0087] If the evaluation error rate is greater than the preset overall error threshold or the evaluation confidence level is less than the preset confidence level threshold, the number of evaluation samples in the sampling sample set is increased, and the process returns to the step of determining the dynamic threshold based on the sampling sample set.

[0088] Before step 102, the overall error threshold can be predetermined. Since the higher the uncertainty value, the greater the possibility that the output information differs from the real security information, for the same sample to be evaluated, the proportion of samples that need to be reviewed by experts can be adjusted by adjusting the size of the dynamic threshold. The smaller the dynamic threshold, the larger the proportion of samples that need to be reviewed by experts, and the lower the evaluation error rate of the corresponding security assessment results.

[0089] As an example, with a preset confidence threshold of (1-α), the overall error rate can be calculated. ,in The security assessment information output by the security assessment model. Label experts. Determine the upper confidence bound (CI) for the overall error rate.

[0090]

[0091] If CI is not greater than the overall error threshold, the assessment confidence level is determined to be not less than the preset confidence level threshold; if CI is greater than the overall error threshold, the assessment confidence level is determined to be less than the preset confidence level threshold. Here, z is the z-value of the normal distribution.

[0092] If the evaluation error rate is greater than the preset overall error threshold or the evaluation confidence level is less than the preset confidence level threshold, the number of evaluation samples in the sampling sample set is increased, and a dynamic threshold is determined based on the sampling sample set to improve the evaluation error rate and evaluation confidence level of the security evaluation results.

[0093] Reference Figure 2 The diagram illustrates a flowchart of another large language model security assessment method provided in this application embodiment.

[0094] In some implementations of this application, determining the dynamic threshold based on the sample set includes: sampling from multiple evaluation samples to obtain multiple sampled samples; determining the expert label corresponding to each input information in the sampled samples; calculating an upper limit of error based on the sampled samples and the expert labels; and determining the dynamic threshold based on the upper limit of error.

[0095] Randomly select evaluation samples covering various risk dimensions as sampling samples, conduct expert review of the sampling samples to obtain expert labels for the sampling samples, and obtain the sampling sample set based on the sampling samples and expert labels. , where m is the number of samples. For sampling, The safety assessment information output by the safety assessment model for the sampled samples. These are real, human-generated labels. This represents the uncertainty value output by the safety assessment model for the sampled data.

[0096] Specifically, for different expert labels of the same sample, the one with the highest percentage is taken as the real human label. For example, if 10,000 evaluation samples are sampled, 500 samples are selected as the sampling sample, and 3 designated units perform cross-labeling on each sampling sample, and the majority result is taken as the expert label.

[0097] Based on the sampled samples and the expert labels corresponding to each sampled sample, the upper limit of error is calculated, and a dynamic threshold is obtained based on the upper limit of error, so that the error rate of the security assessment result in step 104 is not greater than the upper limit of error.

[0098] In some implementations of this application, the step of calculating the upper limit of error based on the sampled samples and the expert labels includes: determining target samples from the sampled samples; the output information of the target samples is different from the expert labels and the uncertainty value is higher than the uncertainty threshold; calculating weighted loss values ​​for each target sample; determining the weighted loss mean and weighted loss standard deviation based on the weighted loss values; determining the product of the normal distribution quantile corresponding to a preset confidence threshold and the weighted loss standard deviation; and determining the sum of the weighted loss mean and the product as the upper limit of error corresponding to the uncertainty threshold.

[0099] A weighted loss formula can be constructed to calculate the weighted loss value for each sample. .

[0100] ,in The uncertainty threshold is used as a constraint to select samples whose uncertainty value is not greater than the uncertainty threshold, and then the weighted loss value is calculated.

[0101] in, It is a binary variable indicating whether sample i was selected. This indicates that you have been selected. 0 indicates that you were not selected. It is the sampling probability of sample i. For an indicator function, if and only if ≠ When the condition is met, the indicator function takes the value 1; otherwise, the indicator function takes the value 0.

[0102] pass and The system achieves a weighted loss of the security assessment information output by the sampling sample set security assessment model compared with the actual human-labeled samples, and evaluates the total error of all evaluation samples.

[0103] Furthermore, the formula for calculating the upper limit of error is as follows: .

[0104] Among them, the weighted average loss Weighted loss standard deviation , It is the confidence threshold quantile of the standard normal distribution.

[0105] In some implementations of this application, determining the dynamic threshold based on the upper limit of error includes: determining multiple risk categories; performing the following steps for each risk category: determining a category error threshold; the category error threshold is used to characterize the proportion of output information error in the same category; determining the dynamic threshold; wherein the dynamic threshold is the smallest of the uncertainty thresholds corresponding to the upper limit of each error value greater than the category error threshold.

[0106] In this embodiment of the application, different risk categories are defined, and these risk categories are further divided into high-risk categories and low-risk categories, resulting in a set of high-risk categories and a set of low-risk categories.

[0107] By setting category error thresholds for different risk categories, and determining the dynamic threshold using a formula: c is used to identify the risk category. The formula above can be used to determine the smallest of the uncertainty thresholds corresponding to the upper limit values ​​of each error that are greater than the category error threshold. The target uncertainty threshold is then used as the dynamic threshold for the risk category.

[0108] In some implementations of this application, if the risk category is a high-risk category, determining the category error threshold includes: determining an error coefficient; the error coefficient is a positive number less than 1; and determining the product of the error coefficient and the overall error threshold as the category error threshold for the high-risk category.

[0109] In the specific implementation, the constraint condition for the overall error threshold is set: . Let c be the class weight of risk category c, satisfying Furthermore, the definition can be predetermined. ,

[0110] To ensure a low error threshold for high-risk categories, a positive number less than 1 can be designated as the error coefficient. The product of this error coefficient and the overall error threshold is then used to obtain the category error threshold for the high-risk category. .

[0111] For example: with an error coefficient of 0.6, the overall error threshold is... ,but .

[0112] For the overall error threshold ϵ, the overall error of all sampled samples ,in .

[0113] Among them, for , H is used to identify the set of high-risk categories. , L is used to identify the set of low-risk categories.

[0114] In some implementations of this application, if the risk category is a low-risk category, determining the category error threshold includes: determining the category weights corresponding to each different risk category based on the sampled samples; determining the sum of the category weights of each low-risk category as the low-risk category weight; determining the sum of the category weights of each high-risk category as the high-risk category weight; determining the category error threshold of the low-risk category based on the low-risk category weights, the high-risk category weights, the error coefficients, and the overall error threshold; wherein the sum of the product of the category error threshold of the low-risk category and the low-risk category weights and the product of the category error threshold of the high-risk category and the high-risk category weights is not greater than the overall error threshold.

[0115] Based on the aforementioned overall error threshold, the category error threshold for high-risk categories, and the relationship between the overall error threshold, the category error thresholds for high-risk categories, and the category error thresholds for low-risk categories, it can be concluded that the sum of the product of the category error threshold for low-risk categories and the weight of low-risk categories, and the sum of the product of the category error threshold for high-risk categories and the weight of high-risk categories, is no greater than the overall error threshold. .

[0116] Therefore, the category error threshold for low-risk categories .because In actual calculations, the following can be taken: This maximizes the allowable error for the low-risk category while satisfying the overall error constraint, where 'a' is the aforementioned error coefficient.

[0117] Therefore, given that the sum of the category weights of each low-risk category is the low-risk category weight, the sum of the category weights of each high-risk category is the high-risk category weight, and the overall error threshold and error coefficient are determined, the category error threshold for the low-risk category is calculated.

[0118] In some implementations of this application, determining the category weights corresponding to each different risk category based on the sampled samples includes:

[0119] Determine the risk severity score and sampling percentage for each risk category; the sampling percentage is the ratio of the number of samples corresponding to the same risk category to the total number of samples.

[0120] The category weights are determined based on the risk severity score, the sample proportion, the first adjustment coefficient corresponding to the risk severity score, and the second adjustment coefficient corresponding to the sample proportion.

[0121] Category weight The risk severity score [0-1] is determined based on the severity of the risk category and its sample proportion. Specifically, a risk severity score is assigned to each risk category. Simultaneously, the sample proportion of each risk category is calculated. The category weights are calculated using the following formula:

[0122]

[0123] For risk category c, a risk severity score is assigned, with λ as the first adjustment coefficient and k as the second adjustment coefficient (e.g., λ=0.6, k=0.4), ensuring weight normalization. =1). Category weights are used to characterize the importance of each risk category in the safety assessment. The larger the sample size of a risk category, the greater its impact on the overall error.

[0124] In some implementations of this application, before determining the expert label of an evaluation sample as the safety assessment information of the evaluation sample for the evaluation sample whose uncertainty value is greater than a dynamic threshold, uncertainty calibration is performed. Uncertainty calibration includes: determining the risk category of the evaluation sample; determining the evaluation sample group to which the evaluation sample belongs and the true error rate corresponding to the evaluation sample group; the evaluation sample group corresponds to the same risk category and the same uncertainty value range; the true error rate is determined based on the sampling sample set; calculating the uncertainty mean of the evaluation sample group; when the difference between the uncertainty mean and the true error rate is greater than a preset deviation tolerance, determining the uncertainty value of the evaluation sample based on the true error tolerance.

[0125] Based on preset compliance standards, multiple risk categories are defined, and these risk categories are clustered to form different evaluation sample groups. Furthermore, the possible value range of uncertainty is divided into multiple uncertainty value intervals, and corresponding evaluation sample groups are set up for different uncertainty value intervals within the same risk category. For example, dividing the possible value range of uncertainty into 10 uncertainty value intervals results in 10 evaluation sample groups for the same risk category, each corresponding to a different uncertainty value interval. For each evaluation sample group, the true error rate is calculated based on the aforementioned sample set. , Let b represent the evaluation sample set of evaluation sample group b with risk category c, where evaluation sample group b corresponds to an uncertainty value range.

[0126] like If the deviation exceeds the preset tolerance value, it means that the uncertainty value of the evaluation sample set of the evaluation sample group b with risk category c needs to be corrected. The uncertainty value of each evaluation sample in the evaluation sample set is corrected to the true error rate so that the uncertainty value of the evaluation sample is more consistent with its actual situation, thereby improving the accuracy of the final safety assessment result.

[0127] By combining the determination of uncertainty thresholds and dynamic thresholds, and uncertainty calibration, mathematical error rate control and confidence guarantees are provided, reducing subjective interference. For example, a clear error rate limit can be set (e.g., the error rate should not exceed 5%), and the reliability of the safety assessment model's judgment can be quantified through uncertainty calibration, ensuring consistent and objective assessment results.

[0128] It can also provide statistical interpretations of the evaluation results. For example, it can clearly explain why a certain evaluation sample was manually evaluated (such as uncertainty exceeding a certain threshold) and provide the confidence interval for the overall error rate, increasing the transparency of expert review of the evaluation samples, thereby enhancing the interpretability and credibility of the evaluation results and making them more practical in high-risk scenarios.

[0129] Furthermore, by combining the determination of uncertainty thresholds and dynamic thresholds, and the expert review triggered by uncertainty calibration, the accuracy of the assessment can be optimized through a dual mechanism of statistical reliability assurance and risk classification control, while avoiding the high cost of full manual review.

[0130] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0131] Reference Figure 3This illustration shows a schematic diagram of a large language model security assessment device provided in an embodiment of this application. Specifically, it may include a model assessment module 301, a dynamic threshold determination module 302, an expert label replacement module 303, and a security assessment result generation module 304, wherein:

[0132] The model evaluation module 301 is used to obtain security evaluation information obtained by evaluating multiple evaluation samples by a preset security evaluation model, and the uncertainty value corresponding to the security evaluation information; the evaluation samples include input information and output information generated by the large model to be evaluated for each input information;

[0133] The dynamic threshold determination module 302 is used to determine a dynamic threshold based on a sample set; the sample set includes several evaluation samples and the uncertainty values ​​and expert labels corresponding to the evaluation samples;

[0134] The expert label replacement module 303 is used to determine the expert label for the evaluation sample as the security assessment information of the evaluation sample for the evaluation sample whose uncertainty value is greater than the dynamic threshold.

[0135] The security assessment result generation module 304 is used to generate the security assessment results of the large model to be evaluated based on each evaluation sample and the corresponding security assessment information.

[0136] In some implementations of this application, the security assessment result includes an assessment error rate and an assessment confidence level of the assessment error rate, and the apparatus further includes:

[0137] The dynamic threshold adjustment module is used to increase the number of evaluation samples in the sampling sample set and re-call the dynamic threshold determination module 302, the expert label replacement module 303, and the security assessment result generation module 304 when the evaluation error rate is greater than the preset overall error threshold or the evaluation confidence is less than the preset confidence threshold.

[0138] In some implementations of this application, the dynamic threshold determination module 302 includes:

[0139] The sampling module is used to sample from multiple evaluation samples to obtain multiple sampling samples;

[0140] The expert label determination submodule is used to determine the expert label corresponding to each piece of input information in the sampled sample.

[0141] The error upper limit calculation submodule is used to calculate the error upper limit based on the sampled sample and the expert label;

[0142] The dynamic threshold determination submodule is used to determine the dynamic threshold based on the upper limit of the error.

[0143] In some implementations of this application, the error upper limit calculation submodule includes:

[0144] A target sample screening unit is used to determine a target sample from the sampled samples; the output information of the target sample is different from the expert label and the uncertainty value is higher than the uncertainty threshold.

[0145] The weighted loss calculation unit is used to calculate the weighted loss value for each target sample.

[0146] The mean and standard deviation determination unit is used to determine the weighted loss mean and the weighted loss standard deviation based on the weighted loss value;

[0147] The product value determination unit is used to determine the product value of the normal distribution quantile corresponding to the preset confidence threshold and the weighted loss standard deviation.

[0148] The error upper limit determination unit is used to determine the sum of the weighted loss mean and the product value as the error upper limit value corresponding to the uncertainty critical value.

[0149] In some implementations of the embodiments of this application, the dynamic threshold determination submodule includes:

[0150] The risk category determination unit is used to determine multiple risk categories; and for each risk category, it calls the following units:

[0151] A category error threshold determination unit is used to determine a category error threshold; the category error threshold is used to characterize the proportion of output information errors within the same category;

[0152] A dynamic threshold determination unit is used to determine a dynamic threshold; the dynamic threshold is the smallest of the uncertainty critical values ​​corresponding to the upper limit values ​​of each error that are greater than the category error threshold.

[0153] In some implementations of the embodiments of this application, the category error threshold determination unit is used to call the following sub-unit if the risk category is a high-risk category:

[0154] An error coefficient determination subunit is used to determine the error coefficient; the error coefficient is a positive number less than 1.

[0155] The category error threshold subunit for high-risk categories is used to determine the product of the error coefficient and the overall error threshold as the category error threshold for high-risk categories.

[0156] In some implementations of the embodiments of this application, the category error threshold determination unit is used to call the following sub-unit if the risk category is a low-risk category:

[0157] The category weight determination subunit is used to determine the category weights corresponding to each different risk category based on the sampled samples;

[0158] The low-risk category weight determination subunit is used to determine the sum of the category weights of each low-risk category as the low-risk category weight;

[0159] The high-risk category weight determination subunit is used to determine the sum of the category weights of each high-risk category as the high-risk category weight;

[0160] The category error threshold determination unit for low-risk categories is used to determine the category error threshold for low-risk categories based on the low-risk category weight, the high-risk category weight, the error coefficient, and the overall error threshold.

[0161] Wherein, the sum of the product of the category error threshold of the low-risk category and the weight of the low-risk category and the product of the category error threshold of the high-risk category and the weight of the high-risk category is not greater than the overall error threshold.

[0162] In some implementations of this application, the category weight determination subunit is specifically used to determine the risk severity score and sampling percentage corresponding to each risk category; the sampling percentage is the ratio of the number of sampling samples corresponding to the same risk category to the total number of sampling samples; the category weight is determined based on the risk severity score, the sampling percentage, the first adjustment coefficient corresponding to the risk severity score, and the second adjustment coefficient corresponding to the sampling percentage.

[0163] In some implementations of this application, before determining the expert label for an evaluation sample as the security assessment information of the evaluation sample for which the uncertainty value is greater than the dynamic threshold, the apparatus further includes:

[0164] The risk category determination module is used to determine the risk category of the evaluation sample;

[0165] The true error rate determination module is used to determine the evaluation sample group to which the evaluation sample belongs and the true error rate corresponding to the evaluation sample group; the evaluation sample group corresponds to the same risk category and the same uncertainty value range; the true error rate is determined based on the sampling sample set;

[0166] The uncertainty mean calculation module is used to calculate the uncertainty mean of the evaluation sample group;

[0167] An uncertainty mean correction module is used to determine the uncertainty value of the evaluation sample based on the true error rate when the difference between the uncertainty mean and the true error rate is greater than a preset deviation tolerance.

[0168] This application provides a large language model security assessment device. By using this device, the steps in the aforementioned method embodiments can be implemented.

[0169] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.

[0170] Reference Figure 4 The diagram illustrates an electronic device according to an embodiment of this application. Figure 4 As shown, the electronic device 400 in this embodiment includes: a processor 410, a memory 420, and a computer program 421 stored in the memory 420 and executable on the processor 410. When the processor 410 executes the computer program 421, it implements the steps in the various embodiments of the large language model security assessment method described above, for example... Figure 1 Steps 101 to 105 are shown. Alternatively, when the processor 410 executes the computer program 421, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 3 The functions of modules 301 to 305 are shown.

[0171] For example, the computer program 421 can be divided into one or more modules / units, which are stored in the memory 420 and executed by the processor 410 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which can be used to describe the execution process of the computer program 421 in the electronic device 400. For example, the computer program 421 can be divided into a model evaluation module, a dynamic threshold determination module, an expert label replacement module, and a security assessment result generation module, with the specific functions of each module as follows:

[0172] The model evaluation module is used to obtain security evaluation information obtained by evaluating multiple evaluation samples by a preset security evaluation model, as well as the uncertainty value corresponding to the security evaluation information; the evaluation samples include input information and output information generated by the large model to be evaluated for each input information;

[0173] A dynamic threshold determination module is used to determine a dynamic threshold based on a sample set; the sample set includes several evaluation samples and the uncertainty values ​​and expert labels corresponding to the evaluation samples;

[0174] The expert label replacement module is used to determine the expert label for the evaluation sample as the security assessment information of the evaluation sample for the evaluation sample whose uncertainty value is greater than the dynamic threshold.

[0175] The security assessment result generation module is used to generate security assessment results for the large model to be evaluated based on each evaluation sample and the corresponding security assessment information.

[0176] The electronic device 400 may be a desktop computer, a cloud server, or other computing device. The electronic device 400 may include, but is not limited to, a processor 410 and a memory 420. Those skilled in the art will understand that... Figure 4 This is merely one example of electronic device 400 and does not constitute a limitation on electronic device 400. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 400 may also include input / output devices, network access devices, buses, etc.

[0177] The processor 410 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0178] The memory 420 can be an internal storage unit of the electronic device 400, such as a hard disk or memory of the electronic device 400. The memory 420 can also be an external storage device of the electronic device 400, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, etc., equipped on the electronic device 400. Furthermore, the memory 420 can include both internal and external storage units of the electronic device 400. The memory 420 is used to store the computer program 421 and other programs and data required by the electronic device 400. The memory 420 can also be used to temporarily store data that has been output or will be output.

[0179] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the large language model security assessment method as described in the foregoing embodiments.

[0180] This application also discloses a computer program product, including a computer program that, when run, causes the large language model security assessment method as described in the foregoing embodiments to be executed.

[0181] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A large language model security evaluation method, characterized in that, The method comprises: obtaining safety evaluation information obtained by a preset safety evaluation model for a plurality of evaluation samples, and an uncertainty value corresponding to the safety evaluation information; the evaluation samples include input information and output information generated by a large model to be evaluated for each input information; determining a dynamic threshold according to a sampling sample set; the sampling sample set includes a plurality of evaluation samples, uncertainty values corresponding to the evaluation samples, and expert labels corresponding to the evaluation samples; for the evaluation samples whose uncertainty values are greater than the dynamic threshold, determining that the expert label corresponding to the evaluation sample is the safety evaluation information of the evaluation sample; generating a safety evaluation result of the large model to be evaluated according to each evaluation sample and the corresponding safety evaluation information; wherein, the dynamic threshold is determined according to the sampling sample set, comprising: sampling a plurality of evaluation samples to obtain a plurality of sampling samples; determining the expert label corresponding to each input information in the sampling sample; calculating an upper limit value of error according to the sampling sample and the expert label; determining a dynamic threshold according to the upper limit value of error; the dynamic threshold is the minimum one of the uncertainty critical values corresponding to each upper limit value of error greater than the category error threshold; the category error threshold is used to represent the output information error proportion of the same category.

2. The method of claim 1, wherein, The safety evaluation result includes an evaluation error rate and an evaluation confidence of the evaluation error rate, and the method further comprises: in the case that the evaluation error rate is greater than a preset overall error threshold or the evaluation confidence is less than a preset confidence threshold, increasing the number of evaluation samples in the sampling sample set, and returning to the step of determining the dynamic threshold according to the sampling sample set.

3. The method of claim 2, wherein, The upper limit value of error is calculated according to the sampling sample and the expert label, comprising: determining a target sample from the sampling sample; the output information of the target sample is different from the expert label and the uncertainty value is higher than the uncertainty critical value; calculating a weighted loss value for each target sample; determining a weighted loss mean value and a weighted loss standard deviation value based on the weighted loss value; determining a product value of a normal distribution quantile corresponding to a preset confidence threshold and the weighted loss standard deviation value; determining the sum of the weighted loss mean value and the product value as the upper limit value of error corresponding to the uncertainty critical value.

4. The method of claim 3, wherein, The dynamic threshold is determined according to the upper limit value of error, comprising: determining a plurality of risk categories; for each risk category, the following steps are performed: determining a category error threshold; determining a dynamic threshold.

5. The method of claim 4, wherein, If the risk category is a high-risk category; the category error threshold is determined, comprising: determining an error coefficient; the error coefficient is a positive number less than 1; determining the product of the error coefficient and the overall error threshold as the category error threshold of the high-risk category.

6. The method of claim 5, wherein, If the risk category is a low-risk category; the category error threshold is determined, comprising: determining a category weight corresponding to each different risk category based on the sampling sample; determining the sum of the category weights of each low-risk category as a low-risk category weight; determining the sum of the category weights of each high-risk category as a high-risk category weight; determining a category error threshold value of the low-risk category based on the low-risk category weight, the high-risk category weight, the error coefficient, and the overall error threshold value; wherein a product of the category error threshold value of the low-risk category and the low-risk category weight is not greater than a sum of a product of a category error threshold value of the high-risk category and the high-risk category weight.

7. The method of claim 6, wherein, The determining, based on the sampling samples, of the category weight corresponding to each different risk category comprises: determining a risk severity score value corresponding to each risk category and a sampling sample proportion; the sampling sample proportion is a proportion of a number of sampling samples corresponding to the same risk category to a total amount of sampling samples; determining the category weight according to the risk severity score value, the sampling sample proportion, a first adjustment coefficient corresponding to the risk severity score value, and a second adjustment coefficient corresponding to the sampling sample proportion.

8. The method of claim 1, wherein, Before the determining, for the evaluation sample whose uncertainty value is greater than the dynamic threshold value, of the expert label for the evaluation sample as the safety evaluation information of the evaluation sample, the method further comprises: determining a risk category of the evaluation sample; determining an evaluation sample group in which the evaluation sample is located and a true error rate corresponding to the evaluation sample group; the evaluation sample group corresponds to the same risk category and the same uncertainty value interval; the true error rate is determined based on the sampling sample set; calculating an uncertainty mean value of the evaluation sample group; when a difference between the uncertainty mean value and the true error rate is greater than a preset deviation tolerance value, determining the uncertainty value of the evaluation sample based on the true error rate.

9. A large language model security evaluation apparatus, characterized by, comprises: a model evaluation module configured to obtain safety evaluation information obtained by a preset safety evaluation model for a plurality of evaluation samples and uncertainty values corresponding to the safety evaluation information; the evaluation samples comprise input information and output information generated by a to-be-evaluated large model for each input information; a dynamic threshold value determination module configured to determine a dynamic threshold value according to a sampling sample set; the sampling sample set comprises a plurality of evaluation samples and uncertainty values and expert labels corresponding to the evaluation samples; an expert label replacement module configured to, for the evaluation sample whose uncertainty value is greater than the dynamic threshold value, determine the expert label for the evaluation sample as the safety evaluation information of the evaluation sample; a safety evaluation result generation module configured to generate a safety evaluation result of the to-be-evaluated large model according to each evaluation sample and the corresponding safety evaluation information; the dynamic threshold value determination module comprises: a sampling submodule configured to sample a plurality of the evaluation samples to obtain a plurality of sampling samples; an expert label determination submodule configured to determine an expert label corresponding to each input information in the sampling samples; an error upper limit value calculation submodule configured to calculate an error upper limit value according to the sampling samples and the expert labels; a dynamic threshold value determination submodule configured to determine a dynamic threshold value according to the error upper limit value; the dynamic threshold value is a smallest one of uncertainty critical values corresponding to each error upper limit value greater than a category error threshold value; the category error threshold value is used to represent an output information error proportion of the same category.

10. An electronic device, comprising: An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, the processor, when executing the computer program, causing the electronic device to implement the method of any one of claims 1-8.

11. A computer program product, characterised in that, A computer program which, when run, causes the method of any one of claims 1-8 to be performed.

Images