AI large model security evaluation method, device, electronic equipment and program product

Abstract

The application relates to the technical field of artificial intelligence, and proposes an AI large model safety evaluation method and device, an electronic device and a computer program product. The method comprises the following steps: acquiring a to-be-evaluated reply text output by a to-be-evaluated AI large model based on a risk problem text; selecting a first evaluation AI large model adapted to a first risk dimension to which the to-be-evaluated reply text belongs from a plurality of evaluation AI large models; and predicting the to-be-evaluated reply text by using the first evaluation AI large model, and outputting a first prediction risk label of the to-be-evaluated reply text. The method can realize directional matching of risk dimensions and model capabilities by directionally assigning the most suitable evaluation AI large model to the to-be-evaluated reply text of different risk dimensions, so that the evaluation AI large model that performs better in a specific risk dimension is used to process the evaluation task of the corresponding risk dimension, and finally, the evaluation efficiency, the evaluation accuracy and the evaluation cost are considered.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and computer program product for the safety evaluation of large AI models. Background Technology

[0002] Security evaluation of AI large-scale models refers to assessing the security of the responses output by AI large-scale models based on risk questions. The process mainly includes: inputting risk question text into the AI ​​large-scale model under test and obtaining the response text output by the AI ​​large-scale model under test; analyzing the response text manually by experts or by evaluating the AI ​​large-scale model and outputting a judgment result on whether the response text is risky text; determining the risk pass rate of the AI ​​large-scale model under test by counting the number of risky texts contained in all response texts, thereby measuring the security level of the AI ​​large-scale model under test.

[0003] Currently, security evaluation of large AI models is mainly conducted manually by experts or through evaluation of the AI ​​models themselves. While manual evaluation boasts high accuracy, it suffers from low processing efficiency, while large AI model evaluation offers high processing efficiency but insufficient accuracy. To address this issue, existing technologies have proposed a multi-model ensemble evaluation method. This method combines the outputs of multiple large AI models to improve the accuracy of security evaluation. Typically, this approach employs voting, weighted averaging, or other aggregation strategies to integrate the scores or judgments of different large AI models, thereby fully leveraging the strengths of each model and reducing the predictive bias and errors that may exist with a single model.

[0004] However, this method requires calling the API interfaces of multiple large AI models, and the API call costs and computational resource consumption increase linearly with the number of models and the amount of evaluation data, ultimately leading to a significant increase in evaluation costs. Therefore, existing methods for evaluating the safety of large AI models cannot simultaneously balance evaluation efficiency, accuracy, and cost. Summary of the Invention

[0005] In view of this, embodiments of this application provide an AI large-scale model security evaluation method, apparatus, electronic device, and computer program product. By assigning the most suitable evaluation AI large-scale model to the response text to be tested for different risk dimensions, it can balance evaluation efficiency, evaluation accuracy, and evaluation cost.

[0006] The first aspect of this application provides a method for security evaluation of large AI models, including:

[0007] Obtain the response text to be tested, which is output by the AI ​​model under test based on the risk question text;

[0008] From multiple large-scale evaluation AI models, the first large-scale evaluation AI model that matches the first risk dimension of the response text to be tested is selected.

[0009] The first evaluation AI model predicts the response text to be tested and outputs the first predicted risk label of the response text.

[0010] In the technical solution of this application embodiment, the response text to be tested, output by the AI ​​model to be tested based on the risk question text, is first obtained. Then, from multiple evaluation AI models, a first evaluation AI model that matches the first risk dimension of the response text to be tested is selected. Finally, the first evaluation AI model is used to predict the response text to be tested, and the first predicted risk label of the response text to be tested is output. The above process uses the first evaluation AI model to complete the automatic evaluation, which can achieve high evaluation efficiency. By assigning the most suitable evaluation AI model to the response text to be tested for different risk dimensions, the above process can achieve targeted matching between risk dimensions and model capabilities, allowing the evaluation AI model that performs better in a specific risk dimension to specialize in the evaluation task of the corresponding risk dimension, thereby effectively improving the evaluation accuracy. The above process only needs to call a single evaluation AI model, which can avoid the problem of API call costs and computing resource consumption increasing linearly with the number of models caused by running multiple models simultaneously, thereby significantly reducing the evaluation cost. Therefore, the technical solution of this application embodiment can balance evaluation efficiency, evaluation accuracy, and evaluation cost.

[0011] In one implementation of this application, a first evaluation AI model that matches the first risk dimension of the response text to be tested is selected from multiple evaluation AI models, including:

[0012] The text to be tested is input into the trained routing model for processing. The routing model outputs the fit weight between each evaluation AI model and the text to be tested. The routing model is a machine learning model that calculates the fit weight between each evaluation AI model and the text to be tested based on the historical evaluation data of each evaluation AI model on the first risk dimension.

[0013] From multiple evaluation AI models, the one with the highest fit weight to the response text to be tested was selected as the first evaluation AI model.

[0014] In one implementation of this application, the training dataset for the routing model is constructed in the following manner:

[0015] The process involves acquiring a sample text dataset, an expert label dataset, a model prediction result dataset, and a model prediction uncertainty score dataset. The sample text dataset includes multiple sample response texts covering various preset risk dimensions. The expert label dataset includes the true risk labels of a subset of the sample response texts. The model prediction result dataset includes the predicted risk labels of each evaluation AI model for each sample response text. The model prediction uncertainty score dataset includes the uncertainty score predicted by each evaluation AI model for each sample response text. Multiple preset risk dimensions are divided into different risk levels, each corresponding to a different risk coefficient. Each sample response text determines its respective risk level based on its preset risk dimension.

[0016] The interface call cost and expert annotation cost of each large-scale AI model being evaluated are calculated separately.

[0017] A training dataset is constructed based on the sample text dataset, expert label dataset, model prediction result dataset, model prediction uncertainty score dataset, as well as the interface call cost and expert annotation cost of each large evaluation AI model.

[0018] In one implementation of this application, the second risk dimension is any one of a plurality of preset risk dimensions, the second evaluation AI model is any one of a plurality of evaluation AI models, and the target score range is any one of a plurality of preset score range ranges. Before constructing the training dataset based on the sample text dataset, expert label dataset, model prediction result dataset, model prediction uncertainty score dataset, and the interface call cost and expert annotation cost of each evaluation AI model, the method further includes:

[0019] Obtain the set of uncertainty scores for the second evaluation AI model to predict sample response texts belonging to the second risk dimension;

[0020] Arrange the uncertainty scores in the uncertainty score set from smallest to largest, and divide them into multiple preset score range intervals;

[0021] By combining the expert label dataset, the actual error rate of the second evaluation AI model in predicting the target score range corresponding to the second risk dimension is calculated.

[0022] The score correction value for the target score range is determined based on the actual error rate and the average value of each uncertainty score in the target score range.

[0023] If the score correction value exceeds the preset tolerance threshold, then each uncertain score in the target score range is corrected according to the score correction value, and the process returns to the step of arranging each uncertain score in the uncertain score set from smallest to largest and dividing them into multiple preset score ranges, until the score correction value does not exceed the tolerance threshold.

[0024] In one implementation of this application, the objective function of the routing model is used to minimize the total labeling cost while constraining the overall error. The objective function introduces a risk coefficient as an error control coefficient, and the parameters optimized by the objective function include network weights and uncertainty score thresholds.

[0025] In one implementation of this application, the training process of the routing model includes:

[0026] Randomly initialize the network weights and initialize the uncertainty score threshold to the median of the uncertainty scores contained in the model's prediction uncertainty score dataset;

[0027] An alternating optimization approach is adopted, with a fixed uncertainty score threshold. The network weights are updated using gradient descent to minimize the objective function. Based on the updated network weights, the uncertainty score threshold is recalculated to constrain the overall error until the convergence condition is met, thus obtaining the final network weights and uncertainty score threshold.

[0028] In one implementation of this application, after predicting the text to be tested using a first evaluation AI model and outputting a first predicted risk label for the text to be tested, the method further includes:

[0029] Determine the target uncertainty score for the first evaluation AI model to predict the response text to be tested;

[0030] If the target uncertainty score does not exceed the uncertainty score threshold, the first predicted risk label will be determined as the risk prediction result of the response text to be tested.

[0031] If the target uncertainty score exceeds the uncertainty score threshold, the second predicted risk label predicted by the expert for the response text to be tested is obtained as the risk prediction result of the response text to be tested.

[0032] A second aspect of this application provides an AI large model security evaluation device, comprising:

[0033] The response text acquisition module is used to acquire the response text to be tested, which is output by the AI ​​model under test based on the risk question text.

[0034] The evaluation model selection module is used to select the first evaluation AI model that matches the first risk dimension of the response text to be tested from multiple evaluation AI models.

[0035] The risk prediction module is used to predict the response text to be tested using the first evaluation AI model, and output the first predicted risk label of the response text to be tested.

[0036] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the AI ​​large model security evaluation method provided in the first aspect of this application.

[0037] A fourth aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the AI ​​large model security evaluation method provided in the first aspect of this application.

[0038] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the AI ​​large model security evaluation method provided in the first aspect of this application.

[0039] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0040] Figure 1 This is a flowchart of a method for security evaluation of large AI models provided in an embodiment of this application;

[0041] Figure 2 This is a schematic diagram illustrating the operational principle of the AI ​​large model security evaluation method provided in this application embodiment in a practical application scenario;

[0042] Figure 3 This is a schematic diagram of the structure of an AI large model security evaluation device provided in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail. Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0045] The security evaluation process for large AI models is crucial for their standardized use. Currently, security evaluation primarily involves two methods: manual evaluation by experts and evaluation of large AI models. The first method relies on experts manually scoring each model output. With large datasets, manual scoring is inefficient and costly, making expert resource requirements a significant bottleneck. The second method utilizes a single large AI model for automated scoring. While this effectively reduces costs, it suffers from insufficient prediction accuracy, especially in high-risk scenarios where misjudgments can lead to serious consequences. In addition to these two methods, existing technologies have proposed multi-model ensemble evaluation methods. By combining the scores or judgments of different large AI models, the advantages of each model can be fully utilized, reducing potential prediction biases and errors from individual models, thereby improving the accuracy of security evaluation. However, this method requires calling the API interfaces of multiple large AI models, and API call costs and computational resource consumption increase linearly with the number of models and the amount of evaluation data, ultimately leading to a significant increase in evaluation costs. Furthermore, when using multiple large AI models for security evaluation, this method typically employs only simple aggregation strategies such as voting or weighted averaging. This fails to effectively eliminate value biases among different models across risk dimensions and may even lead to systematic errors in certain risk dimensions, thus affecting the consistency and fairness of the evaluation results. Additionally, this method usually uses fixed model combinations and aggregation strategies, making it difficult to dynamically adjust based on the risk level or data characteristics of specific tasks. Therefore, existing large AI model security evaluation methods suffer from the inability to simultaneously balance evaluation efficiency, accuracy, and cost.

[0046] To address the aforementioned issues, this application proposes an AI large-scale model security evaluation method, apparatus, electronic device, and computer program product. By assigning the most suitable evaluation AI large-scale model to the test response text for different risk dimensions, it can balance evaluation efficiency, evaluation accuracy, and evaluation cost. For more specific technical implementation details of this application's embodiments, please refer to the various method embodiments described below.

[0047] It should be understood that the implementing entity of the various method embodiments proposed in this application can be various types of electronic devices, such as mobile phones, tablets, desktop computers, wearable devices, medical devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc. The embodiments of this application do not impose any restrictions on the specific type of electronic device.

[0048] Please see Figure 1 This application illustrates an AI large model security evaluation method provided by an embodiment of the present application, including:

[0049] 101. Obtain the response text to be tested, which is output by the AI ​​model under test based on the risk question text;

[0050] First, obtain the response text to be tested, output by the AI ​​model under test based on the risk question text. Here, the AI ​​model under test refers to any AI model that needs to be security evaluated; the risk question text refers to the text describing a problem containing risky content; and the response text to be tested refers to the response text output by the AI ​​model under test based on the risk question text, representing the risk to be assessed. Each risk question text has its own corresponding response text to be tested.

[0051] The source of the risk question texts can be a set of question texts covering multiple preset risk dimensions. For example, in the context of generative AI safety guidelines, the safety assessment needs to cover up to 31 safety indicators, each corresponding to a risk dimension. Therefore, the question text set needs to include multiple risk question texts covering all 31 risk dimensions, meaning there must be at least one risk question text for each risk dimension. By setting up this question text set, the safety of the large AI model under test across the 31 risk dimensions can be comprehensively evaluated.

[0052] 102. From multiple evaluation AI models, select the first evaluation AI model that matches the first risk dimension of the response text to be tested;

[0053] After obtaining the response text to be tested, since the risk question text corresponding to the response text to be tested is known, and the risk dimension corresponding to the risk question text is also known, the risk dimension to which the response text to be tested belongs can be determined, and it is denoted as the first risk dimension.

[0054] In the technical solution of this application embodiment, a candidate evaluation model set is pre-set, which includes multiple large AI models with different types and characteristics used to perform security evaluations. These large AI models can be referred to as evaluation AI models. As an example, the candidate evaluation model set can be represented as follows: , which means k Different evaluation AI models, influenced by training data, algorithm design, and optimization objectives during training, may exhibit inherent value biases in certain risk dimensions, ultimately leading to different evaluation AI models excelling at handling different risk dimensions. For example, evaluation AI model A might perform exceptionally well in detecting politically sensitive content, while another evaluation AI model B might be better at identifying misinformation or malicious content.

[0055] From multiple evaluation AI models, the one that best matches the first risk dimension of the response text to be tested can be selected and designated as the first evaluation AI model. This model excels at evaluating tasks related to the first risk dimension. In practice, the risk dimensions that each evaluation AI model is best suited for can be determined and stored based on its historical evaluation results for different risk dimensions. For example, evaluation AI model A might be best suited for risk dimensions D1, D3, and D5, while evaluation AI model B might be best suited for risk dimensions D1, D2, and D4, and so on. Each evaluation AI model can be best suited for one or more risk dimensions, and different evaluation AI models can be best suited for the same or different risk dimensions. If there are multiple evaluation AI models best suited for the first risk dimension, one of these best suited models can be selected as the first evaluation AI model according to a set method (e.g., randomly or sequentially). By assigning the most suitable evaluation AI model to the response text to be tested for different risk dimensions, it is possible to achieve targeted matching between risk dimensions and model capabilities. This allows the evaluation AI model that performs better in a specific risk dimension to handle the evaluation task for that risk dimension, thereby effectively improving the evaluation accuracy.

[0056] In one implementation of this application, a first evaluation AI model that matches the first risk dimension of the response text to be tested is selected from multiple evaluation AI models, including:

[0057] (1) Input the response text to be tested into the trained routing model for processing, and output the fit weight between each evaluation AI model and the response text to be tested through the routing model; wherein, the routing model is a machine learning model that calculates the fit weight between each evaluation AI model and the response text to be tested based on the historical evaluation data of each evaluation AI model on the first risk dimension.

[0058] (2) Select the AI ​​model with the highest fit weight to the response text to be tested from multiple evaluation AI models as the first evaluation AI model.

[0059] To more accurately select the first evaluation AI model suitable for the current evaluation task from multiple evaluation AI models, this application introduces a risk-weighted multi-model dynamic routing mechanism. The core of this mechanism is to train a routing model that, through intelligent routing decisions, assigns the most suitable evaluation AI model to each response text to be tested, significantly optimizing the overall evaluation cost while meeting prediction accuracy requirements. This routing model is a machine learning model that calculates the fit weight between each evaluation AI model and the response text to be tested based on historical evaluation data for the first risk dimension. After the response text to be tested is input into this routing model for processing, it outputs the fit weight between each evaluation AI model and the response text. Finally, the evaluation AI model with the highest fit weight to the response text to be tested can be selected as the first evaluation AI model. The specific training process of this routing model is described below.

[0060] In one implementation of this application, the training dataset for the routing model is constructed in the following manner:

[0061] (1) Obtain the sample text dataset, expert label dataset, model prediction result dataset and model prediction uncertainty score dataset; wherein, the sample text dataset includes multiple sample response texts covering multiple preset risk dimensions, the expert label dataset includes the true risk labels of a portion of the sample response texts, the model prediction result dataset includes the predicted risk labels of each evaluation AI model for each sample response text, the model prediction uncertainty score dataset includes the uncertainty score predicted by each evaluation AI model for each sample response text, multiple preset risk dimensions are divided into multiple different risk levels, different risk levels correspond to different risk coefficients, and each sample response text determines its corresponding risk level according to its respective preset risk dimension;

[0062] (2) Calculate the interface call cost and expert annotation cost for each large-scale AI model being evaluated;

[0063] (3) Construct a training dataset based on the sample text dataset, expert label dataset, model prediction result dataset, model prediction uncertainty score dataset, and the interface call cost and expert annotation cost of each large AI model being evaluated.

[0064] Training machine learning models requires high-quality training datasets. To construct the training dataset for the routing model, we first need to acquire a sample text dataset, an expert label dataset, a model prediction result dataset, and a model prediction uncertainty score dataset. The sample text dataset includes multiple sample response texts covering several preset risk dimensions, output by the large AI model under test based on a risk question dataset. For example, the sample text dataset can be represented as follows: It contains n Sample response text, n The sample response texts cover 31 risk dimensions, meaning that each risk dimension has at least one sample response text, that is... n ≥31, under normal circumstances n The values ​​can reach several thousand or even tens of thousands.

[0065] The expert-labeled dataset includes true risk labels for a subset of sample response texts from multiple sample response texts. These true risk labels are obtained through manual annotation by experts and are considered to have 100% accuracy. The labeled dataset can be represented as follows: It contains m The true risk label of the sample response text. m ≤ n The true risk label is used to calibrate the routing model error. Its value is 0 or 1, representing the absence of risk / presentation of risk in the text, respectively. A small subset of sample response texts are labeled with true risk labels by experts. These labels can be used to train the routing model. The trained model is then used to evaluate response texts that are not labeled by experts, determining which require expert labeling (higher prediction cost but higher accuracy) and which only require labeling by a large evaluation AI model (lower prediction cost and accuracy than experts). This achieves the goal of saving prediction costs while maintaining prediction accuracy.

[0066] The model prediction results dataset includes the predicted risk labels for each sample response text by each large evaluation AI model, such as the candidate evaluation model set mentioned above. The model prediction result dataset can be represented as , To evaluate large AI models Sample response text The predicted risk label has a value of 0 or 1.

[0067] The model prediction uncertainty score dataset includes the uncertainty score of each evaluation AI model for predicting each sample response text, which can be represented as: , To evaluate large AI models Sample response text The uncertainty score used for prediction ranges from [0, 1]. Each evaluation AI model, while predicting a risk label for a sample response text, outputs an uncertainty score corresponding to the prediction result. The closer the uncertainty score is to 0, the more accurate the model's prediction result is; otherwise, the less accurate the model's prediction result is.

[0068] Multiple preset risk dimensions are divided into multiple different risk levels, each corresponding to a different risk coefficient, with higher risk levels corresponding to larger risk coefficients. As an example, the 31 risk dimensions can be categorized into sets of high-risk levels based on actual circumstances. K H A collection of medium-risk levels K M and low-risk groups K L This makes the high-risk level correspond to the risk coefficient. Medium risk level corresponds to risk coefficient Low risk level corresponds to risk coefficient For example, the risk dimensions "violation of core values" and "containment of discriminatory content" can be categorized into a high-risk level group. K H The risk dimension of "commercial violations and irregularities" can be classified into the set of medium-risk levels. K M The risk dimensions of "infringement of the legitimate rights and interests of others" and "inability to meet specific service types" can be classified into the low-risk category. K L .

[0069] In the data preprocessing stage, each sample response text can be assigned a risk level based on its respective preset risk dimension, thereby... n The sample response texts are grouped according to risk level. This process can be represented as follows: By classifying each risk dimension to its corresponding risk level, the routing model can not only select the appropriate evaluation AI model according to the risk dimension, but also select the appropriate evaluation AI model according to the risk level.

[0070] The interface call cost and expert annotation cost for each evaluation AI model per test sample were calculated separately. For a given evaluation AI model... The cost of an interface call can be expressed as The cost of expert annotation can be expressed as The cost of API calls can be determined by querying the official API pricing of the large-scale AI model being evaluated. Generally, the official API documentation will specify the price per million tokens. Therefore, by calculating the average number of tokens for each test sample, the cost of using the corresponding large-scale model API per test sample can be calculated as the API call cost. Expert annotation costs can use a preset cost value, such as 0.6 yuan per test sample.

[0071] Based on the sample text dataset, expert label dataset, model prediction result dataset, model prediction uncertainty score dataset described above, and the interface call cost and expert annotation cost of each evaluation AI model, a training dataset can be constructed. Subsequently, using basic machine learning model training methods, a neural network can be trained based on this training dataset to obtain the aforementioned routing model.

[0072] In the aforementioned model prediction uncertainty score dataset, the uncertainty scores predicted by each evaluation AI model for each sample response text may exhibit a "mismatch between confidence level and actual error," such as model overconfidence or underconfidence. To address this issue, this application introduces a risk-dimension-based uncertainty calibration mechanism, which can perform targeted calibration of the aforementioned model prediction uncertainty score dataset according to risk level. The specific processing procedure of this uncertainty calibration mechanism is described below.

[0073] In one implementation of this application, the second risk dimension is any one of a plurality of preset risk dimensions, the second evaluation AI model is any one of a plurality of evaluation AI models, and the target score range is any one of a plurality of preset score range ranges. Before constructing the training dataset based on the sample text dataset, expert label dataset, model prediction result dataset, model prediction uncertainty score dataset, and the interface call cost and expert annotation cost of each evaluation AI model, the method further includes:

[0074] (1) Obtain the set of uncertainty scores for the second evaluation AI model to predict sample response texts belonging to the second risk dimension;

[0075] (2) Arrange the uncertain scores in the uncertain score set from smallest to largest, and divide them into multiple preset score range intervals;

[0076] (3) Combine the expert label dataset to calculate the actual error rate of the second evaluation AI model in the case of the second risk dimension and the target score range;

[0077] (4) Determine the score correction value for the target score range based on the actual error rate and the average value of each uncertain score in the target score range;

[0078] (5) If the score correction value exceeds the preset tolerance threshold, then each uncertain score in the target score range interval is corrected according to the score correction value, and the process is returned to execute the step of arranging each uncertain score in the uncertain score set from small to large and dividing them into multiple preset score range intervals until the score correction value does not exceed the tolerance threshold.

[0079] When implementing the uncertainty calibration mechanism, group calibration is performed, processing the uncertainty score data separately for each risk dimension and each large-scale evaluation AI model. Specifically, assuming the second risk dimension... For any one of the multiple preset risk dimensions, the second evaluation AI model... j For any one of multiple large-scale evaluation AI models, the target score range is [range]. For any score range among multiple preset score ranges, first obtain the second evaluation AI model. j For those belonging to the second risk dimension The uncertainty score set is generated from all sample response texts. Then, the uncertainty scores in this set are arranged in ascending order and divided into multiple predefined score range intervals. The lengths of the different score range intervals can be the same or different, and the different score range intervals should not overlap. For example, 10 score range intervals of equal length can be divided, denoted as... Target score range It can be any one of the 10 score ranges.

[0080] Next, combining the expert label dataset described above, the second evaluation AI model is calculated. j In the corresponding second risk dimension and target score range In this case, the actual error rate of prediction is calculated. This actual error rate mainly measures the percentage of inconsistencies between the predicted labels given by the large AI model and the expert labels. The specific calculation formula is as follows:

[0081]

[0082] in, This indicates the second evaluation of the large AI model. j In the corresponding second risk dimension and target score range The actual error rate of prediction under the given circumstances , This is the risk loss function.

[0083] Subsequently, based on the actual error rate and the target score range... The average of each uncertainty score is used to determine the target score range. The fractional correction value is calculated using the following formula:

[0084]

[0085] in, Indicates the range of target scores The score correction value, Indicates the range of target scores The average value of each uncertainty fraction in the equation, also known as the fraction correction value, is equal to the difference between the actual error rate and the average value of the uncertainty fractions. Its value can be positive or negative.

[0086] Preset a tolerance threshold (For example, it could be 0.05), determine the fraction correction value calculated above. Does it exceed the tolerance threshold? If it exceeds, then adjust the value according to the score. For the target score range Each uncertainty score in the sample is corrected separately, that is, for each sample... Update its uncertainty score During the correction process, it is necessary to ensure that each corrected uncertainty score falls within the range [0, 1]. If the score exceeds this range, it should be truncated; for example, if the corrected uncertainty score is less than 0, it should be set to 0; if the corrected uncertainty score exceeds 1, it should be set to 1. The correction should be performed within the target score range. After calculating each uncertainty score, the process returns to the step of sorting the uncertainty scores in the set from smallest to largest and dividing them into multiple preset score ranges, then proceeds to the next round of convergence judgment. This process is repeated until the calculated score correction value is obtained. ≤ When, output the target score range. Each uncertainty score after calibration.

[0087] It's understandable that each score range can be compared to the target score range. The uncertainty score was calibrated using the same method to obtain the calibrated second evaluation AI model. j For the second risk dimension The uncertainty score is predicted based on the sample response text. Each risk dimension can be calculated according to the second risk dimension. The uncertainty score was calibrated using the same method to obtain the calibrated second evaluation AI model. j The uncertainty score is predicted for the sample response text for each risk dimension. Each evaluation AI model can be compared with the second evaluation AI model. j The same method is used to calibrate the uncertainty scores, thereby obtaining the calibrated uncertainty scores for each evaluation AI model's predictions of sample response texts for each risk dimension. This completes the calibration of the entire model's prediction uncertainty score dataset. Considering that different evaluation AI models may have different value biases for different risk dimensions, this uncertainty calibration mechanism calibrates each risk dimension separately, and its calibration effect is better than uniform calibration of all risk dimensions.

[0088] After completing the aforementioned uncertainty score calibration, the training process of the routing model begins. This routing model can dynamically calculate and allocate the optimal evaluation AI model based on the characteristics of the risk dimension to which the input response text belongs.

[0089] In one implementation of this application, the objective function of the routing model is used to minimize the total labeling cost while constraining the overall error. The objective function introduces a risk coefficient as an error control coefficient, and the parameters optimized by the objective function include network weights and uncertainty score thresholds.

[0090] The routing model can be trained using a lightweight neural network. Its input is the sample response text. The feature vectors, the output is k dimensional vectors, respectively representing k A large evaluation AI model and The fitness weights are defined. The objective function is used to minimize the total annotation cost while constraining the overall error. The total annotation cost includes the interface call cost and expert annotation cost described above. This objective function introduces the risk coefficient described above as an error control coefficient, which can strengthen the error control of key terms. The parameters optimized during model training include network weights and uncertainty score thresholds. As an example, a certain objective function is defined as follows:

[0091]

[0092] in, Indicates network weights, Indicates the threshold of uncertainty score. This refers to a large evaluation AI model. j and The fit weight, This represents an estimate of the uncertainty score threshold. express sigmoid function, This is the error control coefficient. Let the risk loss function be defined, with the constraint that the overall error ≤ .

[0093] Adaptability weighting softmax The function is calculated, and its expression is as follows:

[0094]

[0095] in, Indicating the evaluation of large AI models j For the sample The fitness scores are calculated by randomly initializing the fitness weights of each large AI model and normalizing them to ensure that the sum of the fitness weights of each large AI model is 1.

[0096] In the above objective function, The risk coefficient described above , or By introducing As an error control coefficient, the routing model can select the appropriate large-scale evaluation AI model based on the risk level of the input response text. For example, for a high-risk response text, its risk coefficient... A larger risk coefficient means a higher weighting of the predicted risk in the objective function. This prioritizes large AI models with higher prediction accuracy, ensuring the accuracy of the evaluation results. For example, for low-risk response texts, the risk coefficient... Smaller weighting, meaning reducing the weight of prediction risk in the objective function, will prioritize large AI models with lower interface call costs and expert annotation costs, thereby reducing evaluation costs and improving resource utilization efficiency.

[0097] In one implementation of this application, the training process of the routing model includes:

[0098] (1) Randomly initialize network weights and initialize the uncertainty score threshold to the median of each uncertainty score contained in the model prediction uncertainty score dataset;

[0099] (2) Using an alternating optimization approach, the uncertainty score threshold is fixed, and the network weights are updated by gradient descent to minimize the objective function. Based on the updated network weights, the uncertainty score threshold is recalculated to constrain the overall error until the convergence condition is met, and the final network weights and uncertainty score threshold are obtained.

[0100] During the training of the routing model, the network weights are first initialized. and uncertainty score threshold Network weight Random initialization method, uncertainty score threshold The initial value is the median of all uncertainty scores in the uncertainty score dataset predicted by the above model, i.e., the initial value is... Then, using an alternating optimization approach, the uncertainty score threshold is first fixed. Update network weights using gradient descent. To minimize the objective function, and then based on the updated network weights Recalculate the uncertainty score threshold To constrain the overall error. Determine network weights. and uncertainty score threshold If the convergence condition is not met, then continue to fix the uncertainty score threshold. Update network weights using gradient descent. Minimize the objective function... until the convergence condition is met to obtain the final network weights. and uncertainty score threshold .

[0101] The trained routing model and uncertainty score threshold described above can be applied to new AI large-scale model security evaluation tasks. In practice, the feature vector of the response text to be tested is first extracted. Then The input is fed into the routing model described above, which then calculates the fit weight between each evaluation AI model and the response text to be tested. The evaluation AI model with the highest fit weight was selected as the optimal model. That is, the relevant calculation formula is as follows: Optimal model This is the first large-scale AI model for evaluation described above.

[0102] 103. The first evaluation AI model is used to predict the response text to be tested, and the first predicted risk label of the response text to be tested is output.

[0103] After selecting the first evaluation AI model that best suits the current evaluation task using a routing model, the response text to be tested is input into the first evaluation AI model. The first evaluation AI model then predicts the response text and outputs the predicted risk label of the response text, which is denoted as the first predicted risk label. , The label value is 0 or 1, which respectively indicate that the response text to be tested has no risk / has a risk.

[0104] In one implementation of this application, after predicting the text to be tested using a first evaluation AI model and outputting a first predicted risk label for the text to be tested, the method further includes:

[0105] (1) Determine the target uncertainty score for the first evaluation AI model to predict the response text to be tested;

[0106] (2) If the target uncertainty score does not exceed the uncertainty score threshold, the first predicted risk label is determined as the risk prediction result of the response text to be tested;

[0107] (3) If the target uncertainty score exceeds the uncertainty score threshold, the second predicted risk label of the expert's prediction of the response text to be tested is obtained as the risk prediction result of the response text to be tested.

[0108] To further improve the accuracy of risk prediction, this application introduces a mechanism for determining an uncertainty score threshold. First, the target uncertainty score for the first evaluation AI model to predict the response text is determined. To determine whether it exceeds the final uncertainty score threshold. ;if The uncertainty score threshold was not exceeded. This indicates that the prediction results of the first evaluation AI model are highly reliable, therefore the first prediction risk label is set. The risk prediction result is determined to be the response text to be tested; if Exceeding the uncertainty score threshold If the prediction result of the first evaluation AI model is unreliable, then experts are called in to perform manual prediction on the response text to be tested, and a second prediction risk label provided by the experts is obtained. and the second predicted risk label The risk prediction result for the identified response text is shown in the following formula:

[0109]

[0110] in, This indicates the risk prediction result of the response text to be tested.

[0111] It is understandable that each response text to be tested can obtain a corresponding risk prediction result label in the same way as described above. Assume that the number of test response texts output by the large AI model under test based on risky question texts is... n If you can obtain the following, you can get the following: n A labeled dataset consisting of risk prediction result labels The labeled dataset satisfies the specified error rate and confidence level.

[0112] Using the above-mentioned label dataset It can count the number of risky texts contained in all the response texts to be tested, thereby assessing the risk pass rate of the AI ​​model under test, measuring the security level of the AI ​​model under test, and finally completing the security evaluation process of the AI ​​model under test.

[0113] In the technical solution of this application embodiment, the test response text is first obtained by the test AI model based on the risk question text. Then, from multiple evaluation AI models, a first evaluation AI model that matches the first risk dimension of the test response text is selected. Finally, the first evaluation AI model predicts the test response text and outputs the first predicted risk label of the test response text. The above process uses the first evaluation AI model to complete the automatic evaluation, which can achieve high evaluation efficiency. The above process, by assigning the most suitable evaluation AI model to the test response text for different risk dimensions, can achieve targeted matching between risk dimensions and model capabilities, allowing the evaluation AI model that performs better in a specific risk dimension to handle the evaluation task of the corresponding risk dimension, thereby effectively improving the evaluation accuracy. The above process only needs to call a single evaluation AI model, which can avoid the problem of API call costs and computing resource consumption increasing linearly with the number of models caused by running multiple models simultaneously, thereby significantly reducing the evaluation cost. Therefore, the technical solution of this application embodiment can balance evaluation efficiency, evaluation accuracy, and evaluation cost.

[0114] As an example, Figure 2 This is a schematic diagram illustrating the operational principle of the AI ​​large model security evaluation method provided in this application embodiment in a practical application scenario. Figure 2 The core lies in building a risk-weighted, multi-model dynamic routing system. This system, through intelligent routing decisions, assigns a suitable evaluation AI model or human expert to each response text to be evaluated, significantly optimizing the overall evaluation cost while meeting preset accuracy requirements. Figure 2 The process begins with the routing model training phase: Preprocessing the required sample datasets, calibrating the uncertainty scores of the sample datasets according to the uncertainty calibration mechanism described earlier, and constructing the training dataset for the routing model. During the training process, network weights are initialized. and uncertainty score threshold An alternating optimization approach is adopted, first fixing the uncertainty score threshold. Update network weights using gradient descent. To minimize the objective function, and then based on the updated network weights Recalculate the uncertainty score threshold To constrain the overall error, this process is repeated until the convergence condition is met. After the routing model is trained, it enters the model deployment and application phase: the response text output by the AI ​​model to be tested is input into the routing model. The routing model calculates the fit weight between each evaluation AI model and the response text, and selects the evaluation AI model with the highest fit weight as the optimal model. The optimal model is used to predict the response text. If the predicted uncertainty score does not exceed the uncertainty score threshold, the predicted risk label output by the optimal model is retained as the final result label; if the predicted uncertainty score exceeds the uncertainty score threshold, the expert prediction label is used as the final result label. The above process, through a risk-weighted multi-model dynamic routing mechanism, achieves multi-model collaborative safety assessment, which can significantly improve the assessment accuracy, optimize the use of expert resources, greatly save manual annotation costs, and ensure that the assessment results meet the user-specified error rate and confidence requirements.

[0115] In summary, the AI ​​large-scale model security evaluation method provided in this application has the following advantages: (1) By introducing a dynamic routing mechanism, the most suitable evaluation model is assigned to the evaluation tasks of different risk dimensions, and only a single optimal model or a small number of human experts are selected to participate in the evaluation, avoiding the simultaneous use of multiple evaluation models, which can significantly reduce the evaluation cost; (2) Abandoning simple aggregation strategies such as voting or weighted average, the evaluation division of labor is achieved through the targeted matching of risk dimensions and model capabilities, allowing the evaluation model that performs better in a specific risk dimension to specifically handle the evaluation tasks of the corresponding risk dimension, thereby reducing conflicts caused by the inherent value bias of the model from the source and avoiding Different model biases interfere with each other during aggregation, thereby ensuring that the evaluation results of each risk dimension are more consistent and fair; (3) It has flexible dynamic adjustment capabilities, and can dynamically adjust the model selection and resource allocation strategy according to the risk level and data characteristics of the evaluation task. For high-risk evaluation tasks, priority is given to the evaluation model with high reliability and low uncertainty, and the uncertainty score threshold is reduced to ensure prediction accuracy; for low-risk evaluation tasks, priority is given to the evaluation model with low cost to improve resource utilization efficiency; This dynamic adaptation mechanism based on risk level can solve the contradiction between resource waste and insufficient prediction accuracy in fixed model combination and aggregation strategy.

[0116] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0117] The above mainly describes a method for evaluating the security of large AI models. The following section will describe a device for evaluating the security of large AI models.

[0118] Please see Figure 3This application illustrates an AI large model security evaluation device provided in an embodiment, comprising:

[0119] The response text acquisition module 301 is used to acquire the response text to be tested, which is output by the AI ​​model under test based on the risk question text.

[0120] The evaluation model selection module 302 is used to select the first evaluation AI model that matches the first risk dimension of the response text to be tested from multiple evaluation AI models.

[0121] The risk prediction module 303 is used to predict the response text to be tested using the first evaluation AI model and output the first predicted risk label of the response text to be tested.

[0122] In one implementation of this application, the evaluation model selection module includes:

[0123] The routing model processing unit is used to input the response text to be tested into the trained routing model for processing, and output the fit weight between each evaluation AI model and the response text to be tested through the routing model; wherein, the routing model is a machine learning model that calculates the fit weight between each evaluation AI model and the response text to be tested based on the historical evaluation data of each evaluation AI model on the first risk dimension.

[0124] The evaluation model selection unit is used to select the evaluation AI model with the highest fit weight to the response text to be tested from multiple evaluation AI models, and use it as the first evaluation AI model.

[0125] In one implementation of this application, the AI ​​large model security evaluation device further includes:

[0126] The dataset acquisition module is used to acquire sample text datasets, expert label datasets, model prediction result datasets, and model prediction uncertainty score datasets. The sample text dataset includes multiple sample response texts covering multiple preset risk dimensions; the expert label dataset includes the true risk labels of a portion of the sample response texts; the model prediction result dataset includes the predicted risk labels of each evaluation AI model for each sample response text; and the model prediction uncertainty score dataset includes the uncertainty score predicted by each evaluation AI model for each sample response text. Multiple preset risk dimensions are divided into multiple different risk levels, each corresponding to a different risk coefficient. Each sample response text determines its corresponding risk level based on its respective preset risk dimension.

[0127] The cost statistics module is used to separately calculate the interface call cost and expert annotation cost for each large-scale AI model being evaluated;

[0128] The dataset building module is used to build training datasets based on sample text datasets, expert label datasets, model prediction result datasets, model prediction uncertainty score datasets, as well as the interface call cost and expert annotation cost of each large evaluation AI model.

[0129] In one implementation of this application, the second risk dimension is any one of a plurality of preset risk dimensions, the second evaluation AI model is any one of a plurality of evaluation AI models, and the target score range is any one of a plurality of preset score range ranges; the AI ​​model security evaluation device further includes:

[0130] The uncertainty score set acquisition module is used to acquire the uncertainty score set of the second evaluation AI model for predicting sample response texts belonging to the second risk dimension;

[0131] The uncertainty score sorting module is used to sort the uncertainty scores in the uncertainty score set from smallest to largest and divide them into multiple preset score range intervals;

[0132] The actual error rate calculation module is used to combine the expert label dataset to calculate the actual error rate of the second evaluation AI model in predicting the target score range corresponding to the second risk dimension.

[0133] The score correction value determination module is used to determine the score correction value for the target score range based on the actual error rate and the average value of each uncertain score in the target score range.

[0134] The uncertainty score correction module is used to correct each uncertainty score in the target score range according to the score correction value if the score correction value exceeds the preset tolerance threshold. Then, it returns to the execution of the step of arranging each uncertainty score in the uncertainty score set from smallest to largest and dividing them into multiple preset score ranges until the score correction value does not exceed the tolerance threshold.

[0135] In one implementation of this application, the objective function of the routing model is used to minimize the total labeling cost while constraining the overall error. The objective function introduces a risk coefficient as an error control coefficient, and the parameters optimized by the objective function include network weights and uncertainty score thresholds.

[0136] In one implementation of this application, the AI ​​large model security evaluation device further includes:

[0137] The parameter randomization module is used to randomly initialize network weights and initialize the uncertainty score threshold to the median of each uncertainty score contained in the model's prediction uncertainty score dataset.

[0138] The parameter optimization module is used to employ an alternating optimization approach, fix the uncertainty score threshold, update the network weights using gradient descent to minimize the objective function, and recalculate the uncertainty score threshold based on the updated network weights to constrain the overall error until the convergence condition is met, thus obtaining the final network weights and uncertainty score threshold.

[0139] In one implementation of this application, the AI ​​large model security evaluation device further includes:

[0140] The uncertainty score determination module is used to determine the target uncertainty score for the first evaluation AI model to predict the response text under test;

[0141] The first result determination module is used to determine the first predicted risk label as the risk prediction result of the response text to be tested if the target uncertainty score does not exceed the uncertainty score threshold.

[0142] The second result determination module is used to obtain the second predicted risk label predicted by experts for the response text to be tested if the target uncertainty score exceeds the uncertainty score threshold, and use it as the risk prediction result of the response text to be tested.

[0143] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the AI ​​large model security evaluation method described in any of the above embodiments.

[0144] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute the AI ​​large model security evaluation method as described in any of the above embodiments.

[0145] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 4 in this embodiment includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, it implements the steps in the embodiments of the various AI large-scale model security evaluation methods described above, for example... Figure 1 Steps 101-103 are shown. Alternatively, when the processor 40 executes the computer program 42, it implements the functions of each module / unit in the above-described device embodiments, for example, implementing... Figure 3The functions of modules 301-303 of the device shown.

[0146] The computer program 42 can be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 42 in the electronic device 4.

[0147] The processor 40 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0148] The memory 41 can be an internal storage unit of the electronic device 4, such as a hard disk or memory. The memory 41 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 41 can include both internal and external storage units of the electronic device 4. The memory 41 is used to store the computer program and other programs and data required by the electronic device. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0150] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0151] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0152] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0153] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0157] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for security evaluation of large AI models, characterized in that, include: Obtain the response text to be tested, which is output by the AI ​​model under test based on the risk question text; The text to be tested is input into a trained routing model for processing. The routing model outputs the fit weight between each of the multiple evaluation AI models and the text to be tested. From the multiple evaluation AI models, the evaluation AI model with the highest fit weight to the response text to be tested is selected as the first evaluation AI model that fits the first risk dimension to which the response text to be tested belongs. The first evaluation AI model predicts the response text to be tested and outputs the first predicted risk label of the response text to be tested. The training dataset for the routing model is constructed in the following way: Acquire a sample text dataset, an expert label dataset, a model prediction result dataset, and a model prediction uncertainty score dataset; wherein, the sample text dataset includes multiple sample response texts covering multiple preset risk dimensions, the multiple preset risk dimensions are divided into multiple different risk levels, different risk levels correspond to different risk coefficients, and each sample response text determines its corresponding risk level according to its respective preset risk dimension. The interface call cost and expert annotation cost of each of the aforementioned large-scale evaluation AI models were calculated separately. The training dataset is constructed based on the sample text dataset, the expert label dataset, the model prediction result dataset, the model prediction uncertainty score dataset, and the interface call cost and expert annotation cost of each of the evaluation AI large models. The objective function of the routing model during training is used to minimize the total annotation cost while constraining the overall error. The total annotation cost includes the interface call cost and the expert annotation cost. The objective function introduces the risk coefficient as an error control coefficient.

2. The method as described in claim 1, characterized in that, The routing model is a machine learning model that calculates the fit weight between each of the evaluation AI models and the response text to be tested, based on the historical evaluation data of each of the evaluation AI models for the first risk dimension.

3. The method as described in claim 1, characterized in that, The expert label dataset includes the true risk labels of a portion of the sample response texts from the multiple sample response texts; the model prediction result dataset includes the predicted risk labels of each of the evaluation AI models for each of the sample response texts; and the model prediction uncertainty score dataset includes the uncertainty score predicted by each of the evaluation AI models for each of the sample response texts.

4. The method as described in claim 3, characterized in that, Let the second risk dimension be any one of the plurality of preset risk dimensions, let the second evaluation AI model be any one of the plurality of evaluation AI models, and let the target score range be any one of the plurality of preset score range ranges; before constructing the training dataset based on the sample text dataset, the expert label dataset, the model prediction result dataset, the model prediction uncertainty score dataset, and the interface call cost and expert annotation cost of each of the evaluation AI models, the method further includes: Obtain the set of uncertainty scores for the second evaluation AI model in predicting the sample response text belonging to the second risk dimension; Arrange the uncertainty scores in the uncertainty score set from smallest to largest, and divide them into the multiple preset score range intervals respectively; Based on the expert label dataset, the actual error rate of the second evaluation AI model in predicting the target score range corresponding to the second risk dimension is calculated. The score correction value for the target score range is determined based on the actual error rate and the average value of each uncertainty score in the target score range. If the score correction value exceeds the preset tolerance threshold, then each uncertain score in the target score range interval is corrected according to the score correction value, and the process returns to the step of arranging each uncertain score in the uncertain score set from smallest to largest and dividing them into the multiple preset score range intervals, until the score correction value does not exceed the tolerance threshold.

5. The method according to any one of claims 1 to 4, characterized in that, The parameters optimized by the objective function include network weights and uncertainty score thresholds.

6. The method as described in claim 5, characterized in that, The training process of the routing model includes: The network weights are randomly initialized, and the uncertainty score threshold is initialized to the median of the uncertainty scores contained in the model's predicted uncertainty score dataset. An alternating optimization approach is adopted, with the uncertainty score threshold fixed, and the network weights are updated using gradient descent to minimize the objective function. Based on the updated network weights, the uncertainty score threshold is recalculated to constrain the overall error until the convergence condition is met, thus obtaining the final network weights and uncertainty score threshold.

7. The method as described in claim 6, characterized in that, After predicting the text to be tested using the first evaluation AI model and outputting the first predicted risk label for the text to be tested, the method further includes: Determine the target uncertainty score for the first evaluation AI model to predict the response text to be tested; If the target uncertainty score does not exceed the uncertainty score threshold, then the first predicted risk label is determined as the risk prediction result of the response text to be tested; If the target uncertainty score exceeds the uncertainty score threshold, then the second predicted risk label of the response text to be tested, which is predicted by the expert, is obtained as the risk prediction result of the response text to be tested.

8. A security evaluation device for large AI models, characterized in that, include: The response text acquisition module is used to acquire the response text to be tested, which is output by the AI ​​model under test based on the risk question text. The routing model processing unit is used to input the response text to be tested into the trained routing model for processing, and output the fit weight between each of the multiple evaluation AI models and the response text to be tested through the routing model. The evaluation model selection unit is used to select the evaluation AI model with the highest fit weight to the response text to be tested from the multiple evaluation AI models, and use it as the first evaluation AI model that fits the first risk dimension to which the response text to be tested belongs. The risk prediction module is used to predict the response text to be tested using the first evaluation AI model and output the first predicted risk label of the response text to be tested. The dataset acquisition module is used to acquire sample text dataset, expert label dataset, model prediction result dataset, and model prediction uncertainty score dataset; wherein, the sample text dataset includes multiple sample response texts covering multiple preset risk dimensions, the multiple preset risk dimensions are divided into multiple different risk levels, different risk levels correspond to different risk coefficients, and each sample response text determines its corresponding risk level according to its respective preset risk dimension. The cost statistics module is used to separately calculate the interface call cost and expert annotation cost for each of the aforementioned large-scale evaluation AI models; The dataset construction module is used to construct a training dataset based on the sample text dataset, the expert label dataset, the model prediction result dataset, the model prediction uncertainty score dataset, and the interface call cost and expert annotation cost of each of the evaluation AI large models; the objective function of the routing model during the training process is used to minimize the total annotation cost while constraining the overall error, the total annotation cost includes the interface call cost and the expert annotation cost, and the objective function introduces the risk coefficient as an error control coefficient.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the AI ​​large model security evaluation method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, When the computer program product is run on an electronic device, the electronic device performs the AI ​​large model security evaluation method as described in any one of claims 1 to 7.

Images