Model evaluation method and device, electronic equipment, storage medium and program product

Abstract

One or more embodiments of the present specification provide a model evaluation method, device, electronic equipment, storage medium and program product. The model evaluation method comprises: receiving a model evaluation task, the model evaluation task specifying a target model, an evaluation dataset and an evaluation index; quantifying the capability index of the target model in several dimensions using the evaluation dataset; obtaining an evaluation expression corresponding to the evaluation index, the evaluation expression taking one or more capability indexes as input variables; determining a model evaluation result of the target model based on the evaluation expression and the index value of the corresponding capability index.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and more particularly to a model evaluation method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development and widespread application of artificial intelligence technology, the demand for evaluating model performance is increasing. Currently, various model evaluations mainly rely on manual calculations, which are inefficient and prone to errors. Summary of the Invention

[0003] In view of the above, one or more embodiments of this specification provide the following technical solutions:

[0004] According to a first aspect of one or more embodiments of this specification, a model evaluation method is proposed, comprising:

[0005] Receive a model evaluation task, wherein the model evaluation task specifies a target model, an evaluation dataset, and evaluation metrics;

[0006] The evaluation dataset is used to quantify the capability metrics of the target model across several dimensions.

[0007] Obtain the evaluation expression corresponding to the evaluation index, wherein the evaluation expression uses one or more capability indicators as input variables;

[0008] The evaluation result of the target model is determined based on the evaluation expression and the corresponding capability index values.

[0009] Optionally, the process of quantifying the capability metrics of the target model across several dimensions using the evaluation dataset includes:

[0010] Taking each evaluation sample in the evaluation dataset as a unit, the capability index of the target model is quantified in the several dimensions to obtain the capability index value corresponding to each evaluation sample;

[0011] The process of determining the model evaluation result of the target model based on the evaluation expression and the index values ​​of the corresponding capability indicators includes:

[0012] Using the evaluation sample as a unit, calculate the evaluation index value corresponding to the evaluation sample based on the evaluation expression and the corresponding ability index value;

[0013] The evaluation index values ​​corresponding to the evaluation samples are summarized to obtain the model evaluation results of the target model.

[0014] Optionally, the process of summarizing the evaluation index values ​​corresponding to the evaluation samples to obtain the evaluation results of the target model includes:

[0015] For each scene unit under the model evaluation task, the evaluation index values ​​matching the scene unit are summarized according to the annotation information of the evaluation samples to obtain the unit evaluation result corresponding to the scene unit.

[0016] Obtain the target scene specified in the model evaluation task, and determine the scene units associated with the target scene;

[0017] The unit evaluation results corresponding to the scene units associated with the target scene are summarized to obtain the model evaluation results of the target model.

[0018] Optionally, after obtaining the model evaluation results of the target model, the method further includes:

[0019] Receive an update task for the model evaluation results, wherein the update task specifies a new scenario;

[0020] The scene units associated with the new scene are determined, and the unit evaluation results corresponding to the scene units associated with the new scene are summarized to obtain the updated model evaluation results.

[0021] Optional, also includes:

[0022] If no target scenario is specified in the model evaluation task, the unit evaluation results corresponding to all scenario units are summarized to obtain the model evaluation result of the target model.

[0023] Optionally, the process of summarizing the evaluation index values ​​includes:

[0024] Obtain the statistical type of the evaluation indicators;

[0025] The evaluation index values ​​are summarized using the algorithm corresponding to the statistical type.

[0026] Optional, also includes:

[0027] An evaluation report is generated for the target model based on the model evaluation results.

[0028] The evaluation report is returned to the initiator of the model evaluation task.

[0029] Optionally, the process of generating an evaluation report for the target model based on the model evaluation results includes:

[0030] Obtain the display chart type corresponding to the evaluation indicators;

[0031] When generating the evaluation report, the evaluation results are displayed using the aforementioned chart type.

[0032] Optionally, the evaluation metrics include conclusion-type metrics and problem attribution-type metrics, and the evaluation report includes a conclusion-type metric area, a problem attribution-type metric area, and a detail area, wherein:

[0033] The conclusion indicator area is used to display the evaluation results of conclusion-type indicators;

[0034] The problem attribution metric area is used to display the evaluation results of problem attribution metrics;

[0035] The details area is used to display the indicator values ​​corresponding to each evaluation sample in the evaluation dataset.

[0036] Optionally, before receiving the model evaluation task, the method further includes:

[0037] Based on the recommendation strategy, several historical evaluation indicators are selected and recommended to the initiator of the model evaluation task;

[0038] The evaluation metrics specified in the model evaluation task are either defined by the initiator or selected from the historical evaluation metrics.

[0039] According to a second aspect of one or more embodiments of this specification, a model evaluation apparatus is provided, comprising:

[0040] The task receiving unit receives model evaluation tasks, which specify a target model, an evaluation dataset, and evaluation metrics.

[0041] The capability quantification unit uses the evaluation dataset to quantify the capability indicators of the target model across several dimensions.

[0042] The expression acquisition unit acquires the evaluation expression corresponding to the evaluation index, wherein the evaluation expression uses one or more capability indicators as input variables;

[0043] The model evaluation unit determines the model evaluation result of the target model based on the evaluation expression and the index values ​​of the corresponding capability indicators.

[0044] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the aforementioned method by executing the executable instructions.

[0045] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the aforementioned method.

[0046] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned method.

[0047] As can be seen from the above embodiments, after receiving the model evaluation task, the specification can use the evaluation dataset specified by the model evaluation task to quantify the capability indicators of the target model in several dimensions. Then, based on the evaluation expression specified by the model evaluation task and the index values ​​of the corresponding capability indicators, the model evaluation result of the target model can be determined, thereby realizing the automated evaluation of the target model. Compared with manual evaluation, this greatly improves the efficiency and accuracy of model evaluation. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the architecture of a model evaluation service system provided in an exemplary embodiment.

[0049] Figure 2 This is a flowchart of a model evaluation method provided in an exemplary embodiment.

[0050] Figure 3 This is a flowchart of another model evaluation method provided in an exemplary embodiment.

[0051] Figure 4 This is a flowchart of an exemplary embodiment of a method for summarizing evaluation index values.

[0052] Figure 5 This is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0053] Figure 6 This is a block diagram of a model evaluation apparatus provided in an exemplary embodiment. Detailed Implementation

[0054] With the rapid development and widespread application of artificial intelligence technology, the demand for evaluating model performance is increasing. Currently, various model evaluations mainly rely on manual calculations, which are inefficient and prone to errors.

[0055] This specification provides a model evaluation scheme that allows users to specify model evaluation indicators and automatically evaluate the model based on these indicators to obtain model evaluation results. Compared with manual calculation, this scheme can greatly improve the efficiency and accuracy of model evaluation.

[0056] Figure 1 This is a schematic diagram of the architecture of a model evaluation service system provided in an exemplary embodiment. For example... Figure 1As shown, the system may include a server 11, a network 12, and several electronic devices, such as a PC (Personal Computer) 13, a mobile phone 14, etc.

[0057] Server 11 can be a physical server containing an independent host, or it can be a virtual server hosted in a host cluster. During operation, server 11 can run server-side programs for a certain application to implement the relevant functions of that application. For example, when server 11 runs a model evaluation service program, it can be implemented as a corresponding model evaluation service platform.

[0058] PC13 and mobile phone14 are just some of the types of electronic devices that users can use. In reality, users can obviously also use electronic devices such as tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc., and one or more embodiments in this specification do not limit this. During operation, the electronic device can run a client-side program of an application to achieve the relevant functions of that application. For example, when the electronic device runs a model evaluation service program, it can act as a client for that model evaluation service. The client application of the aforementioned model evaluation service can be launched and run on the electronic device. This client-side program can be a native application installed on the electronic device, or it can be a mini-program, quick app, or other similar form. Of course, when using web technologies such as HTML5 or similar, the relevant functions can be achieved through a page displayed by a browser. This browser can be a standalone browser application or a browser module embedded in some applications.

[0059] As for the network 12 that enables interaction between electronic devices such as PC13 and mobile phone 14 and server 11, communication can be achieved using either wired or wireless networks, depending on the communication methods supported by the respective electronic devices. This specification does not impose any restrictions on this. For example, PC13 can support both wired and wireless communication, so it can use either wired or wireless networks as needed. Mobile phone 14 typically only supports wireless communication, so it can use a wireless network for communication.

[0060] Figure 2 This is a flowchart of a model evaluation method provided in an exemplary embodiment.

[0061] Please refer to Figure 2 The model evaluation method described above can be applied to the aforementioned Figure 1 The server 11 shown, which provides model evaluation services, includes the following steps:

[0062] Step 202: Receive the model evaluation task, which specifies the target model, evaluation dataset, and evaluation metrics.

[0063] In this embodiment, the initiating user can initiate a model evaluation task through the client page provided by the model evaluation service. For example, the initiator can specify the target model to be evaluated this time through the client page, and use the evaluation dataset and evaluation metrics of the target model to evaluate the target model.

[0064] In this embodiment, in a model evaluation task, the initiator can specify one or more target models to be evaluated. If multiple target models are specified, the initiator needs to specify an evaluation dataset and evaluation metrics for each target model. This embodiment will subsequently use the example of specifying one target model in the model evaluation task for illustration.

[0065] The number of evaluation datasets can be one or more. Each evaluation dataset can include multiple evaluation samples used for model evaluation. The model evaluation service can provide some publicly available evaluation datasets for the initiator to choose from, or the initiator can provide their own evaluation datasets; this specification does not impose any special restrictions on this.

[0066] The number of evaluation metrics can be one or more. Each evaluation metric can correspond to an evaluation expression, and each evaluation expression uses one or more model capability metrics as input variables. These capability metrics quantify the model's performance when processing data and include various types, such as accuracy, relevance, reasonableness, and robustness. They typically cover various common model quantification dimensions; of course, the initiator can also define custom capability metrics.

[0067] The evaluation metrics are customized by the initiator based on the business scenario and evaluation requirements of the target model. For example, the initiator can customize the evaluation metric name and edit the corresponding evaluation expression. The evaluation expression supports basic operators such as addition, subtraction, multiplication, and division, as well as scenario-specific logical operators, bracket operations, and nested expressions, offering extremely high flexibility.

[0068] In this embodiment, the evaluation metrics may include conclusion-type metrics and problem-attribution-type metrics. The conclusion-type metrics can be used to measure the overall performance of the target model, while the problem-attribution-type metrics can be used to analyze the reasons for model performance problems and can assist in conducting deeper results analysis.

[0069] Step 204: Use the evaluation dataset to quantify the capability indicators of the target model across several dimensions.

[0070] In this embodiment, the capability index of the target model can be quantified based on the evaluation samples in the evaluation dataset to obtain the capability index value of the target model.

[0071] For example, the capability indicators of the target model can be quantified on a sample-by-sample basis to obtain a capability indicator value corresponding to each sample. This capability indicator value represents the capability exhibited by the target model when processing the corresponding sample. For instance, when using sample 1 to quantify the relevance of the target model, the capability indicator value of the target model in the relevance dimension when processing sample 1 can be obtained; similarly, when using sample 5 to quantify the accuracy of the target model, the capability indicator value of the target model in the accuracy dimension when processing sample 5 can be obtained.

[0072] Step 206: Obtain the evaluation expression corresponding to the evaluation index, wherein the evaluation expression uses one or more capability indicators as input variables.

[0073] Step 208: Determine the model evaluation result of the target model based on the evaluation expression and the index values ​​of the corresponding capability indicators.

[0074] In this embodiment, after obtaining the capability indicators of the target model, the evaluation result of the target model can be calculated according to the evaluation expression corresponding to the evaluation indicators specified in the model evaluation task.

[0075] In one example, for each capability dimension, the capability index values ​​corresponding to each evaluation sample obtained in step 204 above can be summarized to obtain the capability index value of the target model in the corresponding dimension. Then, based on the evaluation expression, the required capability index values ​​are input into the evaluation expression to obtain the evaluation result of the target model.

[0076] In another example, each evaluation sample can be considered as a unit. Based on the evaluation expression, the capability index value of the corresponding evaluation sample is input into the evaluation expression to calculate the evaluation index value corresponding to the evaluation sample. Then, the evaluation index values ​​corresponding to each evaluation sample are summarized to obtain the evaluation result of the target model. By calculating the evaluation index value on a sample-by-sample basis, the evaluation index values ​​corresponding to the evaluation samples in the target scenario specified by the initiator can be summarized according to the annotation information of the evaluation samples to obtain the evaluation result of the target model in the target scenario, thereby achieving refined model evaluation.

[0077] As can be seen from the above description, after receiving a model evaluation task, this manual can use the evaluation dataset specified by the model evaluation task to quantify the capability indicators of the target model in several dimensions. Then, based on the evaluation expression specified by the model evaluation task and the corresponding capability indicator values, the model evaluation result of the target model can be determined, thereby realizing the automated evaluation of the target model. Compared with manual evaluation, this greatly improves the efficiency and accuracy of model evaluation.

[0078] The following sections will describe in detail the specific implementation of this manual from three aspects: initiation of model evaluation tasks, model evaluation, and report generation.

[0079] I. Initiation of Model Evaluation Task

[0080] In this embodiment, the initiating user can initiate a model evaluation task through the client page provided by the model evaluation service, and can specify the target model, evaluation dataset, evaluation index and target scenario to be evaluated for the model evaluation task.

[0081] 1. Target Model

[0082] In this embodiment, the target model is the model that needs to be evaluated. The target model can be an image processing model, such as an image classification model. The target model can also be an NLP (Natural Language Processing) model, such as a text classification model, a question answering model, etc., and this specification does not impose any special restrictions on it.

[0083] In this embodiment, a model evaluation task can specify one target model or multiple target models. This embodiment will be described below using a single target model as an example.

[0084] 2. Evaluation Dataset

[0085] In this embodiment, the evaluation dataset is a collection of data used to evaluate the target model. The evaluation dataset includes multiple samples used to evaluate the target model, referred to as evaluation samples. The number of evaluation datasets can be one or more.

[0086] When specifying an evaluation dataset for a model evaluation task, the initiator can add annotation information representing the capability metric type to the evaluation dataset, indicating that the evaluation dataset is used to quantify the corresponding capability metric of the target model. For example, annotation information representing the capability metric type as accuracy can be added to evaluation dataset 1 and evaluation dataset 2, indicating that the evaluation samples in evaluation dataset 1 and evaluation dataset 2 are used to quantify the target model's capability metric in accuracy; if the annotation information for the capability metric type of evaluation dataset 3 is correlation, it indicates that evaluation dataset 3 is used to quantify the target model's capability metric in correlation, and so on.

[0087] The initiator can also add annotation information with multiple annotation dimensions to each evaluation sample in the evaluation dataset. These annotation dimensions can include business scenarios, sample sources, difficulty levels, etc. The aforementioned capability metric types added to the evaluation dataset are also a type of annotation dimension for the evaluation samples, meaning that the capability metric type annotation information is the same for different evaluation samples in the same evaluation dataset.

[0088] Evaluation Samples Business Scenarios Sample source Difficulty level Types of ability indicators Evaluation Sample 1 finance Self-constructed Level 1 Correlation Evaluation Sample 2 finance **Tool Generation Level 2 Correlation Evaluation Sample 3 transportation Self-constructed Level 1 Correlation Evaluation Sample 4 Medical Self-constructed Level 3 Correlation

[0089] Table 1

[0090] Please refer to the examples in Table 1. Evaluation sample 1 is a financial scenario, is self-constructed, and has a difficulty level of 1. It is used to evaluate the relevance of the target model. Similarly, evaluation sample 2 is a financial scenario, is generated by a tool, has a difficulty level of 2, and is also used to evaluate the relevance of the target model.

[0091] This embodiment allows the initiator to add annotation information of multiple dimensions to the evaluation samples according to the evaluation needs. Subsequently, the evaluation results of the target model can be viewed from different annotation dimensions, which greatly improves the flexibility of model evaluation and realizes refined model evaluation.

[0092] In this embodiment, the model evaluation service can provide some publicly available evaluation datasets for the initiator to choose from, or the initiator can provide its own evaluation datasets. This specification does not impose any special restrictions on this.

[0093] 3. Evaluation Indicators

[0094] In this embodiment, the evaluation metrics can be defined by the initiator based on the business scenario of the target model and its evaluation requirements. Specifically, the initiator can customize the name, expression, display chart type, and statistical type of the evaluation metrics.

[0095] The expression, referred to as the evaluation expression, uses one or more model capability metrics as input variables. In other words, the corresponding capability metric values ​​can be substituted into the evaluation expression to calculate the corresponding evaluation metric values. The evaluation expression supports basic operators such as "+", "-", "×", and " / ", logical operators such as "&&", "||", and "!", and supports nested expressions and parenthesis operations, which can greatly improve the complexity and flexibility of the evaluation expression.

[0096] The capability metrics are indicators that quantify the performance of a model when processing data. They include various types, such as accuracy, relevance, reasonableness, and robustness, and typically cover various common model quantification dimensions. Of course, the initiator can also customize capability metrics and their quantification methods.

[0097] The chart type mentioned refers to the display style of the corresponding evaluation results in the model evaluation report, which may include tables, bar charts, pie charts, etc., and the initiator can choose according to its needs. Of course, in other examples, the initiator may not choose a display chart type, and the model evaluation report can be displayed based on the default display style.

[0098] The statistical type represents the algorithm type used when summarizing evaluation indicators, and may include a distribution type and a comprehensive type. The distribution type is used to count the percentage of evaluation indicators that hit a specified rule, while the comprehensive type is used to summarize evaluation indicators and calculate scores (e.g., sums or averages). Of course, in other examples, the initiator may not select a statistical type, and the comprehensive type algorithm may be used by default to calculate the sum or average when summing evaluation indicators.

[0099] In this embodiment, the initiator can specify one or more evaluation metrics for the model evaluation task. More comprehensively, evaluation metrics can be divided into two categories: conclusion-based metrics and problem-attribution-based metrics. Conclusion-based metrics can be used to measure the overall performance of the target model, while problem-attribution-based metrics can be used to analyze the reasons for model performance problems and can assist in deeper results analysis. For example, the initiator can specify two evaluation metrics for the model evaluation task, one of which is a conclusion-based metric and the other is a problem-attribution-based metric. Through the conclusion-based metric, the initiator can understand the performance of the target model; through the problem-attribution-based metric, the initiator can understand the specific reasons for the poor performance of the target model.

[0100] In another example, evaluation metrics can be recommended to the initiator for selection, saving customization time. Specifically, several historical evaluation metrics can be selected based on the recommendation strategy. The initiator can directly choose from the recommended historical evaluation metrics. If the required metrics are not among the recommended historical metrics, the initiator can also customize evaluation metrics using the aforementioned method.

[0101] The historical evaluation metrics refer to evaluation metrics that have been used in the past. When making recommendations, metrics that the initiator has previously used can be recommended, such as the N metrics that the initiator used most frequently. Of course, the initiator is not limited; the M metrics that have been used most frequently by all initiators in history can be recommended to the initiator, etc. This specification does not impose special restrictions on this. The values ​​of M and N can also be defined by the initiator.

[0102] When recommending historical evaluation metrics to the initiator, the name and evaluation expression of the historical evaluation metrics can be displayed to the initiator. Of course, the chart type and statistical type can also be displayed at the same time. The initiator can modify all of this information. For example, if the initiator finds that the evaluation expression of a certain historical evaluation metric is the same as their needs, they can select that historical evaluation metric and then modify its name, display chart type, and statistical type according to their needs, thereby saving the initiator time in customizing evaluation metrics.

[0103] Therefore, the model evaluation scheme provided in this manual can support the initiator to customize evaluation indicators, which is highly flexible and can support different initiators for different evaluation needs of different business models.

[0104] 4. Target scenario

[0105] In this embodiment, the target scenario is used to define the context in which the evaluation results of the target model are viewed. The initiator can customize which target scenarios to view the evaluation results of the target model under according to their needs. For example, the initiator can select annotation information of one or more evaluation samples under different annotation dimensions as the target scenario.

[0106] Taking the evaluation sample annotation information shown in Table 1 as an example, the initiator can define the target scenario as: Finance + Sample Difficulty Level 1, Transportation + Self-constructed Source + Relevance Capability Indicator Type, Finance + Self-constructed Source + Sample Difficulty Level 2, etc. It should be noted that if the initiator does not select a specific annotation dimension, it means there are no restrictions on the annotation information under that dimension. For example, if the target scenario is "Finance + Sample Difficulty Level 1," it means that the target scenario does not restrict the sample source.

[0107] In other examples, the initiator may not specify the target scenario. In this case, there is no need to consider the annotation information of the evaluation samples. When determining the model evaluation results, the evaluation index values ​​corresponding to all evaluation samples are summarized.

[0108] Therefore, the model evaluation scheme provided in this manual can support the initiator to add annotation information to the evaluation samples and support the initiator to customize the target scene. Subsequently, the model evaluation results corresponding to the target scene can be generated. It is highly flexible and can meet the various evaluation needs of the initiator through such high customization.

[0109] II. Model Evaluation

[0110] Figure 3 This is a flowchart of another model evaluation method provided in an exemplary embodiment.

[0111] Please refer to Figure 3 The model evaluation method provided in this embodiment may include the following steps:

[0112] Step 302: Receive the model evaluation task, which specifies the target model, evaluation dataset, and evaluation metrics.

[0113] In this embodiment, the initiation of the model evaluation task can be referred to the aforementioned embodiments, and will not be repeated here.

[0114] Step 304: Using each evaluation sample in the evaluation dataset as a unit, quantify the capability indicators of the target model in several dimensions to obtain the capability indicator value corresponding to each evaluation sample.

[0115] In this embodiment, the capability metrics of the target model can be quantified on a per-sample basis within the evaluation dataset. Specifically, each evaluation sample can be input into the target model to obtain the prediction result output by the target model. Then, based on the difference between the prediction result and the label of the evaluation sample, various capability metric values ​​of the target model can be quantified, such as the accuracy metric value, the relevance metric value, and the reasonableness metric value of the target model.

[0116] The quantification methods for the aforementioned common capability indicators can be found in relevant technologies. If the initiator defines its own capability indicators when launching the model evaluation task, the corresponding indicator values ​​can be calculated according to the initiator's defined capability indicator calculation method.

[0117] In this embodiment, after obtaining the capability index value corresponding to each evaluation sample, the capability index value can be stored in the database for subsequent use or as a report detail to be fed back to the initiator.

[0118] Step 306: Using the evaluation sample as a unit, calculate the evaluation index value corresponding to the evaluation sample based on the evaluation expression and the corresponding capability index value.

[0119] Based on the aforementioned step 304, after quantifying the capability index value corresponding to each evaluation sample, the evaluation index value corresponding to the evaluation sample can be calculated based on the evaluation expression, with the evaluation sample as the unit. The evaluation index value represents the performance or performance problem cause exhibited by the target model when processing the evaluation sample (depending on whether the evaluation index type is a conclusion type or a problem attribution type).

[0120] In this step, the evaluation expression corresponding to the evaluation index can be obtained. Then, the capability index value corresponding to the evaluation expression is input into the evaluation expression on a per-sample basis to obtain the evaluation index value corresponding to the evaluation sample.

[0121] For example, taking evaluation sample 1 as an example, if the evaluation expression uses the accuracy indicator value (hereinafter referred to as the accuracy indicator value) and the correlation indicator value (hereinafter referred to as the correlation indicator value) as input variables, then the accuracy indicator value and the correlation indicator value corresponding to evaluation sample 1 can be input into the evaluation expression to calculate the evaluation indicator value corresponding to evaluation sample 1.

[0122] Similarly, the evaluation index values ​​corresponding to all evaluation samples can be calculated, and the evaluation index values ​​can also be stored in the database.

[0123] In this embodiment, if the model evaluation task specifies multiple evaluation indicators, the evaluation indicator value corresponding to each evaluation indicator needs to be calculated separately for each evaluation sample.

[0124] In this embodiment, since the evaluation expression is defined by the initiator, when calculating the evaluation index value using the evaluation expression, lexical analysis can be performed first to identify numerical values, operators, variables, etc. Then, syntactic analysis can be performed, and a parse tree can be constructed using grammar rules to structure the parsed evaluation expression for subsequent calculation. Next, the value of the evaluation expression can be calculated based on the parse tree using a depth-first or breadth-first traversal approach. The logical operation part can employ a short-circuit evaluation strategy to improve computational efficiency.

[0125] In this embodiment, if the evaluation expression cannot be parsed, a message indicating an expression error can be returned to the initiator.

[0126] It is worth noting that before calculating the evaluation metric values, the values ​​should be deduplicated to ensure that identical evaluation metrics are calculated only once. Specifically, when multiple evaluation metrics are specified in the model evaluation task, metrics with the same name and evaluation expression should be deduplicated. During the deduplication process, the statistical type and chart type of the evaluation metric do not need to be considered. In other words, even if two evaluation metrics have different chart types, if their names and evaluation expressions are the same, deduplication is still required, retaining only one of the evaluation metrics for subsequent calculations. This avoids duplicate calculations caused by the initiator repeatedly setting evaluation metrics.

[0127] Step 308: Summarize the evaluation index values ​​corresponding to the evaluation samples to obtain the model evaluation results of the target model.

[0128] Based on the aforementioned step 306, after calculating the evaluation index value corresponding to each evaluation sample, the evaluation index values ​​can be summarized to obtain the evaluation result of the target model.

[0129] Please refer to Figure 4 The method for summarizing the evaluation index values ​​in step 308 may include the following steps:

[0130] Step 3082: For each scene unit under the model evaluation task, the evaluation index values ​​matching the scene unit are summarized according to the annotation information of the evaluation samples to obtain the unit evaluation result corresponding to the scene unit.

[0131] In this embodiment, when summarizing the evaluation index values ​​corresponding to the evaluation samples, the evaluation index values ​​can be summarized by scene unit to obtain the unit evaluation results corresponding to each scene unit.

[0132] The scene unit is the finest-grained scene in this model evaluation task. Each scene unit has limitations on each annotation dimension of the evaluation sample. The combination of different annotation information under all annotation dimensions constitutes the set of scene units for this model evaluation task. In other words, a scene unit is the Cartesian product of annotation information under different annotation dimensions. Assuming the initiator adds 3 annotation dimensions to the evaluation sample, the first annotation dimension includes m different annotation information under that dimension, the second annotation dimension includes n different annotation information under that dimension, and the third annotation dimension includes p different annotation information under that dimension, then the number of scene units in this model evaluation task is: m × n × p.

[0133] For example, taking an evaluation sample that includes business scenario annotations and difficulty level annotations, assuming the business scenario annotations include three business scenarios: finance, transportation, and healthcare, and the difficulty levels include two levels: Level 1 and Level 2, the number of scenario units is the product of the number of business scenarios and the number of difficulty levels. That is, there are 3 × 2 scenario units, totaling 6 types: finance + difficulty level 1, finance + difficulty level 2, transportation + difficulty level 1, transportation + difficulty level 2, healthcare + difficulty level 1, and healthcare + difficulty level 2. It can be understood that as the annotation dimensions or the annotation information under a certain annotation dimension increases, the number of scenario units can grow exponentially.

[0134] In this embodiment, for each scenario unit, the evaluation index values ​​of the evaluation samples matching the scenario unit can be summarized. Taking the scenario unit "Finance + Difficulty Level 1" as an example, the evaluation index values ​​corresponding to all evaluation samples whose business scenario in the annotation information is finance and whose difficulty level is 1 can be summarized to obtain the unit evaluation result corresponding to the scenario unit "Finance + Difficulty Level 1".

[0135] In the process of summarizing the evaluation index values ​​of each evaluation sample in the scene unit, the statistical type of the evaluation index can be obtained, and then the algorithm corresponding to the statistical type can be used to summarize the evaluation index values.

[0136] If the statistical type is a distribution type, the percentage of evaluation indicator values ​​that match a specified rule can be calculated as the summary result of the evaluation indicator values ​​(i.e., the unit evaluation result). The specified rule can be determined by the initiator when setting the statistical type. For example, if a scene unit has 100 evaluation indicator values, the percentage of these 100 evaluation indicator values ​​whose results are greater than a threshold can be calculated as the unit evaluation result for that scene unit. In actual implementation, the number of evaluation indicators corresponding to that scene unit, and the number of evaluation indicators whose results are greater than the threshold, can be saved.

[0137] If the statistical type is comprehensive, the average value of the evaluation indicators can be calculated as the unit evaluation result. Assuming that a certain scene unit has 100 evaluation indicator values, the average value of these 100 evaluation indicator values ​​can be calculated as the unit evaluation result for that scene unit.

[0138] Of course, if the initiator does not specify the statistical type of the evaluation indicators, it can be assumed to be the comprehensive type, and the average value of the evaluation indicators can be calculated as the unit evaluation result.

[0139] In this embodiment, the unit evaluation results corresponding to each scene unit can be saved for later viewing or use.

[0140] Step 3084: Obtain the target scene specified by the model evaluation task, and determine the scene unit associated with the target scene.

[0141] In this embodiment, when determining the model evaluation result, the target scenario specified by the initiator when initiating the model evaluation task can be obtained, and then the scene units associated with the target scenario can be determined. The scene units associated with the target scenario include all scene units related to any information within the target scenario.

[0142] For example, assuming the target scenario is "finance + difficulty level 1", the associated scenario units include all scenario units with financial business scenarios and all scenario units with difficulty level 1. Assuming the target scenario is "transportation + self-constructed source + difficulty level 3", the associated scenario units include all scenario units with transportation business scenarios, all scenario units with self-constructed sources, and all scenario units with difficulty level 3.

[0143] Step 3086: Summarize the unit evaluation results corresponding to the scene units associated with the target scene to obtain the model evaluation results of the target model.

[0144] In this embodiment, after determining the scene units associated with the target scene, the unit evaluation results corresponding to these associated scene units can be summarized. When summarizing the unit evaluation results, similar to summarizing the evaluation index values, the algorithm corresponding to the statistical type can still be used to summarize the evaluation results of each unit.

[0145]

[0146]

[0147] Table 2

[0148] If the statistical type is distribution, the percentage of evaluation metrics that match a specified rule can be calculated. For example, suppose the evaluation results of three units are summarized, as shown in Table 2. When summarizing, the ratio of the sum of the number of evaluation metrics with results greater than a threshold to the total number of evaluation metrics can be calculated as the model evaluation result of the target model in the target scenario, i.e., (90+85+42) / (100+120+67). Of course, in other examples, if the saved unit evaluation results are specific ratios, such as 80%, the average of these unit evaluation results can be calculated to obtain the model evaluation result.

[0149] If the statistical type is comprehensive, the average value of the evaluation results of each unit can be calculated as the target evaluation result.

[0150] In other examples, if the model evaluation task does not specify any target scenario, the unit evaluation results corresponding to all scenario units can be summarized to obtain the model evaluation result of the target model.

[0151] As described above, the model evaluation scheme provided in this embodiment can evaluate the target model on a sample-by-sample basis, obtaining the evaluation index value of the target model corresponding to the evaluation sample. Then, based on the evaluation index value, the unit evaluation result corresponding to each scene unit is determined. Next, the unit evaluation results of the scene units associated with the target scene specified in the model evaluation task are summarized to obtain the evaluation result of the target model. The entire process is completed automatically after receiving the model evaluation task, without manual intervention, which can greatly improve the efficiency and accuracy of model evaluation, providing real-time feedback on the performance of the target model and significantly improving the efficiency of target model development and optimization. Furthermore, this embodiment supports the customization of target scenes, allowing the initiator to obtain the evaluation results of the target model under a specified scene as needed, providing greater flexibility.

[0152] In another example of this embodiment, after obtaining the model evaluation results of the target model, the model evaluation results can be recalculated. Specifically, after obtaining the model evaluation results, the initiator can recalculate the model evaluation results by initiating an update task for the model evaluation results. For example, the initiator can specify a different target scene (referred to as a new scene for easy distinction) in the update task. In response to receiving the update task, the scene units associated with the new scene can be determined, and then the unit evaluation results corresponding to the scene units associated with the new scene can be summarized according to the stored unit evaluation results to obtain the updated model evaluation results.

[0153] Of course, in other examples, if no new scenario is specified in the update task, the model evaluation results can be recalculated based on the original target scenario.

[0154] In another example of this embodiment, to improve model evaluation efficiency, a distributed architecture can be used to implement the model evaluation service provided in this specification. For example, multiple computing nodes can be deployed to provide the model evaluation service. When a computing node receives a model evaluation task, it can distribute the evaluation samples in the evaluation dataset to other normally running computing nodes in the cluster. Multiple computing nodes then distribute the calculation of the corresponding capability index values ​​and evaluation index values ​​for the evaluation samples, thereby improving evaluation efficiency. After calculating the capability index values ​​and evaluation index values, these computing nodes can store these index values ​​in a database. Then, the computing node that received the model evaluation task can determine the corresponding unit evaluation result for each scene unit and determine the model evaluation result based on the unit evaluation result.

[0155] For example, after receiving a model evaluation task, compute node 1 can scan the normally operating compute nodes in the cluster, and then distribute the evaluation samples evenly to the normally operating compute nodes, including itself, to achieve distributed computation of capability index values ​​and evaluation index values.

[0156] III. Report Generation

[0157] In this embodiment, after obtaining the model evaluation results of the target model, a corresponding evaluation report can be generated and returned to the initiator of the model evaluation task for the initiator to view.

[0158] In this embodiment, the evaluation report may include three areas: a conclusion indicator area for displaying conclusion-type evaluation indicators, a problem attribution indicator area for displaying problem attribution-type evaluation indicators, and a detail area for displaying details.

[0159] The model evaluation results in both the conclusion indicator area and the problem attribution indicator area can be displayed based on the chart type corresponding to the evaluation indicator. For example, if the chart type specified by the initiator for the problem attribution indicator is a bar chart, a bar chart corresponding to the model evaluation results can be generated in the problem attribution indicator area for the initiator to view.

[0160] Of course, if the initiator does not specify conclusion-type indicators or problem attribution-type indicators, there is no need to display the corresponding area.

[0161] In the model evaluation report provided in this embodiment, both conclusion-type indicators and problem attribution-type indicators support sorting and downloading. For example, when the initiator specifies multiple conclusion-type indicators, they can be sorted in descending order of evaluation results for the initiator to view and download. Similarly, when the initiator specifies multiple problem attribution-type indicators, the evaluation results of these indicators can be sorted in descending order of problem severity.

[0162] The detailed area can be used to display information such as the indicator values ​​corresponding to each evaluation sample in the evaluation dataset. For example, the detailed area can display each evaluation sample, the prediction results of the target model for the evaluation sample, the label of the evaluation sample, the corresponding capability indicator values ​​of the evaluation sample, and the corresponding evaluation indicator values. Through the detailed area, the initiator can view the detailed information of each evaluation sample.

[0163] It should be noted that the initiator can implement the model evaluation result update function described in the aforementioned model evaluation embodiments through the controls provided in the evaluation report. For example, an update control can be provided in the model evaluation report. After the initiator triggers the update control, the initiator can specify a new scenario and initiate an update task, etc.

[0164] As can be seen from the above description, in this embodiment, after obtaining the evaluation results of the target model, a corresponding evaluation report can be automatically generated for the initiator to view, seamlessly connecting model evaluation and report generation, which greatly improves the efficiency of report generation compared to manual generation of evaluation reports.

[0165] Figure 5 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 502 reads the corresponding computer program from the non-volatile memory 510 into memory 508 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0166] Please refer to Figure 6 The model evaluation device 600 can be applied to, for example... Figure 5 The device shown is used to implement the technical solution of this specification. The model evaluation device 600 may include:

[0167] The task receiving unit 602 receives a model evaluation task, wherein the model evaluation task specifies a target model, an evaluation dataset, and evaluation metrics.

[0168] The capability quantification unit 604 uses the evaluation dataset to quantify the capability indicators of the target model in several dimensions.

[0169] The expression acquisition unit 606 acquires the evaluation expression corresponding to the evaluation index, wherein the evaluation expression uses one or more capability indicators as input variables;

[0170] The model evaluation unit 608 determines the model evaluation result of the target model based on the evaluation expression and the index values ​​of the corresponding capability indicators.

[0171] Optionally, the process of quantifying the capability metrics of the target model across several dimensions using the evaluation dataset includes:

[0172] Taking each evaluation sample in the evaluation dataset as a unit, the capability index of the target model is quantified in the several dimensions to obtain the capability index value corresponding to each evaluation sample;

[0173] The process of determining the model evaluation result of the target model based on the evaluation expression and the index values ​​of the corresponding capability indicators includes:

[0174] Using the evaluation sample as a unit, calculate the evaluation index value corresponding to the evaluation sample based on the evaluation expression and the corresponding ability index value;

[0175] The evaluation index values ​​corresponding to the evaluation samples are summarized to obtain the model evaluation results of the target model.

[0176] Optionally, the process of summarizing the evaluation index values ​​corresponding to the evaluation samples to obtain the evaluation results of the target model includes:

[0177] For each scene unit under the model evaluation task, the evaluation index values ​​matching the scene unit are summarized according to the annotation information of the evaluation samples to obtain the unit evaluation result corresponding to the scene unit.

[0178] Obtain the target scene specified in the model evaluation task, and determine the scene units associated with the target scene;

[0179] The unit evaluation results corresponding to the scene units associated with the target scene are summarized to obtain the model evaluation results of the target model.

[0180] Optionally, after obtaining the model evaluation results of the target model,

[0181] The model evaluation unit 608 receives an update task for the model evaluation result, wherein the update task specifies a new scene; determines the scene unit associated with the new scene, and summarizes the unit evaluation results corresponding to the scene unit associated with the new scene to obtain the updated model evaluation result.

[0182] Optionally, if the model evaluation unit 608 does not specify a target scenario in the model evaluation task, it can summarize the unit evaluation results corresponding to all scenario units to obtain the model evaluation result of the target model.

[0183] Optionally, the process of summarizing the evaluation index values ​​includes:

[0184] Obtain the statistical type of the evaluation indicators;

[0185] The evaluation index values ​​are summarized using the algorithm corresponding to the statistical type.

[0186] Optionally, the model evaluation unit 608 generates an evaluation report for the target model based on the model evaluation results and returns the evaluation report to the initiator of the model evaluation task.

[0187] Optionally, the process of generating an evaluation report for the target model based on the model evaluation results includes:

[0188] Obtain the display chart type corresponding to the evaluation indicators;

[0189] When generating the evaluation report, the evaluation results are displayed using the aforementioned chart type.

[0190] Optionally, the evaluation metrics include conclusion-type metrics and problem attribution-type metrics, and the evaluation report includes a conclusion-type metric area, a problem attribution-type metric area, and a detail area, wherein:

[0191] The conclusion indicator area is used to display the evaluation results of conclusion-type indicators;

[0192] The problem attribution metric area is used to display the evaluation results of problem attribution metrics;

[0193] The details area is used to display the indicator values ​​corresponding to each evaluation sample in the evaluation dataset.

[0194] Optionally, before receiving the model evaluation task, the model evaluation device is further configured to:

[0195] Based on the recommendation strategy, several historical evaluation indicators are selected and recommended to the initiator of the model evaluation task;

[0196] The evaluation metrics specified in the model evaluation task are either defined by the initiator or selected from the historical evaluation metrics.

[0197] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0198] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0199] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

Claims

1. A model evaluation method, comprising: Receive a model evaluation task, wherein the model evaluation task specifies a target model, an evaluation dataset, and evaluation metrics; The evaluation dataset is used to quantify the capability metrics of the target model across different dimensions. Obtain the evaluation expression corresponding to the evaluation index, wherein the evaluation expression uses one or more capability indicators as input variables; The evaluation result of the target model is determined based on the evaluation expression and the corresponding capability index values. The process of determining the model evaluation result of the target model based on the evaluation expression and the corresponding capability index values ​​includes: Calculate the evaluation index values ​​corresponding to the evaluation samples in the evaluation dataset; The evaluation index values ​​corresponding to the evaluation samples are summarized to obtain the model evaluation results of the target model; The process of summarizing the evaluation index values ​​corresponding to the evaluation samples to obtain the model evaluation results of the target model includes: For each scene unit under the model evaluation task, the evaluation index values ​​matching the scene unit are summarized according to the annotation information of the evaluation samples to obtain the unit evaluation result corresponding to the scene unit. Obtain the target scene specified in the model evaluation task, and determine the scene units associated with the target scene; The unit evaluation results corresponding to the scene units associated with the target scene are summarized to obtain the model evaluation results of the target model.

2. The method according to claim 1, wherein the process of quantifying the capability indicators of the target model using the evaluation dataset on different dimensions includes: Taking each evaluation sample in the evaluation dataset as a unit, the capability index of the target model is quantified on the different dimensions to obtain the capability index value corresponding to each evaluation sample; The process of calculating the evaluation index value corresponding to the evaluation sample in the evaluation dataset includes: Using the evaluation sample as a unit, calculate the evaluation index value corresponding to the evaluation sample based on the evaluation expression and the corresponding capability index value.

3. The method according to claim 1, after obtaining the model evaluation result of the target model, the method further includes: Receive an update task for the model evaluation results, wherein the update task specifies a new scenario; The scene units associated with the new scene are determined, and the unit evaluation results corresponding to the scene units associated with the new scene are summarized to obtain the updated model evaluation results.

4. The method according to claim 1, further comprising: If no target scenario is specified in the model evaluation task, the unit evaluation results corresponding to all scenario units are summarized to obtain the model evaluation result of the target model.

5. The method according to claim 1, wherein the process of summarizing the evaluation index values ​​includes: Obtain the statistical type of the evaluation indicators; The evaluation index values ​​are summarized using the algorithm corresponding to the statistical type.

6. The method according to claim 1, further comprising: An evaluation report is generated for the target model based on the model evaluation results. The evaluation report is returned to the initiator of the model evaluation task.

7. The method according to claim 6, wherein the process of generating an evaluation report for the target model based on the model evaluation results includes: Obtain the display chart type corresponding to the evaluation indicators; When generating the evaluation report, the evaluation results are displayed using the aforementioned chart type.

8. The method according to claim 6, wherein the evaluation indicators include conclusion-type indicators and problem attribution-type indicators, and the evaluation report includes a conclusion indicator area, a problem attribution indicator area, and a detail area, wherein: The conclusion indicator area is used to display the evaluation results of conclusion-type indicators; The problem attribution metric area is used to display the evaluation results of problem attribution metrics; The details area is used to display the indicator values ​​corresponding to each evaluation sample in the evaluation dataset.

9. The method according to claim 1, further comprising, before receiving the model evaluation task: Based on the recommendation strategy, several historical evaluation indicators are selected and recommended to the initiator of the model evaluation task; The evaluation metrics specified in the model evaluation task are either defined by the initiator or selected from the historical evaluation metrics.

10. A model evaluation device, comprising: The task receiving unit receives model evaluation tasks, which specify a target model, an evaluation dataset, and evaluation metrics. The capability quantification unit uses the evaluation dataset to quantify the capability indicators of the target model across different dimensions. The expression acquisition unit acquires the evaluation expression corresponding to the evaluation index, wherein the evaluation expression uses one or more capability indicators as input variables; The model evaluation unit determines the model evaluation result of the target model based on the evaluation expression and the index values ​​of the corresponding capability indicators; The process of determining the model evaluation result of the target model based on the evaluation expression and the corresponding capability index values ​​includes: Calculate the evaluation index values ​​corresponding to the evaluation samples in the evaluation dataset; The evaluation index values ​​corresponding to the evaluation samples are summarized to obtain the model evaluation results of the target model; The process of summarizing the evaluation index values ​​corresponding to the evaluation samples to obtain the model evaluation results of the target model includes: For each scene unit under the model evaluation task, the evaluation index values ​​matching the scene unit are summarized according to the annotation information of the evaluation samples to obtain the unit evaluation result corresponding to the scene unit. Obtain the target scene specified in the model evaluation task, and determine the scene units associated with the target scene; The unit evaluation results corresponding to the scene units associated with the target scene are summarized to obtain the model evaluation results of the target model.

11. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-9 by executing the executable instructions.

12. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-9.

13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-9.

Images