Multi-evaluator AI evaluation methods, devices, storage media and products
By determining the combination of evaluators and performing multi-dimensional evaluation and result fusion of AI model outputs, the problem of inconsistent AI model evaluation results was solved, and standardized evaluation was achieved under different tasks and business lines.
Patent Information
- Application Number
- CN202611353978.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-02
- Publication Date
- 2026-11-17
AI Technical Summary
Existing AI model evaluation methods cannot make unified comparisons across different task types and business lines, resulting in different evaluation dimensions and failing to achieve accurate evaluation.
By obtaining the model output, task type, and user configuration of the AI model to be evaluated, the combination of evaluators is determined, multiple evaluators are used to evaluate the model output, and the evaluation results are fused to obtain standardized evaluation results.
It achieves unified evaluation results across different business types and business lines, ensuring the standardization and accuracy of the evaluation results and enabling effective comparison of the performance of AI models.
Smart Images

Figure CN123116156A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to multi-evaluator AI evaluation methods, devices, storage media, and products. Background Technology
[0002] As the application of artificial intelligence deepens and AI projects and skills development continue to grow, it is crucial to accurately evaluate AI models in order to identify better ones.
[0003] Current AI model evaluation methods employ different evaluation methods for different task types such as classification, generation, and retrieval, as well as for different business lines. The resulting evaluation results for AI models vary in dimensions, making it impossible to compare AI models based on evaluation results from different dimensions. Consequently, current methods cannot accurately evaluate AI models. Summary of the Invention
[0004] The main purpose of this application is to provide a multi-evaluator AI evaluation method, device, storage medium and product, which aims to solve the technical problem that current methods cannot accurately evaluate AI models.
[0005] To achieve the above objectives, this application proposes a multi-evaluator AI evaluation method, the method comprising: Obtain the model output, task type, and user configuration of the AI model to be evaluated, and determine the evaluator combination for this evaluation based on the task type and the user configuration; Based on the combination of evaluators, the model output is evaluated to obtain the evaluation results of multiple evaluators; The evaluation results are then integrated and calculated to obtain a standardized evaluation result. The model is evaluated based on the standardized evaluation results.
[0006] In one embodiment, the evaluator combination further includes evaluation rules corresponding to each evaluator, and the step of evaluating the model output based on the evaluator combination to obtain the evaluation results of multiple evaluators includes: Based on the evaluator and the corresponding evaluation rule, each output data in the model output is evaluated to obtain the output data evaluation result corresponding to each evaluator. The evaluation results of the output data corresponding to the same evaluator are combined to obtain the evaluation results corresponding to each evaluator.
[0007] In one embodiment, the evaluator includes a multi-model ensemble evaluator, a vertical specialty evaluator, a custom evaluator, and a logical rule evaluator. The step of evaluating each output data in the model output based on the evaluator and the corresponding evaluation rules to obtain the output data evaluation result corresponding to each evaluator includes: If the evaluator is the multi-model ensemble evaluator, then the output data is evaluated based on multiple preset heterogeneous large models to obtain an ensemble evaluation result; If the evaluator is the vertical specialization evaluator, then based on the locally deployed lightweight model and the corresponding task type evaluation settings, the output data is evaluated locally to obtain the vertical evaluation result. If the evaluator is the custom evaluator, then the output data is evaluated based on the preset custom evaluation template and the custom prompts modified by the user to obtain a custom evaluation result; If the evaluator is the logic rule evaluator, then the output data is matched according to the corresponding evaluation rules to obtain the rule evaluation result.
[0008] In one embodiment, if the evaluator is the multi-model ensemble evaluator, the step of evaluating the output data based on multiple preset heterogeneous large models to obtain the ensemble evaluation result includes: Determine whether this is the first time the output data has been evaluated; If this is the first evaluation, the output data is evaluated concurrently based on each of the heterogeneous large models to obtain multiple large model evaluation results; Based on the credibility weights corresponding to each of the heterogeneous large models, the evaluation results of the large models are weighted and fused to obtain the integrated evaluation result.
[0009] In one embodiment, after determining whether this is the first time the output data has been evaluated, the method further includes: If this is not the first evaluation, then retrieve the feedback record from the previous evaluation; The feedback summary agent summarizes the feedback records based on the preset feedback summary; Obtain historical evaluation data for each of the heterogeneous large models, and adjust the credibility weights of each of the heterogeneous large models based on the historical evaluation data and the feedback summary to obtain the adjusted credibility weights. Based on the adjusted credibility weights, the evaluation results of the large model are weighted and fused to obtain the integrated evaluation results.
[0010] In one embodiment, the step of determining the evaluator combination for this evaluation based on the task type and the user configuration includes: Based on the task type and the user configuration, determine the evaluator to be used in this evaluation, and determine the evaluation indicators, evaluation thresholds and evaluation logic corresponding to each evaluator; Based on the evaluation metrics, the evaluation thresholds, and the evaluation logic, the evaluation rules for each of the evaluators are determined; The evaluator combination is determined based on each of the evaluators and the corresponding evaluation rules.
[0011] In one embodiment, the step of fusing and calculating the evaluation results to obtain a standardized evaluation result includes: Obtain user-defined fusion computing rules; Based on the fusion calculation rules, the evaluation results are weighted and fused to obtain the standardized evaluation results, wherein the standardized evaluation results include one or more of accuracy, consistency and robustness.
[0012] In addition, to achieve the above objectives, this application also proposes a multi-evaluator AI evaluation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-evaluator AI evaluation method as described above.
[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the multi-evaluator AI evaluation method described above.
[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the multi-evaluator AI evaluation method described above.
[0015] One or more technical solutions proposed in this application have at least the following technical effects: Obtain the model output, task type, and user configuration of the AI model to be evaluated. Based on the task type and user configuration, determine the evaluator combination for this evaluation. Based on the evaluator combination, evaluate the model output to obtain the evaluation results of multiple evaluators. Perform fusion calculation on the evaluation results to obtain a standardized evaluation result. Evaluate the model based on the standardized evaluation result.
[0016] Current AI model evaluation methods employ different methods for different task types and business lines, resulting in varying evaluation dimensions. This makes it impossible to compare AI models based on evaluation results across different dimensions, hindering accurate model evaluation. Therefore, this application determines the evaluator combination based on the acquired task type and user configuration. By fusing multiple evaluation results, a standardized evaluation result is obtained to assess the model. In other words, for the multi-dimensional evaluation results corresponding to different business types and business lines, this application merges the different dimensions into a standardized evaluation result. This ensures that regardless of changes in business type and business line, the obtained evaluation result is standardized and consistent, allowing for accurate AI model comparison and evaluation. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart illustrating an embodiment of the multi-evaluator AI evaluation method of this application. Figure 2 This is a schematic diagram of the system architecture provided in Embodiment 1 of the multi-evaluator AI evaluation method of this application; Figure 3 This is a schematic diagram of the evaluation process provided in Embodiment 1 of the multi-evaluator AI evaluation method of this application; Figure 4 This is a flowchart illustrating Embodiment 2 of the multi-evaluator AI evaluation method of this application; Figure 5 This is a schematic diagram of the recommended metrics for the specific evaluators provided in Embodiment 2 of the multi-evaluator AI evaluation method of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the multi-evaluator AI evaluation method in the embodiments of this application.
[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0023] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or multi-evaluator AI evaluation device capable of performing the above functions. The following description uses a multi-evaluator AI evaluation device as an example to illustrate this embodiment and the subsequent embodiments.
[0024] As the application of artificial intelligence deepens and AI projects and skills development continue to grow, it is crucial to accurately evaluate AI models in order to identify better ones.
[0025] Current AI model evaluation methods employ different evaluation methods for different task types such as classification, generation, and retrieval, as well as for different business lines. The resulting evaluation results for AI models vary in dimensions, making it impossible to compare AI models based on evaluation results from different dimensions. Consequently, current methods cannot accurately evaluate AI models.
[0026] In addition, traditional manual evaluation methods are time-consuming and labor-intensive, making it difficult to cope with the needs of large-scale model iteration, especially in scenarios that require frequent regression testing. If a single LLM (Large Language Model) is used for evaluation, there will be systemic bias and solidified preferences, resulting in unstable evaluation results and limited evaluation capabilities, which cannot guarantee the objectivity and consistency of the evaluation.
[0027] Furthermore, the current method still suffers from incomplete process records, making it difficult to locate problems and optimize models, and it cannot conduct backtracking analysis when evaluation disputes arise.
[0028] Based on this, embodiments of this application provide a multi-evaluator AI evaluation method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the multi-evaluator AI evaluation method of this application.
[0029] In this embodiment, the multi-evaluator AI evaluation method includes steps S10 to S40: Step S10: Obtain the model output, task type, and user configuration of the AI model to be evaluated; and determine the evaluator combination for this evaluation based on the task type and the user configuration. It should be noted that the AI model under evaluation is the evaluation object of this evaluation activity, i.e., the AI model being evaluated. During the evaluation process, this model generates output in response to the input. This evaluation scheme uses this model as its service object and focuses on evaluating the quality of its output. Model output refers to the result data generated by the AI model under evaluation in response to the input. This result data includes classification results, generated text, retrieval results, etc., and is the direct object of quality assessment in this evaluation scheme. Task type refers to the category to which the task performed by the AI model under evaluation belongs. This category includes classification, generation, retrieval, sequence labeling, regression, clustering, and agent functions, etc. Different types of tasks correspond to different evaluation dimensions and evaluation methods.
[0030] User configuration refers to the configuration information provided by the user for this evaluation. This configuration information includes settings such as evaluator selection, evaluation weights, and evaluation parameters, reflecting the user's specific requirements for this evaluation activity. An evaluator is a functional unit used to evaluate the model output of the AI model to be evaluated. Each evaluator corresponds to an evaluation method, including evaluation based on large model adjudication, rule-matching based evaluation, task-specific evaluation, and custom evaluation. This evaluation scheme provides multiple evaluators for selection. The evaluator set is a collection of evaluators determined for this evaluation. This set consists of multiple evaluators working together to comprehensively evaluate the model output of the AI model to be evaluated from different dimensions.
[0031] It is understood that this embodiment obtains the model output, task type, and user configuration of the AI model to be evaluated, and determines the evaluator combination for this evaluation based on the task type and user configuration. That is, it selects an evaluator that matches the task type from a set of preset evaluators according to the task type, and determines the specific composition of the evaluator combination according to the user configuration. This allows the evaluator combination to adaptively adjust with changes in task type and user configuration, thereby improving the adaptability of the evaluator combination to the evaluation scenario, enhancing the relevance and effectiveness of the evaluation results, and improving the evaluation scheme's coverage of diverse evaluation scenarios.
[0032] In the implementation process, the system in this embodiment adopts a three-layer architecture, follows the principles of separation of concerns and domain-driven design, and realizes the reuse of AI capabilities and engineering implementation by clearly defining responsibilities, so as to improve the scalability and maintainability of the system.
[0033] The system architecture of this embodiment can be referred to Figure 3Specifically, it includes the application layer, core capability layer, and data layer. The application layer is used for decoupling business capabilities and providing service exit points. This layer serves as the carrier of business functions, maintaining their isolation and providing a unified RESTful (Representational State Transfer) external interface via FastAPI (Fast Application Programming Interface). This layer primarily implements the evaluation of four evaluators; the specific evaluation process can be found in [reference needed]. Figure 4 .
[0034] The core capability layer is used to build a standardized AI capability hub. This layer is based on the LangChain Agentic framework and industry scaffolding, and uses middleware to achieve end-to-end metadata tracking. With an asynchronous and concurrent architecture, the application layer can continuously call the evaluation rules and result judgment engine on the backend as long as the interface does not time out. The data layer is used to store data related to the evaluation process and final results.
[0035] In one feasible implementation, the specific implementation of determining the evaluator combination for this evaluation based on the task type and the user configuration can also be: Based on the task type and the user configuration, determine the evaluator to be used in this evaluation, and determine the evaluation indicators, evaluation thresholds and evaluation logic corresponding to each evaluator; Based on the evaluation metrics, the evaluation thresholds, and the evaluation logic, the evaluation rules for each of the evaluators are determined; The evaluator combination is determined based on each of the evaluators and the corresponding evaluation rules.
[0036] It should be noted that evaluation metrics are quantitative dimensions used to measure the quality of the AI model's output. These metrics include accuracy, consistency, robustness, etc., and are used to characterize the performance level of the model's output from different aspects. Evaluation thresholds are the boundary values used to determine whether the evaluation results meet the requirements. These boundary values correspond to the evaluation metrics and are the critical standards for determining whether the model output meets the standards on the corresponding evaluation metrics, used to distinguish between qualified and unqualified evaluation results.
[0037] Evaluation logic refers to the processing rules for judging evaluation results. These rules specify how the evaluator makes comprehensive judgments based on evaluation metrics and their results, such as combining multiple judgment conditions using AND or OR logic. Evaluation rules are the judgment criteria followed by the evaluator when performing evaluations. These criteria consist of evaluation metrics, evaluation thresholds, and evaluation logic, and are used to guide the evaluator in specifically evaluating the model output.
[0038] It is understood that this embodiment determines the evaluators to be used in this evaluation, as well as the corresponding evaluation metrics, thresholds, and logic for each evaluator, based on the task type and user configuration. It then determines the evaluation rules for each evaluator based on these metrics, thresholds, and logic. Finally, it determines the evaluator combination based on each evaluator and its corresponding evaluation rules. This ensures that each evaluator in the combination possesses clearly defined evaluation rules consisting of evaluation metrics, thresholds, and logic, giving the evaluator combination complete executable decision-making capabilities from the moment it is determined. Since the evaluation metrics, thresholds, and logic are all determined based on the task type and user configuration, the matching of the evaluation rules to the evaluation scenario is guaranteed, improving the reliability of the evaluation results.
[0039] Step S20: Based on the combination of evaluators, evaluate the model output to obtain the evaluation results of multiple evaluators; It should be noted that the evaluation result is the judgment result obtained by the evaluator after evaluating the model output. This result reflects the performance of the model output on the corresponding evaluation dimension, such as whether it meets the requirements of the corresponding evaluation rules, or the accuracy of the model output.
[0040] It is understood that this embodiment evaluates the model output based on an evaluator combination. The evaluator combination contains multiple evaluators, and each evaluator has evaluation rules consisting of evaluation indicators, evaluation thresholds, and evaluation logic. This allows multiple evaluators to evaluate the model output according to their respective evaluation rules, resulting in evaluation results from multiple evaluators. This avoids the evaluation bias caused by a single evaluator due to a single evaluation perspective, examines the model output from multiple dimensions, and thus improves the comprehensiveness of the evaluation results.
[0041] Step S30: Perform a fusion calculation on the evaluation results to obtain a standardized evaluation result; It should be noted that fusion computation is a computational process that integrates multiple evaluation results. This process combines evaluation results from multiple evaluators into a single overall result using a unified processing method, such as comprehensive weighting or comprehensive judgment of each evaluation result, to eliminate dimensional differences between multiple evaluation results. Standardized evaluation results are normalized results obtained after fusion computation of multiple evaluation results. This result represents the overall performance level of the model output in a unified form, facilitating comparison and analysis in different evaluation scenarios.
[0042] In one feasible implementation, the specific implementation of fusing and calculating the evaluation results to obtain standardized evaluation results may also be: Obtain user-defined fusion computing rules; Based on the fusion calculation rules, the evaluation results are weighted and fused to obtain the standardized evaluation results, wherein the standardized evaluation results include one or more of accuracy, consistency and robustness.
[0043] It should be noted that the fusion calculation rules are the fusion processing regulations set by the user for this evaluation. These regulations clarify how multiple evaluation results are processed during the fusion process, such as the weight allocation and weighting method of each evaluation result, and are used to guide the execution of weighted fusion calculation. Weighted fusion calculation is a calculation method that combines multiple evaluation results according to the fusion calculation rules, merging them into a single result based on the weights corresponding to each evaluation result.
[0044] Accuracy is an evaluation metric that measures the correctness of a model's output. It reflects the proportion of the model's output that matches the standard answer or expected result, and is used to characterize the model's performance level in terms of correctness.
[0045] Consistency is an evaluation metric that measures the stability of model output. It reflects the degree to which the same model output maintains consistent results under different evaluation conditions, and is used to characterize the stability performance of the model output. Robustness is an evaluation metric that measures the reliability of model output. It reflects the ability of the model output to maintain stable performance under conditions of input variation or disturbance, and is used to characterize the reliability performance of the model output.
[0046] It is understood that this embodiment obtains user-defined fusion calculation rules and performs weighted fusion calculations on each evaluation result based on these rules. This allows the weight allocation of each evaluation result during the fusion process to be determined according to the user-defined fusion calculation rules, avoiding the problem of a fixed and singular fusion calculation method that cannot adapt to different evaluation needs, thereby improving the flexibility of standardized evaluation results. Furthermore, since standardized evaluation results include one or more of accuracy, consistency, and robustness, they are presented in the form of specific quantitative indicators, enhancing the intuitiveness of the standardized evaluation results and making it easier for users to directly use and understand the evaluation conclusions according to their own evaluation needs.
[0047] Step S40: Evaluate the model based on the standardized evaluation results.
[0048] It is understood that this embodiment evaluates the model based on standardized evaluation results. The standardized evaluation results are normalized results obtained by fusing and calculating multiple evaluation results. They have integrated information from multiple evaluators evaluating the model output from different evaluation dimensions, so that the evaluation of the model is based on a comprehensive and unified data foundation. This avoids the inconsistency caused by evaluating based on a single evaluation result or unfused scattered evaluation data, thus achieving an effective evaluation of the overall performance of the AI model being evaluated.
[0049] In one embodiment, the step of fusing and calculating the various evaluation results to obtain a standardized evaluation result further includes: The consistency of the evaluation results for each evaluator is checked to obtain the consistency check results. Based on the consistency detection results, it is determined whether there is any evaluation discrepancy among the evaluation results corresponding to each evaluator; If there is a discrepancy in the evaluation, an arbitration process is performed on the evaluation results corresponding to each evaluator to obtain an arbitration evaluation result; Based on the arbitration evaluation results, the credibility weights of each evaluator are updated to obtain the updated credibility weights; Based on the updated credibility weights, the evaluation results are fused and calculated to obtain the standardized evaluation results.
[0050] It should be noted that consistency testing is a process that detects the degree of consistency between the evaluation results of different evaluators. This process can be implemented by statistically analyzing the deviations between evaluation results, calculating the degree of difference between evaluation results, or using consistency indicators such as standard deviation, to quantify the degree of consistency between evaluation results. The consistency test result is the result obtained after performing the consistency test. This result quantitatively represents the degree of consistency between the evaluation results of different evaluators, such as the degree of difference or consistency rate between evaluation results, and is used as a basis for determining whether there are evaluation discrepancies.
[0051] Evaluation discrepancies are inconsistencies between the evaluation results of different evaluators. This situation manifests as significant differences in the numerical values or judgment conclusions of the evaluation results. For example, some evaluators may determine that the evaluation results meet the standards, while others may determine that the evaluation results do not meet the standards. This is an abnormal state that the evaluation system needs to resolve.
[0052] Arbitration is a process of adjudicating the evaluation results of different evaluators when there are disagreements. This process can be implemented through multi-model concurrent arbitration, voting, or a comprehensive judgment combining historical evaluation data. It aims to eliminate discrepancies between evaluation results and reach a unified ruling. The arbitration evaluation result is the ruling obtained after the arbitration process. This result integrates the discrepancies in the evaluation results of each evaluator, forming a unified ruling on the evaluation disagreements, which guides the updating of the evaluator confidence levels.
[0053] The evaluator confidence score is the weight corresponding to each evaluator. This weight reflects the reliability of the evaluator's evaluation result in the fusion calculation. Specifically, it can be determined by statistically analyzing the evaluation performance of each evaluator based on historical evaluation data, or by updating it in conjunction with arbitration evaluation results. It is used to adjust the contribution of each evaluator's evaluation result to the standardized evaluation result. The updated evaluator confidence score is the weight obtained by updating the evaluator confidence scores of each evaluator based on the arbitration evaluation results. This weight reflects the reliability of each evaluator after arbitration processing and is used to replace the original evaluator confidence scores in the fusion calculation.
[0054] Understandably, this embodiment first performs a consistency check on the evaluation results corresponding to each evaluator to obtain a consistency check result. Based on the consistency check result, it determines whether there is an evaluation discrepancy between the evaluation results corresponding to each evaluator. Only when there is an evaluation discrepancy is the evaluation results corresponding to each evaluator arbitrated to obtain an arbitrated evaluation result. Based on the arbitrated evaluation result, the evaluator confidence of each evaluator is updated to obtain an updated evaluator confidence. Then, based on the updated evaluator confidence, the evaluation results are fused to obtain a standardized evaluation result. This ensures that inconsistencies in the evaluation results between evaluators are identified and unified through arbitration before fusion calculation. At the same time, the arbitrated evaluation result obtained from the arbitration process is fed back to the update of the evaluator confidence, so that the evaluator confidence on which the fusion calculation is based can reflect the evaluation information after arbitration. This avoids the problem of instability and unreliability of the standardized evaluation result caused by direct fusion calculation when there is a discrepancy in the evaluation results corresponding to each evaluator, thereby improving the reliability and stability of the standardized evaluation result.
[0055] In one embodiment, the step of performing arbitration processing on the evaluation results corresponding to each evaluator to obtain the arbitration evaluation result further includes: Obtain the evaluation basis information generated by each evaluator during the evaluation process; Based on the evaluation criteria information, the evaluation results corresponding to each evaluator are cross-validated to obtain the support level of the evaluation results corresponding to each evaluator. Based on the support level of the evaluation results corresponding to each evaluator, an arbitration decision is made on the evaluation results corresponding to each evaluator to obtain the arbitration evaluation result.
[0056] It should be noted that the evaluation basis information refers to the intermediate judgment information generated by each evaluator during the evaluation process, supporting its evaluation results. This information can specifically include the judgment reasons recorded by the evaluator during the evaluation, intermediate calculated values during the evaluation process, or detailed records of rule hits, reflecting the basis upon which each evaluator arrives at its evaluation results. Cross-validation is a process of comparing and verifying the evaluation results and their corresponding evaluation basis information for each evaluator. Specifically, this process involves comparing each evaluator's evaluation results with the evaluation basis information generated during the evaluation process item by item, analyzing the correlations and contradictions between the various evaluation basis information, and using this to determine the credibility of each evaluator's evaluation results.
[0057] Support is a quantitative value derived from cross-validation, representing the degree of support received by the evaluation results corresponding to each evaluator. Specifically, this value can be the percentage of evaluation results from other evaluators that are consistent with the current evaluation result, or the degree to which the evaluation result matches the information on which it was evaluated. It serves as the basis for arbitration decisions. Arbitration decisions are made based on the support levels of the evaluation results from each evaluator. This process can be implemented by selecting the evaluation result from the evaluator with the highest support as the final decision, or by weighting and synthesizing the evaluation results from each evaluator according to their support levels to form the final decision, thus obtaining the arbitration evaluation result.
[0058] It is understood that this embodiment acquires the evaluation basis information generated by each evaluator during the evaluation process, and cross-verifies the evaluation results corresponding to each evaluator based on the evaluation basis information to obtain the support level of the evaluation results corresponding to each evaluator. Then, based on the support level of the evaluation results corresponding to each evaluator, an arbitration decision is made to obtain the arbitration evaluation result. This embodiment uses support level for arbitration, so that the arbitration process is no longer limited to the superficial differences in numerical or conclusion levels of the evaluation results corresponding to each evaluator, but rather cross-verifies the evaluation basis information supporting each evaluation result to support the quantification of the credibility of the evaluation results corresponding to each evaluator and make a decision accordingly, thereby improving the objectivity and interpretability of the arbitration evaluation result.
[0059] In summary, this embodiment obtains the model output, task type, and user configuration of the AI model to be evaluated. Based on the task type and user configuration, it determines the combination of evaluators for this evaluation. Based on the combination of evaluators, it evaluates the model output to obtain the evaluation results of multiple evaluators. It then performs a fusion calculation on the evaluation results to obtain a standardized evaluation result. Finally, it evaluates the model based on the standardized evaluation result.
[0060] Current AI model evaluation methods employ different methods for different task types and business lines, resulting in varying evaluation dimensions. This makes it impossible to compare AI models based on evaluation results across different dimensions, hindering accurate model evaluation. Therefore, this embodiment determines the evaluator combination based on the acquired task type and user configuration. By fusing multiple evaluation results, a standardized evaluation result is obtained to assess the model. In other words, for the multi-dimensional evaluation results corresponding to different business types and business lines, this embodiment merges the different dimensional evaluation results into a standardized evaluation result. This ensures that regardless of changes in business type and business line, the obtained evaluation result is standardized and consistent, allowing for accurate AI model evaluation through comparison using the standardized result.
[0061] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 The evaluator combination further includes evaluation rules corresponding to each evaluator. Step S20, and the multi-evaluator AI evaluation method further includes steps S21-S22: Step S21: Based on the evaluator and the corresponding evaluation rule, evaluate each piece of output data in the model output to obtain the output data evaluation result corresponding to each evaluator; It should be noted that the output data is a single data unit in the model's output, which is the result generated by the AI model to be evaluated for a single input. The output data evaluation result is the result obtained by the evaluator after evaluating each single output data in the model's output. This result reflects the performance of that output data in the corresponding evaluation dimension and is the evaluation basis formed by the evaluator after evaluating each output data one by one.
[0062] In one feasible implementation, the evaluator includes a multi-model ensemble evaluator, a vertical specialty evaluator, a custom evaluator, and a logical rule evaluator. The specific implementation of evaluating each output data point in the model output based on the evaluator and the corresponding evaluation rules to obtain the output data evaluation results corresponding to each evaluator can also be: If the evaluator is the multi-model ensemble evaluator, then the output data is evaluated based on multiple preset heterogeneous large models to obtain an ensemble evaluation result; If the evaluator is the vertical specialization evaluator, then based on the locally deployed lightweight model and the corresponding task type evaluation settings, the output data is evaluated locally to obtain the vertical evaluation result. If the evaluator is the custom evaluator, then the output data is evaluated based on the preset custom evaluation template and the custom prompts modified by the user to obtain a custom evaluation result; If the evaluator is the logic rule evaluator, then the output data is matched according to the corresponding evaluation rules to obtain the rule evaluation result.
[0063] It should be noted that a multi-model ensemble evaluator is a type of evaluator that evaluates output data based on multiple pre-defined heterogeneous large models. Each of these heterogeneous large models evaluates the same output data independently, making it a general semantic-level evaluator. Heterogeneous large models are multiple pre-defined, distinct large models used to independently evaluate the output data, each providing its own perspective. The ensemble evaluation result is the result obtained by the multi-model ensemble evaluator after evaluating the output data, integrating the evaluation information from multiple heterogeneous large models.
[0064] A vertical-specific evaluator is a type of evaluator that evaluates output data locally based on a locally deployed lightweight model and corresponding task-type evaluation settings. It is a specialized evaluator designed for vertical tasks. The lightweight model is a locally deployed, dedicated model configured and deployed in a local environment for evaluating output data. The task-type evaluation settings are the specific evaluation methods used by the lightweight model to evaluate the output data, adapting to the evaluation requirements of different task types.
[0065] A custom evaluator is a type of evaluator that evaluates output data based on a preset custom evaluation template and user-modified custom prompts. It is used to meet the evaluation requirements of user-defined evaluation logic. The custom evaluation template is a preset standard input / output template for the evaluation task, specifying the input / output format for the task. Users can modify this template to create custom prompts. Custom prompts are the prompts obtained by users after modifying the preset custom evaluation template. These prompts contain user-defined evaluation requirements and logic, guiding the custom evaluator in evaluating the output data.
[0066] A logic rule evaluator is a type of evaluator that performs rule matching on the output data based on corresponding evaluation rules. It belongs to the category of deterministic logic rule evaluators and is used to verify output data with clearly defined judgment criteria. Rule matching is the matching process performed on the output data according to the evaluation rules. This process determines whether the output data meets the corresponding rule requirements based on the evaluation metrics, evaluation thresholds, and evaluation logic in the evaluation rules.
[0067] It should also be noted that the vertical specialization evaluator in this embodiment includes recommended functional metrics, performance metrics, and security compliance metrics for tasks such as classification, generation, retrieval, sequence labeling, and regression, which can be referred to... Figure 5 Furthermore, the vertical specialization evaluator in this embodiment encapsulates standardized evaluation capabilities as a Skill, supporting expansion at any time.
[0068] This embodiment's custom evaluator input / output templates include templates for comparative evaluation, rating evaluation, and preference evaluation. Users can quickly define new evaluation logic based on these templates by writing or modifying prompts using natural language. The system will automatically encapsulate this into an executable evaluation unit and incorporate it into a unified scheduling and reporting system.
[0069] It is understood that the evaluator in this embodiment includes a multi-model integrated evaluator, a vertical specialization evaluator, a custom evaluator, and a logic rule evaluator. By using different types of evaluators to evaluate the output data one by one in a manner appropriate to their functional positioning, integrated evaluation results, vertical evaluation results, custom evaluation results, and rule evaluation results are obtained respectively. This achieves multi-dimensional coverage evaluation of the output data from general semantics, vertical specialization, custom logic to deterministic rules, avoiding the problem of missing evaluation dimensions and difficulty in comprehensively evaluating the output data caused by using a single evaluation method, thereby improving the comprehensiveness of the evaluation.
[0070] In one feasible implementation, if the evaluator is the multi-model ensemble evaluator, then the specific implementation of evaluating the output data based on multiple preset heterogeneous large models to obtain the ensemble evaluation result can also be: Determine whether this is the first time the output data has been evaluated; If this is the first evaluation, the output data is evaluated concurrently based on each of the heterogeneous large models to obtain multiple large model evaluation results; Based on the credibility weights corresponding to each of the heterogeneous large models, the evaluation results of the large models are weighted and fused to obtain the integrated evaluation result.
[0071] It should be noted that the first evaluation is the initial evaluation of the output data. This evaluation is used to obtain the evaluation results of each heterogeneous large model on the output data when there is no historical evaluation data available. It serves as the starting point for subsequent evaluation processes. Concurrent evaluation is a method in which multiple heterogeneous large models evaluate the output data simultaneously. This method allows each heterogeneous large model to complete its evaluation of the output data in parallel within the same round of evaluation, and the evaluation processes of each heterogeneous large model are independent of each other.
[0072] The large model evaluation result is the result obtained after each heterogeneous large model evaluates the output data separately. Each heterogeneous large model corresponds to a large model evaluation result, reflecting the evaluation opinion of that heterogeneous large model on the output data. The credibility weight is the weight corresponding to each heterogeneous large model. This weight reflects the credibility of the evaluation opinion of the corresponding heterogeneous large model and is used to adjust the contribution of the evaluation results of each heterogeneous large model to the integrated evaluation result during the fusion process.
[0073] Understandably, this embodiment first determines whether it is the first time the output data is evaluated. During the first evaluation, the output data is concurrently evaluated based on various heterogeneous large models, resulting in multiple large model evaluation results. Then, based on the credibility weights corresponding to each heterogeneous large model, the large model evaluation results are weighted and fused to obtain an integrated evaluation result. This allows the first evaluation to obtain an integrated evaluation result derived from the concurrent evaluation of multiple heterogeneous large models and weighted fusion according to credibility weights, avoiding the bias and instability introduced by relying solely on a single model. Furthermore, since the weighted fusion is based on the credibility weights corresponding to each heterogeneous large model, the evaluation results of heterogeneous large models with different credibility levels are treated differently during the fusion process, thereby improving the accuracy of the integrated evaluation result.
[0074] In one feasible implementation, the specific implementation following the step of determining whether it is the first time the output data is evaluated may also be: If this is not the first evaluation, then retrieve the feedback record from the previous evaluation; The feedback summary agent summarizes the feedback records based on the preset feedback summary; Obtain historical evaluation data for each of the heterogeneous large models, and adjust the credibility weights of each of the heterogeneous large models based on the historical evaluation data and the feedback summary to obtain the adjusted credibility weights. Based on the adjusted credibility weights, the evaluation results of the large model are weighted and fused to obtain the integrated evaluation results.
[0075] It should be noted that the feedback record refers to the user feedback information recorded during the previous evaluation process. This record contains user opinions on the evaluation results or model output. The feedback summary agent is a pre-defined agent used to automatically summarize the feedback record and extract structured feedback points from it. The feedback summary is the result obtained by the feedback summary agent after summarizing the feedback record. This result presents the key information in the feedback record in a structured form to guide the adjustment of credibility weights.
[0076] Historical evaluation data consists of evaluation data generated during the historical evaluation process of each heterogeneous large model. This data reflects the performance of each heterogeneous large model in past evaluations and is used to assess the credibility of each heterogeneous large model. The adjusted credibility weight is a weight obtained by adjusting the credibility weight. This weight integrates the information reflected by historical evaluation data and feedback summaries, and is used to replace the original credibility weight in performing weighted fusion.
[0077] It is understood that in this embodiment, when it is not the first evaluation, the feedback record of the previous evaluation is obtained, and the feedback record is summarized based on the preset feedback summary agent to obtain the feedback summary. At the same time, the historical evaluation data of each heterogeneous large model is obtained, and the credibility weight of each heterogeneous large model is adjusted based on the historical evaluation data and the feedback summary to obtain the adjusted credibility weight. Then, the evaluation results of the large model are weighted and fused based on the adjusted credibility weight to obtain the integrated evaluation result. This allows the credibility weight on which the weighted fusion is based to be dynamically adjusted by comprehensively considering the information reflected by the historical evaluation data and the feedback summary.
[0078] This embodiment avoids the inability to reflect the latest credibility level of each heterogeneous large model by adjusting the credibility weight. This allows the integrated evaluation results to be gradually optimized as historical evaluation data and feedback information are continuously accumulated, improving the accuracy and adaptability of the integrated evaluation results. As a result, the evaluation capability of the multi-model integrated evaluator is continuously improved as evaluation feedback accumulates.
[0079] Step S22: The evaluation results of the output data corresponding to the same evaluator are combined to obtain the evaluation results corresponding to each evaluator.
[0080] It is understood that this embodiment integrates the evaluation results of the output data corresponding to the same evaluator to obtain the evaluation results corresponding to each evaluator. The output data evaluation results are obtained by the evaluator after evaluating each output data separately. This allows the fine-grained evaluation results of the evaluator for a single output data to be summarized into the overall evaluation result of the evaluator, providing complete evaluation data for fusion calculation based on the evaluation results corresponding to each evaluator.
[0081] In summary, this embodiment evaluates each piece of output data in the model output based on the evaluator and the corresponding evaluation rules. The evaluator includes a multi-model integration evaluator, a vertical specialization evaluator, a custom evaluator, and a logic rule evaluator. This allows each type of evaluator to evaluate each piece of output data in a way that is appropriate to its functional positioning. This achieves multi-dimensional coverage evaluation of output data from general semantics, vertical specialization, custom logic to deterministic rules, avoiding the bias and instability problems introduced by relying solely on a single model for evaluation.
[0082] Furthermore, this embodiment enables the credibility weight to be dynamically adjusted as historical evaluation data and feedback information accumulate, avoiding the situation where a fixed credibility weight cannot reflect the latest credibility level of each heterogeneous large model. This allows the integrated evaluation results to be gradually optimized as historical evaluation data and feedback information are continuously accumulated, thereby improving the accuracy of the integrated evaluation results.
[0083] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the multi-evaluator AI evaluation method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.
[0084] This application provides a multi-evaluator AI evaluation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the multi-evaluator AI evaluation method in Embodiment 1 above.
[0085] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing the multi-evaluator AI evaluation device in the embodiments of this application. The multi-evaluator AI evaluation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, tablets, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The multi-evaluator AI evaluation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0086] like Figure 6As shown, the multi-evaluator AI evaluation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the multi-evaluator AI evaluation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the multi-evaluator AI evaluation device to communicate wirelessly or wiredly with other devices to exchange data. Although a multi-evaluator AI evaluation device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.
[0087] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0088] The multi-evaluator AI evaluation device provided in this application, employing the multi-evaluator AI evaluation method described in the above embodiments, can solve the technical problem that current methods cannot accurately evaluate AI models. Compared with the prior art, the beneficial effects of the multi-evaluator AI evaluation device provided in this application are the same as those of the multi-evaluator AI evaluation method described in the above embodiments, and other technical features of this multi-evaluator AI evaluation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0089] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0090] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0091] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the multi-evaluator AI evaluation method described in the above embodiments.
[0092] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0093] The aforementioned computer-readable storage medium may be included in a multi-evaluator AI evaluation device; or it may exist independently and not be assembled into a multi-evaluator AI evaluation device.
[0094] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a multi-evaluator AI evaluation device, cause the multi-evaluator AI evaluation device to perform the aforementioned multi-evaluator AI evaluation method.
[0095] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0097] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0098] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described multi-evaluator AI evaluation method, thereby solving the technical problem that current methods cannot accurately evaluate AI models. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the multi-evaluator AI evaluation method provided in the above embodiments, and will not be repeated here.
[0099] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multi-evaluator AI evaluation method described above.
[0100] The computer program product provided in this application can solve the technical problem that current methods cannot accurately evaluate AI models. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the multi-evaluator AI evaluation method provided in the above embodiments, and will not be repeated here.
[0101] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A multi-evaluator AI evaluation method, characterized in that, The method includes: Obtain the model output, task type, and user configuration of the AI model to be evaluated, and determine the evaluator combination for this evaluation based on the task type and the user configuration; Based on the combination of evaluators, the model output is evaluated to obtain the evaluation results of multiple evaluators; The evaluation results are then integrated and calculated to obtain a standardized evaluation result. The model is evaluated based on the standardized evaluation results.
2. The method as described in claim 1, characterized in that, The evaluator combination also includes evaluation rules corresponding to each evaluator. The step of evaluating the model output based on the evaluator combination to obtain the evaluation results of multiple evaluators includes: Based on the evaluator and the corresponding evaluation rule, each output data in the model output is evaluated to obtain the output data evaluation result corresponding to each evaluator. The evaluation results of the output data corresponding to the same evaluator are combined to obtain the evaluation results corresponding to each evaluator.
3. The method as described in claim 2, characterized in that, The evaluator includes a multi-model ensemble evaluator, a vertical specialty evaluator, a custom evaluator, and a logical rule evaluator. The step of evaluating each piece of output data in the model output based on the evaluator and the corresponding evaluation rules to obtain the output data evaluation results corresponding to each evaluator includes: If the evaluator is the multi-model ensemble evaluator, then the output data is evaluated based on multiple preset heterogeneous large models to obtain an ensemble evaluation result; If the evaluator is the vertical specialization evaluator, then based on the locally deployed lightweight model and the corresponding task type evaluation settings, the output data is evaluated locally to obtain the vertical evaluation result. If the evaluator is the custom evaluator, then the output data is evaluated based on the preset custom evaluation template and the custom prompts modified by the user to obtain a custom evaluation result; If the evaluator is the logic rule evaluator, then the output data is matched according to the corresponding evaluation rules to obtain the rule evaluation result.
4. The method as described in claim 3, characterized in that, If the evaluator is the multi-model ensemble evaluator, then the step of evaluating the output data based on multiple preset heterogeneous large models to obtain the ensemble evaluation result includes: Determine whether this is the first time the output data has been evaluated; If this is the first evaluation, the output data is evaluated concurrently based on each of the heterogeneous large models to obtain multiple large model evaluation results; Based on the credibility weights corresponding to each of the heterogeneous large models, the evaluation results of the large models are weighted and fused to obtain the integrated evaluation result.
5. The method as described in claim 4, characterized in that, After determining whether this is the first time the output data has been evaluated, the method further includes: If this is not the first evaluation, then retrieve the feedback record from the previous evaluation; The feedback summary agent summarizes the feedback records based on the preset feedback summary; Obtain historical evaluation data for each of the heterogeneous large models, and adjust the credibility weights of each of the heterogeneous large models based on the historical evaluation data and the feedback summary to obtain the adjusted credibility weights. Based on the adjusted credibility weights, the evaluation results of the large model are weighted and fused to obtain the integrated evaluation results.
6. The method as described in claim 1, characterized in that, The step of determining the evaluator combination for this evaluation based on the task type and the user configuration includes: Based on the task type and the user configuration, determine the evaluator to be used in this evaluation, and determine the evaluation indicators, evaluation thresholds and evaluation logic corresponding to each evaluator; Based on the evaluation metrics, the evaluation thresholds, and the evaluation logic, the evaluation rules for each of the evaluators are determined; The evaluator combination is determined based on each of the evaluators and the corresponding evaluation rules.
7. The method as described in claim 1, characterized in that, The step of fusing and calculating the evaluation results to obtain the standardized evaluation result includes: Obtain user-defined fusion computing rules; Based on the fusion calculation rules, the evaluation results are weighted and fused to obtain the standardized evaluation results, wherein the standardized evaluation results include one or more of accuracy, consistency and robustness.
8. A multi-evaluator AI evaluation device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-evaluator AI evaluation method as described in any one of claims 1 to 7.
9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the multi-evaluator AI evaluation method as described in any one of claims 1 to 7.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the multi-evaluator AI evaluation method as described in any one of claims 1 to 7.
