AI model dynamic evaluation method and system based on multi-agent collaboration

Abstract

The invention relates to an AI model dynamic evaluation method and system based on multi-agent collaboration, and belongs to the technical field of artificial intelligence. Scheduling a group of pre-configured evaluation agents from an agent database according to the output content when the AI model to be evaluated executes the target task; based on a group of scheduled pre-configured evaluation agents, performing multiple times of independent evaluation on sampling content in output content when the AI model to be evaluated executes the target task, and generating an evaluation score of each evaluation agent corresponding to each time of sampling; aggregating evaluation scores of all evaluation agents to all sampling outputs, and determining a stability coefficient of each evaluation agent in combination with a statistical analysis method; and according to the stability coefficient and the evaluation score of each evaluation agent corresponding to each sampling, determining an effective score of each evaluation agent on the sampling content, and obtaining a dynamic evaluation result of the AI model, thereby facilitating more comprehensive and more accurate evaluation of the true capability of the AI model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for dynamic evaluation of AI models based on multi-agent collaboration. Background Technology

[0002] AI evaluation refers to the process of systematically measuring, analyzing, and judging the performance, capabilities, reliability, and safety of artificial intelligence models. As AI (especially generative AI) becomes increasingly complex, evaluation has evolved from simple accuracy calculations into a comprehensive, multi-dimensional, and multimodal challenge. However, traditional AI evaluation methods often employ single-dimensional assessments, lacking flexible evaluation systems tailored to different application scenarios, and are easily affected by the randomness of model outputs, leading to large fluctuations in evaluation results that fail to reflect the model's true capabilities. Summary of the Invention

[0003] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a dynamic evaluation method and system for AI models based on multi-agent collaboration, which solves the technical problem that the AI ​​evaluation dimension in the prior art is too single and cannot reflect the true ability of the model.

[0004] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0005] The first aspect of this invention provides a method for dynamic evaluation of AI models based on multi-agent collaboration.

[0006] The AI ​​model dynamic evaluation method based on multi-agent collaboration proposed in this embodiment includes:

[0007] Receive an evaluation task, which includes the output of the AI ​​model to be evaluated when it performs the target task;

[0008] Based on the output of the AI ​​model to be evaluated when performing the target task, a set of pre-configured evaluation agents are scheduled from the agent database; wherein each evaluation agent in the set of pre-configured evaluation agents is encapsulated with a prompt word template and an evaluation algorithm corresponding to a predetermined test performance; wherein the predetermined test performance is different for different evaluation agents;

[0009] Based on the pre-configured set of evaluation agents, the sampled content in the output of the AI ​​model to be evaluated when it performs the target task is evaluated multiple times independently, and an evaluation score is generated for each evaluation agent corresponding to each sample.

[0010] Aggregate the evaluation scores of all evaluation agents for all sampled outputs, and combine statistical analysis methods to determine the stability coefficient of each evaluation agent;

[0011] Based on the stability coefficient and the evaluation score of each evaluation agent corresponding to each sample, the effective score of each evaluation agent for the sampled content is determined, and the dynamic evaluation result of the AI ​​model is obtained based on the effective scores of each evaluation agent for the sampled content.

[0012] In some instances, the step of scheduling a set of pre-configured evaluation agents from an agent database based on the output of the AI ​​model to be evaluated when performing the target task includes:

[0013] The agent group is determined based on the output of the AI ​​model to be evaluated when it performs the target task;

[0014] Based on the agent group, retrieve all evaluation agents belonging to the agent group from the agent database, and analyze the input-output dependencies between the retrieved evaluation agents;

[0015] Based on the input-output dependencies, a dynamic evaluation execution mode is constructed, resulting in a set of evaluation agents with either parallel or sequential execution modes.

[0016] In some instances, before the set of pre-configured evaluation agents based on scheduling performs multiple independent evaluations of the sampled content in the output of the AI ​​model to be evaluated when performing the target task, the method includes:

[0017] The output of the AI ​​model to be evaluated when performing the target task is sampled quickly an initial number of times, and the evaluation result of the quick sampling is obtained based on the quick sampling of the initial number of times.

[0018] The initial standard deviation is calculated based on the evaluation results of the initial number of rapid samplings, and the total number of samplings is dynamically adjusted according to the initial standard deviation.

[0019] The output of the AI ​​model to be evaluated when performing the target task is sampled based on the total number of samplings.

[0020] In some instances, the dynamic adjustment of the total number of samples based on the preliminary standard deviation includes:

[0021] If the preliminary standard deviation is less than the first threshold, then the total number of samples is reduced;

[0022] If the initial standard deviation is greater than the second threshold, the total number of samples is increased; wherein the second threshold is greater than the first threshold.

[0023] In some instances, the aggregation of evaluation scores from all evaluation agents for all sampled outputs, combined with statistical analysis methods, determines the stability coefficient of each evaluation agent, including:

[0024] Based on the evaluation score of each evaluation agent corresponding to each sample, the coefficient of variation of the evaluation agent corresponding to multiple evaluation scores is determined;

[0025] The stability coefficient of each evaluation agent is determined using the exponential decay function and the coefficient of variation.

[0026] In some instances, determining the effective score for the sampled content by each evaluation agent based on the stability coefficient and the evaluation score of each evaluation agent corresponding to each sample includes:

[0027] Based on the stability coefficient and the effective score evaluation model, the effective score for each evaluation agent on the sampled content is determined; wherein, the effective score evaluation model is:

[0028] Where Ei is the effective score for each evaluator, α is the stability coefficient, Mi is the mean of the evaluation scores of a single evaluator, and G is the global mean of the evaluation scores of all scheduled evaluators.

[0029] In some instances, the dynamic evaluation result of the AI ​​model is obtained based on the effective scores given by each evaluation agent to the sampled content, including:

[0030] Determine the assigned weights for each evaluation agent;

[0031] Based on the effective scores and weights assigned to the sampled content by each evaluation agent, a weighted average is calculated to obtain the dynamic evaluation result of the AI ​​model.

[0032] A second aspect of this invention provides an AI model dynamic evaluation system based on multi-agent collaboration, comprising:

[0033] The task receiving unit is used to receive evaluation tasks, which include the output content of the AI ​​model to be evaluated when it performs the target task.

[0034] An agent scheduling unit is used to schedule a set of pre-configured evaluation agents from an agent database based on the output of the AI ​​model to be evaluated when it performs the target task; wherein each evaluation agent in the set of pre-configured evaluation agents is encapsulated with a prompt word template and an evaluation algorithm corresponding to a predetermined test performance; wherein the predetermined test performance is different for different evaluation agents;

[0035] The sampling evaluation unit is used to independently evaluate the sampled content in the output of the AI ​​model to be evaluated when it performs the target task, based on the set of pre-configured evaluation agents, and generate an evaluation score for each evaluation agent corresponding to each sampling.

[0036] The stability coefficient determination unit is used to aggregate the evaluation scores of all evaluation agents for all sampled outputs and combine statistical analysis methods to determine the stability coefficient of each evaluation agent.

[0037] The evaluation result determination unit is used to determine the effective score of each evaluation agent for the sampled content based on the stability coefficient and the evaluation score of each evaluation agent for each sample, and to obtain the dynamic evaluation result of the AI ​​model based on the effective scores of each evaluation agent for the sampled content.

[0038] A third aspect of the present invention provides a computer-readable storage medium storing a dynamic evaluation program for an AI model based on multi-agent collaboration. When the dynamic evaluation program for an AI model based on multi-agent collaboration is executed by a processor, it implements the dynamic evaluation method for an AI model based on multi-agent collaboration described in the first aspect.

[0039] A fourth aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the method described in the first aspect above.

[0040] This invention discloses a dynamic evaluation method for AI models based on multi-agent collaboration, comprising: receiving an evaluation task, the evaluation task including the output content of the AI ​​model to be evaluated when performing a target task; scheduling a set of pre-configured evaluation agents from an agent database according to the output content of the AI ​​model to be evaluated when performing the target task; wherein each evaluation agent in the set of pre-configured evaluation agents encapsulates a prompt word template and an evaluation algorithm corresponding to a predetermined test performance; wherein the predetermined test performance corresponding to different evaluation agents is different; performing multiple independent evaluations on sampled content in the output content of the AI ​​model to be evaluated when performing the target task based on the scheduled set of pre-configured evaluation agents, generating an evaluation score for each evaluation agent corresponding to each sample; aggregating the evaluation scores of all evaluation agents for all sampled outputs, and determining the stability coefficient of each evaluation agent by combining statistical analysis methods; determining the effective score of each evaluation agent for the sampled content based on the stability coefficient and the evaluation score of each evaluation agent corresponding to each sample, and obtaining the dynamic evaluation result of the AI ​​model based on the effective scores of each evaluation agent for the sampled content. In this application, multiple different evaluation agents are used to conduct performance tests and evaluations on the AI ​​model to be evaluated from multiple dimensions and aspects. The effective score is determined comprehensively, and the dynamic evaluation result of the AI ​​model is obtained, which is conducive to a more comprehensive and accurate evaluation of the true capabilities of the AI ​​model. Attached Figure Description

[0041] Figure 1A flowchart of a dynamic evaluation method for AI models based on multi-agent collaboration is provided for an embodiment of the present invention.

[0042] Figure 2 This invention provides a dynamic evaluation process for AI models based on multi-agent collaboration. Figure 1 ;

[0043] Figure 3 This invention provides a dynamic evaluation process for AI models based on multi-agent collaboration. Figure 2 ;

[0044] Figure 4 This invention provides a dynamic evaluation process for AI models based on multi-agent collaboration. Figure 3 ;

[0045] Figure 5 This is a schematic diagram of the structure of an AI model dynamic evaluation system based on multi-agent collaboration, provided as an embodiment of the present invention. Detailed Implementation

[0046] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] The AI ​​model dynamic evaluation method based on multi-agent collaboration proposed in this invention addresses the problem that existing AI evaluation methods have a single dimension and cannot reflect the true capabilities of the model. By using multiple different evaluation agents to perform performance testing and evaluation on the AI ​​model to be evaluated from multiple dimensions and aspects, a comprehensive and effective score is determined, resulting in a dynamic evaluation result of the AI ​​model. This facilitates a more comprehensive and accurate evaluation of the true capabilities of the AI ​​model.

[0048] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0049] Figure 1 A flowchart illustrating a dynamic evaluation method for AI models based on multi-agent collaboration, providing insights into existing technologies. (For example...) Figure 1 As shown in the embodiments of the present invention, the dynamic evaluation method for AI models based on multi-agent collaboration includes:

[0050] Step 100: Receive the evaluation task, which includes the output of the AI ​​model to be evaluated when it performs the target task;

[0051] Step 110: Based on the output of the AI ​​model to be evaluated when performing the target task, schedule a set of pre-configured evaluation agents from the agent database; wherein each evaluation agent in the set of pre-configured evaluation agents is encapsulated with a prompt word template and evaluation algorithm corresponding to a predetermined test performance; wherein the predetermined test performance is different for different evaluation agents;

[0052] Step 120: Based on the scheduling of the set of pre-configured evaluation agents, the sampled content in the output of the AI ​​model to be evaluated when it performs the target task is evaluated multiple times independently, and an evaluation score corresponding to each sample is generated for each evaluation agent.

[0053] Step 130: Aggregate the evaluation scores of all evaluation agents for all sampled outputs, and determine the stability coefficient of each evaluation agent by combining statistical analysis methods;

[0054] Step 140: Based on the stability coefficient and the evaluation score of each evaluation agent corresponding to each sample, determine the effective score of each evaluation agent for the sampled content, and obtain the dynamic evaluation result of the AI ​​model based on the effective scores of each evaluation agent for the sampled content.

[0055] In this exemplary embodiment, the predetermined test performance may include content integrity testing, numerical consistency testing, logical coherence testing, etc. The prompt word template provides prompt words that embed domain knowledge (such as validation rules, cross-reference formulas) for specific evaluation dimensions (such as integrity and consistency). For example, it may embed validation rules, formulas, algorithms, etc. Evaluation algorithms may include statistical analysis-based stability quantification algorithms for calculating confidence intervals, coefficients of variation, etc.; rule- and logic-based domain knowledge validation algorithms, such as cross-reference formulas and regular expressions; semantic understanding-based similarity calculation algorithms; machine learning-based specialized classification algorithms for toxicity detection, fact-finding error identification, etc.; and advanced evaluation algorithms based on large model introspection, etc.

[0056] In this application, multiple different evaluation agents are used to conduct performance tests and evaluations on the AI ​​model to be evaluated from multiple dimensions and aspects. The effective score is determined comprehensively, and the dynamic evaluation result of the AI ​​model is obtained, which is conducive to a more comprehensive and accurate evaluation of the true capabilities of the AI ​​model.

[0057] In some instances, the step of scheduling a set of pre-configured evaluation agents from an agent database based on the output of the AI ​​model to be evaluated when performing the target task includes:

[0058] The agent group is determined based on the output of the AI ​​model to be evaluated when it performs the target task;

[0059] Based on the agent group, retrieve all evaluation agents belonging to the agent group from the agent database, and analyze the input-output dependencies between the retrieved evaluation agents;

[0060] Based on the input-output dependencies, a dynamic evaluation execution mode is constructed, resulting in a set of evaluation agents with either parallel or sequential execution modes.

[0061] In this exemplary embodiment, when analyzing the input-output dependencies between agents, the system analyzes the configuration of these agents and determines whether there is a dependency on their execution order (e.g., the input of one agent requires the output of another agent). The execution order of the agents is automatically established based on the group relationship.

[0062] Based on the dependency analysis results, the system determines whether these agents execute in parallel or in a specific order, forming an evaluation pipeline for this task. The system dynamically generates the optimal evaluation pipeline by analyzing the input-output dependencies between agents (e.g., numerical consistency evaluation must be performed after content completeness is achieved). The output of a preceding agent is automatically injected into the prompt word template placeholders of subsequent agents (e.g., {{previous result}}), achieving cross-agent context inheritance.

[0063] Scenario Example: Multi-dimensional verification of AI-generated company financial reports is crucial. Traditionally, this involves manually comparing results one by one. Furthermore, AI generation inherently involves randomness, meaning results may vary each time, posing a challenge for testing. Our system, however, can simultaneously verify content completeness, numerical consistency, and logical coherence.

[0064] The combination process is as follows: Different types of agents can be added through the agent configuration interface. A new agent named "Content Integrity Assessment" is added, using any pre-configured large model, with a temperature of 0.3, a maximum token count (MaxToken) of 8192, and categorized as an evaluator. A prompt message including disclosure list verification rules is also written. Simultaneously, a new agent named "Numerical Consistency Assessment" is added, using any pre-configured large model, with a temperature of 0.3, a MaxToken of 8192, and categorized as an evaluator. A prompt message embedding financial statement reconciliation formulas is also written. Finally, a new agent named "Logical Coherence" is added, using any pre-configured large model, with a temperature of 0.3, a MaxToken of 8192, and categorized as an evaluator. A prompt message defining the growth rate tolerance threshold algorithm is also written. When a user submits the AI ​​agent system for evaluation, the user can select an evaluator: The system retrieves related agents by group = evaluator. If no strict sequential dependency is detected among the three agents, a parallel evaluation mode is initiated, with each agent assigned an independent thread and the original financial statement data injected synchronously.

[0065] In some instances, before the set of pre-configured evaluation agents based on scheduling performs multiple independent evaluations of the sampled content in the output of the AI ​​model to be evaluated when performing the target task, the method includes:

[0066] The output of the AI ​​model to be evaluated when performing the target task is sampled quickly an initial number of times, and the evaluation result of the quick sampling is obtained based on the quick sampling of the initial number of times.

[0067] The initial standard deviation is calculated based on the evaluation results of the initial number of rapid samplings, and the total number of samplings is dynamically adjusted according to the initial standard deviation.

[0068] The output of the AI ​​model to be evaluated when performing the target task is sampled based on the total number of samplings.

[0069] In this exemplary embodiment, the step of dynamically adjusting the total number of samples based on the preliminary standard deviation includes:

[0070] If the preliminary standard deviation is less than the first threshold, then the total number of samples is reduced;

[0071] If the initial standard deviation is greater than the second threshold, the total number of samples is increased; wherein the second threshold is greater than the first threshold.

[0072] In this exemplary embodiment, 10 initial rapid samplings are performed first to calculate the initial standard deviation σ', and small-batch sampling is used to initially detect the output stability. If σ' < 0.1 (high stability), the total number of samplings is automatically reduced to 30. If σ' > 0.2 (low stability), the number of samplings is increased to 100 and an alarm is triggered. At the same time, sequential analysis technology is used to terminate the process early when the standard deviation of the scores for 20 consecutive samplings is < 0.03.

[0073] In some instances, the aggregation of evaluation scores from all evaluation agents for all sampled outputs, combined with statistical analysis methods, determines the stability coefficient of each evaluation agent, including:

[0074] Based on the evaluation score of each evaluation agent corresponding to each sample, the coefficient of variation of the evaluation agent corresponding to multiple evaluation scores is determined;

[0075] The stability coefficient of each evaluation agent is determined using the exponential decay function and the coefficient of variation.

[0076] In this exemplary embodiment, the coefficient of variation (CV) is the ratio of the standard deviation to the mean of the evaluation score for each evaluation agent corresponding to each sample. It measures relative volatility and is a core indicator of stability.

[0077] The stability coefficient α is calculated using the exponential decay function, including:

[0078] The stability coefficient is calculated using the formula α=e^(-k·CV); where the smaller the CV value (the more stable), the closer α is to 1; the larger the CV value (the less stable), the closer α is to 0, and k is a coefficient.

[0079] Adjusting effective scores based on stability coefficients allows for the smoothing of raw scores, thus determining the effective scores for each evaluation agent on the sampled content.

[0080] In some instances, determining the effective score for the sampled content by each evaluation agent based on the stability coefficient and the evaluation score of each evaluation agent corresponding to each sample includes:

[0081] Based on the stability coefficient and the effective score evaluation model, the effective score for each evaluation agent on the sampled content is determined; wherein, the effective score evaluation model is:

[0082] ;

[0083] in, To evaluate intelligent agents Valid score, To evaluate intelligent agents The stability coefficient; To evaluate intelligent agents The mean of the assessment scores; This is the global mean of the evaluation scores of all scheduled evaluation agents.

[0084] In this exemplary embodiment, the stable agent ( ≈1): Effective ratings are close to their own average scores, and their judgments are completely trusted;

[0085] Unstable intelligent agents ( ≈0): Effective scores are "pulled" toward the global average, suppressing unreliable opinions.

[0086] In some instances, the dynamic evaluation result of the AI ​​model is obtained based on the effective scores given by each evaluation agent to the sampled content, including:

[0087] Determine the assigned weights for each evaluation agent;

[0088] Based on the effective scores and weights assigned to the sampled content by each evaluation agent, a weighted average is calculated to obtain the dynamic evaluation result of the AI ​​model.

[0089] In this exemplary embodiment, the effective scores of each agent are weighted and averaged using their initial weights (representing the business importance of the evaluation dimension) to obtain the final comprehensive score and generate a complete report including details such as stability analysis.

[0090] In this exemplary embodiment, the comprehensive score and detailed evaluation report are transformed into easily understandable visual formats such as charts and scorecards, and the visualized report is ultimately presented to the user, completing the entire evaluation process. As shown in Table 1, the data content related to the valid score is displayed.

[0091] Table 1 Valid Scoring Table

[0092] intelligent agent Equal distribution within itself (Mᵢ) Stability coefficient (αᵢ) Global equal distribution (G) Valid score (Eᵢ) calculation Valid rating (Eᵢ) Content completeness 0.85 0.84 (stable) 0.817 0.84×0.85+0.16×0.817 0.845 Numerical consistency 0.9 0.49 (Medium) 0.817 0.49×0.90+0.51×0.817 0.858 Logical coherence 0.7 0.22 (Unstable) 0.817 0.22×0.70+0.78×0.817 0.791

[0093] Figure 2 This invention provides a dynamic evaluation process for AI models based on multi-agent collaboration. Figure 1 .like Figure 2 As shown, the dynamic evaluation process of an AI model based on multi-agent collaboration includes:

[0094] Step 20: The administrator logs into the system;

[0095] Step 21: Enter the agent configuration interface;

[0096] Step 22: Add / edit the evaluation agent;

[0097] Step 23: Fill in the agent parameters;

[0098] Step 24: Create a dedicated prompt word template;

[0099] Step 25: Set categories and groups;

[0100] Step 26: Save the configuration to the agent database;

[0101] Step 27: The user submits the evaluation task and selects the group;

[0102] Step 28, Task Scheduling Center;

[0103] Step 29: Retrieve the list of associated agents from the database by group;

[0104] Step 30: Analyze the input-output dependencies between agents;

[0105] Step 31: Dynamically generate the evaluation pipeline;

[0106] Step 32, End.

[0107] The administrator login system is used to authorize users to enter the management backend and access the unified intelligent agent management page.

[0108] The AI ​​agent configuration page provides unified management and configuration of AI agents, supporting multi-dimensional management of model parameters, grouping, classification, and other aspects, helping enterprises upgrade their intelligence.

[0109] Create or modify an agent instance. The backend creates a database table with fields including (name, code, model, temperature, MaxToken, category, group, and prompt word).

[0110] The front-end allows users to input agent parameters, such as configuring core agent parameters including name, code, selected base model, set temperature, and maximum token count. The category field is used to count the number of agents of different types, and the group field is used to call agents by group, facilitating flexible combinations of different agents.

[0111] Among them, prompt words for constructing evaluation requests are used to write prompt words that embed domain knowledge (such as verification rules and cross-reference formulas) for specific evaluation dimensions (such as completeness and consistency).

[0112] Set categories and groups, for example, set "categories" for intelligent agents for statistics, and "groups" for flexible combination when calling tasks, and persist all configuration information to a database table.

[0113] At the same time, select the content to be evaluated (such as AI-generated financial reports) and the intelligent agent group on the front-end interface.

[0114] The task scheduling center is the core scheduling module that receives user task requests. Based on the group selected by the user, the scheduling center retrieves all relevant agents from the database. Each agent is assigned a "group" field in the database, and the system uses group identifiers to enable clustered agent invocation. For example, all agents marked as evaluators will be automatically grouped into the same execution pipeline.

[0115] Figure 3 This invention provides a dynamic evaluation process for AI models based on multi-agent collaboration. Figure 2 .like Figure 3 As shown, the dynamic evaluation process of an AI model based on multi-agent collaboration includes:

[0116] Step 33, Task Scheduling Center;

[0117] Step 34: Invoke the dynamic sampling mechanism;

[0118] Step 35: Perform initial fast sampling;

[0119] Step 36: Calculate the preliminary standard deviation;

[0120] Step 37: Reduce the total number of samples;

[0121] Step 38: Increase the total number of samples;

[0122] Step 39: Keep the default number of attempts;

[0123] Step 40: Continue to complete the remaining sampling;

[0124] Step 41: Assign an evaluation thread for each sample;

[0125] Step 42: Parallel evaluation by a cluster of dedicated evaluation agents;

[0126] Step 43: Content integrity assessment;

[0127] Step 44: Numerical consistency assessment;

[0128] Step 45: Logical coherence assessment;

[0129] Step 46: Result Aggregation Engine.

[0130] Among them, the stability-based scoring system can adjust the effective scores using stability coefficients, intelligently suppress the impact of high volatility assessment results, and ensure the robustness of the scoring results.

[0131] The architecture separates the front-end and back-end. The front-end provides a visual configuration interface to support the configuration of evaluation tasks and the display of results. The back-end adopts a microservice architecture, in which each evaluation agent runs as an independent service and the agents collaborate through message queues.

[0132] Figure 4 This invention provides a dynamic evaluation process for AI models based on multi-agent collaboration. Figure 3 .like Figure 4 As shown, the dynamic evaluation process of an AI model based on multi-agent collaboration includes:

[0133] Step 46, Result Aggregation Engine;

[0134] Step 47: Calculate the statistical indicators for each agent;

[0135] Step 48: Calculate the mean, standard deviation, and CV.

[0136] Step 49: Calculate the stability coefficient using the exponential decay function;

[0137] Step 50: Adjust the effective score based on the stability coefficient;

[0138] Step 51: Determine the valid scores;

[0139] Step 52: Generate a comprehensive score and evaluation report;

[0140] Step 53: Visualize the report;

[0141] Step 54: Front-end interface display.

[0142] This invention provides a dynamic evaluation system for AI models based on multi-agent collaboration. Figure 5 This is a schematic diagram of an AI model dynamic evaluation system based on multi-agent collaboration, provided as an embodiment of the present invention. Figure 5 As shown, it includes:

[0143] The task receiving unit 500 is used to receive an evaluation task, which includes the output content of the AI ​​model to be evaluated when it performs the target task.

[0144] The agent scheduling unit 510 is used to schedule a set of pre-configured evaluation agents from the agent database according to the output content of the AI ​​model to be evaluated when it performs the target task; wherein each evaluation agent in the set of pre-configured evaluation agents is encapsulated with a prompt word template and an evaluation algorithm corresponding to a predetermined test performance; wherein the predetermined test performance is different for different evaluation agents.

[0145] The sampling evaluation unit 520 is used to independently evaluate the sampled content in the output content of the AI ​​model to be evaluated when it performs the target task, based on the set of pre-configured evaluation agents, and generate an evaluation score for each evaluation agent corresponding to each sampling.

[0146] The stability coefficient determination unit 530 is used to aggregate the evaluation scores of all evaluation agents for all sampled outputs and combine statistical analysis methods to determine the stability coefficient of each evaluation agent.

[0147] The evaluation result determination unit 540 is used to determine the effective score of each evaluation agent for the sampled content based on the stability coefficient and the evaluation score of each evaluation agent for each sample, and to obtain the dynamic evaluation result of the AI ​​model based on the effective scores of each evaluation agent for the sampled content.

[0148] In this exemplary embodiment, the predetermined test performance may include content integrity testing, numerical consistency testing, logical coherence testing, etc. The prompt word template provides prompt words that embed domain knowledge (such as validation rules, cross-reference formulas) for specific evaluation dimensions (such as integrity and consistency). For example, it may embed validation rules, formulas, algorithms, etc. Evaluation algorithms may include statistical analysis-based stability quantification algorithms for calculating confidence intervals, coefficients of variation, etc.; rule- and logic-based domain knowledge validation algorithms, such as cross-reference formulas and regular expressions; semantic understanding-based similarity calculation algorithms; machine learning-based specialized classification algorithms for toxicity detection, fact-finding error identification, etc.; and advanced evaluation algorithms based on large model introspection, etc.

[0149] In this application, multiple different evaluation agents are used to conduct performance tests and evaluations on the AI ​​model to be evaluated from multiple dimensions and aspects. The effective score is determined comprehensively, and the dynamic evaluation result of the AI ​​model is obtained, which is conducive to a more comprehensive and accurate evaluation of the true capabilities of the AI ​​model.

[0150] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0151] This invention provides a computer-readable storage medium, characterized in that it stores a dynamic evaluation program for AI models based on multi-agent collaboration. When the dynamic evaluation program for AI models based on multi-agent collaboration is executed by a processor, it implements the dynamic evaluation method for AI models based on multi-agent collaboration described in the above embodiments.

[0152] This invention provides an electronic device, characterized in that it includes a memory, a processor, and a dynamic evaluation program for AI models based on multi-agent collaboration stored in the memory and executable on the processor. When the processor executes the dynamic evaluation program for AI models based on multi-agent collaboration, it implements the dynamic evaluation method for AI models based on multi-agent collaboration described in the above embodiments.

[0153] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0154] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0155] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that they are in indirect contact through an intermediate medium. Furthermore, "above," "over," or "on top" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," or "beneath" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0156] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0157] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A dynamic evaluation method for AI models based on multi-agent collaboration, characterized in that, include: Receive an evaluation task, which includes the output of the AI ​​model to be evaluated when it performs the target task; Based on the output of the AI ​​model to be evaluated when performing the target task, a set of pre-configured evaluation agents are scheduled from the agent database; wherein each evaluation agent in the set of pre-configured evaluation agents is encapsulated with a prompt word template and an evaluation algorithm corresponding to a predetermined test performance; wherein the predetermined test performance is different for different evaluation agents; Based on the pre-configured set of evaluation agents, the sampled content in the output of the AI ​​model to be evaluated when it performs the target task is evaluated multiple times independently, and an evaluation score is generated for each evaluation agent corresponding to each sample. Aggregate the evaluation scores of all evaluation agents for all sampled outputs, and combine statistical analysis methods to determine the stability coefficient of each evaluation agent; Based on the stability coefficient and the evaluation score of each evaluation agent corresponding to each sample, the effective score of each evaluation agent for the sampled content is determined, and the dynamic evaluation result of the AI ​​model is obtained based on the effective scores of each evaluation agent for the sampled content.

2. The dynamic evaluation method for AI models based on multi-agent collaboration according to claim 1, characterized in that, The step of scheduling a set of pre-configured evaluation agents from the agent database based on the output of the AI ​​model to be evaluated when performing the target task includes: The agent group is determined based on the output of the AI ​​model to be evaluated when it performs the target task; Based on the agent group, retrieve all evaluation agents belonging to the agent group from the agent database, and analyze the input-output dependencies between the retrieved evaluation agents; Based on the input-output dependencies, a dynamic evaluation execution mode is constructed, resulting in a set of evaluation agents with either parallel or sequential execution modes.

3. The dynamic evaluation method for AI models based on multi-agent collaboration according to claim 1, characterized in that, Before the set of pre-configured evaluation agents based on scheduling independently evaluates multiple samples of the output content of the AI ​​model to be evaluated when it performs the target task, the method includes: The output of the AI ​​model to be evaluated when performing the target task is sampled quickly an initial number of times, and the evaluation result of the quick sampling is obtained based on the quick sampling of the initial number of times. The initial standard deviation is calculated based on the evaluation results of the initial number of rapid samplings, and the total number of samplings is dynamically adjusted according to the initial standard deviation. The output of the AI ​​model to be evaluated when performing the target task is sampled based on the total number of samplings.

4. The dynamic evaluation method for AI models based on multi-agent collaboration according to claim 3, characterized in that, The step of dynamically adjusting the total number of samples based on the preliminary standard deviation includes: If the preliminary standard deviation is less than the first threshold, then the total number of samples is reduced; If the initial standard deviation is greater than the second threshold, the total number of samples is increased; wherein the second threshold is greater than the first threshold.

5. The dynamic evaluation method for AI models based on multi-agent collaboration according to claim 1, characterized in that, The aggregation of evaluation scores from all evaluation agents for all sampled outputs, combined with statistical analysis methods, determines the stability coefficient of each evaluation agent, including: Based on the evaluation score of each evaluation agent corresponding to each sample, the coefficient of variation of the evaluation agent corresponding to multiple evaluation scores is determined; The stability coefficient of each evaluation agent is determined using the exponential decay function and the coefficient of variation.

6. The dynamic evaluation method for AI models based on multi-agent collaboration according to claim 1, characterized in that, The step of determining the effective score of each evaluation agent for the sampled content based on the stability coefficient and the evaluation score of each evaluation agent corresponding to each sample includes: Based on the stability coefficient and the effective score evaluation model, the effective score for each evaluation agent on the sampled content is determined; wherein, the effective score evaluation model is: Where Ei is the effective score for each evaluator, α is the stability coefficient, Mi is the mean of the evaluation scores of a single evaluator, and G is the global mean of the evaluation scores of all scheduled evaluators.

7. The dynamic evaluation method for AI models based on multi-agent collaboration according to claim 1, characterized in that, The dynamic evaluation result of the AI ​​model, obtained based on the effective scores given by each evaluation agent to the sampled content, includes: Determine the assigned weights for each evaluation agent; Based on the effective scores and weights assigned to the sampled content by each evaluation agent, a weighted average is calculated to obtain the dynamic evaluation result of the AI ​​model.

8. A dynamic evaluation system for AI models based on multi-agent collaboration, characterized in that, include: The task receiving unit is used to receive evaluation tasks, which include the output content of the AI ​​model to be evaluated when it performs the target task. An agent scheduling unit is used to schedule a set of pre-configured evaluation agents from an agent database based on the output of the AI ​​model to be evaluated when it performs the target task; wherein each evaluation agent in the set of pre-configured evaluation agents is encapsulated with a prompt word template and an evaluation algorithm corresponding to a predetermined test performance; wherein the predetermined test performance is different for different evaluation agents; The sampling evaluation unit is used to independently evaluate the sampled content in the output of the AI ​​model to be evaluated when it performs the target task, based on the set of pre-configured evaluation agents, and generate an evaluation score for each evaluation agent corresponding to each sampling. The stability coefficient determination unit is used to aggregate the evaluation scores of all evaluation agents for all sampled outputs and combine statistical analysis methods to determine the stability coefficient of each evaluation agent. The evaluation result determination unit is used to determine the effective score of each evaluation agent for the sampled content based on the stability coefficient and the evaluation score of each evaluation agent for each sample, and to obtain the dynamic evaluation result of the AI ​​model based on the effective scores of each evaluation agent for the sampled content.

9. A computer-readable storage medium, characterized in that, It stores a dynamic evaluation program for AI models based on multi-agent collaboration. When the processor executes the dynamic evaluation program for AI models based on multi-agent collaboration, it implements the dynamic evaluation method for AI models based on multi-agent collaboration as described in any one of claims 1-7.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images