Performance evaluation method suitable for large model agent

Abstract

The invention provides a performance evaluation method suitable for a large-model agent, and the method comprises the steps: obtaining the sample data size of a to-be-evaluated agent, and obtaining the task output type of the to-be-evaluated agent; when the task output type of the to-be-evaluated agent is a continuous output result, obtaining a comparison evaluation result of the to-be-evaluated agent through a pairing t inspection mode; when the task output type of the to-be-evaluated agent is a dichotomous output result, obtaining a comparison evaluation result of the to-be-evaluated agent through a pairing chi-square test mode; wherein the comparison evaluation result comprises a comparison experiment evaluation result and an ablation experiment evaluation result. According to the method, the performance difference between the large-model intelligent agent and the original LLM is quantified through a contrast experiment, the contribution of each module to the overall performance is analyzed through an ablation experiment, and data support and improvement basis are provided for further optimization of the intelligent agent. The performance of the intelligent agent is systematically and comprehensively evaluated, and the accuracy and repeatability of performance evaluation are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of authenticity detection of network information, and in particular to a performance evaluation method suitable for large-model intelligent agents. Background Art

[0002] Since the release of GPT-3.5, the Large Language Model (LLM) has received widespread attention for its excellent natural language processing capabilities. With the opening of LLM programmable interfaces, researchers not only use it for text generation, but also explore its potential as an intelligent tool in various applications. Agents based on LLM have gradually become a research hotspot. Compared with traditional intelligent neural networks, Agents based on LLM, with their powerful natural language processing capabilities and rich prior knowledge, do not need to rely on large data sets and high-performance computing resources for model training. By designing reasonable prompt words and combining modules such as "action", "plan", and "memory", they can efficiently complete natural language-related tasks in specific application scenarios, especially in the field of authenticity detection of network information, showing great potential.

[0003] However, how to systematically and comprehensively evaluate its performance remains a challenge. This not only involves how to quantify the performance of the agent in a specific task, but also requires ensuring the accuracy and repeatability of the performance evaluation. Summary of the invention

[0004] The present invention provides a performance evaluation method suitable for a large model intelligent agent, which is used to achieve a systematic and comprehensive evaluation of the performance of the large model intelligent agent and ensure the accuracy and repeatability of the performance evaluation.

[0005] The present invention provides a performance evaluation method applicable to a large model intelligent agent, comprising: Obtaining the sample data volume of the agent to be evaluated and the task output type of the agent to be evaluated; When the task output type of the subject to be evaluated is a continuous output result, a comparative evaluation result of the subject to be evaluated is obtained by a paired t-test method; When the task output type of the subject to be evaluated is a binary classification output result, a comparative evaluation result of the subject to be evaluated is obtained by a paired chi-square test; The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

[0006] The performance evaluation method applicable to a large model agent provided by the present invention also includes: When the comparative evaluation results of the intelligent agent to be evaluated show significant performance differences, according to the data type of the sample data volume, either the ROC curve or the PR curve is selected to evaluate the intelligent agent to be evaluated to obtain the comprehensive performance evaluation result of the intelligent agent to be evaluated.

[0007] According to the performance evaluation method for a large model agent provided by the present invention, according to the data type of the sample data volume, any one of the ROC curves or the PR curve is selected to evaluate the agent to be evaluated, and a comprehensive performance evaluation result of the agent to be evaluated is obtained, including: When the proportion of positive sample data in the sample data volume is greater than a preset proportion threshold, a PR curve is selected to evaluate the subject to be evaluated to obtain a comprehensive performance evaluation result of the subject to be evaluated.

[0008] According to the performance evaluation method for a large model agent provided by the present invention, according to the data type of the sample data volume, any one of the ROC curves or the PR curve is selected to evaluate the agent to be evaluated, and a comprehensive performance evaluation result of the agent to be evaluated is obtained, including: When the proportion of positive sample data in the sample data volume is less than or equal to a preset proportion threshold, the ROC curve is selected to evaluate the intelligent agent to be evaluated to obtain a comprehensive performance evaluation result of the intelligent agent to be evaluated.

[0009] The performance evaluation method applicable to a large model agent provided by the present invention also includes: When any evaluation result of the comparative experiment evaluation result or the ablation experiment evaluation result shows a significant performance difference, it is determined that the comparative evaluation result of the to-be-evaluated intelligent agent shows a significant performance difference.

[0010] According to any of the above performance evaluation methods applicable to large-model intelligent agents provided by the present invention, obtaining the sample data volume of the intelligent agent to be evaluated includes: The amount of sample data of the agent to be evaluated is determined based on the standard effect size, significance level and statistical power.

[0011] The present invention also provides a performance evaluation device suitable for a large model intelligent agent, comprising: A model data acquisition module is used to obtain the sample data volume of the intelligent agent to be evaluated and the task output type of the intelligent agent to be evaluated; A first comparative evaluation module, used for obtaining comparative evaluation results of the subject to be evaluated by a paired t-test when the task output type of the subject to be evaluated is a continuous output result; A second comparative evaluation module is used to obtain a comparative evaluation result of the intelligent agent to be evaluated by a paired chi-square test when the task output type of the intelligent agent to be evaluated is a binary classification output result; The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the performance evaluation method applicable to a large model intelligent agent as described in any one of the above-mentioned methods is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the performance evaluation method applicable to a large model intelligent agent as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the performance evaluation methods described above that are applicable to large model intelligent agents.

[0015] The performance evaluation method, device and electronic device applicable to a large model intelligent agent provided by the present invention introduce a paired t-test method to quantify the performance difference between the intelligent agent to be evaluated and the standard large language model by outputting the credibility results of continuous values; for the binary classification task scenario, a paired chi-square test method is used to analyze the classification results of the intelligent agent to be evaluated and the standard large language model. In particular, in the task of detecting the authenticity of network information, the performance of the intelligent agent to be evaluated is comprehensively evaluated through scientific method design and statistical theory. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is one of the flow charts of the performance evaluation method applicable to a large model intelligent agent provided by the present invention.

[0018] Figure 2 This is the second flow chart of the performance evaluation method applicable to large model intelligent agents provided by the present invention.

[0019] Figure 3It is a structural schematic diagram of a performance evaluation device suitable for a large model intelligent agent provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the descriptions used in this way can be interchanged where appropriate, so that the embodiments can be implemented in a sequence other than that illustrated or described in the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps that appear in the present application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed in multiple circuit units, and some or all of the units may be selected according to actual needs to achieve the purpose of the present application.

[0023] Combine the following Figure 1-Figure 4 The specific contents of the present invention are described.

[0024] Figure 1It is a flow chart of a performance evaluation method applicable to a large model intelligent agent provided by the present invention, which is applied in a server and includes steps S101-S104.

[0025] Step S101, obtaining the sample data volume of the agent to be evaluated and obtaining the task output type of the agent to be evaluated.

[0026] In an embodiment of the present invention, the intelligent agent to be evaluated is constructed based on LLM, and the task output types of the intelligent agent to be evaluated include two types of task results: continuous output and binary classification output.

[0027] Step S102, when the task output type of the subject to be evaluated is a continuous output result, a comparative evaluation result of the subject to be evaluated is obtained by a paired t-test method.

[0028] In the embodiments of the present invention, the continuous output result is usually expressed as a numerical value within a range, such as a predicted value in a regression task. The paired t-test is suitable for comparing whether there is a significant difference between the means of two sets of continuous data (such as the outputs of two different models or the same model under different conditions).

[0029] Step S103, when the task output type of the subject to be evaluated is a binary classification output result, a comparative evaluation result of the subject to be evaluated is obtained by a paired chi-square test.

[0030] In an embodiment of the present invention, the binary classification output result is usually represented as one of two categories (such as 0 and 1, or positive and negative). The paired chi-square test is suitable for comparing whether there is a significant difference in classification accuracy or error rate between two groups of binary classification data.

[0031] The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

[0032] In a possible implementation, the performance evaluation method applicable to a large model agent also includes: Step S104: When any evaluation result of the comparative experiment evaluation result or the ablation experiment evaluation result shows a significant performance difference, it is determined that the comparative evaluation result of the subject to be evaluated shows a significant performance difference.

[0033] In one possible implementation, step S104 includes steps S104A and S104B.

[0034] Step S104A, obtaining comparative experiment evaluation results through comparative experiments to determine the comparative evaluation result performance of the subject to be evaluated.

[0035] Among them, the comparative evaluation results of the intelligent agent to be evaluated are obtained through the paired t-test method.

[0036] Specifically, for the results of the continuous distribution on [0, 1], we can calculate the following statistics: Mean of difference: Standard deviation of the difference: Degrees of Freedom: Statistics: Where n is the current sample number, represents the predicted value of the i-th data in the first experiment, Represents the predicted value of the i-th data in the second experiment.

[0037] In the embodiment of the present invention, the two experiments include a test experiment for the agent to be evaluated and a test experiment for a standard large language model.

[0038] According to the t value and degree of freedom, the corresponding probability p can be obtained by looking up the t distribution table. If p<α, α is the significance level, it means that there is a significant performance difference between the two experimental results. That is, it is determined that the comparative evaluation results of the intelligent agent to be evaluated show a significant performance difference.

[0039] Among them, the comparative evaluation results of the intelligent agent to be evaluated are obtained through the paired chi-square test method.

[0040] The details are as follows: It is applicable to binary classification tasks. By constructing the following confusion matrix, the chi-square statistic can be calculated.

[0041]

[0042] in,

[0043] TP: means that the first experiment is true, and the second experiment is also true. is the corresponding sample number; FN: means the first experiment is true, but the second experiment is false. is the corresponding sample number; FP: means the first experiment is false, but the second experiment is true. is the corresponding sample number; TN: means the first experiment is false, and the second experiment is also false. is the corresponding sample number; According to the confusion matrix, we can calculate: Chi-square statistic calculation formula:

[0044] In the embodiment of the present invention, for any parameter among TP, TN, FP, and FN, in the comparative experiment, each time it appears, its corresponding value is increased by 1. This is until all sample sizes in the sample size n are traversed. For example, each time TP appears, the corresponding value is increased by 1. The value of is increased by 1.

[0045] Then, according to the chi-square statistic and the degree of freedom (equal to 1), the chi-square distribution table is searched to find the corresponding critical value. If the chi-square statistic is greater than or equal to the critical value, it means that there is a significant performance difference between the two groups of experimental results in the comparative experiment. That is, it is determined that the comparative evaluation results of the agent to be evaluated show a significant performance difference.

[0046] Step S104B, obtaining comparative experimental evaluation results through ablation experiments to determine the comparative evaluation result performance of the subject to be evaluated.

[0047] In the embodiment of the present invention, in step S104B and step S104A, the model types of the two groups of experiments are different.

[0048] In step S104A, the model types for the two groups of experiments are: the performance difference of the to-be-evaluated intelligent agent and other models with different to-be-evaluated intelligent agent configurations under the same task.

[0049] In step S104B, the contribution of each module (such as memory module, planning module, etc.) in the agent to be evaluated to the overall performance is analyzed.

[0050] In step S104B, the following steps are specifically included: Full Model Evaluation: We first evaluate the performance of the complete agent with all modules.

[0051] Module Removal: Gradually remove modules from the agent to be evaluated and re-evaluate the performance.

[0052] Performance comparison: Paired t-test or chi-square test was used to analyze the performance difference before and after module removal.

[0053] For example, if the intelligent agent to be evaluated includes module A, module B, and module C.

[0054] In step S104B, two groups of experiments are conducted using the following model types: a model with module A removed from the agent to be evaluated, and a complete agent to be evaluated.

[0055] In step S104B, the comparative evaluation results of the subject to be evaluated are obtained by paired t-test, or the comparative evaluation results of the subject to be evaluated are obtained by paired t-test. Please refer to the content of step S104A, and the present invention will not be repeated in detail.

[0056] In a possible implementation, Figure 2 As shown in Figure 2, performance evaluation methods applicable to large model agents also include: Step S105: When the comparative evaluation results of the subject to be evaluated show significant performance differences, select either the ROC curve or the PR curve to evaluate the subject to be evaluated according to the data type of the sample data volume to obtain the comprehensive performance evaluation results of the subject to be evaluated.

[0057] In the embodiments of the present invention, when the comparative evaluation results have shown significant performance differences, further use of the ROC curve or the PR curve can provide a deeper understanding of the performance characteristics of the model. These two curves provide an intuitive representation of the model performance in different scenarios.

[0058] The ROC curve (Receiver Operating Characteristic Curve) is insensitive to the ratio of positive and negative samples. It plots the relationship between the true positive rate (TPR) and the false positive rate (FPR). Therefore, when the ratio of positive and negative samples is unbalanced, the ROC curve can still provide a stable model performance evaluation.

[0059] The PR curve (precision-recall curve) focuses more on the prediction performance of positive samples. It plots the relationship between precision and recall. When the ratio of positive and negative samples is unbalanced, especially when there are few positive samples, the PR curve can more accurately reflect the model's ability to recognize positive samples.

[0060] In a possible implementation, in step S105, according to the data type of the sample data volume, either the ROC curve or the PR curve is selected to evaluate the subject to be evaluated, and a comprehensive performance evaluation result of the subject to be evaluated is obtained, including: Step S105A, when the proportion of positive sample data in the sample data volume is greater than the preset proportion threshold, select the PR curve to evaluate the intelligent agent to be evaluated to obtain the comprehensive performance evaluation result of the intelligent agent to be evaluated.

[0061] Specifically, after step S104, when the proportion of positive sample data in the sample data volume is greater than the preset proportion threshold, it indicates that the data distribution in the sample volume is unbalanced (for example, there is less false information and more real information). The accuracy and recall rate of the classifier under different thresholds are compared by drawing a PR curve, and the optimal threshold is determined in combination with the F1 score.

[0062] In a possible implementation, in step S105, according to the data type of the sample data volume, either the ROC curve or the PR curve is selected to evaluate the subject to be evaluated, and a comprehensive performance evaluation result of the subject to be evaluated is obtained, including: Step S105B, when the proportion of positive sample data in the sample data volume is less than or equal to the preset proportion threshold, select the ROC curve to evaluate the intelligent agent to be evaluated to obtain the comprehensive performance evaluation result of the intelligent agent to be evaluated.

[0063] Specifically, after step S104, when the proportion of positive sample data in the sample data volume is less than or equal to the preset proportion threshold, it means that the data is relatively balanced or requires global performance evaluation, and the overall performance of the classifier is evaluated by drawing the ROC curve and calculating the area under the curve (AUC).

[0064] In a possible implementation, in step S101, the amount of sample data of the subject to be evaluated is obtained, including: determining the amount of sample data of the subject to be evaluated according to the standard effect size, significance level and statistical power.

[0065] Specifically, in the present invention, obtaining the sample data volume of the agent to be evaluated is crucial for the subsequent steps, which directly affects the statistical significance and reliability of the experimental results and the applicability of the evaluation method. Therefore, before conducting performance evaluation, the appropriate sample size must be determined based on statistical principles and factors such as standard effect size, significance level and statistical power.

[0066] Specifically, the determination of the sample size must be based on statistical principles to ensure the accuracy and reliability of the experimental results. In step S101, through the statistical calculation method mentioned above, the significance level α is first set to 0.05 and the statistical power (1-β) is set to 0.8 (i.e., the second type error rate is 20%, or β=0.2).

[0067] The effect size (ES) needs to be estimated based on the actual situation. For the task of detecting the authenticity of online information, a reasonable effect size is set based on prior knowledge or pre-experimental results.

[0068] In a possible implementation, the amount of sample data of the agent to be evaluated is determined according to the standard effect size, significance level and statistical power, specifically including: In the process of obtaining the comparative evaluation results of the intelligent agent to be evaluated by paired t-test, the sample size Determined according to the following formula: Sample size: The effect size (ES) was determined according to the following formula: ES

[0069] in, Indicates the significance level The corresponding quantile value of the standard normal distribution, It represents the quantile value of the standard normal distribution corresponding to the statistical power 1-β. is the adjustment factor.

[0070] In a possible implementation, the amount of sample data of the agent to be evaluated is determined according to the standard effect size, significance level and statistical power, specifically including: In the process of obtaining the comparative evaluation results of the intelligent agent to be evaluated through the paired chi-square test, the sample size Determined according to the following formula: Sample size: The effect size (ES) was determined according to the following formula: ES

[0071] in, Indicates the significance level The corresponding quantile value of the standard normal distribution, It represents the quantile value of the standard normal distribution corresponding to the statistical power 1-β. and Represent the classification accuracy of the first experiment and the second experiment respectively, is the adjustment factor.

[0072] According to the above sample size The calculation formula will calculate a reasonable sample size. For example, if the sample size If the value is 500, then at least 500 data samples for network information authenticity detection need to be prepared.

[0073] The present invention can achieve the following beneficial effects by adopting the above method: The performance evaluation method, device and electronic device applicable to a large model intelligent agent provided by the present invention introduce a paired t-test method to quantify the performance difference between the intelligent agent to be evaluated and the standard large language model by outputting the credibility results of continuous values; for the binary classification task scenario, a paired chi-square test method is used to analyze the classification results of the intelligent agent to be evaluated and the standard large language model. In particular, in the task of detecting the authenticity of network information, the performance of the intelligent agent to be evaluated is comprehensively evaluated through scientific method design and statistical theory.

[0074] The performance evaluation device applicable to a large model intelligent agent provided by the present invention is described below. The performance evaluation device applicable to a large model intelligent agent described below and the performance evaluation method applicable to a large model intelligent agent described above can be referenced to each other.

[0075] Figure 3A schematic diagram of a performance evaluation device for a large model intelligent agent provided by the present invention includes the following parts: The model data acquisition module 310 is used to obtain the sample data volume of the intelligent agent to be evaluated and the task output type of the intelligent agent to be evaluated; The first comparative evaluation module 320 is used to obtain comparative evaluation results of the subject to be evaluated by a paired t-test when the task output type of the subject to be evaluated is a continuous output result; The second comparative evaluation module 330 is used to obtain comparative evaluation results of the subject to be evaluated by a paired chi-square test when the task output type of the subject to be evaluated is a binary classification output result; The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

[0076] In one possible implementation, the performance evaluation device applicable to large-model intelligent entities also includes a comprehensive evaluation module 340, which is used to select either the ROC curve or the PR curve to evaluate the intelligent entity to be evaluated according to the data type of the sample data volume when the comparative evaluation results of the intelligent entity to be evaluated show significant performance differences, so as to obtain a comprehensive performance evaluation result of the intelligent entity to be evaluated.

[0077] In a possible implementation, the comprehensive evaluation module 340 is specifically used to select a PR curve to evaluate the subject to be evaluated when the proportion of positive sample data in the sample data volume is greater than a preset proportion threshold, so as to obtain a comprehensive performance evaluation result of the subject to be evaluated.

[0078] In a possible implementation, the comprehensive evaluation module 340 is specifically used to select the ROC curve to evaluate the subject to be evaluated when the proportion of positive sample data in the sample data volume is less than or equal to a preset proportion threshold, so as to obtain a comprehensive performance evaluation result of the subject to be evaluated.

[0079] In one possible implementation, the performance evaluation device applicable to a large model intelligent agent also includes an experimental result acquisition module, which is used to determine that the comparative evaluation result of the intelligent agent to be evaluated shows a significant performance difference when any evaluation result of the comparative experiment evaluation result or the ablation experiment evaluation result shows a significant performance difference.

[0080] In a possible implementation, the model data acquisition module 310 is specifically used to determine the amount of sample data of the agent to be evaluated based on the standard effect size, significance level and statistical power.

[0081] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communications interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a performance evaluation method applicable to a large model intelligent agent, the method comprising: Obtain the sample data volume of the agent to be evaluated and the task output type of the agent to be evaluated; When the task output type of the agent to be evaluated is a continuous output result, the comparative evaluation results of the agent to be evaluated are obtained through the paired t-test method; When the task output type of the agent to be evaluated is a binary classification output result, the comparative evaluation result of the agent to be evaluated is obtained through the paired chi-square test method; The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

[0082] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0083] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute a performance evaluation method applicable to a large model intelligent agent provided by the above methods, the method comprising: Obtain the sample data volume of the agent to be evaluated and the task output type of the agent to be evaluated; When the task output type of the agent to be evaluated is a continuous output result, the comparative evaluation results of the agent to be evaluated are obtained through the paired t-test method; When the task output type of the agent to be evaluated is a binary classification output result, the comparative evaluation result of the agent to be evaluated is obtained through the paired chi-square test method; The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

[0084] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform a performance evaluation method applicable to a large model intelligent agent provided by the above methods, the method comprising: Obtain the sample data volume of the agent to be evaluated and the task output type of the agent to be evaluated; When the task output type of the agent to be evaluated is a continuous output result, the comparative evaluation results of the agent to be evaluated are obtained through the paired t-test method; When the task output type of the agent to be evaluated is a binary classification output result, the comparative evaluation result of the agent to be evaluated is obtained through the paired chi-square test method; The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

[0085] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A performance evaluation method suitable for large model agents, characterized in that: include: Obtaining the sample data volume of the agent to be evaluated and the task output type of the agent to be evaluated; When the task output type of the subject to be evaluated is a continuous output result, a comparative evaluation result of the subject to be evaluated is obtained by a paired t-test method; When the task output type of the subject to be evaluated is a binary classification output result, a comparative evaluation result of the subject to be evaluated is obtained by a paired chi-square test; The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

2. The performance evaluation method applicable to a large model agent according to claim 1, characterized in that: Also includes: When the comparative evaluation results of the intelligent agent to be evaluated show significant performance differences, according to the data type of the sample data volume, either the ROC curve or the PR curve is selected to evaluate the intelligent agent to be evaluated to obtain the comprehensive performance evaluation result of the intelligent agent to be evaluated.

3. The performance evaluation method applicable to a large model agent according to claim 2, characterized in that: According to the data type of the sample data volume, any one of the ROC curve and the PR curve is selected to evaluate the subject to be evaluated, and a comprehensive performance evaluation result of the subject to be evaluated is obtained, including: When the proportion of positive sample data in the sample data volume is greater than a preset proportion threshold, a PR curve is selected to evaluate the intelligent agent to be evaluated to obtain a comprehensive performance evaluation result of the intelligent agent to be evaluated.

4. The performance evaluation method applicable to a large model agent according to claim 2, characterized in that: According to the data type of the sample data volume, any one of the ROC curve and the PR curve is selected to evaluate the subject to be evaluated, and a comprehensive performance evaluation result of the subject to be evaluated is obtained, including: When the proportion of positive sample data in the sample data volume is less than or equal to a preset proportion threshold, the ROC curve is selected to evaluate the subject to be evaluated to obtain a comprehensive performance evaluation result of the subject to be evaluated.

5. The performance evaluation method applicable to a large model agent according to claim 1, characterized in that: Also includes: When any evaluation result of the comparative experiment evaluation result or the ablation experiment evaluation result shows a significant performance difference, it is determined that the comparative evaluation result of the to-be-evaluated intelligent agent shows a significant performance difference.

6. The performance evaluation method for a large model agent according to any one of claims 1 to 5, characterized in that: Get the sample data volume of the agent to be evaluated, including: The amount of sample data of the agent to be evaluated is determined based on the standard effect size, significance level and statistical power.

7. A performance evaluation device suitable for a large model intelligent agent, characterized in that: include: A model data acquisition module is used to obtain the sample data volume of the intelligent agent to be evaluated and the task output type of the intelligent agent to be evaluated; A first comparative evaluation module, used for obtaining comparative evaluation results of the subject to be evaluated by a paired t-test when the task output type of the subject to be evaluated is a continuous output result; A second comparative evaluation module is used to obtain a comparative evaluation result of the intelligent agent to be evaluated by a paired chi-square test when the task output type of the intelligent agent to be evaluated is a binary classification output result; The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the performance evaluation method applicable to a large model intelligent agent as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the performance evaluation method applicable to a large model intelligent agent as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the performance evaluation method applicable to a large model intelligent agent as described in any one of claims 1 to 6 is implemented.

Images