Performance evaluation method suitable for large model agent

Abstract

The application provides a performance evaluation method suitable for a large model agent, comprising: obtaining sample data quantity of an agent to be evaluated, and obtaining task output type of the agent to be evaluated; when the task output type of the agent to be evaluated is continuous output result, obtaining a comparative evaluation result of the agent to be evaluated through a paired t-test method; when the task output type of the agent to be evaluated is binary classification output result, obtaining the comparative evaluation result of the agent to be evaluated through a paired chi-square test method; wherein the comparative evaluation result comprises a comparative experiment evaluation result and an ablation experiment evaluation result. Through comparative experiments, the application quantifies the performance difference between the large model agent and the original LLM, analyzes the contribution of each module to the overall performance through ablation experiments, and provides data support and improvement basis for further optimization of the agent. The performance of the agent is systematically and comprehensively evaluated, and the accuracy and repeatability of the performance evaluation are ensured.

Description

Technical Field

[0001] This invention relates to the field of network information authenticity detection technology, and in particular to a performance evaluation method applicable to large-scale intelligent agents. Background Technology

[0002] Since the release of GPT-3.5, Large Language Models (LLMs) have garnered widespread attention due to their superior natural language processing capabilities. With the opening of LLM programmable interfaces, researchers have not only used them for text generation but also explored their potential as intelligent tools in various applications, making LLM-based agents a research hotspot. Compared to traditional intelligent neural networks, LLM-based agents, leveraging their powerful natural language processing capabilities and rich prior knowledge, do not require massive datasets or high-performance computing resources for model training. By designing appropriate prompts and combining modules such as "action," "plan," and "memory," they can efficiently complete natural language-related tasks in specific application scenarios, demonstrating significant potential, particularly in the field of online information authenticity detection.

[0003] However, systematically and comprehensively evaluating its performance remains a challenge. This involves not only quantifying the agent's performance in a specific task, but also ensuring the accuracy and repeatability of the performance evaluation. Summary of the Invention

[0004] This invention provides a performance evaluation method applicable to large-scale intelligent agents, which enables a systematic and comprehensive evaluation of the performance of large-scale intelligent agents, ensuring the accuracy and repeatability of the performance evaluation.

[0005] This invention provides a performance evaluation method applicable to large-scale intelligent agents, comprising:

[0006] Obtain the sample data volume of the intelligent agent to be evaluated, and obtain the task output type of the intelligent agent to be evaluated;

[0007] When the task output type of the agent to be evaluated is continuous output results, the comparative evaluation results of the agent to be evaluated are obtained by pairwise t-test.

[0008] When the task output type of the agent to be evaluated is a binary classification output result, the comparative evaluation result of the agent to be evaluated is obtained by the paired chi-square test.

[0009] The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

[0010] The performance evaluation method for large-scale intelligent agents provided by the present invention further includes:

[0011] When the comparative evaluation results of the agent to be evaluated show significant performance differences, the agent to be evaluated is evaluated by selecting either the ROC curve or the PR curve, depending on the data type of the sample data, to obtain the comprehensive performance evaluation result of the agent to be evaluated.

[0012] According to the performance evaluation method for large-scale intelligent agents provided by the present invention, based on the data type of the sample data, either a ROC curve or a PR curve is selected to evaluate the intelligent agent to be evaluated, thereby obtaining a comprehensive performance evaluation result of the intelligent agent to be evaluated, including:

[0013] When the proportion of positive sample data in the sample data is greater than a preset proportion threshold, the PR curve is selected to evaluate the agent to be evaluated, and the comprehensive performance evaluation result of the agent to be evaluated is obtained.

[0014] According to the performance evaluation method for large-scale intelligent agents provided by the present invention, based on the data type of the sample data, either a ROC curve or a PR curve is selected to evaluate the intelligent agent to be evaluated, thereby obtaining a comprehensive performance evaluation result of the intelligent agent to be evaluated, including:

[0015] When the proportion of positive sample data in the sample data is less than or equal to a preset proportion threshold, the ROC curve is selected to evaluate the agent to be evaluated, and the comprehensive performance evaluation result of the agent to be evaluated is obtained.

[0016] The performance evaluation method for large-scale intelligent agents provided by the present invention further includes:

[0017] When either the comparative experimental evaluation result or the ablation experimental evaluation result shows a significant performance difference, the comparative evaluation result of the intelligent agent to be evaluated is determined to show a significant performance difference.

[0018] According to any of the above-described performance evaluation methods for large-scale intelligent agents provided by the present invention, obtaining the sample data volume of the intelligent agent to be evaluated includes:

[0019] The sample data size of the agent to be evaluated is determined based on the standard effect size, significance level, and statistical power.

[0020] The present invention also provides a performance evaluation device suitable for large-scale intelligent agents, comprising:

[0021] The model data acquisition module is used to acquire the sample data volume of the agent to be evaluated and to acquire the task output type of the agent to be evaluated.

[0022] The first comparison and evaluation module is used to obtain the comparison and evaluation results of the intelligent agent to be evaluated by means of a paired t-test when the task output type of the intelligent agent to be evaluated is a continuous output result.

[0023] The second comparison and evaluation module is used to obtain the comparison and evaluation results of the intelligent agent to be evaluated by means of a paired chi-square test when the task output type of the intelligent agent to be evaluated is a binary classification output result.

[0024] The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

[0025] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the performance evaluation method applicable to large model intelligent agents as described above.

[0026] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the performance evaluation method applicable to large model intelligent agents as described above.

[0027] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the performance evaluation method applicable to large model intelligent agents as described above.

[0028] The present invention provides a performance evaluation method, device, and electronic device applicable to large-scale intelligent agents. By outputting the credibility results for continuous values, it introduces a paired t-test to quantify the performance difference between the intelligent agent to be evaluated and the standard large language model. For binary classification tasks, it uses a paired chi-square test to analyze the classification results of the intelligent agent to be evaluated and the standard large language model. Especially in the task of detecting the authenticity of network information, it comprehensively evaluates the performance of the intelligent agent to be evaluated through scientific method design and statistical theory. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 This is one of the flowcharts illustrating the performance evaluation method for large-scale intelligent agents provided by this invention.

[0031] Figure 2 This is the second flowchart of the performance evaluation method for large-scale intelligent agents provided by the present invention.

[0032] Figure 3 This is a schematic diagram of the performance evaluation device for large-scale intelligent agents provided by the present invention.

[0033] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The module division described in this application is a logical division. In practical applications, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed in multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0036] The following is combined with Figures 1-4 The specific contents of this invention are described below.

[0037] Figure 1 This is a flowchart illustrating a performance evaluation method for large-scale intelligent agents provided by the present invention, applied in a server, including steps S101-S104.

[0038] Step S101: Obtain the sample data volume of the agent to be evaluated and obtain the task output type of the agent to be evaluated.

[0039] In the embodiments of the present invention, the agent to be evaluated is constructed based on LLM, and the task output types of the agent to be evaluated include two types of task results: continuous output and binary classification output.

[0040] Step S102: When the task output type of the agent to be evaluated is continuous output results, the comparative evaluation results of the agent to be evaluated are obtained by pairwise t-test.

[0041] In embodiments of the present invention, continuous output results are typically represented as numerical values ​​within a range, such as predicted values ​​in a regression task. Paired t-tests are suitable for comparing whether there is a significant difference between the means of two sets of continuous data (such as the outputs of two different models or the same model under different conditions).

[0042] Step S103: When the task output type of the agent to be evaluated is a binary classification output result, the comparative evaluation result of the agent to be evaluated is obtained by the paired chi-square test.

[0043] In embodiments of the present invention, the binary classification output is typically represented as one of two categories (such as 0 and 1, or positive and negative). The paired chi-square test is suitable for comparing whether there is a significant difference in classification accuracy or error rate between two sets of binary classification data.

[0044] The comparative evaluation results include the comparative experimental evaluation results and the ablation experimental evaluation results.

[0045] In one possible implementation, the performance evaluation method applicable to large model agents also includes:

[0046] Step S104: When either the comparative experimental evaluation result or the ablation experimental evaluation result shows a significant performance difference, the comparative evaluation result of the intelligent agent to be evaluated is determined to show a significant performance difference.

[0047] In one possible implementation, step S104 includes steps S104A and S104B.

[0048] Step S104A: Obtain the comparative experiment evaluation results through comparative experiments to determine the comparative evaluation performance of the agent to be evaluated.

[0049] Among them, the comparative evaluation results of the agent to be evaluated are obtained by using the paired t-test method.

[0050] Specifically, for the results of a continuous distribution on [0, 1], we can calculate the following statistics:

[0051] Mean of difference:

[0052] Standard deviation of the difference:

[0053] Degrees of freedom:

[0054] Statistic:

[0055] Where n is the current number of samples, This represents the predicted value of the i-th data point in the first experiment. This represents the predicted value of the i-th data point in the second experiment.

[0056] In embodiments of the present invention, the two experiments include a test experiment for the agent to be evaluated and a test experiment for a standard large language model.

[0057] By looking up the t-distribution table using the t-value and degrees of freedom, the corresponding probability p can be obtained. If p < α, where α is the significance level, it indicates a significant performance difference between the two experimental results. This confirms that the comparative evaluation results of the agent under evaluation show a significant performance difference.

[0058] Among them, the comparative evaluation results of the intelligent agent to be evaluated are obtained by using the paired chi-square test.

[0059] Specifically, the following applies to binary classification tasks. By constructing the following confusion matrix, the chi-square statistic can be calculated.

[0060]

[0061] in,

[0062] TP: Indicates that the first experiment is true and the second experiment is also true. This corresponds to the number of samples;

[0063] FN: indicates that the first experiment is true, but the second experiment is false. This corresponds to the number of samples;

[0064] FP: Indicates that the first experiment was false, but the second experiment was true. This corresponds to the number of samples;

[0065] TN: Indicates that the first experiment was false and the second experiment was also false. This corresponds to the number of samples;

[0066] Based on the confusion matrix, we can calculate:

[0067] Chi-square statistic calculation formula:

[0068]

[0069] In embodiments of the present invention, for any one of the parameters TP, TN, FP, and FN, in the comparative experiment, its corresponding value is incremented by 1 each time it appears. This continues until all samples in sample size n have been traversed. For example, each time TP appears, its corresponding value is incremented by 1. Add 1 to the value.

[0070] Then, based on the chi-square statistic and degrees of freedom (equal to 1), the corresponding critical value is found in the chi-square distribution table. If the chi-square statistic is greater than or equal to the critical value, it indicates that there is a significant performance difference between the two experimental groups in the comparative experiment. That is, it is determined that the comparative evaluation results of the agent to be evaluated show a significant performance difference.

[0071] Step S104B: Obtain the comparative evaluation results through ablation experiments to determine the comparative evaluation performance of the intelligent agent to be evaluated.

[0072] In an embodiment of the present invention, the model types used in the two sets of experiments in steps S104B and S104A are different.

[0073] In step S104A, the model types for the two sets of experiments are: the performance difference of the agent to be evaluated and other models with different configurations of the agent to be evaluated under the same task.

[0074] In step S104B, the contribution of each module (such as the memory module, planning module, etc.) in the agent to be evaluated to the overall performance is analyzed.

[0075] Step S104B specifically includes the following steps:

[0076] Complete model evaluation: First, evaluate the performance of the complete agent to be evaluated, including all modules.

[0077] Module Removal: Gradually remove each module from the agent to be evaluated and re-evaluate its performance.

[0078] Performance comparison: Use paired t-tests or chi-square tests to analyze the performance differences before and after module removal.

[0079] For example, the agent to be evaluated includes module A, module B, and module C.

[0080] In step S104B, the model types for the two sets of experiments are: the model after removing module A from the agent to be evaluated, and the complete agent to be evaluated.

[0081] In step S104B, the comparative evaluation results of the agent to be evaluated are obtained through a paired t-test, or the comparative evaluation results of the agent to be evaluated are obtained through a paired t-test. See step S104A for details; this invention will not elaborate further.

[0082] In one possible implementation, such as Figure 2 As shown, performance evaluation methods suitable for large-scale intelligent agents also include:

[0083] Step S105: When the comparative evaluation results of the agent to be evaluated show significant performance differences, select either the ROC curve or the PR curve to evaluate the agent to be evaluated based on the data type of the sample data, and obtain the comprehensive performance evaluation results of the agent to be evaluated.

[0084] In embodiments of the present invention, when comparative evaluation results already show significant performance differences, further use of ROC curves or PR curves can provide a deeper understanding of the model's performance characteristics. These two types of curves provide intuitive representations of model performance in different scenarios.

[0085] The ROC curve (Receiver Operating Characteristic) is insensitive to the ratio of positive to negative samples; it plots the relationship between the true positive rate (TPR) and the false positive rate (FPR). Therefore, even with an imbalanced ratio of positive to negative samples, the ROC curve can still provide a stable assessment of model performance.

[0086] The PR curve (precision-recall curve) focuses more on the predictive performance of positive samples, plotting the relationship between precision and recall. When the ratio of positive to negative samples is unbalanced, especially when there are fewer positive samples, the PR curve can more accurately reflect the model's ability to identify positive samples.

[0087] In one possible implementation, step S105 involves selecting either a ROC curve or a PR curve to evaluate the agent based on the data type of the sample data, thereby obtaining a comprehensive performance evaluation result for the agent, including:

[0088] Step S105A: When the proportion of positive sample data in the sample data is greater than the preset proportion threshold, the PR curve is selected to evaluate the agent to be evaluated, and the comprehensive performance evaluation result of the agent to be evaluated is obtained.

[0089] Specifically, after step S104, when the proportion of positive sample data in the sample data is greater than the preset proportion threshold, it indicates that the data distribution in the sample is unbalanced (e.g., less false information and more real information). The accuracy and recall of the classifier under different thresholds are compared by plotting PR curves, and the optimal threshold is determined by combining the F1 score.

[0090] In one possible implementation, step S105 involves selecting either a ROC curve or a PR curve to evaluate the agent based on the data type of the sample data, thereby obtaining a comprehensive performance evaluation result for the agent, including:

[0091] Step S105B: When the proportion of positive sample data in the sample data is less than or equal to the preset proportion threshold, select the ROC curve to evaluate the agent to be evaluated and obtain the comprehensive performance evaluation result of the agent to be evaluated.

[0092] Specifically, after step S104, when the proportion of positive sample data in the sample data is less than or equal to the preset proportion threshold, it indicates that the data is relatively balanced or that a global performance evaluation is needed. The overall performance of the classifier is evaluated by plotting the ROC curve and calculating the area under the curve (AUC).

[0093] In one possible implementation, in step S101, the sample data size of the agent to be evaluated is obtained, including: determining the sample data size of the agent to be evaluated based on the standard effect size, significance level, and statistical power.

[0094] Specifically, in this invention, the amount of sample data obtained for the agent to be evaluated is crucial for subsequent steps, directly affecting the statistical significance, reliability, and applicability of the evaluation methods. Therefore, before performance evaluation, an appropriate sample size must be determined based on statistical principles and factors such as standard effect size, significance level, and statistical power.

[0095] Specifically, the determination of the sample size must be based on statistical principles to ensure the accuracy and reliability of the experimental results. In step S101, using the statistical calculation method mentioned above, the significance level α is first set to 0.05, and the statistical power (1-β) is set to 0.8 (i.e., the Type II error rate is 20%, or β=0.2).

[0096] The effect size (ES) needs to be estimated based on the actual situation. For online information authenticity detection tasks, a reasonable effect size should be set based on prior knowledge or pre-experiment results.

[0097] In one possible implementation, the sample data size of the agent to be evaluated is determined based on the standard effect size, significance level, and statistical power, specifically including:

[0098] In the process of obtaining the comparative evaluation results of the agent to be evaluated through paired t-tests, the sample size... Determined according to the following formula:

[0099] Sample size:

[0100] The effect size (ES) is determined according to the following formula:

[0101] ES

[0102] in, Indicates significance level The corresponding quantile values ​​of the standard normal distribution. This represents the quantile value of the standard normal distribution corresponding to the statistical power 1-β. This is for adjusting the coefficient.

[0103] In one possible implementation, the sample data size of the agent to be evaluated is determined based on the standard effect size, significance level, and statistical power, specifically including:

[0104] In the process of obtaining the comparative evaluation results of the intelligent agent to be evaluated through the paired chi-square test, the sample size... Determined according to the following formula:

[0105] Sample size:

[0106] The effect size (ES) is determined according to the following formula:

[0107] ES

[0108] in, Indicates significance level The corresponding quantile values ​​of the standard normal distribution. This represents the quantile value of the standard normal distribution corresponding to the statistical power 1-β. and These represent the classification accuracy rates in the first and second experiments, respectively. This is for adjusting the coefficient.

[0109] Based on the above sample size The calculation formula will determine a reasonable sample size. For example, if the sample size... If the value is 500, then at least 500 data samples for verifying the authenticity of online information need to be prepared.

[0110] By employing the above method, the present invention can achieve the following beneficial effects:

[0111] The present invention provides a performance evaluation method, device, and electronic device applicable to large-scale intelligent agents. By outputting the credibility results for continuous values, it introduces a paired t-test to quantify the performance difference between the intelligent agent to be evaluated and the standard large language model. For binary classification tasks, it uses a paired chi-square test to analyze the classification results of the intelligent agent to be evaluated and the standard large language model. Especially in the task of detecting the authenticity of network information, it comprehensively evaluates the performance of the intelligent agent to be evaluated through scientific method design and statistical theory.

[0112] The performance evaluation device for large-scale intelligent agents provided by the present invention is described below. The performance evaluation device for large-scale intelligent agents described below can be referred to in correspondence with the performance evaluation method for large-scale intelligent agents described above.

[0113] Figure 3 A schematic diagram of a performance evaluation device for large-scale intelligent agents provided by the present invention includes the following parts:

[0114] The model data acquisition module 310 is used to acquire the sample data volume of the agent to be evaluated and to acquire the task output type of the agent to be evaluated.

[0115] The first comparison and evaluation module 320 is used to obtain the comparison and evaluation results of the intelligent agent to be evaluated by means of a paired t-test when the task output type of the intelligent agent to be evaluated is a continuous output result.

[0116] The second comparison and evaluation module 330 is used to obtain the comparison and evaluation results of the intelligent agent to be evaluated by means of the paired chi-square test when the task output type of the intelligent agent to be evaluated is a binary classification output result.

[0117] The comparative evaluation results include the comparative experimental evaluation results and the ablation experimental evaluation results.

[0118] In one possible implementation, the performance evaluation device for large model agents further includes a comprehensive evaluation module 340, which is used to select either the ROC curve or the PR curve to evaluate the agent when the comparative evaluation results of the agent to be evaluated show significant performance differences, based on the data type of the sample data, and obtain the comprehensive performance evaluation result of the agent to be evaluated.

[0119] In one possible implementation, the comprehensive evaluation module 340 is specifically used to select the PR curve to evaluate the agent to be evaluated when the proportion of positive sample data in the sample data is greater than a preset proportion threshold, so as to obtain the comprehensive performance evaluation result of the agent to be evaluated.

[0120] In one possible implementation, the comprehensive evaluation module 340 is specifically used to select the ROC curve to evaluate the agent to be evaluated when the proportion of positive sample data in the sample data is less than or equal to a preset proportion threshold, so as to obtain the comprehensive performance evaluation result of the agent to be evaluated.

[0121] In one possible implementation, the performance evaluation device for large model agents further includes an experimental result acquisition module, used to determine that the comparative evaluation result of the agent to be evaluated shows a significant performance difference when either the comparative experimental evaluation result or the ablation experimental evaluation result shows a significant performance difference.

[0122] In one possible implementation, the model data acquisition module 310 is specifically used to determine the amount of sample data of the agent to be evaluated based on the standard effect size, significance level, and statistical power.

[0123] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a performance evaluation method suitable for large-scale intelligent agents, the method including:

[0124] Obtain the sample data volume of the agent to be evaluated, and obtain the task output type of the agent to be evaluated;

[0125] When the task output type of the agent to be evaluated is continuous output results, the comparative evaluation results of the agent to be evaluated are obtained by pairwise t-test.

[0126] When the task output type of the agent to be evaluated is a binary classification output result, the comparative evaluation result of the agent to be evaluated is obtained by the paired chi-square test.

[0127] The comparative evaluation results include the comparative experimental evaluation results and the ablation experimental evaluation results.

[0128] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute a performance evaluation method applicable to large-scale intelligent agents provided by the above methods, the method comprising:

[0130] Obtain the sample data volume of the agent to be evaluated, and obtain the task output type of the agent to be evaluated;

[0131] When the task output type of the agent to be evaluated is continuous output results, the comparative evaluation results of the agent to be evaluated are obtained by pairwise t-test.

[0132] When the task output type of the agent to be evaluated is a binary classification output result, the comparative evaluation result of the agent to be evaluated is obtained by the paired chi-square test.

[0133] The comparative evaluation results include the comparative experimental evaluation results and the ablation experimental evaluation results.

[0134] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform a performance evaluation method for large-scale intelligent agents provided by the methods described above, the method comprising:

[0135] Obtain the sample data volume of the agent to be evaluated, and obtain the task output type of the agent to be evaluated;

[0136] When the task output type of the agent to be evaluated is continuous output results, the comparative evaluation results of the agent to be evaluated are obtained by pairwise t-test.

[0137] When the task output type of the agent to be evaluated is a binary classification output result, the comparative evaluation result of the agent to be evaluated is obtained by the paired chi-square test.

[0138] The comparative evaluation results include the comparative experimental evaluation results and the ablation experimental evaluation results.

[0139] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A performance evaluation method applicable to large-scale intelligent agents, characterized in that, include: Obtain the sample data volume for network information authenticity detection of the agent to be evaluated, and obtain the task output type of the agent to be evaluated; When the task output type of the agent to be evaluated is a continuous output result representing the true value, the comparative evaluation result of the agent to be evaluated is obtained by pairwise t-test. When the task output type of the agent to be evaluated is a binary classification output result representing the truth category, the comparative evaluation result of the agent to be evaluated is obtained by the paired chi-square test. The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

2. The performance evaluation method for large-scale intelligent agents according to claim 1, characterized in that, Also includes: When the comparative evaluation results of the agent to be evaluated show significant performance differences, the agent to be evaluated is evaluated by selecting either the ROC curve or the PR curve, depending on the data type of the sample data, to obtain the comprehensive performance evaluation result of the agent to be evaluated.

3. The performance evaluation method for large-scale intelligent agents according to claim 2, characterized in that, Based on the data type of the sample data, either a ROC curve or a PR curve is selected to evaluate the agent to be evaluated, resulting in a comprehensive performance evaluation result for the agent, including: When the proportion of positive sample data in the sample data is greater than a preset proportion threshold, the PR curve is selected to evaluate the agent to be evaluated, and the comprehensive performance evaluation result of the agent to be evaluated is obtained.

4. The performance evaluation method applicable to large-scale intelligent agents according to claim 2, characterized in that, Based on the data type of the sample data, either a ROC curve or a PR curve is selected to evaluate the agent to be evaluated, resulting in a comprehensive performance evaluation result for the agent, including: When the proportion of positive sample data in the sample data is less than or equal to a preset proportion threshold, the ROC curve is selected to evaluate the agent to be evaluated, and the comprehensive performance evaluation result of the agent to be evaluated is obtained.

5. The performance evaluation method for large-scale intelligent agents according to claim 1, characterized in that, Also includes: When either the comparative experimental evaluation result or the ablation experimental evaluation result shows a significant performance difference, the comparative evaluation result of the intelligent agent to be evaluated is determined to show a significant performance difference.

6. The performance evaluation method for large-scale intelligent agents according to any one of claims 1 to 5, characterized in that, The amount of sample data obtained for the network information authenticity detection of the agent to be evaluated includes: The sample data size for detecting the authenticity of network information of the agent to be evaluated is determined based on the standard effect size, significance level, and statistical power.

7. A performance evaluation device suitable for large-scale intelligent agents, characterized in that, include: The model data acquisition module is used to acquire the amount of sample data for the network information authenticity detection of the agent to be evaluated, and to acquire the task output type of the agent to be evaluated. The first comparison and evaluation module is used to obtain the comparison and evaluation results of the intelligent agent to be evaluated by means of a paired t-test when the task output type of the intelligent agent to be evaluated is a continuous output result representing the true value. The second comparison and evaluation module is used to obtain the comparison and evaluation results of the intelligent agent to be evaluated by means of a paired chi-square test when the task output type of the intelligent agent to be evaluated is a binary classification output result representing the true category. The comparative evaluation results include comparative experimental evaluation results and ablation experimental evaluation results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the performance evaluation method for large-scale intelligent agents as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the performance evaluation method for large-scale intelligent agents as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the performance evaluation method for large-scale intelligent agents as described in any one of claims 1 to 6.

Images