Intelligent agent evaluation method and apparatus

Abstract

The present disclosure relates to the technical field of agent evaluation, and specifically discloses an agent evaluation method and device, wherein the agent evaluation method comprises: obtaining at least two configuration groups of an agent and an evaluation test task pool; for each configuration group of the agent, respectively executing multiple rounds of evaluation test tasks based on the evaluation test task pool in parallel, and obtaining evaluation test results of each round; determining a configuration evaluation result corresponding to each configuration group based on the evaluation test results of multiple rounds of each configuration group; and determining a contrast evaluation result of the agent based on the configuration evaluation results of multiple configuration groups. The present disclosure can execute evaluation test tasks based on test cases asynchronously and in parallel according to multiple configuration groups to be evaluated of the agent, perform multi-dimensional analysis on each configuration group, and perform contrast evaluation between different configuration groups, so that a user can accurately obtain application capabilities of the agent under different configuration groups.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent agent evaluation technology, and in particular to an intelligent agent evaluation method and apparatus. Background Technology

[0002] With the rapid development of artificial intelligence technology, large models (such as large language models and multimodal generative models) are increasingly widely used in various fields. Agents built based on large models are widely used in application scenarios such as business data analysis, content insight, ad delivery assistance, and intelligent voice customer service.

[0003] Unlike the direct application of large models, intelligent agents, as applications or complex systems built upon large models and capable of autonomously calling various tools to execute required tasks, are not limited to a simple "input → output" question-and-answer process. Instead, they can autonomously plan multiple task execution steps, invoke multiple external tools (such as database queries, retrieval, and computation), and produce the final intelligent agent output through multiple rounds of interaction. Therefore, compared to traditional large model evaluation, the intelligent agent evaluation process requires consideration of more dimensions and factors, and its implementation is relatively more complex. There is an urgent need for a dedicated evaluation method and system for intelligent agents to assist users in conducting targeted evaluation and testing of the intelligent agent's optimization process. Summary of the Invention

[0004] The purpose of this disclosure is to provide an agent evaluation method and apparatus that can asynchronously and concurrently execute test case-based evaluation tasks according to multiple configuration groups to be evaluated of the agent during the agent evaluation process, thereby performing multi-dimensional analysis of the evaluation test situation of each configuration group.

[0005] Specifically, the first aspect of this disclosure provides an agent evaluation method, which may include the following steps: obtaining at least two configuration groups of the agent and an evaluation test task pool; for each configuration group of the agent, concurrently executing multiple rounds of evaluation test tasks based on the evaluation test task pool, and obtaining the evaluation test results for each round; determining the configuration evaluation result corresponding to the configuration group based on the evaluation test results of multiple rounds for each configuration group; and determining the comparison evaluation result of the agent based on the configuration evaluation results of multiple configuration groups.

[0006] In some embodiments, for each configuration group of the agent, multiple rounds of evaluation test tasks are executed concurrently based on the evaluation test task pool, and the evaluation test results of each round are obtained, including: creating a corresponding evaluation test data space based on the configuration group; after all evaluation test data spaces have been created, controlling each configuration group to independently select an evaluation test task from the evaluation test task pool based on a concurrent semaphore; and controlling each configuration group to concurrently execute the evaluation test task; wherein, the evaluation test task includes at least one preset test case and the repeated execution rounds corresponding to the test case.

[0007] In some embodiments, the evaluation test task includes a test case execution phase and a test result evaluation phase; controlling the concurrent execution of the evaluation test task for each configuration group includes: based on a preset first concurrent semaphore, invoking an agent to execute the test case execution phase based on the configuration group; based on a preset second concurrent semaphore, invoking a preset evaluator to execute the test result evaluation phase based on the test case execution result and / or the test case execution process; wherein the first concurrent semaphore and the second concurrent semaphore are independent of each other.

[0008] In some embodiments, the invoked agent performs the test case execution phase based on the configured group, including: creating a test case execution subtask and a timeout sentinel subtask; outputting the test case execution result and / or test case execution process when the test case execution subtask completes before the timeout sentinel subtask; and determining that the test case execution subtask has timed out and abandoning the test case execution subtask when the timeout sentinel subtask completes before the test case execution subtask.

[0009] In some embodiments, the evaluator includes one or any combination of tool correctness dimension evaluator, rule dimension evaluator, efficiency dimension evaluator, cost dimension evaluator, lexical dimension evaluator, delay dimension evaluator, and large language model evaluator: invoking a preset evaluator to perform a test result evaluation phase based on the test case execution results and / or test case execution process, including: invoking the evaluator to obtain the dimension evaluation results of one or more evaluators based on the test case execution results and / or test case execution process, and determining the evaluation test result based on all dimension evaluation results.

[0010] In some embodiments, the configuration evaluation result corresponding to the configuration group is determined based on the evaluation test results of multiple rounds for each configuration group, including: obtaining the statistical results of the evaluation test results of multiple rounds; obtaining the random fluctuation parameter and effect size parameter of the configuration group based on the historical evaluation test baseline and the statistical results, and determining the configuration evaluation result corresponding to the configuration group based on the random fluctuation parameter and effect size parameter.

[0011] In some embodiments, the agent evaluation method provided in this disclosure may further include: determining the evaluation test task that failed to execute based on statistical results, and obtaining the failed test cases corresponding to the evaluation test task that failed to execute; and selecting failed test cases with a similarity lower than a preset threshold to update the historical failed test case set based on the similarity between the failed test cases and the historical failed test case set.

[0012] In some embodiments, the agent evaluation method provided in this disclosure may further include: visually presenting the evaluation test results of each round and / or the evaluation test task execution process of multiple rounds for each configuration group of the agent; and visualizing the comparative evaluation results of the agent based on the configuration evaluation results of multiple configuration groups.

[0013] In some embodiments, the agent evaluation method provided in this disclosure may further include: acquiring the evaluation test results for each round and synchronously writing them into a first data space and a second data space; wherein the first data space is independent of the second data space; the first data space is used to provide the evaluation test results for determining the configuration evaluation results and / or the comparison evaluation results; and the second data space is used to provide the evaluation test results for visualization.

[0014] In some embodiments, the agent evaluation method provided in this disclosure may further include: determining the agent's evaluation mode in response to a natural language instruction; determining the agent's configuration group and evaluation intent form based on the agent's evaluation mode; and determining the evaluation test task pool and the evaluation dimension information corresponding to the evaluation test results in response to a confirmation instruction on the evaluation intent form.

[0015] In some embodiments, the evaluation modes of the agent include a parameter tuning evaluation mode, a script evaluation mode, and a code evaluation mode; wherein, the parameter tuning evaluation mode includes a comparison of the agent's generation results under different calling models, and / or different inference parameters, and / or different prompt word versions; the script evaluation mode includes a comparison of the agent's generation results under different iteration versions; and the code evaluation mode includes a comparison of the agent's generation results under different code fine-tuning scenarios.

[0016] The second aspect of this disclosure provides an agent evaluation apparatus, which may specifically include: an acquisition unit for acquiring at least two configuration groups of an agent and an evaluation test task pool; a testing unit for concurrently executing multiple rounds of evaluation test tasks based on the evaluation test task pool for each configuration group of the agent, and acquiring the evaluation test results for each round; a group evaluation unit for determining the configuration evaluation result corresponding to the configuration group based on the evaluation test results of multiple rounds for each configuration group; and a comparison evaluation unit for determining the comparison evaluation result of the agent based on the configuration evaluation results of multiple configuration groups.

[0017] A third aspect of this disclosure provides an electronic device, which may include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the agent evaluation method provided in the first aspect.

[0018] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the agent evaluation method provided in the first aspect.

[0019] The fifth aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the agent evaluation method provided in the first aspect.

[0020] The beneficial effects of this disclosure may include, but are not limited to: enabling asynchronous and concurrent execution of test case-based evaluation tasks during agent evaluation, based on multiple configuration groups to be evaluated for the agent. This allows for multi-agent-specific dimension evaluation and analysis of the evaluation test results for each configuration group, and enables comparative evaluation between different configuration groups, allowing users to accurately obtain the real application capabilities of the agent under different configuration groups. Furthermore, the technical solution provided by this disclosure employs concurrent semaphore isolation during the specific execution of the evaluation test task. Independent concurrent semaphores are used in the test case execution phase and the test result evaluation phase, achieving decoupling of the test case execution phase and the test result evaluation phase in terms of concurrent semaphores, effectively avoiding abnormal situations such as deadlock and freezing. Furthermore, the technical solution provided by this disclosure also supports a timeout race mechanism during the execution of the evaluation test task, preventing timeout deadlocks in the processing process by setting timeout sentinel tasks. Furthermore, the technical solution provided in this disclosure also supports the full visualization of the evaluation and testing process and the comparison of evaluation results, enabling users to intuitively obtain the evaluation and testing results of the intelligent agent under different configuration groups; at the same time, it supports users to confirm the configuration group information and evaluation intent form through natural language commands, reducing the configuration operations required for users to evaluate the intelligent agent and improving the user experience of intelligent agent evaluation.

[0021] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects may be any one or a combination of the above, or any other possible beneficial effects. Attached Figure Description

[0022] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0023] Figure 1 This is a schematic diagram illustrating an application scenario of an intelligent agent evaluation method provided in an embodiment of this disclosure; Figure 2 This is a flowchart illustrating an agent evaluation method provided according to an embodiment of the present disclosure; Figure 3 This is a flowchart illustrating the concurrent execution of evaluation test tasks for each configuration group of an intelligent agent according to an embodiment of this disclosure; Figure 4 This is a schematic diagram of a process for controlling the concurrent execution of evaluation test tasks for each configuration group according to an embodiment of the present disclosure; Figure 5 This is a flowchart illustrating the execution phase of a test case according to an embodiment of this disclosure; Figure 6 This is a flowchart illustrating a process for determining the configuration evaluation result corresponding to a configuration group, according to an embodiment of this disclosure. Figure 7 This is a schematic diagram of a process for evaluating test failure reflow processing according to an embodiment of this disclosure; Figure 8 This is a schematic diagram of a process for evaluating an intelligent agent based on natural language instructions, according to an embodiment of this disclosure. Figure 9 This is a schematic diagram of the structure of an intelligent agent evaluation device according to an embodiment of the present disclosure; Figure 10 This is an exemplary hardware architecture diagram of a computer device according to some embodiments of the present disclosure. Detailed Implementation

[0024] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the content described below are some examples or embodiments of this disclosure. For those skilled in the art, without creative effort, the technical solutions or means disclosed in this disclosure can be applied to other scenarios based on this technical content.

[0025] It should be understood that the terms "system," "device," "unit," and / or "module" used in this disclosure are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0026] Unless otherwise specified, the technical terms used to describe components, elements, etc. in this disclosure are not specifically singular, but may include plural. Generally speaking, terms such as "comprising" or "including" only indicate that explicitly identified steps, elements, or components are included, and these steps, elements, and components do not constitute an exclusive list, as the described method or apparatus may also include other steps or components.

[0027] This disclosure uses flowcharts to illustrate the operational steps performed by the apparatus or system of related embodiments. However, unless otherwise specified, the order in which these steps are described should not be construed as a limitation on the order of execution. Those skilled in the art can adjust the order of these steps based on the knowledge and information conveyed by the embodiments of this disclosure. Adjustments include, but are not limited to, reversing the order of steps, merging multiple steps, and splitting a step.

[0028] With the rapid development of artificial intelligence technology, large models (such as large language models and multimodal generative models) are increasingly widely used in various fields. Agents built upon these large models are widely applied in business data analysis, content insight, ad placement assistance, and intelligent voice customer service. Unlike the direct application of large models, agents, as applications or complex systems built upon them and capable of autonomously calling various tools to execute required tasks, are not limited to a simple "input → output" question-and-answer process. Instead, they can autonomously plan multiple task execution steps, call multiple external tools (such as database queries, retrieval, and calculations), and produce the final agent output through multiple rounds of interaction. Therefore, compared to traditional large model evaluation, agent evaluation requires consideration of more dimensions and factors, and its implementation is more complex. There is an urgent need for a dedicated evaluation method and system for agents to assist users in conducting targeted evaluation tests of the agent's optimization process.

[0029] In view of this, this disclosure provides an intelligent agent evaluation method and apparatus, which can asynchronously and concurrently execute test case-based evaluation tasks according to multiple configuration groups of the intelligent agent to be evaluated during the intelligent agent evaluation process. This allows for multi-dimensional analysis of the evaluation test results for each configuration group and enables comparative evaluation between different configuration groups, allowing users to accurately obtain the real application capabilities of the intelligent agent under different configuration groups. It is understood that in the specific embodiments of this specification, the collection, use, or processing of data (such as test cases for the intelligent agent) requires the permission or consent of the data subject when one or more embodiments of this disclosure are applied to specific products or technology implementations. Furthermore, the collection, use, or processing of related data must strictly comply with the relevant laws, regulations, and standards of the data source country, implementation country, and other relevant countries and regions. De-identification technology ensures that the final data used is securely processed de-identified data, protecting the rights and interests of the data subject and data security.

[0030] Figure 1 This is a schematic diagram illustrating an application scenario of an agent evaluation method according to some embodiments of this disclosure. In some embodiments, such as... Figure 1 As shown, application scenario 100 may include server 110, network 120, and client 130, wherein server 110 can connect to client 130 through network 120. In some embodiments, client 130 may receive agent configuration instructions and agent evaluation instructions provided by the user, generate corresponding evaluation test tasks, and send them to server 110 through network 120. Server 110 then performs agent invocation for different configurations, executes evaluation test tasks, and provides feedback on evaluation test results. Server 110 feeds back the evaluation test results of each configuration group and the comparative evaluation results between multiple configuration groups to client 130 through network 120, and presents them to the user through the client 130's visual interface, allowing the user to intuitively obtain the specific evaluation status of the agent under different configuration groups. This is not limited here. In some embodiments, the application scenario of agent evaluation may specifically include evaluating the actual application of the agent under different large language models, the actual application of the agent under different iterative versions, and the actual application of the agent's code after adjustment and optimization compared to before adjustment and optimization, etc., which are not limited here.

[0031] In some embodiments, server 110 may be a computer device with high computing performance. In some embodiments, server 110 may be a single computer device or a computing cluster composed of multiple computer devices, thereby providing powerful computing power and efficient processing response for agent evaluation. In some embodiments, server 110 may be a server, which may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Distribution Network), and big data and artificial intelligence platforms.

[0032] In some embodiments, network 120 can be any form of wired or wireless network, or any combination thereof. By way of example only, network 120 can be one or more combinations of wired networks, fiber optic networks, telecommunications networks, internal networks, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, etc. Network 120 can have multiple access points, through which server 110 and client 130 can access network 120.

[0033] In some embodiments, the client 130 may include, but is not limited to, desktop computers, smartphones, laptops, VR (Virtual Reality) devices, and tablets. In some embodiments, users can provide agent evaluation instructions through the client 130, set specific configuration groups for the agent, and obtain comparative evaluation results of multiple configuration groups of the agent through the client 130 to realize human-computer interaction in the agent evaluation process, which is not limited here.

[0034] It should be noted that, Figure 1 The illustrated application scenario diagram of the agent evaluation method is merely an example. The application scenarios described in this disclosure are intended to more clearly illustrate the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided in this disclosure. For example, Figure 1The number of server-side 110 and client-side 130 is merely illustrative and not intended to limit the scope of patent protection of this disclosure. Depending on the actual situation, any number of server-side 110 and client-side 130 may be used. As those skilled in the art will recognize, with the development of agent evaluation technology and the emergence of new business scenarios, the embodiments provided in this disclosure can also be applied to other agent evaluation scenarios, and are not limited here. For example, in some embodiments, the embodiments provided in this disclosure can be applied to local agent evaluation application scenarios, achieving local acquisition of comparative evaluation results through the model evaluator and graphics processing unit locally mounted on the client-side 130; the embodiments provided in this disclosure can also support interconnected, collaborative, and parallel communication between multiple clients-side 130, such as using agent evaluation resources such as graphics processing units of different specifications deployed locally on multiple clients-side 130 for collaborative agent evaluation, etc., and are not limited here.

[0035] Figure 2 is an exemplary flowchart of an agent evaluation method according to some embodiments of the present disclosure. In some embodiments, Figure 2 The process 200 shown can be executed by a terminal device, for example, by a terminal device such as... Figure 1 The client 130 shown is executed; or, Figure 2 The process 200 shown can be executed jointly by a processing device and a terminal device, for example, it can be executed by a processing device and a terminal device. Figure 1 The server 110 and client 130 shown are executed together. In some embodiments, process 200 may be implemented by an agent evaluation device 1000 deployed on a terminal device.

[0036] In some embodiments, such as Figure 2 As shown, process 200 may include the following steps: Step 210: Obtain at least two configuration groups for the agent and an evaluation test task pool. In some embodiments, step 210 can be implemented by acquisition unit 910.

[0037] In some embodiments, an agent refers to an application or system built on a Large Language Model (LLM) that can autonomously call various tools to perform required tasks; this is not limited to any particular type. In some embodiments, the configuration group of an agent may specifically include different large language models used by the agent, information on custom scripts employed by the agent, version iteration information of the agent, and at least some code content within the agent. A set of configuration contents that forms a complete agent can be considered a configuration group of the agent; this is not limited to any particular type. It is understood that in the actual application of an agent, it is usually necessary to optimize and adjust the agent multiple times. During the optimization and adjustment process, the evaluation and testing results of the agent across different configuration groups can inform the user of the actual application status of the agent before and after the current optimization and adjustment, so that the user can decide whether to make further adjustments to the agent or directly implement the agent based on the current optimization and adjustment. In some embodiments, the technical solution provided in this disclosure can also be applied to the evaluation process of a single configuration group of the agent, and can be used to evaluate the specific application of the agent under a single configuration group (e.g., whether the agent can complete the test tasks corresponding to preset test cases, etc.); this is not limited to any particular type. In some embodiments, the evaluation test task pool may specifically include multiple evaluation test tasks. Each evaluation test task may consist of preset test cases and the number of repeated execution rounds corresponding to the test cases, which is not limited here. The specific configuration of the evaluation test task pool will be further explained in conjunction with the embodiments later, and will not be repeated here.

[0038] Step 220: For each configuration group of the agent, execute multiple rounds of evaluation test tasks concurrently based on the evaluation test task pool, and obtain the evaluation test results for each round. In some embodiments, step 220 can be implemented by the test unit 920. In some embodiments, considering the random fluctuations that may occur in the application of an agent equipped with a large language model, a single round of evaluation test cannot reflect the actual application of the agent. It is necessary to evaluate the stability of the agent's output through multiple repeated rounds. The concurrent execution of multiple rounds of evaluation test tasks and the acquisition of the evaluation test results for each round will be further explained and described in conjunction with the embodiments later, and will not be repeated here.

[0039] Step 230: Based on the evaluation test results of multiple rounds for each configuration group, determine the configuration evaluation result corresponding to the configuration group. In some embodiments, step 230 can be implemented by the group evaluation unit 930. In some embodiments, the configuration evaluation result corresponding to the configuration group can be determined by statistical methods or other means based on the evaluation test results of multiple rounds for each configuration group; the specific determination of the configuration evaluation result will be further explained and described in conjunction with the embodiments later, and will not be elaborated here.

[0040] Step 240: Based on the configuration evaluation results of multiple configuration groups, determine the comparative evaluation results of the agent. In some embodiments, step 240 can be implemented by the comparative evaluation unit 940. It is understood that through the above process 200, during the agent evaluation process, evaluation test tasks based on test cases can be executed asynchronously and concurrently according to multiple configuration groups to be evaluated for the agent, thereby performing multi-dimensional analysis of the evaluation test situation of each configuration group and enabling comparative evaluation between different configuration groups, allowing users to accurately obtain the real application capabilities of the agent under different configuration groups. The specific implementation of each step in the above process 200 will be further explained and illustrated below with reference to specific embodiments.

[0041] In some embodiments, Figure 3 This diagram illustrates a process for concurrently executing evaluation test tasks for each configuration group of an agent, such as... Figure 3 As shown, process 300 may specifically include the following steps: Step 310: Based on the configuration group, create the corresponding evaluation test data space. In some embodiments, the evaluation test data space may specifically include a test database corresponding to each configuration group, used to collect and store various data generated during the evaluation test process; in other embodiments, in scenarios where the agent evaluation test process is presented to the user in a visual manner, the evaluation test data space may also include the creation of an observable platform, which is not limited here.

[0042] Step 320: After all evaluation test data spaces have been created, based on concurrent semaphores, control each configuration group to independently select evaluation test tasks from the evaluation test task pool. In some embodiments, a barrier can be initialized to ensure that specific test task execution only begins after all evaluation test data spaces have been created, thereby avoiding possible state errors or other anomalies caused by the interleaving of evaluation test data space creation and specific test task execution between different configuration groups. Each configuration group can rotate in a preset semaphore queue to achieve fair scheduling of task execution among different configuration groups. In some embodiments, the evaluation test task pool can flatten the test cases and corresponding repetitive execution rounds corresponding to the configuration group into a single task pool instead of sequentially executing rounds. Each evaluation test task corresponds to at least one test case and the repetitive execution rounds required for that test case. The corresponding evaluation test task is retrieved and executed asynchronously and concurrently, so that multiple concurrently executing configuration groups can have the same or similar evaluation test progress.

[0043] Step 330: Control each configuration group to execute the evaluation test task concurrently. The execution process of the evaluation test task will be explained and illustrated in detail below with reference to specific embodiments.

[0044] It is understandable that the concurrent execution of evaluation test tasks in each configuration group typically includes a test case execution phase and an evaluation phase of the agent's execution results. Existing agent evaluation schemes, in practical implementation, usually use a single semaphore for concurrent control of the execution and evaluation phases. This means that if the agent evaluation phase experiences slow waiting or rate limiting, subsequent test case execution must wait for the entire evaluation phase to complete before starting, potentially leading to deadlocks or freezes. In some embodiments, to overcome the above problems, Figure 4 A flowchart illustrating a process for controlling the concurrent execution of evaluation test tasks for each configuration group is shown, such as... Figure 4 As shown, process 400 may specifically include: Step 410: Based on the preset first concurrent semaphore, the agent is invoked to execute the test case execution phase according to the configured group. In some embodiments, the first concurrent semaphore may be set by the user according to the actual evaluation needs of the agent, and may be used to characterize the number of test cases that can be executed simultaneously, without limitation here.

[0045] Step 420: Based on the preset second concurrent semaphore, invoke the preset evaluator to perform the test result evaluation phase based on the test case execution results and / or test case execution process. In some embodiments, the second concurrent semaphore can be set by the user according to the actual evaluation needs of the agent, and can be used to characterize the number of output content evaluations that can be performed simultaneously, without limitation. In some embodiments, the first concurrent semaphore and the second concurrent semaphore are independent of each other. Specifically, the first concurrent semaphore can be released after the test case execution phase ends, so that the agent's call slots are not occupied by the subsequent test result evaluation phase, thereby achieving decoupling of the test case execution phase and the test result evaluation phase in terms of concurrent semaphores, thereby avoiding abnormal situations such as deadlock and freezing.

[0046] In some embodiments, the evaluator may specifically include one or any combination of tool correctness dimension evaluators, rule dimension evaluators, efficiency dimension evaluators, cost dimension evaluators, lexical dimension evaluators, latency dimension evaluators, and large language model evaluators. In some embodiments, the evaluator does not participate in the input and output of the agent evaluation and testing process, but is only used to evaluate and score the output process and output content of a single agent. In some embodiments, the tool correctness dimension evaluator can be used to evaluate the agent's tool selection under the configuration group, specifically including the precision of tool selection, the recall of tool selection, the correctness of tool parameter calls, the redundancy of calls, whether errors occur, etc., which are not limited here. In some embodiments, the efficiency dimension evaluator can be used to evaluate the trajectory execution efficiency of the agent under the configuration group, specifically including whether the number of execution steps of the agent exceeds the preset step limit, whether the number of loops exceeds the preset repetition limit, whether it passes through the required task node / prohibited task node, etc., which are not limited here. In some embodiments, the rule-dimensional evaluator can be used to evaluate the agent's matching of format rules under a configuration group, specifically including plain text format evaluation, JSON validity evaluation, and whether it contains preset keyword content / banned word content, etc., without limitation. In some embodiments, the cost-dimensional evaluator can be used to evaluate the agent's resource consumption under a configuration group, specifically including whether the agent's inference implementation cost under the configuration group exceeds the budget target, etc., without limitation. In some embodiments, the lexical-dimensional evaluator can be used to evaluate the agent's lexical usage under a configuration group, specifically including whether the agent's lexical consumption under the configuration group exceeds the budget target, etc., without limitation. In some embodiments, the latency-dimensional evaluator can be used to evaluate the agent's latency under a configuration group, specifically including the comparison between the agent's end-to-end processing latency under the configuration group and the preset latency target, etc., without limitation. In some embodiments, the large language model evaluator can be used to evaluate and score the agent's output semantic content under the configuration group by calling other large language models, referring to the preset expected output content, and generating corresponding comments, etc., without limitation. In some embodiments, each selected evaluator may have a corresponding preset weight. During the test result evaluation phase, when the preset evaluator is invoked based on the test case execution results and / or the test case execution process, the dimensional evaluation results of one or more evaluators can be obtained. Based on all dimensional evaluation results, the evaluation test result is determined by weighted summation using preset weights. This is not limited here.

[0047] In some embodiments, during the execution phase of the test case, the agent's execution process is highly dependent on the remote large model gateway. However, in practical applications, when the gateway only returns a portion of the response headers, or stops sending response body data after completing the response headers, the client may remain in the network read state for an extended period, resulting in a timeout and freezing. Existing timeout configuration methods, such as asyncio.wait_for, employ a cooperative cancellation mechanism. Their timeout handling relies on the event loop obtaining scheduling opportunities and the called task responding to cancellation requests. Therefore, they cannot forcibly terminate running synchronous system calls, background threads, or uninterruptible underlying extensions. If there are simultaneous issues such as missing read timeouts, streaming response reads not being covered by the total timeout, or cancellation exceptions being swallowed by internal code, the outer-configured timeout may not terminate the underlying requests in time, leading to the agent being suspended for a long time during the evaluation of the test task, resulting in a timeout freeze that is unrecoverable. In some embodiments, to overcome the above practical application problems, Figure 5 A flowchart illustrating the execution phase of a test case is shown, such as... Figure 5 As shown, process 500 may specifically include: Step 510: Create test case execution subtasks and timeout sentinel subtasks. In some embodiments, the test case execution subtask corresponds to a specific test case call execution process, and the execution duration of the timeout sentinel subtask corresponds to the execution timeout threshold of the test case execution subtask. The timeout determination of the test case execution process can be achieved by the synchronous race execution of the two subtasks.

[0048] Step 520: If the test case execution subtask completes before the timeout sentinel subtask, output the test case execution result and / or test case execution process. It can be understood that the execution duration of the timeout sentinel subtask corresponds to the execution timeout threshold for the test case execution subtask. Therefore, if the test case execution subtask completes before the timeout sentinel subtask, it indicates that the execution duration of the test case execution subtask is lower than the execution timeout threshold, and the test case execution result and / or test case execution process can be returned normally.

[0049] Step 530: If the timeout sentinel subtask completes before the test case execution subtask, determine that the test case execution subtask has timed out and abandon it. In some embodiments, steps 510 and 520 can be executed either based on the order of completion of the timeout sentinel subtask and the test case execution subtask. It is understood that the execution duration of the timeout sentinel subtask corresponds to the timeout threshold of the test case execution subtask. Therefore, when the timeout sentinel subtask completes before the test case execution subtask, it can be determined that the test case execution subtask has timed out, and the test case execution subtask is abandoned. The timeout status information is returned synchronously, and efforts are made to cancel the execution process related to the test case execution subtask. Through the dual-task timeout race setting, the execution timeout threshold corresponding to the timeout sentinel task is called the set timeout limit, which is independent of whether the underlying socket can be canceled, ensuring that the timeout of a single evaluation test task does not affect the overall evaluation test progress of the agent.

[0050] In some embodiments, further, during multiple rounds of evaluation and testing, a secondary timeout threshold can be set for each evaluation and testing task, as well as a tertiary timeout threshold for the entire agent evaluation process. By setting specific multi-level timeout thresholds, it can be ensured that any deadlock that may occur during the agent evaluation process has a set timeout limit, thereby effectively ensuring the smooth implementation of the entire agent evaluation process.

[0051] Based on the technical solutions provided in the foregoing embodiments, it is possible to achieve single-round evaluation and testing of a single evaluation and testing task for different configuration groups; however, considering the random fluctuations that may occur in the actual application of an agent equipped with a large language model, a single-round evaluation and testing cannot reflect the real application situation of the agent, and further evaluation of the stability of the agent's output content is required through multiple repeated rounds. In some embodiments, further, Figure 6 A flowchart illustrating a process for determining the configuration evaluation result corresponding to a configuration group is shown, such as... Figure 6 As shown, process 600 may specifically include the following steps: Step 610: Obtain statistical results of the evaluation test results across multiple rounds. In some embodiments, the statistical results may specifically include the mean and variance of the evaluation scores corresponding to the evaluation test results across multiple rounds, which is not limited here. In some embodiments, further, if the mean of the evaluation scores across multiple rounds does not reach a preset score threshold, the current evaluation test task can be determined as a failed evaluation test task; if the variance of the evaluation scores across multiple rounds exceeds a preset variance threshold, the current evaluation test task can be determined to be unstable, which is not limited here.

[0052] Step 620: Based on the historical evaluation test baseline and statistical results, obtain the random fluctuation parameter and effect size parameter of the configuration group, and determine the configuration evaluation result corresponding to the configuration group based on the random fluctuation parameter and effect size parameter. In some embodiments, the historical evaluation test baseline may be the currently used agent configuration group, or a validated agent configuration group, or the statistical results of the agent configuration group used as a reference under the current evaluation test task, to provide a reference standard. In some embodiments, the random fluctuation parameter is used to characterize the fluctuation of the evaluation score of the current configuration group. For example, if the average evaluation score of the current configuration group in multiple rounds exceeds the historical evaluation test baseline, if the random fluctuation parameter is less than a preset threshold, it indicates that the evaluation test performance of the current configuration group can stably exceed the historical evaluation test baseline, and the current configuration group is better than the reference standard corresponding to the historical evaluation test baseline; if the random fluctuation parameter exceeds the preset threshold, it indicates that the evaluation test performance of the current configuration group is highly volatile. Although the average evaluation score seems to exceed the historical evaluation test baseline, the output performance of the agent under the current configuration group is not stable, and the user may need to perform further adjustments and optimizations. In some embodiments, the effect size parameter is used to characterize the improvement of the current configuration group relative to the historical evaluation test baseline. It is understood that with a sufficient number of evaluation test tasks, the random fluctuation parameter of the current configuration group can remain at a low level if the generated content is relatively stable. However, the improvement of the current configuration group compared to the historical evaluation test baseline may be very slight, making it not worthwhile to perform a complete application iteration of the agent for the current configuration group. The effect size parameter can intuitively inform the user of the improvement of the current configuration group, helping the user to gain a more specific and in-depth understanding of the agent's specific content generation performance under the configuration group. Furthermore, in some embodiments, the user can also choose other suitable evaluation parameters to determine the configuration evaluation results according to the needs of the actual scenario. For example, nonparametric tests such as the Mann-Whitney U test (comparing each item in the statistical results pairwise with each item in the historical evaluation test baseline, etc.) or substitution of the fluctuation parameters based on the Bootstrap confidence interval can be used, etc., without limitation.

[0053] Based on the foregoing descriptions, comparative evaluation tests of the agent under multiple configuration groups can be implemented, allowing users to select the configuration group that best suits the current application scenario for specific application implementation. It is noted that during the specific execution of the agent evaluation, if an evaluation test task / test case fails, it indicates that the agent's model reasoning ability cannot correctly handle that evaluation test task / test case. In this case, the evaluation test task / test case can be collected and merged to generate a corresponding historical failure test case set. This historical failure test case set can then be used to verify the capabilities of the optimized agent during subsequent optimization and improvement processes. In some embodiments, further... Figure 7 A schematic diagram of a process for evaluating test failure reflow handling is shown, such as... Figure 7 As shown, process 700 may specifically include the following steps: Step 710: Based on the statistical results, identify the evaluation test tasks that failed to execute, and obtain the failed test cases corresponding to the failed evaluation test tasks. In some embodiments, the statistical results can be obtained based on process 600 provided in the foregoing embodiments, and are not limited here. In some embodiments, the evaluation test tasks that failed to execute may specifically include evaluation test tasks that timed out, evaluation test tasks whose test scores were below standard, and intelligent agent output results that deviated from preset test targets, etc., and are not limited here.

[0054] Step 720: Based on the similarity between failed test cases and the historical set of failed test cases, select failed test cases with a similarity lower than a preset threshold to update the historical set of failed test cases. In some embodiments, the historical set of failed test cases stores failed test cases of various execution failures that occurred during the agent's historical evaluation testing process. Before updating the historical set of failed test cases, the similarity between the failed test cases corresponding to the current round of evaluation testing and each test case in the historical set of failed test cases can be compared. For example, semantic embedding clustering can be performed on each test case, or cosine similarity calculation can be performed after text vectorization, etc., which is not limited here. After comparing the similarity, failed test cases with a similarity lower than the preset threshold are selected for updating to avoid adding similar or duplicate failed test cases to the historical set of failed test cases repeatedly. Understandably, through the above process 700, various failed test cases scattered throughout the evaluation and testing process can be collected back into a unified set of historical failed test cases, which can provide test samples for further optimization and improvement of the agent. If the optimized agent can successfully execute the failed test cases in the set of historical failed test cases in the subsequent evaluation and testing, it means that the optimized agent has overcome the system defects that caused the relevant failed test cases and achieved further targeted improvement in application performance.

[0055] Based on the foregoing descriptions of the embodiments, the agent evaluation method provided by this disclosure can achieve multi-dimensional evaluation and comparative evaluation of agents under different configuration groups, and can avoid specific application problems such as timeouts and high-concurrency deadlocks that affect the evaluation process. The following will further explain and illustrate the human-computer interaction process of the agent evaluation method provided by this disclosure in conjunction with specific embodiments, so that users can intuitively understand the specific evaluation status of the agent under different configuration groups. In some embodiments, specifically, for each configuration group of the agent, the agent evaluation method provided by this disclosure supports the visual presentation of the evaluation test results of each round and / or the execution process of evaluation test tasks in multiple rounds, so that users can intuitively obtain the specific process status information of each configuration group of the agent during the evaluation process, as well as the specific evaluation results of each configuration group after each evaluation round. In some embodiments, specifically, implementation can rely on general large model observability platforms, such as Langfuse and LangSmith. These large model observability platforms can record the entire link trajectory of a single call by the agent to a large model during agent evaluation, statistically analyze the number of lexical units and model call cost per call, provide ports for large model referee evaluation or manual annotation and scoring, manage test case datasets, and support the visualization of evaluation test results for each round through dashboard presentation styles. This allows users to intuitively obtain information about the agent's evaluation process and results through a visual interactive interface. Users can also choose other suitable visualization methods for the evaluation test results of each round and / or the execution process of multiple rounds of evaluation test tasks, which are not limited here. Furthermore, in some embodiments, the agent evaluation method provided in this disclosure also supports configuration evaluation results based on multiple configuration groups, visualizing the comparative evaluation results of the agent. This allows users to more intuitively obtain the specific processing performance scores of the agent under different configuration groups, helping to assist users in continuously evaluating, improving, and optimizing the agent's update process for a better user experience.

[0056] In some embodiments, to simultaneously visualize the agent evaluation process / results and analyze the agent evaluation results across multiple rounds of test case execution, a dual-data-path data writing setting is proposed in the agent evaluation method provided in this disclosure. Specifically, while acquiring the evaluation test results for each round, the evaluation test results are synchronously written to a first data space and a second data space. The first data space can be used to provide evaluation test results for determining configuration evaluation results and / or comparison evaluation results. For example, the first data space can be a preset business database used to store the evaluation test results for each round and provide subsequent data processing functions such as structured query, data aggregation, and data comparison, so that the evaluation test results stored in the first data space can be applied to the specific generation process of subsequent configuration evaluation results and / or comparison evaluation results, which is not limited here. In some embodiments, the second data space can be used to provide visualization of the evaluation test results. For example, the second data space can be associated with the large model observability platform provided in the aforementioned embodiments, used for the large model observability platform to acquire and visualize the evaluation test results of the agent in each round under the selected configuration group. In some embodiments, specifically, the first data space is independent of the second data space, and the evaluation test results of each round are synchronously written into the two independent data spaces, which are inaccessible to each other, so that the agent evaluation and visualization can be decoupled and executed independently. Users can also set the specific data flow path and corresponding data storage space of the evaluation test results of the agent in each round under the set configuration group according to the needs of the actual application scenario, which is not limited here.

[0057] Based on the foregoing descriptions of the embodiments, the intelligent agent evaluation method provided in this disclosure supports the visualization of evaluation results through a human-computer interaction interface provided by a terminal device. In some embodiments, the intelligent agent evaluation method further supports receiving natural language commands provided by the user through a human-computer interaction interface and driving the intelligent agent evaluation process according to the natural language commands. In some embodiments, exemplarily... Figure 8 A flowchart illustrating an agent evaluation process driven by natural language instructions is shown, such as... Figure 8 As shown, process 800 may specifically include the following steps: Step 810: In response to a natural language instruction, determine the agent's evaluation mode. In some embodiments, the user can input natural language instructions through a human-computer interaction interface, and the agent's evaluation mode is determined by parsing and semantic analysis of the natural language instructions. The user can choose an appropriate parsing method for the natural language instructions based on the actual application scenario, which is not limited here. In some embodiments, the agent's evaluation mode may specifically include a parameter tuning evaluation mode, a script evaluation mode, and a code evaluation mode. The parameter tuning evaluation mode may include a comparison of the agent's generation results under different calling models, and / or different inference parameters, and / or different prompt word versions. For example, for the user input "compare the agent's dialogue output capabilities under A1 model and A2 model", the semantic parsing and intent analysis of the natural language instruction can determine that the user needs to compare the agent's output under different large model scenarios, and determine the agent's evaluation mode as the parameter tuning evaluation mode. In some embodiments, further, in the parameter tuning and evaluation mode, the user can provide agent configuration information based on the human-computer interaction interface. The agent configuration information may specifically include the model name, prompt word version, inference temperature, etc., which are not limited here. The user does not need to provide other code content, and the required configuration group can be automatically generated by the agent evaluation device provided in this disclosure, which are not limited here.

[0058] In some embodiments, the script evaluation mode may specifically include a comparison of the agent's generation results under different iteration versions. For example, for a user input of "compare the dialogue output capabilities of the agent in versions 7.1 and 7.2", it can be determined that the user needs to compare the agent's generation results under different iteration versions, and the evaluation mode of the agent is determined to be the script evaluation mode. In some embodiments, further, in the script evaluation mode, the user can pass in the running script path corresponding to different iteration versions based on specific evaluation needs. Each configuration group can run completely different code logic, thereby realizing the agent evaluation comparison between multiple iteration versions; in some embodiments, the script evaluation mode can also realize the comparative evaluation test between any agents, which is not limited here.

[0059] In some embodiments, the code evaluation mode may specifically include a comparison of the generated results of the agent under different code fine-tuning scenarios. For example, for the user input "switching function A in the agent to function B and then conducting an evaluation test", it can be determined that the user needs to compare the agent's code-level A / B comparison test under partial code fine-tuning scenarios, and determine that the agent's evaluation mode is the code evaluation mode. In some embodiments, furthermore, based on the code variant information provided by the user, an automated A / B comparison agent can be generated by cloning the code of the target agent; this is not limited here.

[0060] In some embodiments, regardless of the specific agent evaluation mode adopted, the agent evaluation method provided in the foregoing embodiments of this disclosure can be used to access a unified evaluator system for visualized agent evaluation analysis. This allows model differences, configuration differences, and code differences between agents to be evaluated and compared under the same criteria, and supports users to complete the entire agent evaluation process through natural language interaction.

[0061] Step 820: Based on the agent's evaluation mode, determine the agent's configuration group and evaluation intent form. In some embodiments, when the agent's evaluation mode is determined, the user's corresponding agent evaluation intent can be written into the state bus, and the state bus automatically triggers the listener of the human-computer interaction interface. The listener is responsible for switching the view presented by the human-computer interaction interface, filling in the pre-filled values ​​of the evaluation intent form based on user instructions, and opening the dialogue interaction box based on user needs. After the human-computer interaction interface detects the responsive state change, it automatically re-renders. During rendering, the pre-filled values ​​are directly read as the initial values ​​to determine the agent's configuration group and evaluation intent form. This is not limited here.

[0062] Step 830: In response to the confirmation instruction of the evaluation intent form, determine the evaluation test task pool and the evaluation dimension information corresponding to the evaluation test results. In some embodiments, when the user confirms the visually presented evaluation intent form, the corresponding agent evaluation test process can be executed in response to the confirmation instruction of the evaluation intent form, and the agent evaluation method provided in the foregoing embodiments can be invoked to execute the corresponding agent evaluation process, including the establishment of the evaluation test task pool, the selection and setting of the evaluator, and the specific execution of the evaluation process, etc., which are not limited here.

[0063] Figure 9 This is a schematic diagram of the structure of an intelligent agent evaluation device according to some embodiments of the present disclosure. In some embodiments, such as Figure 9 As shown, the intelligent agent evaluation device 900 may include an acquisition unit 910, a testing unit 920, a group evaluation unit 930, and a control evaluation unit 940.

[0064] In some embodiments, such as Figure 9 As shown, the acquisition unit 910 can be used to acquire at least two configuration groups of the agent and an evaluation test task pool. For details regarding the acquisition of the evaluation test task pool and the setting of the configuration groups, please refer to the relevant descriptions in the foregoing embodiments, which will not be repeated here. In some embodiments, such as Figure 9The test unit 920 shown can be used to concurrently execute multiple rounds of evaluation test tasks based on the evaluation test task pool for each configuration group of the agent, and obtain the evaluation test results for each round. In some embodiments, the evaluation test results for each round may specifically include the accuracy of the agent's tool selection, the efficiency of the task execution trajectory, the agent's processing cost, whether the lexical consumption exceeds the budget, and whether the agent's processing latency meets the standard, etc., which are not limited here. In some embodiments, such as Figure 9 The group evaluation unit 930 shown can be used to determine the configuration evaluation result corresponding to each configuration group based on the evaluation test results of multiple rounds for each configuration group. It utilizes multiple rounds of evaluation and comparison with historical baseline processing to minimize random fluctuations during the agent evaluation process. In some embodiments, such as... Figure 9 The comparison evaluation unit 940 shown can be used to determine the comparison evaluation result of the agent based on the configuration evaluation results of multiple configuration groups. This allows users to intuitively and clearly obtain the specific task / test case execution status of the agent under different configuration groups, thereby enabling targeted selection of agent configuration groups according to different application scenarios. This helps users to continuously update and optimize the agent and achieve a better user experience. No further limitations are made here.

[0065] For more information on each unit, please refer to [link / reference]. Figures 2 to 8 The relevant explanations will not be repeated here. It should be understood that, as... Figure 10 The apparatus and modules shown can be implemented in various ways. For example, in some embodiments, they can be implemented by hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the methods, apparatus, and systems described above can be implemented using computer-executable instructions and / or included in the control code of a processor, such as code provided in the memory of a programmable device on a media such as a disk, CD, or DVD-ROM. The apparatus and modules of this disclosure can be implemented not only by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips or transistors, or programmable hardware devices such as field-programmable gate arrays or programmable logic devices, but also by software executed by various types of processors, or by a combination of the aforementioned hardware circuits and software (e.g., firmware).

[0066] It should be noted that the above description of the device and modules is for convenience only and should not be construed as limiting this disclosure to the scope of the embodiments described. It is understood that those skilled in the art, after understanding the principle of the device, can arbitrarily combine the various modules without departing from this principle to form sub-devices connected to other modules. Alternatively, some modules can be split to obtain more modules or multiple units corresponding to a module. Such modifications are all within the scope of this disclosure.

[0067] Figure 10 This is an exemplary hardware architecture diagram of a computer device shown according to some embodiments of the present disclosure. In some embodiments, such as Figure 10 As shown, the computer device 1000 may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the agent evaluation method provided in any of the foregoing embodiments. In some embodiments, the computer device 1000 may be a terminal device such as a smartphone, tablet computer, or personal computer. Figure 1 The client 130 is shown. In other embodiments, the computer device 1000 may be a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), for example, it may be a... Figure 1 The server 110 shown here is not limited to any particular server.

[0068] In some embodiments, such as Figure 10As shown, the computer device 1000 includes, but is not limited to, a memory 1010, a processor 1020, and a network interface 1030 that can communicate with each other via a system bus. The memory 1010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD memory, DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 1010 may be an internal storage module of the computer device 1000, such as the hard disk or memory of the computer device 1100. In other embodiments, the memory 1010 may also be an external storage device of the computer device 1000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 1000. Of course, the memory 1010 may include both the internal storage module of the computer device 1000 and its external storage device. In this embodiment, the memory 1010 is typically used to store the operating system and various application software installed on the computer device 1000, such as the relevant program code of the agent evaluation method provided in any of the foregoing embodiments. In addition, the memory 1010 may also be used to temporarily store various types of data that have been output or will be output, such as test cases used in the agent evaluation process, historical baseline information stored in the agent evaluation process, etc., which are not limited here.

[0069] In some embodiments, the processor 1020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chip. The processor 1020 is typically used to control the overall operation of the computer device 1000, such as performing control and processing related to data interaction or communication with the computer device 1000. In this embodiment, the processor 1020 is used to run program code stored in the memory 1010 or process data.

[0070] In some embodiments, network interface 1030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 1000 and other computer devices. For example, network interface 1030 is used to connect computer device 1000 to an external terminal or server via a network, and to establish a data transmission channel and communication link between computer device 1000 and the external terminal or server. The network may be an intranet, the Internet, Global System for Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 10G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0071] It should be pointed out that, Figure 10 Only a computer device having components 1010 to 1030 is shown; however, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented alternatively. In some embodiments, the relevant program code for the agent evaluation method stored in memory 1010 may also be divided into one or more program modules and executed by one or more processors (such as processor 1020) to implement the embodiments of this disclosure, without limitation herein.

[0072] Some embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, can implement this disclosure. Figures 2 to 8 The agent evaluation method shown.

[0073] One or more embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, can implement the agent evaluation method provided in any of the foregoing embodiments. In some embodiments, the computer program product may relate to a computer program that may be carried on a storage medium or a processing device. In other embodiments, the computer program product may also be a storage medium or a processing device containing the aforementioned computer program. The processing device may include one or more processors, and a storage medium.

[0074] The basic concepts have been described above. It is obvious that the detailed disclosure above is merely illustrative and does not constitute a limitation of this disclosure. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to this disclosure by those skilled in the art. Such modifications, improvements, and corrections are taught in this disclosure and therefore remain within the spirit and scope of the exemplary embodiments of this disclosure.

Claims

1. A method for evaluating an intelligent agent, characterized in that, The method includes: Obtain at least two configuration groups for the agent and a pool of evaluation test tasks; For each configuration group of the intelligent agent, multiple rounds of evaluation and testing tasks are executed concurrently based on the evaluation and testing task pool, and the evaluation and testing results of each round are obtained. Based on the evaluation test results of multiple rounds for each configuration group, the configuration evaluation result corresponding to the configuration group is determined; Based on the configuration evaluation results of multiple configuration groups, the comparison evaluation result of the agent is determined.

2. The agent evaluation method according to claim 1, characterized in that, For each configuration group of the intelligent agent, multiple rounds of evaluation and testing tasks are executed concurrently based on the evaluation and testing task pool, and the evaluation and testing results of each round are obtained, including: Based on the configuration group, create a corresponding evaluation test data space; Once all the evaluation test data spaces have been created, based on concurrent semaphores, each configuration group independently selects an evaluation test task from the evaluation test task pool; and Control each of the aforementioned configuration groups to concurrently execute the evaluation test task; The evaluation test task includes at least one preset test case and the number of repeated execution rounds corresponding to the test case.

3. The agent evaluation method according to claim 2, characterized in that, The evaluation test task includes the test case execution phase and the test result evaluation phase; The control of concurrent execution of the evaluation test task for each of the configuration groups includes: Based on a preset first concurrent semaphore, the intelligent agent is invoked to execute the test case execution phase according to the configured group; Based on a preset second concurrent semaphore, a preset evaluator is invoked to perform the test result evaluation phase based on the test case execution results and / or the test case execution process; The first concurrent semaphore and the second concurrent semaphore are independent of each other.

4. The agent evaluation method according to claim 3, characterized in that, The step of invoking the intelligent agent to execute the test case execution phase based on the configured group includes: Create test case execution subtasks and timeout sentinel subtasks; If the test case execution subtask completes before the timeout sentinel subtask, output the test case execution result and / or the test case execution process: If the timeout sentinel subtask completes before the test case execution subtask, it is determined that the test case execution subtask has timed out, and the test case execution subtask is abandoned.

5. The agent evaluation method according to claim 3, characterized in that, The evaluator includes one or any combination of multiple of the following: tool correctness dimension evaluator, rule dimension evaluator, efficiency dimension evaluator, cost dimension evaluator, lexical dimension evaluator, delay dimension evaluator, and large language model evaluator: The invocation of a preset evaluator, based on the test case execution results and / or the test case execution process, executes the test result evaluation phase, including: The evaluator is invoked to obtain one or more dimension evaluation results of the evaluator based on the test case execution results and / or test case execution process, and the evaluation test result is determined based on all the dimension evaluation results.

6. The agent evaluation method according to claim 1, characterized in that, The determination of the configuration evaluation result corresponding to each configuration group based on the evaluation test results of multiple rounds for each configuration group includes: Obtain statistical results from multiple rounds of evaluation and testing; Based on the historical evaluation test baseline and the statistical results, the random fluctuation parameters and effect size parameters of the configuration group are obtained, and the configuration evaluation result corresponding to the configuration group is determined based on the random fluctuation parameters and effect size parameters.

7. The agent evaluation method according to claim 6, characterized in that, The method further includes: Based on the statistical results, identify the evaluation test tasks that failed to execute, and obtain the failed test cases corresponding to the evaluation test tasks that failed to execute. Based on the similarity between the failed test cases and the historical set of failed test cases, the historical set of failed test cases is updated by selecting the failed test cases with a similarity lower than a preset threshold.

8. The agent evaluation method according to claim 1, characterized in that, The method further includes: For each configuration group of the intelligent agent, the evaluation test results for each round and / or the execution process of the evaluation test tasks for multiple rounds are visualized; and Based on the configuration evaluation results of multiple configuration groups, the comparative evaluation results of the agent are visualized.

9. The agent evaluation method according to claim 8, characterized in that, The method further includes: The evaluation test results for each round are obtained and synchronously written to the first data space and the second data space; The first data space is independent of the second data space; The first data space is used to provide the evaluation test results for determining the configuration evaluation results and / or the control evaluation results; The second data space is used to provide a visual representation of the evaluation test results.

10. The agent evaluation method according to claim 1, characterized in that, The method further includes: In response to natural language instructions, the evaluation mode of the agent is determined; Based on the evaluation mode of the intelligent agent, determine the configuration group of the intelligent agent and the evaluation intent form; In response to the confirmation instruction of the evaluation intent form, the evaluation test task pool and the evaluation dimension information corresponding to the evaluation test results are determined.

11. The agent evaluation method according to claim 10, characterized in that, The evaluation modes of the intelligent agent include parameter tuning evaluation mode, script evaluation mode, and code evaluation mode; The parameter tuning evaluation mode includes a comparison of the generation results of the agent under different invocation models, and / or different inference parameters, and / or different prompt word versions; The script evaluation mode includes a comparison of the generation results of the agent under different iterative versions; The code evaluation mode includes a comparison of the generated results of the agent under different code fine-tuning scenarios.

12. An intelligent agent evaluation device, characterized in that, The device includes: The acquisition unit is used to acquire at least two configuration groups of the agent and an evaluation test task pool; The testing unit is used to concurrently execute multiple rounds of evaluation test tasks based on the evaluation test task pool for each configuration group of the agent, and obtain the evaluation test results for each round. The group evaluation unit is used to determine the configuration evaluation result corresponding to the configuration group based on the evaluation test results of multiple rounds for each configuration group; The comparison evaluation unit is used to determine the comparison evaluation result of the agent based on the configuration evaluation results of multiple configuration groups.

13. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the agent evaluation method according to any one of claims 1 to 10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the agent evaluation method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the agent evaluation method according to any one of claims 1 to 11.

Images